MICRO-502_Aerial_Robotics_NotesIntro (week1)OverviewFixed wing Staying in the AirMaintaining constant speed VMajor Application FieldsFor AgricultureFor EnergyFor Public safety & securityFor DeliveryCategory«Can Drones Deliver?»Last-cm delivery - Dronistics☑️ CheckpointsMulticopters (week1)IntroductionRotorcrafts (helicopters vs multicopters)Pros and ConsPros-Easy to build and maintainConsStructure and PhysicsMain componentsConfigurationRotation speeds / Forces / MomentsHover conditionsFlight mechanicsMoving Up and DownRotating in YawRotation in Roll/PitchSummary of equationsExample-Translated flightTypes of MulticoptersConfigurationFully Actuated MulticoptersFeaturesProsConsEnergeticsEnergy in hoveringEnergy in forward flight of quadcopterIncrease flight time☑️ CheckpointsAttitude representations (week2)3D Attitude representation- Euler AnglesQuaternionConversionComplex examples✖️ CheckpointsControl (week2&3)Cascaded control ArchitectureControl allocationMethodRemarkRate controlAttitude controlFull quaternion based attitude control for a quadrotorControl strategies for multicoptersLinearExamplesFeaturesPID controllerLQR/LQGNonlinearExamplesFeaturesAn introduction to fully actuated multirotor UAVs☑️ CheckpointsState Estimation (week3&4)Introduction to State EstimationWhy State Estimation in Robotics?Estimation for deterministic systemsState Observer (or Luenberger Observer)Dynamical SystemsEstimation errorState Estimation for stochastic systemsRecap of fundamental conceptsKalman filter (KF)PREDICTIONUPDATEsensor fusion on a quadrotorIntro to EKF, UKF and particle filtersEKFOverviewEquationsUKFParticle filtersState Estimation in aerial roboticsin lab settingsOutdoor☑️ Checkpoints🚧 Navigation (week5)Velocity controlWaypoint NavigationDubins PathsVector fields☑️ Checkpoints🚧 VIO & SLAM (week5)VIOSLAM☑️ CheckpointsFixed-wing drones (week6)IntroductionStructureFlight MechanicsControl surfaceStabilityEnergeticsInduced powerProfile and parasite power☑️ CheckpointsAerial Swarms (week7)IntroReynolds flocking algorithm (Reynolds, 1987)Reynolds flocking: modelReynolds flocking with migrationCase: Aerial swarms for disaster mitigationCommunication radius and turning angleVirtual agents for flocking with fixed-wing dronesReynolds flocking with obstacles (Virtual agents)Other modelsVicsek model: particles in confined environments (密闭环境)Olfati-Saber modelDrone SwarmsVisual information in flockingSoria2019IRC-influence of limited visual sensing using ReynoldsSchilling2019RAL-Learning to flock in simulation with visionSchilling2021RAL-Learning to flock outdoor with vision☑️ Check pointsFlapping-Wing (week8)IntroductionStructureFlight mechanics - Lift generationLift generation in hovering flightAsymmetric hoveringSymmetric hoveringLift generation in forward flightFlight mechanics - ManeuveringEnergetics☑️ CheckpointsDrone Regulations (week8)3 Pillar Concept / Drone CategoriesActSpecific CategoryU-Space☑️ CheckpointsUAS Hardware (week9)IntroductionFrame and materialsmaterials comparisonmetric when considering materialsEnergy sourcesCategoryEnergy and power densityLi-Po batteriesDischarge Curves of Li-Po batteryEnergy Curve of Li-Po batteryActuatorsActuators for propulsionElectric motor example-Brushless DC electric motorsActuators for control/maneuveringServomotorsExamples of ServomotorsPropellersCharacteristicsPitch and efficiency at different cruise speedChoose the right combination actuator and propellerSensorsGyroscopesAccelerometersMagnetometersPressure / Altitude sensorsAirspeed sensorsGlobal positioning system (GPS)Power sensorsOptic flow camerasAutopilotsCommunication protocols✖️ CheckpointsInsect-inspired vision (week10)Optical flowFor pure translational motionSensors used for flight controlArchitecture of insect eyes and brainsElementary Motion Detector 初级运动检测器Experiment - Optomotor Response 视运动反应Wide-field, motion-specific neuronsOptic Flow ComputationGradient Descent MethodsImage Interpolation Algorithm –I2AObstacle avoidance with I2A☑️ CheckpointsAdaptive Morphology in Flying Animals and Drones (week10)Bioinspired Mechanical ResilienceHow do insects cope with collisions?The Size ProblemSelf-deployable origami droneOrigami Drone WingAdaptive Morphology☑️ CheckpointsAgile Flight (week11)Autonomous drone racingDrone acrobaticsLow-latency sensingLearning of flight controllers (week11)challenge1: Architecture/Input and Output representationchallenge2: Data collectionLimitations of Imitation Learning -> RLRL limitationschallenge3: Guarantee the platform's safety during training and testingTakeaways

MICRO-502_Aerial_Robotics_Notes

Lecture notes by Yujie He

Last updated on 2021/07/02

All checkpoints summary can be found here!

Intro (week1)

Overview

Fixed wing Staying in the Air

generate a force Lift L equal and opposite to its own weight W

week1_hovering

fixed-wing generates by airflow

Maintaining constant speed V

Major Application Fields

For Agriculture

For Energy

For Public safety & security

For Delivery

Category

Forcast: parcel delivery > air freight in near future

«Can Drones Deliver?»

Last-cm delivery - Dronistics

☑️ Checkpoints

Multicopters (week1)

 Fixed wingFlapping wingrotating wing
Examplesairplanes, glidersnew robotshelicopters, multicopters
ProsFast; EfficientEfficientCan hover;Highly maneuverable
ConsCannot HoverHard to build and controlLess efficient
Features Scale down in sizeVertical take-off and landing (VTOL)

Introduction

Rotorcrafts (helicopters vs multicopters)

generates lift using high speed rotary blades called rotors

Pros and Cons

Pros-Easy to build and maintain

Cons

Structure and Physics

Main components

frame; control board; Motors and motor drivers (ESC, electronic speed controller); Propellers; Battery; Receiver

Configuration

Rotation speeds / Forces / Moments

movement are controlled by changing the rotation speed of the propellers

week1_force

Hover conditions

  1. All forces must be balanced

    move up and down

  2. Lift forces must be parallel to gravity

  3. All moments must be balanced

    pitch and roll

  4. Rotor speeds must be balanced (torque balanced)

    yaw

Flight mechanics

How to move a quadrotor around?

Violating one or more of these conditions implies that the quadcopter starts to move

Moving Up and Down

Rotating in Yaw

week1_rollpitch

Rotation in Roll/Pitch

Summary of equations

week1_speed2motion

Example-Translated flight

Moving Forward

Translated flight requires more thrust than hovering, but not always more power (see section 6)!

Types of Multicopters

main feature; fully actuated multicopters

Configuration

week1_config

Fully Actuated Multicopters

underactuated multicopters -> all propellers are rotated in the same plane

Features

Pros

Cons

Energetics

What is the power consumption of a multicopter during flight?

How to extend multicopters flight time?

Energy in hovering

week1_energetics

Energy in forward flight of quadcopter

Increase flight time

  1. Weight and drag reduction

  2. Increase the specific power of the energy source

    switch from LiPo batteries to gasoline (far more energy stored)

  3. Docking station for charging/battery swapping

  4. Tether for power supply

    reduce battery

  5. Improving efficiency via mechanical

  6. Energy aware motion planning

    reduce acceleration (aggressive flight)

  7. Multi-modal operation

    perching; walking and rolling

☑️ Checkpoints

Attitude representations (week2)

week2_frame

3D Attitude representation- Euler Angles

Quaternion

the union of scalar and vector

week2_quaternion

🚧 Claw Example

rotation + translation using quaternions

Conversion

Complex examples

frame: image/centered -> camera -> gimbal -> body -> local

🚧 注意谁相对谁!!!​

✖️ Checkpoints

Control (week2&3)

结合exercise进行再次查看

Cascaded control Architecture

week2_cascaded_control

Control allocation

convert thrust & torque setpoint into actuator commands

account for different geometries

week2_control_allocation

Method

week2_allocation_method

  1. Compute generated force and torque of each actuator

  2. Build matrix "Actuator Effectiveness"

  3. Compute allocation matrix B as the pseudo-inverse of A

    not always square, so using pseudo-inverse ()

Remark

Rate control

Input body rate setpoint -> Output torque to each angle rate

Attitude control

Input quaternions setpoint -> Output torque to each angle rate

Full quaternion based attitude control for a quadrotor

Fresk, E. and Nikolakopoulos, G., 2013, July. Full quaternion based attitude control for a quadrotor. In 2013 European control conference (ECC) (pp. 3864- 3869). IEEE.

Control strategies for multicopters

Linear

Examples

Features

PID controller

LQR/LQG

Nonlinear

Examples

Features

An introduction to fully actuated multirotor UAVs

☑️ Checkpoints

State Estimation (week3&4)

Introduction to State Estimation

Why State Estimation in Robotics?

Estimation for deterministic systems

State Observer (or Luenberger Observer)

week3_state_observer week3_state_observer2

Dynamical Systems

week3_dynamical_system

Estimation error

week4_observer

State Estimation for stochastic systems

Recap of fundamental concepts

Kalman filter (KF)

use Gaussians to implement a Bayesian filter. That’s all the Kalman filter is - a Bayesian filter that uses Gaussians

PREDICTION

UPDATE

sensor fusion on a quadrotor

use multiply KF to achieve better estimation

Intro to EKF, UKF and particle filters

EKF

Overview

week4_ekf_equ

Equations

UKF

Particle filters

week5_state_estimation

State Estimation in aerial robotics

in lab settings

Outdoor

☑️ Checkpoints

🚧 Navigation (week5)

Velocity control

Waypoint Navigation

Dubins Paths

Vector fields

☑️ Checkpoints

🚧 VIO & SLAM (week5)

VIO

SLAM

☑️ Checkpoints

Fixed-wing drones (week6)

Introduction

Structure

Flight Mechanics

Control surface

Stability

longitudinal -> pitching

lateral -> rolling

directional -> yawing

Energetics

Induced power

Profile and parasite power

week6_power_fixed

☑️ Checkpoints

Aerial Swarms (week7)

Intro

Reynolds flocking algorithm (Reynolds, 1987)

week7_swarm_reynolds

Reynolds flocking: model

week7_swarm_reynolds_model

Reynolds flocking with migration

Case: Aerial swarms for disaster mitigation

week7_swarm_project_example

SMAV platform with control electronics

Communication radius and turning angle

Virtual agents for flocking with fixed-wing drones

Reynolds flocking with obstacles (Virtual agents)

week7_swarm_reynolds_obs

Other models

Vicsek model: particles in confined environments (密闭环境)

Vasarhelyi et al., Optimized flocking of autonomous drones in confined environments, Science Robotics, 2019

DOI: http://doi.org/10.1126/scirobotics.aat3536

Video: https://youtu.be/E4XpyG4eMKE

Project web: http://hal.elte.hu/drones/scirob2018.html

Olfati-Saber model

R. Olfati-Saber, Flocking for multi-agent dynamic systems: algorithms and theory, IEEE Transactions on Automatic Control, 2006

week7_swarm_olfati_model

Drone Swarms

Coppola et al., A Survey on Swarming With Micro Air Vehicles: Fundamental Challenges and Constraints, Front. Robot. AI, ‘20

week7_swarm_survey

The combination of centralized planning/control with external positioning has allowed to fly significantly larger swarms. The numbers are lower for the works featuring decentralized control with external positioning, or centralized control with local sensing

Three categories

  1. Centralized with external positioning

    latest: September 20 2020

    3,051 drones

    News: https://www.guinnessworldrecords.com/news/2020/10/3051-drones-create-spectacular-record-breaking-light-show-in-china (Company: https://www.dmduav.com/)

    YouTube: https://youtu.be/44KvHwRHb3A

    Bilibili: https://www.bilibili.com/video/BV1jt4y1q762

  2. Decentralized with external positioning or centralized with on-board sensing

    Vasarhelyi et al. (2019)

  3. Decentralized with on-board sensing

    Saska et al. (2017)

Visual information in flocking

Soria2019IRC-influence of limited visual sensing using Reynolds

Soria et al., The influence of limited visual sensing on the Reynolds flocking algorithm, 2019

Schilling2019RAL-Learning to flock in simulation with vision

Schilling et al., Learning Vision-Based Flight in Drone Swarms by Imitation, RAL2019

week7_swarm_vision_Schilling19RAL

Schilling2021RAL-Learning to flock outdoor with vision

Schilling et al., Vision-Based Drone Flocking in Outdoor Environments, RAL2021

week7_swarm_vision_Schilling2021RAL

☑️ Check points

Flapping-Wing (week8)

Introduction

Structure

categories were determined to be the tail (1) and wing design (2)

week8_flapping

Flight mechanics - Lift generation

Lift generation in hovering flight

Asymmetric hovering

Symmetric hovering

Lift generation in forward flight

Downstroke/Upstroke

Flight mechanics - Maneuvering

flapping wing MAVs can use the tail and / or the wings for control.

Energetics

☑️ Checkpoints

Drone Regulations (week8)

Author: Markus Farner

https://www.bazl.admin.ch/bazl/en/home/good-to-know/drohnen.html

week9_regulation_uas

3 Pillar Concept / Drone Categories

  1. Open-Within the legal framework (No Authorization required)
  2. Specific-Not sufficiently safe (Authorization required)
  3. Certified-Approved to accepted standards

Act

Specific Category

Application for an operating permit on the basis of the SORA (Specific Operations Risk Assessment)

Operational Volume = Flight Geography + Contingency Volume

week9_regulation_SORA

U-Space

The U-space is a collection of decentralized services that collectively aim to safely and efficiently integrate drones into the airspace and enable drone operations alongside manned flight.

https://www.bazl.admin.ch/bazl/en/home/good-to-know/drohnen/wichtigsten-regeln/uspace.html.html

https://www.skyguide.ch/en/events-media-board/u-space-live-demonstration/

airspace in block to avoid collision and report the location for further path calculation

☑️ Checkpoints

UAS Hardware (week9)

Introduction

main component required

  1. The aerial vehicle

    • Air frame

    • Actuators for propulsion and control

    • Energy source

    • Autopilot

      • Sensors for attitude estimation
      • Electronics for regulation, control and communication
      • Sensor and avoid system
  2. Payload

    • Cameras
    • Environmental sensors (wind, temperature, humidity)
    • Robotic arms for manipulation
  3. Ground Control Station

    • Communication systems
    • Interface to monitor internal parameters and to send commands to the vehicle

Frame and materials

materials comparison

MaterialCompositeABS/PLAWoodFoam
ProsStiff, lightweightEasy to manufacture by 3D printing or injection moldingLightweight and cheapLightweight and soft, resistance to collision
ConsExpensive, complex to manufactureHeavier, less stiffcomplex to work withlimited load
Comment-useful for prototyping-absorb energy, less prone to damage

metric when considering materials

Energy sources

Goal: power the robots to fly

Metric: energy density, power density, charging time and so on

Category

Energy and power density

week9_UAS_Hardware_energy_density

Li-Po batteries

week9_UAS_Hardware_LiPo_battery

Discharge Curves of Li-Po battery

week9_UAS_Hardware_discharge_curve

Book: G. C. H. E. Decroon, M. Perçin, B. D. W. Remes, R. Ruijsink, and C. De Wagter, The delfly: Design, aerodynamics, and artificial intelligence of a flapping wing robot. 2015.

Energy Curve of Li-Po battery

week9_UAS_Hardware_energy_curve

Actuators

Actuators for propulsion

 Electric motorsCombustion engineHybrid
Prosclean and quite; Reliable and easy to maintain; Fast to change operational state (accelerate and decelerate)High weight to power ratio using fuelLong endurance; Suited for fast change of speed
ConsLimited weight to power ratio due to batteryVibration, dirt, and noise; Requires tuning; Not suited for fast change of speedComplex and expensive
  1. Combustion engine is not suited for fast change of speed (problem in controlling quadcopters)

  2. Hybrid systems (fuel generator coupled with electric motor)

    e.g. skyfront drone with 4.5 hour endurance (demonstrated) and 3 kg payload capacity

Electric motor example-Brushless DC electric motors

week9_UAS_Hardware_brushless_motor

 

Actuators for control/maneuvering

Servomotors

need to deflect the control surfaces

week9_UAS_Hardware_servomotor

Examples of Servomotors

Rotary servos with push rodLinear servos
week9_UAS_Hardware_rotary_servoweek9_UAS_Hardware_linear_servo
Weight: 1 to 500 gWeight: 1 to 5 g
-to control elevators, flaps and ailerons

Propellers

to convert power (delivered by a rotating shaft) into thrust

Characteristics

Pitch and efficiency at different cruise speed

Choose the right combination actuator and propeller

match the propeller and the motor to maximize propulsive efficiency

Sensors

Gyroscopes

measure changes in vehicle orientation

Accelerometers

measure acceleration to get the inertial information

week9_UAS_Hardware_accelerometers

Magnetometers

Exteroceptive

Pressure / Altitude sensors

to measure the altitude according the atmosphere pressure

Airspeed sensors

Global positioning system (GPS)

Power sensors

Optic flow cameras

week9_UAS_Hardware_optical_camera

Autopilots

system used to stabilize (e.g. attitude stabilization of a multicopter) or to control the trajectory

receive the input information -> process information -> send actuator commands

week9_UAS_Hardware_Autopilot_connection

Communication protocols

✖️ Checkpoints

Insect-inspired vision (week10)

Optical flow

week10_optical_flow

week10_optical_flow_component

For pure translational motion

, where

Sensors used for flight control

  1. Compound eyes: a large set of eyes to detect in different directions, no color capability -> used to detect optical flow

    • small viewing angle
    • several small eyes
  2. Ocelli: a small set of eyes are sensitive to luminosity to detect contrast; on the head and point upward -> used for stabilization, orientation, and attitude

  3. Halteres (like accelerators) -> used to measure rotational speed and stabilize than visual information

Architecture of insect eyes and brains

Optic flow is detected by neurons in the lamina (椎板), whose response is aggregated and transformed by neurons in the medulla (髓质) and in the lobula plata (小叶平台) of brain regions

Elementary Motion Detector 初级运动检测器

Experiment - Optomotor Response 视运动反应

torque response is not related to optical flow speed

Wide-field, motion-specific neurons

Optic Flow Computation

Gradient Descent Methods

Image Interpolation Algorithm –I2A

computed as the image shift that generates the smallest error between artificially shifted versions of the image at time t and the image at time t + Δt

week10_optical_flow_i2a

Obstacle avoidance with I2A

☑️ Checkpoints

Adaptive Morphology in Flying Animals and Drones (week10)

Bioinspired Mechanical Resilience

How do insects cope with collisions?

Mintchev, de Rivaz, Floreano, Insect-Inspired Mechanical Resilience for Multicopters, IEEE Robotics & Automation Letters, 2017

An active uprighting mechanism for flying robots, Klaptocz et al., IEEE Transactions on Robotics, 2012

Briod, Kornatowski, Zufferey, Floreano, A collision‐resilient flying robot, Journal of Field Robotics, 2014

flyability

The Size Problem

Self-deployable origami drone

Mintchev, Daler, L’Eplattanier, Saint_Raymond, Floreano, Foldable and self-deployable pocket sized quadrotor, ICRA, 2015

Origami Drone Wing

Dufour, Owen, Mintchev, Floreano, A drone with insect-inspired folding wings, IROS 2016

Adaptive Morphology

Daler, Lecoeur, Hählen, Floreano, A flying robot with adaptive morphology for multi-modal locomotion, IROS Proceedings, 2013

A bioinspired multi-modal flying and walking robot, Bioinspiration & Biomimetics, 2015

Ajanic, Feroshkan, Mintchev, Noca, & Floreano (2020) Science Robotics

☑️ Checkpoints

Tucking need base speed to activate

week10_lishawk

Agile Flight (week11)

Pfeiffer, Scaramuzza (2021) Human-piloted drone racing: Perception and control, RAL’21. PDF. Dataset.

Autonomous drone racing

P. Foehn et al., AlphaPilot: Autonomous Drone Racing, RSS 2020, Best System Paper Award. PDF YouTube

[1] Foehn, Scaramuzza, CPC: Complementary Progress Constraints for Time-Optimal Quadrotor Trajectories, arXiv preprint, 2020. PDF. Video.

Falanga, Foehn, Peng, Scaramuzza, PAMPC: Perception-Aware Model Predictive Cotrol, IROS18. PDF. Video. Open Source: https://github.com/uzh-rpg/rpg_quadrotor_mpc

Kaufmann et al., Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing, ICRA’19. PDF. Video Deployed to win the IROS Autonomous Drone Racing Competition, 2018. Video.

Drone acrobatics

Kaufmann, Loquercio, Ranftl, Mueller, Koltun, Scaramuzza, Deep Drone Acrobatics, RSS 2020. PDF. Video.Code Best Paper Award Honorable Mention

week11_tradition_learning_navigationpng

Low-latency sensing

Learning of flight controllers (week11)

challenge1: Architecture/Input and Output representation

challenge2: Data collection

  1. imitate and use data from cars and bicycles

    • already integrated into city environments
    • large publicly available datasets
    • no need for an expert drone pilot

DroNet: Learning to Fly by Driving, Loquercio et al., Robotics and Automation Letters 2018, PDF, Video, Code.

  1. Simulation to Reality Transfer -> Sim2Real: Domain Randomization for Drone Racing

    Deep Drone Racing with Domain Randomization

    Loquercio et al., Agile Autonomy: Learning High-Speed Flight in the Wild, Under Review

    • Transfer learning: Abstraction

      • use abstraction function to make Sim2Real more similar

      week11_transfer_learning

      • How to find the abstraction function

        week11_abstraction_function

    • Imitation learning by using current SOTA method (RRT , A*, MPC, LQR) as GT

Limitations of Imitation Learning -> RL

RL limitations

challenge3: Guarantee the platform's safety during training and testing

Loquercio et al., A General Framework for Uncertainty Estimation in Deep Learning, RA-L 2020

Takeaways

  1. Car-Driving Datasets or Simulation can be used to train deep navigation systems.
  2. Transfer knowledge via input and output abstractions.
  3. Measuring the networks’ uncertainty is necessary to guarantee safety.