Bringing Model Predictive Control to the Crazyflie with TinyMPC

2024-07-15 | Sam Schoedel | 1 Comment

Today we welcome Sam Schoedel and Khai Nguyen from Carnegie Mellon University. Enjoy!

We’re excited to share the research we’ve been doing on model-predictive control (MPC) for tiny robots! Our goal was to find a way to compress an MPC solver to a size that would fit on common microcontrollers like the Crazyflie’s STM32F405 while being fast enough to control the higher frequency dynamics of smaller robots. We came up with a few tricks to make that happen and dubbed the resulting solver TinyMPC. When it came time for hardware experiments, using the Crazyflie just made sense. A tiny solver deserves a tiny robot.

Motivation

Model predictive control is a powerful tool for controlling complex systems, but it is computationally expensive and thus often limited to use cases where the robot can either carry enough computational power or when offboard computing is available. The problem becomes challenging to solve for small robots, especially when we want to perform all of the computation onboard. Smaller robots have inherently faster dynamics which require higher frequency controllers to stabilize, and because of their size they don’t have the capacity to haul around as much computational power as their larger robot counterparts. The computers they can carry are often highly memory-constrained as well. Our question was “how can we shrink the computational complexity and memory costs of MPC down to the scale of tiny robots?”

What We Did

We settled on developing a convex model predictive control solver based on the alternating direction method of multipliers. Convex MPC solvers are limited to reasoning about linear dynamics (on top of any other convex constraints), but have structure that TinyMPC exploits to solve problems efficiently. The tricks we used to achieve this efficiency are described in the paper, but it boils down to rewriting the problem as a constrained linear-quadratic regulator to reduce the memory footprint and then precomputing as many matrices as possible offline so that online calculations are less expensive. These tricks allowed us to fit long-time horizon MPC problems on the Crazyflie and solve them fast enough for real-time use.

What TinyMPC Can Do

We decided to demonstrate the constraint-handling capabilities of TinyMPC by having the Crazyflie avoid a dynamic obstacle. We achieved this by re-computing hyperplane constraints (green planes in the first video) about a spherical obstacle (transparent white ball) for each knot point in the trajectory at every time step, and then by solving the problem with the new constraints assuming they stayed fixed for the duration of the solve.

In the two videos below, the reference trajectory used by the solver is just a hover position at the origin for every time step. Also, the path the robot takes in the real world will never be exactly the same as the trajectory computed by the solver, which can easily result in collisions. To avoid this, we inflated the end of the stick (and the simulated obstacle) to act as a keep-out region.

TinyMPC is restricted to reasoning about linear dynamics because of its convex formulation. However, a simple linearization can be taken pretty far. We experimented with recovering from different starting conditions to push the limits of our linear Crazyflie model and were able to successfully recover from a 90 degree angle while obeying the thrust commands for each motor.

We recently added support for second-order cone constraints as well. These types of constraints allow TinyMPC to reason about friction and thrust cones, for example, which means it can now intelligently control quadrupeds on slippery surfaces and land rockets. To clearly demonstrate the cone constraint, we took long exposure photos of the Crazyflie tracking a cylindrical landing trajectory without any cone constraints (red) and then with a spatial cone constraint that restricts the landing maneuver to a glide slope (blue).

How To Use TinyMPC

All of the information regarding the solver can be found on our website and GitHub org (which is where you can also find the main GitHub repository). TinyMPC currently has a Python wrapper that allows for validating the solver and generating C++ code to run on a robot, and we have a few examples in C++ if you don’t want to use Python. Our website also explains how to linearize your robot and has some examples for setting up the problem with a linear model, solving it an MPC loop, and then generating and running C++ code.

Most importantly to the Crazyflie community, our TinyMPC-integrated firmware is available and should work out of the box. Let us know if you use it and run into issues!

Our accompanying research papers:

Khai Nguyen, Sam Schoedel, Anoushka Alavilli, Brian Plancher, and Zachary Manchester. “TinyMPC: Model-Predictive Control on Resource-Constrained Microcontrollers.” arXiv preprint arXiv:2310.16985 (2023). https://arxiv.org/pdf/2310.16985

Sam Schoedel, Khai Nguyen, Elakhya Nedumaran, Brian Plancher, and Zachary Manchester. “Code Generation for Conic Model-Predictive Control on Microcontrollers with TinyMPC.” arXiv preprint arXiv:2403.18149 (2024). https://arxiv.org/pdf/2403.18149

We would love your feedback and suggestions, and let us know if you use TinyMPC for your tiny platforms!

Discover the Impact of Structural Defects on Nano-Quadcopter Performance: CrazyPAD dataset

2024-06-24 | Tagir Muslimov | Leave a comment

This week we have a guest blogpost by Kamil Masalimov (MSc) and Tagir Muslimov (PhD) of the Ufa University of Science and Technology. Enjoy!

As researchers passionate about UAV technology, we are excited to share our recent findings on how structural defects affect the performance of nano-quadcopters. Our study, titled “CrazyPAD: A Dataset for Assessing the Impact of Structural Defects on Nano-Quadcopter Performance,” offers comprehensive insights that could greatly benefit the Crazyflie community and the broader UAV industry.

The Motivation Behind Our Research

Understanding the nuances of how structural defects impact UAV performance is crucial for advancing the design, testing, and maintenance of these devices. Even minor imperfections can lead to significant changes in flight behavior, affecting stability, power consumption, and control responsiveness. Our goal was to create a robust dataset (CrazyPAD) that documents these effects and can be used for further research and development.

Key Findings from Our Study

We conducted a series of experiments by introducing various defects, such as added weights and propeller cuts (Figure 1), to nano-quadcopters. For the experiments, we used the Lighthouse Positioning System with two SteamVR 2.0 virtual reality stations (Figure 2).

**Figure 1. Propeller with two side defects**

**Figure 2. Schematic of the experimental setup with Lighthouse Positioning System**

Here are some of the pivotal findings from our research:

Stability Impact: We observed that both added weights and propeller cuts lead to noticeable changes in the stability of the quadcopter. Larger defects caused greater instability, emphasizing the importance of precise manufacturing and regular maintenance.
Increased Power Consumption: Our experiments showed that structural defects result in higher power consumption. This insight is vital for optimizing battery life and enhancing energy efficiency during flights.
Variable Control Responsiveness: We used the standard deviation of thrust commands as a measure of control responsiveness. The results indicated that defects increased the variability of control inputs, which could affect maneuverability and flight precision.
Changes in Roll and Pitch Rates: The study also highlighted variations in roll and pitch rates due to structural defects, providing a deeper understanding of how these imperfections impact flight dynamics.

We show Figure 3 as an example of a graph obtained from our dataset. In this figure, you can see the altitude and thrust command over time for different flight conditions. The blue line represents the normal flight, while the orange line represents the flight with additional weight near the M3 propeller. In Figure 4, you can see the 3D flight trajectory of the Crazyflie 2.1 quadcopter under the cut_propeller_M3_2mm condition with the corrected ideal path. The blue line represents the actual flight trajectory, while the red dashed line with markers represents the ideal trajectory. Figure 5 shows the Motor PWM values over time for the add_weight_W1_near_M3 condition. The plot shows the PWM values of each motor (M1, M2, M3, and M4) as they respond to the added weight near the M3 propeller.

More examples of graphs obtained from the CrazyPAD dataset can be found in our research paper specifically describing this dataset: https://doi.org/10.3390/data9060079

**Figure 3. Altitude and thrust command over time for different flight conditions**

**Figure 4. 3D flight trajectory of the Crazyflie 2.1**

**Figure 5. Motor PWM values over time**

Leveraging Research for Diagnostic and Predictive Models

One of the most exciting aspects of our research is its potential application in developing diagnostic and predictive models. The CrazyPAD dataset can be utilized to train machine learning algorithms that detect and predict structural defects in real-time. By analyzing flight data, these models can identify early signs of wear and tear, allowing for proactive maintenance and reducing the risk of in-flight failures.

Diagnostic models can continuously monitor the performance of a UAV, identifying anomalies and pinpointing potential defects. This real-time monitoring can significantly enhance the reliability and safety of UAV operations.

Predictive models can forecast future defects based on historical flight data. By anticipating when and where defects are likely to occur, these models can inform maintenance schedules, ensuring UAVs are serviced before issues become critical.

Why This Matters for the Crazyflie Community

The CrazyPAD dataset and our findings offer valuable resources for the Crazyflie community. By understanding how different defects affect flight performance, developers and enthusiasts can improve design protocols, enhance testing procedures, and ensure higher safety and performance standards for their UAVs.

We believe that sharing our research with the Crazyflie community can lead to significant advancements in UAV technology. The dataset we created is open under the MIT License for further exploration and can serve as a foundation for new innovations and improvements.

Get Involved and Explore Further

We invite community members to explore our full research article and the CrazyPAD dataset. Together, we can drive forward the standards of UAV technology, ensuring that Crazyflie remains at the forefront of innovation and excellence.

Our research paper with a detailed description of this dataset:

Masalimov, K.; Muslimov, T.; Kozlov, E.; Munasypov, R. CrazyPAD: A Dataset for Assessing the Impact of Structural Defects on Nano-Quadcopter Performance. Data 2024, 9, 79. ht tps://doi.org/10.3390/data9060079

Dataset: https://github.com/AerialRoboticsUUST/CrazyPAD

We are eager to collaborate with the Crazyflie community and welcome any feedback or questions regarding our research. Let’s work together to push the boundaries of what’s possible in UAV technology.

CrazySim: A Software-in-the-Loop Simulator for the Crazyflie

2024-04-15 | Christian Llanes | Leave a comment

This week we have a guest blogpost by Christian Llanes, a Robotics PhD of from Formal Methods & Autonomous Control of Transportation Systems Lab of the Georgia Institute of Technology. Enjoy!

Why do we need simulators?

Simulators are one of the most important tools used in robotics research. They usually are designed for different purposes with different levels of complexity. For example, simulators with low computational overhead that are parallelizable are mainly used for either training reinforcement learning algorithms or Monte Carlo sampling for verification of task completion in a nondeterministic environment. Some simulators also use rendering engines for the graphical display of models and the environment or when cameras are intended to be used in the robotics platform. Simulation is also useful for the development and deployment of new robotics firmware features where the firmware is compiled on a test machine and run in the loop with a simulated sensor suite. This simulator configuration is known as software-in-the-loop (SITL) because the vehicle firmware is intended to be run in the loop with the simulated vehicle physics and/or rendering engine. This feature is supported by autopilot suites such as PX4, ArduPilot, CogniPilot, and BetaFlight. This feature is not officially supported yet for Crazyflies because it requires a large overhaul of the firmware to be able to compile on a desktop machine and interact with different simulators such as Gazebo, Webots, PyBullet, CoppeliaSim, Isaac Sim, or Unreal Engine.

CrazySim

Last summer I began working with Crazyflies and noticed this Crazyflie simulator gap. I stumbled on a community-developed project for Crazyflie SITL called sim_cf. This project is exactly what I was looking for. However, the firmware used by the project is from July 2019 and the official firmware has had over 2000 commits made since then. The project also uses ROS 1, Gazebo Classic, and doesn’t support the Crazyflie Python library (CFLib). Using this project as a starting point I set out to develop CrazySim–a Crazyflie SITL project that doesn’t require ROS, uses Gazebo Sim, and supports connectivity through CFLib. Using CFLib we can connect the simulator to external software such as Crazyswarm2 or the Crazyflie ground station client. Users test their control algorithms in the external software using the simulator interface before deploying to real flight hardware.

Using the Crazyflie Client for PID Tuning

We have also provided a modified Crazyflie client for CrazySim support. The Crazyflie client is a cool tool for testing a single drone in hardware. We can perform command based flight control, look at real time plots, save log data, and tune PID values in real time. The PID values are typically tuned for an out of the box Crazyflie. However, when we modify the Crazyflie and add extra weight through batteries, decks, and upgraded thrust motors then the behavior of the Crazyflie will change. If a user wants to tune a custom Crazyflie setup, then they can add additional models in this folder with their own motor and mass properties. Then they just need to add it to the list of supported models in either of the launch scripts. There is already an example model for the thrust upgrade bundle. Documentation for installing the custom client can be found here.

PID tuning a simulated Crazyflie using CrazySim on the Crazyflie PC client.

Crazyswarm2

We can now connect to the simulated Crazyflie firmware using CFLib. Therefore, we can set up a ROS 2 interface through Crazyswarm2 for swarm command and control through ROS 2 topics and services. To do this we first startup the drones using any of the launch scripts.

bash tools/crazyflie-simulation/simulator_files/gazebo/launch/sitl_multiagent_square.sh -n 16 -m crazyflie

Then, we bring up Crazyswarm2 after setting up the configuration file for the number of drones chosen.

ros2 launch crazyflie launch.py backend:=cflib

We demonstrate an example of how we can control a swarm of drones using Crazyswarm2 GoTo service commands.

Crazyswarm2 GoTo service commands using CrazySim.

ICRA 2024

CrazySim is also being presented as a paper at the 2024 IEEE International Conference on Robotics and Automation in Yokohama, Japan. If you are attending this conference and are interested in this work, then I invite you to my presentation and let me know that you are coming from this blog post after. For the paper, I created a multi agent decentralized model predictive controller (MPC) case study on ROS 2 to demonstrate the CrazySim simulation to hardware deployment workflow. Simulating larger swarms with MPC may require a high performance computer. The simulations in this work were performed on an AMD Ryzen 9 5950X desktop processor.

Links

Other Crazyflie SITL projects:

LambdaFlight: Crazyflie Meets Haskell

2024-03-04 | Simon Levy | Leave a comment

Today we have a guest blogpost by Simon D. Levy (Washington and Lee University) about using Haskell to warp parts of the Crazylife-firmware. We also have some announcements from Bitcraze itself at the bottom. Enjoy!

As Richard Feynman famously said, What I cannot create, I do not understand. My less ambitious version of this motto is What I can translate into another language, I might better understand.

In order to better understand the Crazyflie firmware, I first undertook to translate the C-based firmware into C++. By separating out the general-purpose algorithmic code (state estimator, closed-loop control) from the Crazyflie-specific / Hardware Abstraction Layer (HAL) component of the firmware, I ended up with a nice, header-only C++ library of algorithms that can work with both a simulator like Webots and on a Crazyflie 2.1 or Crazyflie Bolt flight controller (with the remaining Crazyflie firmware translated into the standard C++ .h/cpp format).

As a fan of functional languages like OCAML and Haskell (and Rust, which was strongly influenced by their approach), I wondered whether I couldn’t push this idea further, to get an even higher-level implementation of the algorithms. Having played a bit with Rust and encountered issues getting it to work efficiently with C++ (something that is thankfully now being addressed), I thought it might be work trying NASA Copilot, a Haskell extension that compiles Haskell into C with a fixed memory footprint (i.e., no malloc() or garbage-collection).

My efforts paid off, resulting in a Haskell library of algorithms for closed-loop (PID) control and motor mixing that works with both my modified version of the Crazyflie firmware and with a simulator like Webots. For anyone wondering “why Haskell”, I refer you to this excellent discussion on the language’s advantages (purity, elegance), as well as this classic presentation on why the functional programming offers a better solution to complex tasks than object-oriented approach. For example, the code in the LambdaFlight core module cleanly reflects the dataflow (input/output) diagram for a standard flight controller:

Here is the Haskell code corresponding the component labeled Core in the diagram:

— Clock rate is 500Hz for Crazyflie, 100Hz for sim
dt = rateToPeriod clock_rate

— Make a list of PID controllers based on application order
pids = [positionPid resetPids inHoverMode dt,
    pitchRollAnglePid resetPids inHoverMode dt,
    pitchRollRatePid resetPids inHoverMode dt,
    altitudePid inHoverMode dt,
    climbRatePid inHoverMode dt,
    yawAnglePid dt,
    yawRatePid dt]

— Run PID controllers on open-loop (stick) demands to get demands for motor mixer
demands’ = foldl (\demand pid -> pid vehicleState demand) openLoopDemands pids

— Adjust thrust component for hover mode
thrust” = if inHoverMode then ((thrust demands’) * tscale + tbase) else tmin

— Run motor mixer on closed-loop demands to get motor spins, scaled for CF or sim
motors = quadCFMixer $ Demands thrust”
((roll demands’) * prscale)
((pitch demands’) * prscale)
((yaw demands’) * yscale)

By adjusting the values of the clock rate and scaling coefficients (tscale, tbase, prscale, …), the same PID controllers (with same Kp, Ki, Kd constants) can be used for both the simulation and the actual Crazyflie.

Two pieces complete the picture.

First, how can a pure functional language like Haskell, lacking mutation/side-effects, support an algorithm like PID control, which requires keeping state variables (error integral, previous error) across iterations? The answer to this question comes from an ingenious part of the NASA Copilot framework, namely, streams. A stream (which is similar to a lazy list in Haskell), can come from an module of the program written in C (for example, the vehicle state obtained by state estimation) or from a previous value passed into the function. This feature allows Copilot to have functions that possess state while still being “pure” in the sense of being amenable to formal verification (mathematical proof of correctness; see this thread for a discussion). Although I don’t have the mathematical background to do formal verification proofs, the ability to prove the code correct is an extremely powerful feature of languages like Haskell and is what led NASA to develop the Copilot framework for its space missions.

Second, how can Haskell be combined with C/C++ without running into the performance issues typically encountered in calling code in one language from code in another? As I alluded to before, NASA Copilot solves this issue nicely, by compiling the Haskell code to C code with a fixed memory footprint. Declaring certain C/C++ variables to be global makes them accessible to the Haskell code as streams, and causes them to be compiled into the resulting C code, which can then be compiled and linked to the Crazyflie (or simulator) C/C++ code in the usual way (thanks to the Crazyflie Makefile’s use of the Kbuild system). Hence, to compile the whole project into a form suitable for flashing with make cload, requires just the following chain of commands (where make links creates links to some external libraries I wrote for the sensors):

make cf2_defconfig && make links && make copilot && make -j 32

My current work focuses on two directions: first, I am translating Crazyflie’s Kalman filter into Haskell — a more ambitious undertaking, but one that I feel more confident in undertaking now that the core control algorithms are completed. Second, I am looking for ways of modifying the most popular robotics simulators (Webots, Gazbeo) to work with dynamics (physics engine) code custom-written for quadcopters and other aerial vehicles (like the dynamics code I wrote for this simulator), as well as faster control loops.

Announcements

There are some announcements that are not part of this guest blogpost that we’d like to share.

The 350 mAh batteries are out of stock and unfortunately have a very long lead time at our manufacturer, so you can expect them to be back in stock early May. Please take a look this blogpost which also compares this battery to the Tattu alternative, if you can’t wait and need to have a similar battery now.
We have a Bitcraze developer meeting this Wednesday to talk about the supervisor safety functionalities in the crazyflie-firmware. Please keep an eye on the GH discussion thread for information on how to join.

Simulation sim_cf2 for Multi-Agent Algorithm Development

2024-01-22 | Thomas Izycki | Leave a comment

Today we have a guest blogpost by Thomas Izycki (Technische Hochschule Augsburg) and Klaus Kefferpütz (Ingolstadt Technical University of Applied Sciences, former Augsburg Technical University of Applied Sciences) talking about implementing Software-in -the-Loop for swarm simulation of the Crazyflie.

When our cooperative control lab at the Technical University of Applied Sciences Augsburg was founded a few years ago, our goal was to develop distributed algorithms for teams of UAVs. We quickly decided to use Crazyflies with the algorithms directly implemented in the firmware, thus having a platform for a truly decentralized system. Our ongoing projects focus on cooperative path planning, navigation, and communication.

Since working with several drones at once, which have to communicate and coordinate with each other, can quickly become confusing and very time-consuming, a simulation was needed. It should preferably offer the possibility to integrate the firmware directly in the simulation environment and ideally also offer an interface to the crazyflie python API. With the relocation of our laboratory from Augsburg to the Technical University of Applied Sciences Ingolstadt, which however does not yet have a permanently established flying space, the need for a simulation environment was further increased in order to be less dependent on hardware accessibility. A look at the community and the available simulations quickly led us to the sim_cf flight simulator for the Crazyflie. A fantastic project supporting the use of the actual Crazyflie firmware in software-in-the-loop (SITL) mode and even in hardware-in-the-loop (HITL) mode on a real Crazyflie using the FreeRTOS Linux Port together with ROS and Gazebo. Unfortunately, the project has not been maintained for several years and also had no integration with the Crazyflie Python API. After a short chat with Franck Djeumou and his agreement to use the code of the original sim_cf simulation, the project was ready for an upgrade.

sim_cf2 Flight Simulator

With support for ROS coming to an end we decided to migrate the project to ROS2 and additionally support the current version of the Crazyflie firmware, which at this time is release version 2023.11. With the addition of a driver to connect the cflib to the SITL process, the same python cflib-based scripts can now be used with real Crazyflies and for use in the simulation environment. Only the corresponding driver needs to be loaded during initialization. Overall, the focus of the sim_cf2 simulation is now on using the Crazyflie python API instead of commanding the Crazyflies via ROS.

As for the Crazyflie firmware the whole build process has been fully integrated into the firmware’s KBuild build system. This also allows the use of the same code base for simulation and execution on the real Crazyflie. Depending on the configuration, the firmware is compiled for the STM32F on the Crazyflie or the host system running the SITL.

Components of the sim_cf2 Flight Simulator

To run the simulation, the three modules must therefore be started separately. Gazebo is started using the main launch file. The number and initial pose of the simulated Crazyflies is also defined in this file. Once the firmware for the host system has been built, the desired number of SITL instances can be started using the attached script. Finally, a cflib-based script, the Crazyflie client or the multi-agent client presented below is started. With no radio dongles attached to the computer, the simulation driver is initialized automatically and a connection to the simulated Crazyflie can be established.

There are still a few open issues, including the absence of implemented decks for positioning, such as LPS and Lighthouse. Currently, the absolute position is sent from Gazebo to the SITL instance and fused in the estimator. Moreover, Gazebo requires quite a lot of computing power. We were able to run a maximum of four Crazyflies simultaneously on a relatively old laptop. However, a modern desktop CPU with multiple cores allows for simulating a significantly larger number of Crazyflies.

Multi-Agent Client

Independent of the simulation we designed a GUI for controlling and monitoring our multi-agent teams. It currently supports up to eight Crazyflies but could be upgraded for bigger teams in the future. So far it has been enough for our requirements.

A central feature is the interactive map, which makes up about half of the gui. This is a 2D representation of the flight area with a coordinate system drawn in. Connected Crazyflies are displayed as small circles on the map and a new target position can be assigned by clicking on the map after they have been selected by their corresponding button. If required, obstacles or paths to be flown can also be drawn into the map.

An important aspect of a truly decentralized system is peer to peer communication. It allows information to be exchanged directly between agents and ideally takes place without a central entity. Currently peer to peer communication is not available in the Crazyflie ecosystem, but is in development.

For this reason, we have implemented a workaround in our client, enabling a pseudo-decentralized communication system. This involves adding an additional layer to the Crazy RealTime Protocol (CRTP), which we named the Multi-Agent Communication Protocol (MACP). It consists of an additional packet header made of the destination ID, source ID and a port as endpoint identifier. Every ID is unique and directly derived from the Crazyflie’s address.

These MACP packets are sent via the unused CRTP port 0x9. The packet routing mechanism implemented in the client forwards the packets to their destination. It is also possible to send packets as a broadcast or to address the client directly.

On the firmware side, we have added a corresponding interface to simplify the sending and receiving of macp packets. It is analogous to the CRTP implementation and allows the registration of callback functions or queues for incoming packets on corresponding ports. It can be activated via KBuild.

At least for use in the laboratory and the development of distributed algorithms, this method has proven its worth for us.

Project

We hope that the sim_cf2 simulation can also be useful to others. The complete source code is available on GitHub. Further information concerning installation and configuration can be found in the readme files in the respective repositories.

We would also like to point out that other simulations have been created that are based on sim_cf and therefore offer similar functionality. One of these projects is CrazySim, also available on GitHub. Moreover, there are ongoing efforts to officially integrate such a software-in-the-loop simulation into the Crazyflie firmware and ecosystem.

sim_cf2:

https://github.com/CrazyflieTHI/sim_cf2

Multi-Agent Client:

https://github.com/CrazyflieTHI/crazyflie-client-multi-agent-thi

Show-and-tell post of CrazySim:

https://github.com/orgs/bitcraze/discussions/995

For the original sim_cf Flight Simulator for the Crazyflie visit:

https://github.com/wuwushrek/sim_cf

Adaptive Cooperative Flight for Transportation using Crazyflies

2023-11-13 | Dimitrios Chaikalis | Leave a comment

Today, we welcome Dimitrios Chaikalis from New York University to talk about their project of cooperative flight. Enjoy!

For our work in cooperative flight, we developed controllers for many tightly coupled drones to fly as a unit. The idea is that, either in a centralized or decentralized manner, it should be possible to treat drones as thrust force and yaw moment modules, in order to allow many small drones to carry objects too heavy for a single one to lift.

It quickly turned out that the Crazyflies, with their small size, open-source firmware, ROS compatibility, and, as we happily found out after hours upon hours of crashes, amazing durability, would be the perfect platform to test our controllers.

We designed and 3D-printed very lightweight, hollow connecting rods that could latch onto Crazyflies on one side, along with a number of lightweight polygons such as squares and hexagons with housings for the other side of the rods on all their faces. This allowed us to seamlessly change between geometric configurations and test our controllers.

We first tested with some symmetric triangle and quad formations.

The above is probably literally the first time our cooperative configuration achieved full position control

The tests on quad-copter configurations started as we transitioned to fully modular designs

Eventually, to make the controller generic, we developed a simple script that could deduce with some accuracy the placement of drones given a small lexicographic description submitted by the tester as a string, essentially denoting a sequence of rods and polygons utilized in the current configuration. Of course, some parameters such as rod lengths, or additional weights that we added to the system (such as a piece of foam attached to the structure), could not be known in advance, but the adaptive controller design ensured that the overall system could still achieve stable flight.

Strangely, the L shape has become a sort of ‘staple’ configuration in cooperative load transportation

We also proved that with more than 3 drones in a configuration, we could optimize the thrusts of the agents such that additional performance criteria could be met. For example, in an asymmetric configuration of 5 drones, one of them had a significantly more depleted battery. Crazyflies provide real-time battery voltage feedback, so we were able to use that in an optimization node running in Matlab on a ground computer, choosing thrust levels such that the depleted agent could be utilized less. This was a significant help, because in many of those experiments, the Crazyflies had to operate at more than 80% of their thrust capacity, so battery life optimization was of the essence.

We used ROS for all the code written for the above implementations, using the Crazyflie-ROS package in order to get battery and IMU readings from all drones and provide thrust and roll, pitch, and yaw rate commands at up to 100Hz.

The corresponding publication can be found here: https://link.springer.com/article/10.1007/s10846-023-01842-1

In case you want to build on our work, you can cite the above paper as such:

D. Chaikalis, N. Evangeliou, A. Tzes, F. Khorrami, ‘Modular Multi-Copter Structure Control for Cooperative Aerial Cargo Transportation‘, Journal of Intelligent & Robotic Systems, 108(2), 31.

YouTube Link: https://www.youtube.com/watch?v=nA41uJIehH8&t=1s

Tired of crashing? Use Open Gimbal!

2023-10-09 | Suryansh Sharma | Leave a comment

Today, Suryansh Sharma from TU Delft presents the open-source Gimbal they devised. Enjoy!

Crazyflies (and other drones in this weight class) are extremely fun to fly and prototype with! But if you are also a scientist or tinkerer and not a well-skilled drone pilot then you might struggle with flying these platforms especially when testing new control loops or experimental code. While crashing also teaches a lot about the behavior of the system, sometimes we are interested in seeing the system dynamics without breaking the drone.

Currently, doing this for such small drones is not easy. We need something lightweight and still accessible. To solve this, we made Open Gimbal: a specially designed 3 degrees of freedom (DoF) platform that caters to the unique requirements of these tiny drones. We make two versions, (a) Tripod version which can be mounted on a camera / light tripod with a screw thread of sizes 1/4-20 UNC or 3/8-16 UNC (b) Desktop version which can be placed on a table top.

Our approach focuses on simplicity and accessibility. We developed an open-source, 3-D printable electro-mechanical design that has minimal size and low complexity. This design facilitates easy replication and customization, making it widely accessible to researchers and developers. The platform allows for unrestricted and free rotational motion, enabling comprehensive experimentation and evaluation. You can see the movement from the CAD version below:

*Degrees of Rotational freedom that Open Gimbal provides*

You can also check out the interactive CAD model and see how the gimbal moves here. All of the 3D model files as well as the BOM and instructions for assembly can be found in our repository here.

In our publication, we also address the challenges of sensing flight dynamics at a small scale. To do so, we have devised an integrated wireless batteryless sensor subsystem. Our innovative solution eliminates the need for complex wiring and instead uses wireless power transfer for sensor data reception. You can read all about how we do this in our paper here.

If you do end up using the platform for research then you can cite us using the details below:

@ARTICLE{10225720, author={Sharma, Suryansh and Dijkstra, Tristan and Prasad, Ranga Venkatesha}, journal={IEEE Sensors Letters}, title={Open Gimbal: A 3 Degrees of Freedom Open Source Sensing and Testing Platform for Nano- and Micro-UAVs}, year={2023}, volume={7}, number={9}, pages={1-4}, doi={10.1109/LSENS.2023.3307121}}

OpenGimbal at European Robotics Forum 2023 — Open Gimbal at European Robotics Forum 2023 in Denmark

Open Gimbal at European Robotics Forum 2023 in Denmark

I hope that you find the Open Gimbal useful! Feel free to reach out to me at Suryansh.Sharma@tudelft.nl if you have any ideas/questions or if you end up making an Open Gimbal yourself!

AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms

2023-08-21 | Vivek | Leave a comment

Today, Vivek Adajania from Learning Systems and Robotics lab write about a project for a safe motion planning of Crazyflie swarm that was published at ICRA 2023. Enjoy!

Motivation

Quadrotor swarms offer significant potential in applications like search and rescue, environmental mapping, and payload transport due to their flexibility and robustness compared to single quadrotors. The core challenge in these applications is collision-free and kinematically feasible trajectory planning. As the quadrotors share space, they must safely manoeuvre around each other and avoid collisions with static obstacles. Existing solutions [1] [2], while effective for generating collision-free trajectories, often struggle in densely cluttered scenarios due to simplifying approximations.

Background

There are two literature groups in the domain of optimization-based quadrotor swarm motion planning: centralized and distributed approaches. In a centralized setup, a central computer solves a joint optimization problem that computes trajectories for all quadrotors at once. These approaches have broad solution space but quickly become computationally intractable as the number of quadrotors increases. On the other hand, the distributed approach involves each quadrotor independently solving its optimization problem and incorporating trajectories shared by the neighbouring quadrotors. This strategy offers improved scalability, yet existing distributed approaches struggle in cluttered environments.

Fig. Centralized and distributed planning approach to quadrotor swarm motion planning. The arrows indicate the flow of communication.

In this work, we adopt a distributed planning strategy. The independent optimization problem that needs to be solved by each of the quadrotors in the distributed setup is a non-convex quadratically constrained quadratic program (QCQP). This nature of the problem stems from non-convex and quadratic collision avoidance constraints and kinematic constraints.

Existing distributed approaches rely on sequential convex programming (SCP) that performs conservative approximations to obtain a quadratic program (QP). First, linearization of the collision avoidance constraints to obtain affine hyperplane constraints. Second, axis-wise decoupling of the kinematic constraints to obtain affine box constraints. We obtain a QP but with small feasible sets.

Fig. Conservative approximations made by Sequential Convex Programming (SCP) based approaches.

Proposed Approach

In contrast, our proposed approach obtains a QP without relying on the previously mentioned approximations. The first ingredient is the polar reformulation of collision avoidance and kinematic constraints. An example of the 2D polar reformulation of collision avoidance constraints is shown below:

Fig. Example illustration of polar reformulation of 2D collision avoidance constraints.

The second ingredient is to relax the reformulated constraints as l-2 penalties into the cost function and apply Alternating Minimization. Alternating Minimization results in subproblems that are convex QPs, and some have closed-form solutions, thus obtaining a QP form without relying on linearization; further details can be found in our paper [3]. We can also use and reformulate alternative collision avoidance constraints, barrier function (BF) constraints

where h_ij is the Euclidean distance between quadrotor i and quadrotor j, and the parameter γ controls how fast the quadrotor i is allowed to approach the boundary of quadrotor j.

Results

We experimentally demonstrate our approach on a 12 Crazyflie 2.0 swarm testbed in challenging scenes: obstacle-free, obstacle-rich, shared workspace with a human. The experimental video is provided below:

In the simulation, we compare our approach against two SCP approaches: SCP (Continuous) [2] enforces constraints across the entire horizon, while SCP (On-demand) [1] enforces only on the first predicted collision. Our (Axiswise) includes box kinematic constraints, while Our (Quadratic) preserves the original quadratic constraints.

From our simulation results, we see that SCP (On-demand) has a lower compute time than SCP (Continuous), as SCP (On-demand) enforces fewer constraints. But, this compute time trend comes at the expense of success rate. On the contrary, our approaches achieve a high success rate with low compute times. Ours (Quadratic) has a slightly higher success rate than Ours (Axiswise) as it has access to large kinematic bounds.

Fig. Simulation results from 100 start-goal configurations with swarm sizes ranging from 10 to 50 in a cluttered environment with 16 cylindrical static obstacles.

Fig. Simulation results from 100 start-goal configurations with swarm sizes ranging from 10 to 50 and three different γvalues in a cluttered environment with 16 cylindrical static obstacles.

On average, our approaches achieved a 72% success rate improvement, a 36% reduction in mission time, and 42x faster per-agent computation time—our approach trades-off mission time with inter-agent clearance and distance to obstacles via BF constraints.

Outlook

In this work, we presented an online and scalable trajectory planning algorithm for quadrotor swarms in cluttered environments that do not rely on the linearization of collision avoidance constraints and axis-wise decoupling of kinematic constraints. We do so by reformulating the quadratic constraints to a polar form and applying alternating minimization to the resulting problem. Consequently, our planner achieves high scalability and low computation times than existing approaches. We also show that we can reformulate barrier function constraints to introduce safety behaviours in the swarm. One of the future works is to extend the approach to navigate the swarm in a complex 3D environment.

References

[1] Luis, Carlos E., Marijan Vukosavljev, and Angela P. Schoellig. “Online trajectory generation with distributed model predictive control for multi-robot motion planning.” IEEE Robotics and Automation Letters 5.2 (2020): 604-611.

[2] E. Soria, F. Schiano and D. Floreano, “Distributed Predictive Drone Swarms in Cluttered Environments,” in IEEE Robotics and Automation Letters, vol. 7, no. 1, pp. 73-80, Jan. 2022, doi: 10.1109/LRA.2021.3118091.

[3] V. K. Adajania, S. Zhou, A. K. Singh and A. P. Schoellig, “AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments,” 2023 IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023, pp. 1421-1427, doi: 10.1109/ICRA48891.2023.10161063.

Links

Video: https://youtu.be/eIBcOKq5_Jk
Paper: https://arxiv.org/abs/2303.04856
Code: https://github.com/utiasDSL/AMSwarm
Our Lab: http://www.learnsyslab.org

The authors are with the Learning Systems and Robotics Lab at the University of Toronto and the Technical University of Munich. The authors are also affiliated with the Vector Institute for Artificial Intelligence and the University of Toronto Robotics Institute (RI) in Canada and the Munich Institute of Robotics and Machine Intelligence (MIRMI) in Germany.

Feel free to contact us with any questions or ideas: vivek.adajania@robotics.utias.utoronto.ca. Please cite this as:

@INPROCEEDINGS{
adajania2023amswarm, 
author={Adajania, Vivek K. and Zhou, Siqi and Singh, Arun Kumar and Schoellig, Angela P.}, 
booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)}, 
title={AMSwarm: An Alternating Minimization Approach for Safe Motion Planning of Quadrotor Swarms in Cluttered Environments}, 
year={2023}, 
pages={1421-1427}, 
doi={10.1109/ICRA48891.2023.10161063} 
}

High school students learn about electrical engineering with the Crazyflie

2023-07-31 | Hanna Müller | Leave a comment

This early summer my research group (Center for Project-Based Learning at ETH Zürich) was in charge of a special week – high school students from all over Switzerland (actually even the world, they had to speak German though) could apply for a study week at different departments from our university. The departments which joined this initiative were mathematics, physics, biology, environmental sciences, material sciences, and our department, electrical engineering and information technologies (ITET). But how do you show teenagers between 15-19 in one week as much as possible from electrical engineering while also having fun? And best inspire them to study at ITET? Our solution was: drones. More specifically, Crazyflies. With those we had many possibilities to learn about electrical engineering – from sensors, microcontrollers, timers, and motors to LEDs, batteries, embedded systems, FreeRTOS tasks, state estimation, and controller – and all this with a high fun potential and a low risk of accidents, as with their weight of only 30g they hardly ever do any damage. In this blog post, I will guide you through our week, in hopes to help others who also want to use the Crazyflie to teach students about electrical engineering in a fun way.

Monday

We started in the afternoon (in the morning they had a welcoming tour) with a short introduction and splitting the 20 students into groups of two (everyone got a paper slip and had to find the matching one, accelerometer gyroscope, pitch roll, UART SPI, and so on – this gives the lecturer a great opportunity for interaction with the students later on, once their word gets relevant during the week). After a short introduction to programming and microcontrollers we moved on to the most classic beginner task: blink an LED! We chose to use the front left one, as this one is only used when communicating – so as long as we don’t connect to the drone we can observe exactly what we programmed. Most students got the LED turned on rather quickly – however, pulsing the LED to change the intensity took them some more time and forced them to learn how to write loops. They also already learned how PWM works without knowing it yet – setting the intensity of an LED or the strength of a motor is about the same thing in the end after all and this gave us a great start for Tuesday.

Tuesday

On Tuesday we looked at hardware from different perspectives. As you might have guessed, we looked at motors and how to control them with PWM and timers. The students were a bit disappointed that we still didn’t fly, but as soon as they realized that they could play their favorite song on the motors the motivation was high again! We didn’t even have any stray drones, even though we let them mount the propellers (the songs sound much better with propellers).
We also looked at another aspect of electrical engineering: PCB design (this was already done though) and soldering. For this, we prepared a custom deck, with four colored LEDs, which could be populated like in industry with solder paste and then soldered with a hot plate. To make it even more fun (and partly to show off our laser cutter) they also designed a small plastic diffuser that could be mounted on top. So in the end our setup resembles the LED ring – however, it can be mounted on top which is essential if you don’t want to fly with a positioning system (and therefore need to mount a flow deck).

Wednesday

Now that we knew how to blink LEDs and, even more important, how to control motors we had to learn a bit about how the drone actually figures out which motor should be turned on and how much. For this, we first looked at sensors and wired communication protocols, such as I2C (for the IMU and time-of-flight sensor), SPI (for the optical flow), and UART (to the nRF) – due to limited time we didn’t go into details here though. We briefly touched wireless communication, to explain how the commands they will later send to the drone (and the firmware) are actually sent to the right drone.

We moved on to what state estimation is in general – again jumping over all details of an extended Kalman Filter, but had a closer look at the logging and parameter system. We then spent a bit more time on the PID controller again – which was also a bit hard to explain, as half the teenagers hadn’t learned how to integrate and differentiate yet. However, they learned fast and we could move to the part they waited for all week long: Flying (and tuning the PID).

Thursday

This was the day that was meant for creativity – the students could choose themselves which project they want to achieve. We proposed them some ideas, such as blinking LEDs depending on the height, flying through a gate (the challenge here is to filter the height measurements when the gate border is below the drone), steering the drone with the keyboard, soldering an own sensor on a break-out board, …

In the end, we saw many cool projects, from a song played as a canon on multiple motors to a transporter drone, flying successfully through the gate, doing a successful looping (unfortunately no successful landing yet…), racing against each other (possibly with disco-lights on the deck) and trials for reaching max speed in the hallway.

The hallway was very popular to fly, as the distance to accelerate was longer…

The most popular base project was to steer the drone with the keyboard – unfortunately (or fortunately? they sure had fun with it once it was running and they might have learned enough in the remaining week…), this was very easy after we showed them where Marcus’ script lives (here) and which 8 lines (170-177) they have to remove for it to work without an AI-deck (and don’t forget to adapt the URI)…

Friday

Flying was forbidden – but playing music with motors was not!

On Friday it was presentation day – in the morning they could still work on their projects, but in the afternoon all the 120 students (and most of their parents) came together in a huge lecture hall to present what they did during this week. And, as at a real conference, they had posters and their drones (which we, unfortunately, were not allowed to fly without a fireproof net… Will organize this next time) to show their projects to family, friends, and even random tourists (the entrance hall of the ETH main building is on many sightseeing tours).

At the end of the week I doubted the robustness of Crazyflies for a moment – however, Monday morning once I had peace and quiet once again I figured out what was wrong with all hardware which ended up on the “not working for unknown reasons” stack in less than an hour (and fixed almost all of it). Notes to all others and my future self for the next time I give 10 drones to 20 teenagers:

If you show them how to tune a PID, also explain that “persistent” means exactly what it says – if you mess up the PID values and persist them they will stay this way until you reset them, no matter how often you reflash the drone.
Explain how fragile connectors are and that you NEVER should pull at cables. Also, mention 10x more to be careful when plugging in decks. And radios.
Keep one “private” drone no one is allowed to mess with – it will help greatly to figure out if they only broke the flow deck connectors or something more serious (which actually never happened)
Doing only warm boots with setting individual addresses with the CLOAD_CMDS while flashing saved us a lot of trouble, randomly connecting to drones only happened once they discovered the app for the phone…

The coding tasks (and at least some minimal solutions) can be found on my fork: Tasks and solutions. They are kept short on purpose – we at the Center for Project-Based Learning believe in our name – we believe the most learning (and fun) happens when you rather freely explore what you can do with the basic tools you just learned.

P.S. For completeness – I cut out all the parts which really had nothing to do with Crazyflies, we also did lab tours in the high-voltage laboratory and the laboratory for optical communication – and of course had some social events with actual university students. As much as we like the Crazyflie, even we have to admit that the field of electrical engineering is even bigger than what we can show with those tiny drones ;)

TU Delft Science Centre Swarming Lab

2023-07-10 | Lennart | 5 Comments

Today, Lennart Bult from Emergent Swarns presents us with this project of a 24/7 swarming demo. Enjoy!

Over the last few months our team has been working on creating a 24/7 swarming demo. Initially tasked by Guido de Croon and Chris Verhoeven from TU Delft MAVLab and the TU Delft Robotics Institute, we set out to find our way within the Crazyflie ecosystem to gradually increase the size and capabilities of the swarm. In this article we will first talk about some of the work and methods that we used. After that, we will introduce the TU Delft Science Centre Swarming Lab and talk about some applications of swarming drones.

Developing the 24/7 swarm

The project started in February with the goal of creating a physical swarm capable of real-time collision avoidance with drones and static obstacles. We started out with three drones equipped with the Flow Deck, and by setting them up in a clever way we could perform the first collision avoidance and landing tests. We were impressed with the performance we got out of the Flow Deck, however, eventually, it is mostly a battle against the drift of the position estimate, that is, we could increase some of the margins on the collision avoidance only so far before we would either fly out of the test zone or collide with another drone. Luckily with short test flights, we were able to see some of the flaws in our algorithms and correct them before testing with the new setup.

*Setup after the first expansion to eight drones.*

After a few weeks of testing we got approved for the first swarm expansion, five more drones and a Lighthouse positioning setup. This is when we could do our first real tests with the collision avoidance algorithm, which, much to our own surprise, worked on the first try. This is also when we first posted a project update on LinkedIn. There were however a lot of bugs that still needed to be worked out, and a lot of system experience still to be gained. After flying for a bit longer we noticed that some of the drones would flip quite often, which is when we discovered that we needed the thrust upgrade to control the additional weight of the larger battery and charging deck.

For the charging setup we took inspiration from the Bitcraze IROS 2022 demo; we 3D printed sloped landing pads that we tape onto a wireless charger. After a few iterations we landed on a design that uses minimal printer resources and allows the Crazyflie to land a bit off-center. This last feature turned out to be quite useful considering the large amount of destabilizing airflow that is generated by 40 drones. After receiving the last order of drones we also expanded the charging setup, which at this point takes up quite a bit of floor space. There are some ideas to create a vertical landing pad stack, which would bring the additional challenge of missing the landing pad not being an option.

*All 40 drones recharging before their next flight.*

After prototyping the charging setup and building confidence with the initial setup, we were confident enough in our system capabilities to expand it to the point where a continuous demo of 5-8 drones is possible. Although the system integration of the previous expansion went without much trouble, we did encounter a few issues when expanding to 40 drones. The first issue of which was radio communication, we noticed that a delay in the radio communication would be present if we increased the update rate above a certain level for a specific number of drones per radio. The second issue we encountered were performance drops related to the violation of certain bounds in the collision avoidance algorithm. These two issues were very difficult to debug since it was not immediately obvious where the source of the issue was.

The third and last major issue was the increase in destabilizing airflow of 40 drones compared to 8. With 40 drones there is a noticeable breeze when you stand next to the drone cage, which is nice for summertime, but not so nice when drones need to land in a tight-packed configuration. To combat this issue there is a limit to the amount of drones that can land at the same time. There is also a minimum separation distance between two active landing pads, which reduced the severity of the induced turbulence. There are still ongoing efforts to increase the landing success rate, which is currently affected by drones running out of power during the landing procedure.

To control and monitor the swarm we designed a custom GUI, an impression of which you can see below. Although some of the buttons are still a work in progress, there are a lot of features that have already proven very useful, especially when testing a new feature.

V1 of the graphical user interface developed for the 24/7 swarm.

The code base that we created for the swarm will be largely open-sourced (only the collision avoidance will not be open-source) to provide researchers all around the world with the possibility to setup their own Crazyflie swarm for research. You can find the repository through this link. Note that the documentation and code base are still under development and might contain bugs/errors.

Human interaction

After creating all functionality to provide a continuously operating swarm demo, it was time to work on some of our stretch goals: 1. walking through the swarm whilst it is operating and 2. controlling the swarm using our arms. In the image below you can see an impression of precisely this functionality. The drones are following the operator’s gesture commands whilst performing live collision avoidance with an operator.

*Team member Seppe directing the 40 drone swarm, see the full video here.*

This demo requires multiple techniques and hardware elements working together to create a relatively low-latency, human-controlled swarm. We used a Kinect-like 3D sensor to perform human pose estimation, we subsequently used this data to create a dynamic obstacle in our collision avoidance software. An important element to consider here is the synchronization of the Lighthouse- and 3D sensor coordinate frames, i.e. without proper calibration the human will not be correctly positioned with respect to the drones and the drones will crash into the human. The interaction between the swarm control software and the human gesture commands also requires careful consideration, proper tuning is required to ensure a responsive system that is reliable and not too aggressive.

TUD Science Centre Swarming Lab

The next step in this project will be to set up the swarm at its new location, the TU Delft Science Centre. Here, the swarm will first and foremost be visible as a public demo, showcasing the capabilities of TU Delft state-of-the-art swarming research. There will also be a focus on developing the swarm as a research platform. This will allow TU Delft students and researchers to extend swarm functionalities and test their theory on a physical swarming system. Besides demos and academic research, there will also be worked on developing educational applications across the full educational board (primary school, high school and applied education). If you are interested in working on, or collaborating with the swarming lab on any of the above-mentioned tasks, feel free to email the lab management at operations.swarminglab@tudelft.nl.

*The TU Delft Swarming Lab setup with 40 drones and charging pads for continuous operations and research.*

Applications of Swarming

There are a lot of potential use-cases for fully autonomous drone swarms, ranging from indoor applications such as warehouse monitoring and factory inspection to outdoor applications such as search and rescue and surveillance. In our opinion, the true potential of drone swarms lies in applications where there is a significant need for a scalable system with a lot of built-in redundancy. A lot of additional use cases open up when we consider fully onboard autonomous systems, where the full benefits of decentralized swarming can be utilized. Currently, the size of drones needed to achieve such feats is quite large, though maybe in a few years, we could see more and more being done on drone platforms such as the Crazyflie.

*A swarm inspection of an F-16 Fighting Falcon at Deltion College in Zwolle, the Netherlands.*

An interesting area of application for drone swarms could be in the inspection of aircraft. Drone swarms provide a scalable and flexible means to perform a fast inspection of aircraft across an entire airfield or military base. To showcase that this can be done with any size of drone, we went to Deltion College in Zwolle to perform a mock inspection of an F-16 fighter jet. Above you can see an impression of the inspection. Another area of application is search and rescue, where there is a need for systems that can find people or objects of interest in unknown and cluttered environments. Furthermore, the area that needs to be searched is usually very large and sometimes difficult to travel on foot. A drone swarm could provide fast and reliable coverage of the area of interest, whilst providing full data traceability. Seppe and Lennart will work on creating drone swarms for these use cases with the start-up Emergent Swarms.