Flapper Drones – bioinspired robots powered by Crazyflie Bolt

2022-07-11 | Matěj Karásek | Leave a comment

As the Crazyflie ecosystem expands, more and more novel aerial (but also ground or hybrid) robots are being built with one of the Crazyflie controllers onboard. For recent examples, you can check e.g. the recent blogpost about ICRA 2022.

In this post, I will introduce yet another Crazyflie-Bolt-powered aerial robot, the Flapper Nimble+ from our company Flapper Drones, which unlike other flying robots doesn’t have any propellers but uses flapping wings instead.

The best aerial robot design is…

Small drones, or micro air vehicles, have seen a lot of progress and new developments in the last 20 years. The most widespread design nowadays is a quadcopter, such as the Crazyflie 2.1. But is a quadcopter the ultimate (micro) drone solution? At Flapper Drones, we believe nature might provide even better designs… For some applications at least! 😊

Flying like a bird…

Flapper Drones is a spinoff of the MAVLab of the Delft University of Technology. At the MAVLab, we have been researching bio-inspired flight as part of the DelFly project since 2005. From the beginning, the goal has been to develop a lightweight, mission capable micro air vehicle, the design of which would draw inspiration from nature. Over the years, many such MAV concepts have been designed, built and tested, including the DelFly Micro, the world’s smallest camera-equipped MAV, or the DelFly Explorer, the first autonomous flapping-wing MAV equipped with a stereovision system. All these designs were propelled by a pair of flapping wings, while being controlled (and passively stabilized) by a tail such as birds or men-made airplanes.

… or an insect

The latest design, the DelFly Nimble is insect-inspired instead. What does that mean? The Nimble has no tail, which would provide the damping needed for stable flight. Instead, it is stabilized actively, by adjustments of the motion of its flapping wings. This is what all flying insects and also hummingbirds do. Flies, for example, sense their body motions with their halteres, drum-stick like biological gyroscopes, and adapt their wing motion accordingly to stay balanced…. or to be agile, when someone is trying to swat them!

And while the Nimble was originally built just to demonstrate that an insect-inspired flying robot can be built, eventually we could also use it to learn more about the flight of insects:

Flapper Drones – how do they work?

The Flapper Nimble+ is the commercial (and enlarged) version of the DelFly Nimble, developed and produced by Flapper Drones. To our knowledge, it is the first, and so far the only hover-capable tailless flapping-wing drone available!

The thrust keeping the Nimble+ airborne is created by its four flapping wings, which flap back and forth horizontally, about 10 to 12 times per second.

The wing actuation mechanism allows to adjust the flapping frequency of the left and right wing pairs independently, which enables control of the roll rotation. Pitch rotation is controlled by adjusting the mean wing position within the stroke plane, which shifts the mean thrust force forward or backward with respect to the center of mass, and also introduces a stabilizing dihedral angle in forward flight. Finally, yawing motion is achieved by tilting the wing roots of the left and right wing pair asymmetrically:

Advantages of flapping wings

The use of flapping-wing drones such as the Flapper Nimble+ brings several advantages. Next to their attractive biological appearance, the soft flapping wings produce less intrusive, low frequency sound and are safer, compared to propellers. As the wings move back and forth, minor mid-air collisions are not a problem. The wings bounce off objects leaving no damage, and the drone keeps flying as this only represents a minor disturbance:

The aerial drag characteristic is also different and helps with precise indoor flight. As soon as zero attitude is commanded, the Nimble+ goes into halt in a matter of several wingbeats, making it an ideal choice for novice drone pilots as well as in constrained or cluttered indoor spaces. Finally, the flapping wings can provide additional lift force as they also glide in forward flight. This can improve the power efficiency by over 20%, compared to hovering.

Otherwise, Flapper Drones can be operated as any other drone, with vertical take offs and landings, quick maneuvers and flight in any direction:

Crazyflie Bolt & compatibility

The Flapper Nimble+ is powered by the Crazyflie Bolt 1.1, where the Bolt’s BMI088 IMU and STM32F4 MCU are suitable substitutes to the halteres and brains of the real fly. We made this choice, because this enables compatibility with most of the Crazyflie ecosystem, but also, because we felt the only way a Crazyflie would do justice to its name is if it had flapping wings😊

Currently, the Nimble+ uses a fork of the Crazyflie firmware, which is of course open source. Moreover, with the recently introduced platform functionality, we will be able to include the Flapper platform into the official crazyflie firmware very soon (expected still in July 2022). This means that the Flapper remains compatible with the official Python libraries, the PC client or the smartphone app. But also third-party projects like the Crazyswarm or the Skybrush should only require minor adjustments, if any, to operate a swarm of Flappers. Thus, for the existing Crazyflie users, switching from a Crazyflie to the Flapper should be a breeze!

The Flapper Nimble+ is hardware compatible with most of the Crazyflie expansion decks. While software support remains experimental (the Flapper Nimble+ is not a native Crazyflie product, after all), many of the decks work out of the box and others might need just minor firmware modifications. Would you like to fly the Nimble+ autonomously? Add an LPS or Lighthouse deck and you’re good to go!

For more details regarding deck compatibility, you can check this overview.

Applications

While the Nimble+ was originally designed for drone shows and similar entertainment applications, the open-source firmware and expansion decks enabling autonomous flight make it ideal also as for academic research and, in general, as a development platform. Are you researching swarming, and would you like to make your swarm even more bio-inspired? Are you developing new sensors, or new controllers (possibly even bioinspired), which you would like to test on a new type of flying platform? Are you interested in the aerodynamics of flapping wings, or the flight dynamics of insect-like flight? Or are you just curious and would you like to learn more about bioinspired flight? In all these cases, a Flapper might be what you are looking for!

The 114-g and 49-cm wide Flapper Nimble+ has been designed as a modular system where any part can easily be replaced. Flapper Drones provides all the spares, which are available upon request. If you are interested in using the Nimble+ for entertainment, rather than research, you can modify the appearance by creating your own body shells, which can also be illuminated by RGB Leds (a suitable interface and power supply is already integrated). Or even by altering the design of the wings. Finally, you can easily extend the Flapper with your own sensors, or other devices. Would you like to add a tail? A gripper? A perching device? This is all possible, as long as these additions fit within the payload limit of about 25 grams.

Available soon in the webstore!

Did you get (bio)inspired, and would you like to try an insect-like flying robot yourself? Then we have some good news! The Flapper Nimble+ will soon be available for sale in an exclusive partnership with Bitcraze and their webstore. Checkout the product description and leave your email address behind, such that you get a notification when the Flappers are in stock and ready to ship. The first batch of 10 units is expected to be available at the end of summer, so do not wait too long 😉

Want to learn more?

To learn more about Flapper Drones, you can check our website, or watch the talk I gave at the last miniBAM:

Updates on Simulation work

2022-03-14 | Kimberly McGuire | Leave a comment

In December we had a blogpost where we gave an overview of existing simulation models that were out there. In the mean time, I have done some work during my Fun Fridays to get this to work even further. Currently I moved the efforts from my personal Github repo to the Bitcraze organization github called crazyflie-simulation. It is all still very much work in progress but in this blogpost I will explain the content of the repository and what these elements can already do.

Low Poly CAD model

The first thing that you will need to have for any simulation, is a 3D model of the Crazyflie. There is of course already great models available from the CrazyS project, the sim_cf project and the multi_uav_simulator, which are completely fine to use as well. But since we have direct access to the exact geometries of the real crazyflie itself, I wanted to see if I could abstract the shapes myself. And also I would like to improve my Blender skills, so this seemed to be a nice project to work with! Moreover, it might be handy to have a central place if anybody is looking for a 3D simulation model of the Crazyflie.

For simulations with only one or a few Crazyflie, the higher resolution models from the other repository are absolutely sufficient, especially if you are not using a very complicated physics geometry model (because that is where most of the computation is). But if you would like to simulate very big swarms, then the polygon count will have more influences on the speed of the simulation. So I managed to make it to 1970 vertices with the below Crazyflie model, which is not too bad! I am sure that we can make it even with lesser polygons but this is perhaps a good place to start out with for now.

In the crazyflie-simulation, you can find the Blender, stl files and collada files under the folder ‘meshes’.

Webots model

We implemented the above model in a Webots simulator, which was much easier to implement than I thought! The tutorials they provide are great so I was able to get the model flying within a day or two. By combining the propeller node and rotational motor, and adjusting the thrust and drag coefficient to be a bit more ‘Crazyflie like’, it was able to take off. It would be nice to perhaps base these coefficients on the system identification of the Crazyflie, like what was done for this bachelor thesis, but for now our goal is just to make it fly!

The webots model can found in the same simulation repository under /webots/. You can try out the model by

webots webots/world/crazyfly_world.wbt

It would then be possible to control the pitch and roll with the arrow keys of your keyboard while it is maintaining a current height of 1 meter. This is current state of the code as of commit 79640a.

Ignition Gazebo model

Ignition will be the replacement for Gazebo Classic, which is already a well known simulator for many of you. Writing controllers and plugins is slightly more challenging as it is only in C++ but it is such a landmark in the world of simulation, it only makes sense that we will try to make a Gazebo model of the Crazyflie as well! In the previous blogpost I mentioned that I already experimented a bit with Ignition Gazebo, as it has the nice multicopter motor model plugin standard within the framework now. Then I tried to make it controllable with the intergrated multicopter velocity control plugin but I wasn’t super successful, probably because I didn’t have the right coefficients and gains! I will rekindle these efforts another time, but if anybody would like to try that out, please do so!

First I made my own controller plugin for the gazebo model, which can be found in the repository in a different branch under /gazebo-ignition/. This controller plugin needs to be built first and it’s bin file added to the path IGN_GAZEBO_SYSTEM_PLUGIN_PATH, and the Crazyflie model in IGN_GAZEBO_RESOURCE_PATH , but then if you try to fly the model with the following:

 ign gazebo crazyflie_world.sdfCode language: CSS (css)

It will take off and hover nicely. Unfortunately, if you try out the key publisher widget with the arrow keys, you see that the Crazyflie immediately crashes. So there is still something fishy there! Please check out the issue list of the repo to check the state on that.

Controllers

So the reason why I made my own controller plugins for the above mentioned simulation models, is that I want to experiment with a way that we can separate the crazyflie firmware controllers, make a code wrapper for them, and use those controllers directly in the simulator. So this way it will become a hybrid software in the loop without having to compile the entire firmware that contains all kinds of extra things that the simulation probably does not need. We can’t do this hybrid SITL yet, but at least it would be nice to have the elements in place to make it possible.

Currently I’m only experimenting with a simple fixed height and attitude PID controller written in C, and some extra files to make it possible to make a python wrapper for those. The C-controller itself you can try out in Webots as of this commit 79640a, but hopefully we will have the python version of it working too.

What is next?

As you probably noticed, the simulation work are still very much work in process and there is still a lot enhancements to add or fix. Currently this is only done on available Fridays so the progress is not super fast unfortunately, but at least there is one model flying.

Some other elements that we would like to work on:

Velocity controller, so that the models are able to react on twist messages.
Crazyflie firmware bindings of controllers
Better system variables (at least so that the ign gazebo model and the webots model are more similar)
CFlib integration
Add a multiranger and/or camera.
and more!

I might turn a couple of these into topics that would be good for contribution, so that any community members can help out with. Please keep an eye on the issue list, and we are communicating on the Crazyswarm2 Discussion page about simulations if you want to share your thoughts on this as well.

A Modified EKF for Constant Altitude Flights

2022-01-10 | Dominik | Leave a comment

Today we will have a guest blogpost by Dominik Natter, working in the Robotics & Control group at SINTEF in Trondheim, Norway. Enjoy!!

In this blogpost we will teach you how to fly the Crazyflie beyond edges without crashing, using only on-board sensors. Come join in!

flying over edges — Safe flights across edges are achievable!

Introduction

UAVs have seen tremendous progress in the last decades and have since moved from research labs to various real-world environments. Small UAVs (so-called micro air vehicles, MAVs) like the Crazyflie open up even more possibilities. For example, their size allows them to traverse narrow passages or fly in cluttered environments (as recently showcased in this blog post). However, in order to achieve these complex tasks the community must further improve the cognitive ability of these MAVs in order to avoid crashes.

One task on this list and today’s topic is the possibility to fly at constant altitude irrespective of the terrain. This feature has been discussed in the community already two years ago. To understand the problem, let’s look at the currently implemented solution: With the Flow deck mounted the Crazyflie uses a 1D lidar sensor to estimate its vertical position. This vertical position (more or less) equals the current sensor reading. On flat floors this solution works very well. However, if the Crazyflie shall traverse through a narrow window or fly above irregular terrain its altitude will change based on the sensor readings. This can lead to unstable flights, as in the following video, or even crashes!

You might wonder: why not use any of the other great tools from the Bitcraze universe? Indeed, the Lighthouse positioning system and the Loco positioning system work well for absolute positioning (as we have seen earlier, e.g., in this blog post). However, the required setups are often not available in difficult environments. Alternatively, the barometer could be used to achieve a solution based solely on on-board sensors. In fact, Bitcraze has proposed an altitude hold functionality a few years ago. This is a cool feature, but its positioning accuracy of “roughly ±15cm” is not fully satisfying. Finally, relying on the on-board IMU alone will inevitably lead to drifting over time.

Thus, we propose a solution based on the Flow deck and the Multiranger deck. This approach, only based on on-board sensors, allows to fly at constant altitude with obstacles above, below, or even both above and below the Crazyflie. Kristoffer Skare developed this solution when he worked with us as an intern in 2021.

Technical Description

As a first step, the upward-facing lidar of the Multiranger deck is incorporated in the same way the downward-facing lidar of the Flow deck is used in the firmware. This additional measurement can then be used in the extended Kalman filter (EKF) to improve the state estimation. Currently, the EKF estimates and outputs 1 value for the altitude. For our purpose two more states are added to the EKF: one state is defined as the height of the object under the Crazyflie compared to the height where the altitude state is defined as 0. Similarly, the other state is defined as the height of the object above the Crazyflie compared to the same reference height. The Crazyflie keeps therefore track of the environment in order to keep its own altitude constant. To achieve this, an edge detection was implemented: The errors between the predicted and measured distance are tracked in both the upward or downward range measurement. If either of these errors is too large the algorithm assumes that the floor or roof has changed (while the original EKF would think the drone’s position has changed, triggering a change in thrust). Thus, the corresponding state gets updated. For more details on the technical implementation and the code itself, check out our pull request.

Results

To analyze our approach we have used a Qualisys motion capture system. We have conducted many different tests: flying over different obstacles, flying at different velocities, flying at different altitudes, or even flying under different lighting conditions. Exemplarily, in this post we will have a look at a baseline example, a good estimate, and a bad estimate. In each picture you can see the altitude (in meters) over time (in seconds) for different flight speeds (in centimeters per second). You will see three lines: The motion capture ground truth (blue), the altitude estimated by our code (orange), and the new state keeping track of the floor height (green). For each plot, the Crazyflie takes off, flies in positive x direction, and lands.

In the baseline experiment, it flies over a flat floor. Clearly, the altitude estimates follow the ground truth values well, and the floor is correctly estimated to be flat.

In the next example, we have added a box with an approximate height of 0.225 m and made the Crazyflie fly over it. Despite the obstacle the altitude estimates follow the ground truth values well. Note how the floor estimates indicates the shape of the box.

experiment with box — Experiment flying over a box

Because the algorithm is based on an edge detection, we had a hunch that smoothly changing obstacles will pose a problem. Indeed, the estimates can be messy as we see in the next example. Here, the Crazyflie flies over an orthogonal triangle, with the short leg at 0.23 m pointing upwards and the long leg with 0.65 m pointing in flight direction (thus forming a slope). For different flight speeds different the estimates turn out quite differently.

experiment with slope — Experiment flying over a slope

If you don’t like looking at plots, check out this video with some cool shots instead!

Conclusion

To summarize, we propose a solution for constant altitude flight with Crazyflies, using the Flow deck and the Multiranger deck. We have tested it successfully under various circumstances. Still, we see some potential for improvement, e.g. when dealing with slopes. In addition, the current implementation is quite a change to the original EKF, which poses a problem for integration.

Thus, a way forward can be an out-of-tree build to ease the use of the solution for the community. At SINTEF we certainly plan to deploy this code in all of our tests in 2022, which will hopefully allow us to gather more experience and thus find further ways to improve or tune the system.

We want to emphasize that this is not a perfect solution. That means a) you should use it with care and b) you are very much welcomed to contribute. E.g. feel free to chime in in the pull request, test the code in your environments, propose improvements, or implement an out-of-tree build! :) Maybe you can even come up with an alternative approach for constant altitude flights?

If you want to check out more of our work, visit our website. Also, keep reading this amazing blog from Bitcraze as we try to be back some day (if Bitcraze wants us hehe)!

Looking back at 2021

2021-12-27 | Barbara | Leave a comment

2021 is coming to an end. As we’re about to flip the page on a new calendar, let’s take a look at what happened during this past year.

Community

It’s no news to you, but keeping in touch with our community during a pandemic has forced us to try new ways to meet and interact. The main event for us this year has been the BAM days: our first conference held entirely by ourselves, full of exciting talks and fun, and we’re very happy with how it went down! You can still watch the talks we hosted during those three days in our dedicated Youtube playlist.

Guest blogposts

Once again, we’ve had the honor to host some awesome guests on our blog, which you can read (or re-read) here:

Software

We had 3 releases this year, (one in January, one in March and one in June). We worked a lot on improving things like the Crazyflie logging and parameter interface or wondered how to deal with our API. We’ve also implemented a new way to store things, and we now have new, powerful persistent parameters.

Thanks to Jonas’ hardwork, we also set up a “crazy lab” here in Malmö. During Fun Friday, Arnaud has been experimenting with Rust, working on a webclient.

Hardware

We’ve been dealing since 2020 with a hardware crisis. The component shortage has made production erratic and difficult, and for the first time in Bitcraze’s history, we’ve had to increase our prices.

Lighthouse

The first part of the year was dedicated to the Lighthouse system. At the beginning of the year, we finally got it out of early access. We documented the new Lighthouse Functionality and even wrote a paper for ICRA about the Lighthouse accuracy with Wolfgang’s help. We created the Lighthouse swarm bundle, getting every element to fly a swarm with this positioning system.

AI deck

We worked a lot on the AI deck this year. We upgraded to the AI deck 1.1, with a gray-scale camera and a newer version of the GAP-8. Part of this work was also to improve the documention and informations we had on it. We had a workshop with PULP, and dreamed of a mega-tutorial.

Documentation

As usual, we’ve been trying to improve our documentation. This year, it included an API reference in our Python library , and rethinking our structure

Bitcraze

Bitcraze has once again increased in numbers, as we welcomed into our ranks Jonas. We were also really happy to add Wolfgang to the team for a few months, as well as Rik, an intern from MAV-lab.

The biggest event this year for us was our 10 year anniversary. That’s right, Bitcraze turned 10 in September, and we tried to celebrate it as much as possible ! With a nice outing and with the BAM days, but also with a video trying to show what 10 years of Bitcraze looks like:

Santa’s flying helpers

2021-12-20 | Barbara | Leave a comment

Christmas is just around the corner, and it’s time for the traditionnal Christmas video! This year, we wanted to use the AI deck as we’ve been working hard on this deck for some time now. Showcasing its new feature in this festive video seemed the best idea.

Santa this year needed help to find and get the presents delivered, so he asked for help from Bitcraze! Let’s see how it played out:

I’m sure you’re wondering how we managed to set this up, so let’s discuss how we did it!

Picking up the packages

The goal was to pick up some packages and place them in the sleigh, and it all worked out pretty well. All the flying in the video is scripted using the python lib and positioning is done using Qualisys’ motion capture system with Active marker decks. The trajectories are hard coded and with some careful adjustments we managed to lift and fly the 4 crazyflies attached to the same present, even though it was a bit wobbly from time to time. Getting the present into the sleigh was not as easy, and we might have taken some short cuts here as well as when attaching/removing lines.

Sometimes detangling the wires was a teamwork

Picking up the second package needed some precision, and it went incredibly well, on the first try! The present was very light and needed someone to hold it, to prevent it from moving when the Crazyflie approached.

Getting the sleigh off

Our first tries included 5 strings attached to the sleigh, but it was difficult to get the right tension at the right time to have the UAVs actually pull the weight. Here came the “rubber band solution”: we just attached all the strings to a rubber band, that was itself attached to the sleight. That way, the tension could get even when all the Crazyflies were in the air and ready to pull the sleight.

The AI deck camera/streamer

When we started this project, the intention was to run a neural network in the AI-deck to identify or classify the presents. We did not manage to get to a point where we had something that actually added value to the story, so we settled for just streaming the video from the AI-deck camera on the scouting drone instead.

The AI-deck example with color camera viewer is still under development, but if you want to give it a try you can take a look at the readme in the github repo.

Bloopers

And finally, as an added bonus, if you ever wonder how many tries it takes to make 5 Crazyflies pull a sleigh, here is a little behind-the-scenes video too!

Where do I add my code?

2021-11-22 | Kristoffer Richardsson | Leave a comment

Sometimes we get the question of where to modify or add code to change some behavior of the Crazyflie. There is no quick answer to this question but we thought that we should write a post to clear up some question marks and give a better idea of how to approach the problem.

There are quite a few repositories on the Bitcraze github page but there are two that are the main focal point for almost any Crazyflie work, those are the crazyflie-firmware and the crazyflie-lib-python. The crazyflie-firmware contains the source code (written in C) for the firmware that runs in the Crazyflie, that is the code responsible for flying, blinking LEDs, communicating with the radio, scanning sensors and so on. The crazyflie-lib-python (often called the python lib) on the other hand is running on the PC side and is the API to use to communicate with the Crazyflie from a script. The crazyflie-lib-python is also used by the python client which means that anything you see in the client can be done by a script using the python lib.

Let’s assume we have a system of one Crazyflie connected to a computer using a Crazyradio. Now we want to control the Crazyflie and make it take off for instance, how should this be done?

The easiest way would be to use the python lib. The python lib is used to communicate with the Crazyflie and we can use it to send instructions to the Crazyflie, for instance to take off or fly a trajectory. It is also possible to use the parameter framework to change values in the Crazyflie. The main way of monitoring what is going on in the Crazyflie is to use the log framework to read variables from the Crazyflie. The python lib is perfect for controlling the Crazyflie or prototype ideas as it is very fast to make changes and try things out. The best to get started with the python lib is to start from an example that already uses functionalities you want to use.

Another option is to add code in the firmware. Originally this has been quite hard since the firmware has not been initially designed to accept user code. This means that unless you want to modify an already existing code, it is quite hard to find where to add your code so that it runs in the Crazyflie firmware and you would have to make a fork of the firmware which can he hard to maintain and keep up to date in the long run. This is one of the things the out-of-tree build and the app layer is solving, it is now quite easy to add and run your own C files to the firmware in your own project without having to fork the Crazyflie firmware. There is a bunch of examples in the firmware that shows how to implement autonomous behavior as an app. The easiest is to start with the hello world example. When it comes to modifying exiting functionality in the firmware code, most of the time forking and modifying the official firmware unfortunately is the only solution. We are however working our way to make more and more of the firmware modular so that it can be expanded out of tree. For example there has been work to make an out of tree estimator possible to implement.

A prototype written as a python script is often pretty easy to move to the Crazyflie firmware. This is a good pattern when writing an application, rapid prototyping in python and then finalizing in firmware if needed. The best example of that is the push demo. It is a demo where the Crazyflie can be pushed-around with the help of the flow deck for autonomous flight and the multiranger deck for detecting obstacle/hands. We have a python cflib push demo as well as a Crazyflie firmware push demo app.

There is some support in the python lib for interacting with multiple Crazyflies and it is probably a good start point for simple swarms. For more advanced swarm work Crazyswarm may be a better option.

If you would like to see some of the process in action, we have made a workshop during our BAM days about implementing functionality both using the python lib and in the firmware as an out of tree app:

Real-time control of a swarm of Crazyflies in artificial forests

2021-11-08 | Enrica | 4 Comments

This week we have a guest blog post from Enrica Soria from the Laboratory of Intelligent Systems Faculty of Ecole Polytechnique Fédérale de Lausanne (EPFL) . Enjoy!

From Star Wars to Black Mirror, sci-fi movies predict a future where thousands of drones will fill our sky. Curving sharply around trees or soaring over buildings, they fly just like a flock of starlings. To turn this vision into a reality, real drone swarms need to increase their autonomy and operate in a decentralized fashion. In a decentralized swarm, each robot makes its own decision based only on local information. Decentralization not only allows the swarm to be more robust to the failure of single individuals, but also removes the dependency from a single computing unit, thus making the swarm more scalable in terms of size.

We at LIS (EPFL) have shown that predictive controllers can improve the safety of aerial swarms by predicting and optimizing the agents’ future behavior in an iterative process. However, the centralized nature of this method allowed us to only control five drones and prevented us from scaling up to a large number of drones. For this reason, we have worked on a novel decentralized and scalable swarm controller that allows the safe and cohesive flight of aerial swarms in cluttered environments. In our latest article, published in IEEE Robotics and Automation Letters (RA-L), we describe how we designed the controller, show its scalability in size, and demonstrate its robustness to noise. We studied the swarms’ performance and compared how it changes in two different environments: a forest and funnel-like environment.

The Crazyflie 2.1 was the perfect platform for our experiments. They are lightweight, modular, and tough. This quadcopter can survive big hits when things don’t go as planned… and, if you work on swarms, things can go wrong!

*The fleet of Crazyflies equipped with a single marker.*

With our algorithm, sixteen robots were able to fly through an artificial forest that we set up in our indoor motion capture arena. In our previous work, we installed four markers on each quadcopter and used the rigid body tracking from Motive (the Optitrack software). The large volume of our experimental room required the usage of big markers for long-distance detection, which added considerable weight to the drone. Hence, in our new work, we use a single marker per drone. Tracking is supported by the ‘crazyswarm’ package and communication with the entire swarm only requires two radio links. However, despite our model being decentralized, in our implementation robots relay the information to an external brain, which does the computations for them. In the future, all the necessary code will be embedded onboard, removing the dependency on external infrastructure.

*Our predictive swarm of Crazyflies flying among obstacles in our indoor experimental room.*

Video about the article

This work is a step forward towards the fully autonomous deployment of drone swarms in our cities. By enabling safe navigation in cluttered environments, drone fleets will be able to integrate with conventional air traffic, search for missing people, inspect dangerous areas, transport injured people to hospitals quicker, and deliver important packages right to our doors.

For further details, check out our article here!

Sniffy Bug: A Fully Autonomous Swarm of Gas-Seeking Nano Quadcopters in Cluttered Environments

2021-07-14 | Bart Duisterhof | 1 Comment

This week we have a guest blog post from Bart Duisterhof and Prof. Guido de Croon from the MAVlab, Faculty of Aerospace Engineering from the Delft University of Technology. Enjoy!

Tiny drones are ideal candidates for fully autonomous jobs that are too dangerous or time-consuming for humans. A commonly shared dream would be to have swarms of such drones help in search-and-rescue scenarios, for instance to localize gas leaks without endangering human lives. Drones like the CrazyFlie are ideal for such tasks, since they are small enough to navigate in narrow spaces, safe, agile, and very inexpensive. However, their small footprint also makes the design of an autonomous swarm extremely challenging, both from a software and hardware perspective.

From a software perspective, it is really challenging to come up with an algorithm capable of autonomous and collaborative navigation within such tight resource constraints. State-of-the-art solutions like SLAM require too much memory and processing power. A promising line of work is to use bug algorithms [1], which can be implemented as computationally efficient finite state machines (FSMs), and can navigate around obstacles without requiring a map.

A downside of using FSMs is that the resulting behavior can be very sensitive to their hyperparameters, and therefore may not generalize outside of the tested environments. This is especially true for the problem of gas source localization (GSL), as wind conditions and obstacle configurations drastically change the problem. In this blog post, we show how we tackled the complex problem of swarm GSL in cluttered environments by using a simple bug algorithm with evolved parameters, and then tested it onboard a fully autonomous swarm of CrazyFlies. We will focus on the problems that were encountered along the way, and the design choices we made as a result. At the end of this post, we will also add a short discussion about the future of nano drones.

Why gas source localization?

Overall we are interested in finding novel ways to enable autonomy on constrained devices, like CrazyFlies. Two years ago, we showed that a swarm of CrazyFlie drones was able to explore unknown, cluttered environments and come back to the base station. Since then, we have been working on an even more complex task: using such a swarm for Gas Source Localization (GSL).

There has been a lot of research focussing on autonomous GSL in robotics, since it is an important but very hard problem. The difficulty of the task comes from the complexity of how odor can spread in an environment. In an empty room without wind, a gas will slowly diffuse from the source. This can allow a robot to find it by moving up gradient, just like small bacteria like E. Coli do. However, if the environment becomes larger with many obstacles and walls, and wind comes into play, the spreading of gas is much less regular. Large parts of the environment may have no gas or wind at all, while at the same time there may be pockets of gas away from the source. Moreover, chemical sensors for robots are much less capable than the smelling organs of animals. Available chemical sensors for robots are typically less sensitive, noisier, and much slower.

Due to these difficulties, most work in the GSL field has focused on a single robot that has to find a gas source in environments that are relatively small and without obstacles. Relatively recently, there have been studies in which groups of robots solve this task in a collaborative fashion, for example with Particle Swarm Optimization (PSO). This allows robots to find the source and escape local maxima when present. Until now this concept has been shown in simulation [2] and on large outdoor drones equipped with LiDAR and GPS [3], but never before on tiny drones in complex, GPS-denied, indoor environments.

Required Infrastructure

In our project, we introduce a new bug algorithm, Sniffy Bug, which uses PSO for gas source localization. In order to tune the FSM of Sniffy Bug, we used an artificial evolution. For time reasons, evolution typically takes place in simulation. However, early in the project, we realized that this would be a challenge, as no end-to-end gas modeling pipeline existed yet. It is important to have an easy-to-use pipeline that does not require any aerodynamics domain knowledge, such that as many researchers as possible can generate environments to test their algorithms. It would also make it easier to compare contributions and to better understand in which conditions certain algorithms work or don’t work. The GADEN ROS package [4] is a great open-source tool for modeling gas distribution when you have an environment and flow field, but for our objective, we needed a fully automated tool that could generate a great variety of random environments on-demand with just a few parameters. Below is an overview of our simulation pipeline: AutoGDM.

AutoGDM, a fully automated gas dispersion modeling (GDM) simulation pipeline.

First, we use a procedural environment generator proposed in [5] to generate random walls and obstacles inside of the environment. An important next step is to generate a 3D flowfield by means of computational fluid dynamics (CFD). A hard requirement for us was that AutoGDM needed to be free to use, so we chose to use the open-source CFD tool OpenFOAM. It’s used for cutting-edge aerodynamics research, and also the tool suggested by the authors of GADEN. Usually, using OpenFOAM isn’t trivial, as a large number of parameters need to be selected that require field expertise, resulting in a complicated process. Next, we integrate GADEN into our pipeline, to go from environment definition (CAD files) and a flow field to a gas concentration field. Other parts that needed to be automated were the random selection of boundary conditions, which has a large impact on the actual flow field, and source placement, which has an equally large impact on the concentration field.

After we built this pipeline, we started looking for a robot simulator to couple it to. Since we weren’t planning on using a camera, our main requirement was for the simulator to be efficient (preferably in 2D) so that evolutions would take relatively little time. We decided to use Swarmulator [6], a lightweight C++ robot simulator designed for swarming and we plugged in our gas data.

Algorithm Design

Roughly speaking, we considered two categories of algorithms for controlling the drones: 1) a neural network, and 2) an FSM that included PSO, with evolved parameters. Since we used a tiny neural network for light seeking with a CrazyFlie in our previous work, we first evolved neural networks in simulation. One of the first experiments is shown below.

A single agent in simulation seeking a light source using a tiny neural network.

While it worked pretty well in simple environments with few obstacles, it seemed challenging to make this work in real life with complex obstacles and multiple agents that need to collaborate. Given the time constraints of the project, we have opted for evolving the FSM. This also facilitated crossing the reality gap, as the simulated evolution could build on basic behaviors that we developed and validated on the real platform, including obstacle avoidance with four tiny laser rangers, while communicating with and avoiding other drones. An additional advantage of PSO with respect to the reality gap is that it only needs gas concentration and no gradient of the gas concentration or wind direction (which many algorithms in literature use). On a real robot at this scale, estimating the gas concentration gradient or the direction of a light breeze is hard if not impossible.

Hardware

Our CrazyFlie needs to be able to avoid obstacles, execute velocity commands, sense gas, and estimate the other agent’s position in its own frame. For navigation, we added the flow deck and laser rangers, whereas for gas sensing we used a TGS8100 gas sensor that was used on a CrazyFlie before in previous work [7]. The sensor is lightweight and inexpensive, but accurately estimating gas concentrations can be difficult because of its size. It tends to drift and needs time to recover after a spike in concentration is observed. Another thing we noticed is that it is possible to break them, a crash can definitely destroy the sensor.

To estimate the relative position between agents, we use a Decawave Ultra-Wideband (UWB) module and communicate states, as proposed in [8]. We also use the UWB module to communicate gas information between agents and collaboratively seek the source. The complete configuration is visible below.

A 37.5 g nano quadcopter, capable of fully autonomous waypoint tracking, obstacle avoidance, relative localization, communication and gas sensing.

Evaluation in Simulation

After we optimized the parameters of our model using Swarmulator and AutoGDM, and of course trying many different versions of our algorithm, we ended up with the final Sniffy Bug algorithm. Below is a video that shows evolved Sniffy Bug evaluated in six different environments. The red dots are an agent’s personal target waypoint, whereas the yellow dot is the best-known position for the swarm.

Sniffy Bug evaluated in Swarmulator environments.

Simulation showed that Sniffy Bug is effective at locating the gas source in randomly generated environments. The drones successfully collaborate by means of PSO.

Real Flight Testing

After observing Sniffy Bug in simulation we were optimistic, but unsure about performance in real life. First, inspired by previous works, we disperse alcohol through the air by placing liquid alcohol into a can which is then dispersed using a computer fan.

Dispersion of liquid alcohol in flight tests.

We test Sniffy Bug in our flight arena of size 10 x 10 meters with large obstacles that are shaped like walls and orange poles. The image below shows four flight tests of Sniffy Bug in cluttered environments, flying fully autonomously, i.e., without the help from any external infrastructure.

Time-lapse images of real-world experiments in our flight arena. Sniffy was evaluated on four distinct environments, 10 x 10 meters in size, seeking a real isopropyl alcohol source. The trajectories of the nano quadcopters are clearly visible due to their blue lights.

In the total of 24 runs we executed, we compared Sniffy Bug with manually selected and evolved parameters. The figure below shows that the evolved parameters are more efficient in locating the source as compared to the manual parameters.

Maximum recorded gas reading by the swarm, for each time step for each run.

This does not only show that our system can successfully locate a gas source in challenging environments, but it also demonstrates the usefulness of the simulation pipeline. The parameters that were learned in simulation yield a high-performance model, validating the environment generation, randomization, and gas modeling parts of our pipeline.

Conclusion and Discussion

With this work, we believe we have made an important step towards swarms of gas-seeking drones. The proposed solution is shown to work in real flight tests with obstacles, and without any external systems to help in localization or communication. We believe this methodology can be extended to larger environments or even to 3 dimensions, since PSO is a robust, multi-dimensional heuristic search method. Moreover, we hope that AutoGDM will help the community to better compare gas seeking algorithms, and to more easily learn parameters or models in simulation, and deploy them in the real world.

To improve Sniffy Bug’s performance, adding more laser rangers will definitely help. When working with only four laser rangers you realize how little information it actually provides. If one of the rangers senses a low value it is unclear if a slim pole or a massive wall is detected, adding inefficiency to the algorithm. Adding more laser rangers or using other sensor modalities like vision will help to avoid also more complex obstacles than walls and poles in a reliable manner.

Another interesting discussion can be held on the hardware required for real deployment. When working with 40 grams of maximum take-off weight, the sensors and actuators that can be selected are limited. For example, the low-power and lightweight flow deck works great but fails in low-light scenarios or with smoke. Future work exploring novel sensors for highly constrained nano robots could really help increase the Technological Readiness Level (TRL) of these systems.

Finally, this has been a really fun project to work on for us and we can’t wait to hear your thoughts on Sniffy Bug!

References

[1] K. N. McGuire, C. De Wagter, K. Tuyls, H. J. Kappen, and G. C. H. E.de Croon, “Minimal navigation solution for a swarm of tiny flying robotsto explore an unknown environment,”Science Robotics, vol. 4, no. 35,2019.

[2] W. Jatmiko, K. Sekiyama and T. Fukuda, “A pso-based mobile robot for odor source localization in dynamic advection-diffusion with obstacles environment: theory, simulation and measurement,” in IEEE Computational Intelligence Magazine, vol. 2, no. 2, pp. 37-51, May 2007, doi: 10.1109/MCI.2007.353419.

[3] Steiner, JA, Bourne, JR, He, X, Cropek, DM, & Leang, KK. “Chemical-Source Localization Using a Swarm of Decentralized Unmanned Aerial Vehicles for Urban/Suburban Environments.” Proceedings of the ASME 2019 Dynamic Systems and Control Conference. Volume 3, Park City, Utah, USA. October 8–11, 2019. V003T21A006. ASME. https://doi.org/10.1115/DSCC2019-9099

[4] . Monroy, V. Hernandez-Bennetts, H. Fan, A. Lilienthal, andJ. Gonzalez-Jimenez, “Gaden: A 3d gas dispersion simulator for mobilerobot olfaction in realistic environments,”MDPI Sensors, vol. 17, no.7: 1479, pp. 1–16, 2017.

[5] K. McGuire, G. de Croon, and K. Tuyls, “A comparative study of bug algorithms for robot navigation,”Robotics and Autonomous Systems, vol.121, p. 103261, 2019.

[6] https://github.com/coppolam/swarmulator

[7] J. Burgues, V. Hern ́andez, A. J. Lilienthal, and S. Marco, “Smellingnano aerial vehicle for gas source localization and mapping,”Sensors(Switzerland), vol. 19, no. 3, 2019.[8] S. Li, M. Coppola, C. D. Wagter, and G. C. H. E. de Croon, “An autonomous swarm of micro flying robots with range-based relative localization,” Arxiv, 2020.

[8] S. Li, M. Coppola, C. D. Wagter, and G. C. H. E. de Croon, “An autonomous swarm of micro flying robots with range-based relative localization,” Arxiv, 2020.

Links

ArXiv: https://arxiv.org/abs/2107.05490

Code: https://github.com/tudelft/sniffy-bug

Video:

Please reach out if you have any questions or ideas, you can reach us at: b.p.duisterhof@gmail.com or g.c.h.e.decroon@tudelft.nl

The Lighthouse Swarm bundle

2021-06-28 | Barbara | Leave a comment

If you haven’t visited our store in a while, you may have missed our new addition: the Lighthouse Swarm bundle!

We’ve been working for some time now on improving the Lighthouse decks and its positioning system. Earlier in the year, we have brought the Lighthouse deck out of early access. While working with it, we have seen the great possibilities and the accuracy of this new positioning system. Thanks to Steam’s VR base station that we use as an optical beacon, the Crazyflie calculates its position with an accuracy better than a decimeter and millimeter precision. It gives a tracking volume of up to 5x5x2 meters with sub-millimetre jitter and below 10 cm accuracy while flying. It’s perfect for a swarm, as it’s accurate, precise and autonomous. We’ve flown our Crazyflies with it a number of time and seen some awesome stuff with it!

As an example, here is a demo we’ve shown on a conference back in October. We’ve used 8 Crazyflies equipped with Lighthouse decks and Qi chargers, to make a spiraling swarm. A computer orchestrates the Crazyflies and make sure one is flying at all times, while the others re-charge their batteries on their pads. After a pre-programmed trajectory is finished or when the battery of the flying Crazyflie is depleted, it goes back to its pad while another one takes over. The demo had an all-in mode that runs the trajectory on all Crazyflie with sufficient charge at once, the result is quite impressive and demonstrate the great relative precision of the Lighthouse system:

After the launch signal is sent to the Crazyflies, the computer is not required anymore: the Crazyflie will autonomously estimate its position from the lighthouse’s signals. The Crazyflie can estimate its own X, Y and Z in a global coordinate system.

What’s great with the Lighthouse Swarm is that it allows you to do drone research even if you’re on a tighter budget.

And when we got the opportunity to acquire our own base stations (that are also available in the shop by the way), it seemed only logical to offer a Swarm bundle similar to our Loco swarm bundle. So what’s in it ?

While the positioning will work with one base station, two base stations will allow better coverage of the flight space and better stability; as Kimberly can attest, it’s even possible to set it in your kitchen. The Crazyradios allow communication between the Crazyflies and your computer.

We dedicated a lot of time to the Lighthouse this winter, writing a paper with the help of Wolgangs’ calibration expertise. In this paper, we compared both Lighthouse V1 and V2 with the MoCap system. In all cases, the mean and median Euclidean error of the Lighthouse positioning system are about 2-4 centimeters compared to our MoCap system as ground truth. You can check the paper here, but here is a brief summary we used for our ICRA workshop:

We are now quite excited to get to see what you will do with this exciting new swarm bundle !

And if you don’t know how to set up the Swarm, you can get started at least with your Lighthouse system in this tutorial or watch Kristoffer explain it in this video:

Ultra-Wideband Swarm Ranging

2021-06-21 | Feng Shan | Leave a comment

This week we have a guest blog post from Dr Feng Shan at School of Computer Science and Engineering
Southeast University, China. Enjoy!

It is possible to utilize tens and thousands of Crazyflies to form a swarm to complete complicated cooperative tasks, such as searching and mapping. These Crazyflies are in short distance to each other and may move dynamically, so we study the dynamic and dense swarms. The ultra-wideband (UWB) technology is proposed to serve as the fundamental technique for both networking and localization, because UWB is so time sensitive that an accurate distance can be calculated using the transmission and receive timestamps of data packets. We have therefore designed a UWB Swarm Ranging Protocol with key features: simple yet efficient, adaptive and robust, scalable and supportive. It is implemented on Crazyflie 2.1 with onboard UWB wireless transceiver chips DW1000.

Fig.1. Nine Crazyflies are in a compact space ranging the distance with each other.

The Basic Idea

The basic idea of the swarm ranging protocol was inspired by Double Sided-Two Way Ranging (DS-TWR), as shown below.

Fig.2. The exsiting Double Sided-Two Way Ranging (DS-TWR) protocol.

There are four types of message in DS-TWR, i.e., poll, response, final and report message, exchanging between the two sides, A and B. We define their transmission and receive timestamps are T_p, R_p, T_r, R_r, T_f, and R_f, respectively. We define the reply and round time duration for the two sides as follows.

Let t_p be the time of flight (ToF), namely radio signal propagation time. ToF can be calculated as Eq. (2).

Then, the distance can be estimated by the ToF.

In our proposed Swarm Ranging Protocol, instead of four types of messages, we use only one type of message, which we call the ranging message.

Fig.3. The basic idea of the proposed Swarm Ranging Protocol.

Three sides A, B and C take turns to transmit six messages, namely A₁, B₁, C₁, A₂, B₂, and C₂. Each message can be received by the other two sides because of the broadcast nature of wireless communication. Then every message generates three timestamps, i.e., one transmission and two receive timestamps, as shown in Fig.3(a). We can see that each pair has two rounds of message exchange as shown in Fig.3(b). Hence, there are sufficient timestamps to calculate the ToF for each pair, that means all three pairs can be ranged with each side transmitting only two messages. This observation inspires us to design our ranging protocol.

Protocol Design

The formal definition of the i-th ranging message that broadcasted by Crazyflie X is as follows.

X_i is the message identification, e.g., sender and sequence number; T_{x_i-1} is the transmission timestamp of X_i-1, i.e., the last sent message; RxM is the set of receive timestamps and their message identification, e.g., RxM = {(A₂, R_A₂), (B₂, R_B₂)}; v is the velocity of X when it generates message X_i.

As mentioned above, six timestamps (T_p, R_p, T_r, R_r, T_f, R_f,) are needed to calculate the ToF. Therefore, for each neighbor, an additional data structure is designed to store these timestamps which we named it the ranging table, as shown in Fig.4. Each device maintains one ranging table for each known neighbor to store the timestamps required for ranging.

Fig.4. The *ranging table*, one for each neighbor.

Let’s focus on a simple scenario where there are a number of Crazyflies, A, B, C, etc, in a short distance. Each one of them transmit a message that can be heard by all others, and they broadcast ranging messages at the same pace. As a result, between any two consecutive message transmission, a Crazyflie can hear messages from all others. The message exchange between A and Y is as follows.

Fig.5. Message exchange between A and Y.

The following steps show how the ranging messages are generated and the ranging tables are updated to correctly compute the distance between A and Y.

Fig.6. How the ranging message and ranging table works to compute distance.

The message exchange between A and Y could be also A and B, A and C, etc, because they are equal, that’s means A could perform the ranging process above with all of it’s neighbors at the same time.

To handle dense and dynamic swarm, we improved the data structure of ranging table.

Fig.7. The improved ranging table for dense and dynamic swarm.

There are three new notations P, t_n, t_s, denoting the newest ranging period, the next (expected) delivery time and the expiration time, respectively.

For any Crazyflie, we allow it to have different ranging period for different neighbors, instead of setting a constant period for all neighbors. So, not all neighbors’ timestamps are required to be carried in every ranging message, e.g., the receive timestamp to a far apart and motionless neighbor is required less often. t_n is used to measure the priority of neighbors. Also, when a neighbor is not heard for a certain duration, we set it as expired and will remove its ranging table.

If you are interested in our protocol, you can find much more details in our paper, that has just been published on IEEE International Conference on Computer Communications (INFOCOM) 2021. Please refer the links at the bottom of this article for our paper.

Implementation

We have implemented our swarm ranging protocol for Crazyflie and it is now open-source. Note that we have also implemented the Optimized Link State Routing (OLSR) protocol, and the ranging messages are one of the OLSR messages type. So the “Timestamp Message” in the source file is the ranging message introduced in this article.

The procedure that handles the ranging messages is triggered by the hardware interruption of DW1000. During such procedure, timestamps in ranging tables are updated accordingly. Once a neighbor’s ranging table is complete, the distance is calculated and then the ranging table is rearranged.

All our codes are stored in the folder crazyflie-firmware/src/deck/drivers/src/swarming.

The following figure is a ranging performance comparison between our ranging protocol and token-ring based TWR protocol. It’s clear that our protocol handles the large number of drones smoothly.

We also conduct a collision avoidance experiment to test the real time ranging accuracy. In this experiment, 8 Crazyflie drones hover at the height 70cm in a compact area less than 3m by 3m. While a ninth Crazyflie drone is manually controlled to fly into this area. Thanks to the swarm ranging protocol, a drone detects the coming drone by ranging distance, and lower its height to avoid collision once the distance is small than a threshold, 30cm.

Build & Run

Clone our repository

git clone --recursive https://github.com/SEU-NetSI/crazyflie-firmware.git

Go to the swarming folder.

cd crazyflie-firmware/src/deck/drivers/src/swarming

Then build the firmware.

make clean
make

Flash the cf2.bin.

cfloader flash path/to/cf2.bin stm32-fw

Open the client, connect to one of the drones and add log variables. (We use radio channel as the address of the drone) Our swarm ranging protocol allows the drones to ranging with multiple targets at the same time. The following shows that our swarm ranging protocol works very efficiently.

Summary

We designed a ranging protocol specially for dense and dynamic swarms. Only a single type of message is used in our protocol which is broadcasted periodically. Timestamps are carried by this message so that the distance can be calculated. Also, we implemented our proposed ranging protocol on Crazyflie drones. Experiment shows that our protocol works very efficiently.

Category: Video