Author: Wolfgang Hoenig

Communicating with your Crazyflie is an important pillar for its operation. As more robots are controlled, the reliability of this communication link becomes more and more important, as the probability that there is no failure at any of the Crazyflies decreases exponentially with the number of robots. We have written about the low-level radio link before. Today, we focus on past, on-going, and future improvements to make the communication more reliable.

Reliability Challenges

As part of doing research with the Crazyswarm, I noticed several issues:

  1. Sometimes commands do not seem to arrive at the Crazyflie, especially when using unicast (i.e., direct) communication with a specific Crazyflie.
  2. Sending ‘too much’ data while the Crazyflie is flying can cause catastrophic crashes. One example is trying to upload a trajectory, while flying in a motion capture space. However, this occasionally even happens if just trying to update a parameter.

The radio link has a feature called Safelink, which essentially guarantees that packets are send and received in order and no packet gets dropped. This feature never worked reliably in the crazyflie_cpp implementation and is therefore not used in the Crazyswarm. However, it is the default mode for cflib. The Crazy RealTime Protocol (CRTP) also has the notion of different communication ports in order to prioritize important messages such as control commands over less important ones such as trajectory upload. However, this prioritization was never implemented in any of the clients.

Native Link Implementations

Another, non reliability related, issue always was that the performance of cflib is not stellar when connected to many Crazyflies. This is mostly because (C)Python has a Global Interpreter Lock (GIL), which prevents true multi-threaded operation. A common solution to allow true parallelism is multiprocessing or implementations in a language that compiles to machine code (native code).

Crazyflie-link-Cpp

The first native implementation is written in C++ and includes Python bindings using pybind11. The overall API is simple: A connection can be created given a URI, and data can be send and received. Internally, this library implements Safelink to guarantee packets are in order, uses priority queues to prioritize messages on important CRTP ports, and handles all multi-radio and multi-thread related synchronization issues. Unlike prior implementations, the bandwidth is shared uniformly between all connections, i.e., there is no race between different Crazyflies communicating over the same radio. We measured a 50% lower CPU utilization when connecting with the CF client, a 20% higher bandwidth, and a 20% lower latency compared to the pure Python implementation.

For now, this feature is experimental and needs to be enabled using an environment variable. If you want to give it a try, you can use

pip install cflinkcpp
USE_CFLINK=cpp cfclient

once everything is merged to master in the next couple of days.

Crazyflie-link-rs

The second native implementation is written in Rust. As it turns out, writing multi-threaded code in C++ is very difficult and error-prone, because it is up to the user to use the various synchronization primitives correctly. Rust, on the other hand, provides extensive compile-time checks to increase reliability by design. This implementation does not have quite as many features as the C++ version yet and requires a bit more work to be used with the client but it is going to be worked-on on Fridays. The goal is to reach functional parity with the other link implementations and be able to use the rust-implemented link with “USE_CFLINK=rs cfclient” in the future.

Future Work

As part of the development process, we also wrote many system tests that verify correctness and measure performance of the various link implementations. We already found and fixed several firmware bugs (681 and 688). The major open issue is that there is no flow control on the system link (the connection between the NRF51 and the STM32), which still causes packet loss in some cases.

Hi, fellow Crayzflie’ers! I am Wolfgang Hönig and I am a visiting researcher at Bitcraze until early April. I am excited to be part of the team and will be working on making the Crazyflie (and its ecosystem) even more accessible, more robust, and versatile for other researchers and users.

Wolfgang in California.

I started working with the Crazyflie (1) back in 2014 as preparation for grad school. It was the only affordable, small, and open-source robot that I could find and helped me a lot to gain practical knowledge for controls, state estimation, and realtime systems. Now, 7 years later, it is still an exceptional product for the same reasons. During my PhD at the ACT Lab at the University of Southern California, I developed crazyflie_cpp, crazyflie_ros and, together with James A. Preiss, the Crazyswarm. The Crazyswarm in particular has helped me (& many other researchers) to validate robotics research (I worked mostly on multi-robot motion planning). After my PhD, I joined ARCL at Caltech as a postdoctoral researcher, where we used the Crazyflies for research in novel techniques that combine machine learning, motion planning, and controls (see GLAS and Neural-Swarm). Starting May 2021, I’ll be leading my own research group at TU Berlin.

At Bitcraze, I am looking forward to a close collaboration with the team. So far, we are planning to improve the low-level communication over the radio (joint work with Arnaud) and data collection and improvements to the state estimator by using the unique flight arena that has a motion capture, Lighthouse, and Loco Positioning system (joint work with Kimberly). If you are a researcher and have any other ideas, feel free to reach out via email or a comment!

What’s better than a single Crazyflie? A swarm of them! Over a year ago our research group at the University of Southern California posted a blog post with the title “Towards CrazySwarms“, explaining how to fly six Crazyflies at the same time. Since then, we’ve expanded our fleet to 49 Crazyflies. It turns out that flying 49 requires a completely different approach. We will outline the additional challenges, and of course show a fun video!

Why is flying many Crazyflies hard? It comes down to two different categories:

  1. Communication Limitations: The standard Crazyflie software does not support controlling more than one crazyflie per radio. Putting 49 radios on a PC is possible, but would cause very high latencies because the Universal Serial Bus (USB) operates, as the name suggests, serially in 1 ms intervals. Earlier, we showed that we can share a radio for two Crazyflies by using different addresses, but 25 radios are still too much to be handled on one PC reasonably. We can overcome this issue by reducing the amount of data to be transferred. However, this forces us to increase the autonomy of the Crazyflie. Instead of sending attitude control input for each Crazyflie at a high rate, we move the controller on-board and send high-level trajectory descriptions and external position information at a low-rate. In particular, we need to:
    1. Move the position controller on-board, and
    2. Be able to handle packet losses more gracefully.

    i) is relatively easy, apart from the testing and tuning. For ii) we use an Extended Kalman Filter to estimate the state on-board. This state, consisting of the position, angle, and the translational velocities, is estimated by combining the on-board sensors (gyroscope, accelerometer) with external position information. Even if we are not able to send the external position for a while due to packet drops, the on-board sensors will keep the estimated state correct for a while.
    Finally, we implemented broadcasts (rather than 1-to-1 communication between PC and each Crazyflie) and used a number of compression tricks in order to limit the required bandwidth further. We are able to broadcast the pose (position and rotation) for all 49 Crazyflies using just three Crazyradios 100 times per second. Each Crazyflie can handle several packet drops in a row before the state estimate becomes too unreliable to fly.

  2. External Position Feedback: The on-board sensors of the Crazyflie are not sufficient to determine its position, so we need some external position feedback. In academia, optical motion capture systems are frequently used. They consist of a number of specialized, synchronized, high-speed infrared cameras. Each object to track is equipped with at least three retroreflective spheres (so-called markers), which reflect infrared light sent out by the IR light sources next to the cameras. If we know the pose of all cameras, we can use triangulation to determine the 3D positions of all retroreflective markers.Traditionally, motion capture systems require that each object has a unique arrangement of markers; this allows to determine each object’s position from a single frame of marker data by searching for its unique pattern. Unfortunately,  the Crazyflie is too small to have 49 unique marker arrangements that can be reliably distinguished. To solve that issue, we put the Crazyflies at known positions initially and use their marker arrangement to track their position and pose over time, at 100 Hz. This allows us to use the same marker arrangement for each Crazyflie.

 

Putting that together (combined with an improved controller), allows us to create nice formations:

So what is next? Eventually, we will integrate our changes into the various projects (including the firmwares and the ROS driver), allowing everyone to work on and with CrazySwarms.

Have fun flieing!

Wolfgang Hönig
PhD Student
Automatic Coordination of Teams Laboratory
University of Southern California
James Preiss
PhD Student
Robotics Embedded Systems Laboratory
University of Southern California