Author: Jonas Danielsson

The Crazyflie firmware is the result of a lot of engineering (Making It Work) effort over a long period of time, the first commit in the GitHub repository is from 2013.

There are issues, inconsistencies and pain points in our firmware that have been found along the way, things that are difficult to fix because they have, in some sense, become load-bearing.

An examples of an issue like this is our inconsistency in what units are used in the Crazyflie firmware, we would like to use SI units everywhere but today that is not the case. And converting everything is proving to be tough.

More examples can be found in Appendix A of the paper Development of a multi-agent quadrotor research platform with distributed computational capabilities (Ian Scott Mcinerney, 2017).

We would like to fix all of these, but, the problem is that we have APIs to consider. There are programs written that assume that the data received from the Crazyflie is in a certain way. If we were to switch to the, more correct, aerospace axes these programs might break. And worse, they might break silently, meaning there will be no error message.

Our APIs constitute some kind of promise between the Crazyflie ecosystem (firmware, python library, documentation) and our users.

Examples include the parameter- and logging API which today has ideas about which way the coordinate system points and what units are used for values. As well as the Commander Framework / API, since this API has opinions regarding what way X, Y or Z grows in relation to reality.

Our APIs exists both between the Crazyflie firmware and our Python library code and between our Python library code and our users programs, as visualized in the diagram below.

CF - API

If we were to implement solutions to the known issues we need to take these API into consideration. There are applications, demos, scripts and research code out there that rely on these APIs to behave a certain way.

So what can we do? What are our options? We have tried to enumerate them below.

1. Go Ahead And Change Units and Coordinates – But Do Not Update APIs

We fix as much as we can of the unit, correctness and conventions bugs in the firmware, but we leave the APIs as they are. This means that we keep the name of the parameter- and logging variables and we keep the name of all Python classes and functions. This is the less amount of work, it only needs updates to the crazyflie-firmware repository.

All examples and programs will continue to run. But they might now operate under false pretenses, perhaps with no error messages. The programs might stop working because of the implicit change of API.

We will have to communicate loud and clear that this will happen beforehand.

2. Go Ahead And Change Units and Coordinates – And Update the APIs

We fix as much as we can of the unit, correctness and conventions bugs in the firmware, and we also rework our APIs! We will gather learnings from our years with the current CRTP and Python API and we update them to be more developer ergonomic as well as update the parameter- and logging variables to be more consistent in naming and SI units.

This will require quite a bit of work, both in the crazyflie-firmware repository and possibly in the the crazyflie-lib-python repository.

A big benefit of this would be that we are then open to change where the boundaries for our API lies, what we provide as libraries and what is applications responsibility. And we can address pain points that are deep and structural to todays APIs.

This will break all existing programs out there, they will no longer run. They will get error messages and will need to be updated to the new APIs. We might also get weird issues and scenarios where old library code try to interface with newer firmware. We could create policys on for how long we maintain the old API and try to have them side-by-side for a bit.

3. Go Ahead And Change Units and Coordinates – Keep Old API Forever and Add a New One

We fix as much as we can of the unit, correctness and conventions bugs in the firmware, and we also rework our APIs! We will gather learnings from our years with the current CRTP and Python API and we update them to be more developer ergonomic as well as update the parameter- and logging variables to be more consistent in naming and SI units. And! We will also keep the old API as a legacy API.

It is unclear if this is possible, if we have enough memory in the firmware as well as enough bandwidth to maintain two APIs.

We could in theory keep all of the user-facing Python API and attempt to translate between the old and new one in Python code. This might take a lot of work and effort to do, but it might be possible.

But. Fixing all the correctness, convention and units in the firmware might make it very difficult to support both APIs. It might also limit how creative and how different a new API could be, while still being translatable to the old API.

We could also investigate if it would be possible to keep the API between the firmware and the libraries (CRTP and paramter- and logging variables) but translate to be more correct in regards to Units and Conventions in our library.

Even if this was deemed possible it would not help anyone developing directly against the firmware, without using a Bitcraze library.

4. Do Not Change Units and Coordinates – Keep Old API

Accept that the crazyflie-firmware will not reach correctness in respect to units and conventions and close the issues above. We can make sure all new features going in respect SI units and convention, but consider what we have as a legacy. This is by far the one that requires the less work. zero. But it will keep the pain points and inconsistencies of the Crazyflie platform.

Conclusions

There are trade offs between all solutions. We need to decide what our current APIs are worth to us. And how painful it would be for our users to handle an API break.

Maintaining multiple APIs might be to big of an burden for both us and our code base though. This is very very tricky.

It would help us a lot to hear from our users!

If you are someone who develops or have developed against the Crazyflie ecosystem please get in touch, we have created an GitHub issue here. Tell us if you have hit any of the pain-points described in this post or how an API break would affect you.

At Bitcraze, the team is starting to return from vacations even if it will take some weeks still until we are full force again. Work will pick up pace more and more from now, so if we have been slow to answering any issue, pull-request or forum post we ask you to have patience, we will get to you!

Poplinks

Last week we managed to land more improvements to our documentation, by adding poplinks to all references to expansion decks on our website. Poplinks is our name for links that when clicked will popup a box with links to information about the thing you clicked.

Example poplink from the Positioning Systems Overview page

Our aim is to add more poplinks, in areas where they make sense.

Lighthouse decks soon back in stock

We are working hard on completing the next batch of Lighthouse decks and expect them to be back in stock shortly, keep an eye on our store!

BAM days

The planning for the Bitcraze Awesome Meetup days (19th to the 21th of October 2021) is ongoing and we have decided on a online platform to use: Welkom ! This platform comes from the Netherlands and offers some cool functionalities, especially for social gatherings, that we can’t wait for you to see.

We have begun to shape up a nice program to fill these 3 days with talks, demos, workshops and fun things, so don’t forget to fill in the interest form if you want to be a part of it:

We like to describe the Crazyflie as a versatile open source flying development platform. It is something that enables you to do cool stuff. It is not a finished, polished end product in its own.

The platform offers hardware extensions in form of decks, that you make your self or buy from Bitcraze. And the platform offers software extension, either by altering the firmware on the device or by writing programs that communicate with the platform.

This approach makes determining the expectations and requirements for the platform and the surrounding ecosystem a bit tricky. It is dependent on what you as a user plan to create using our products. And since the ecosystem is growing we need a, scalable, way to handle these fuzzy expectations during development and maintenance.

We think testing is big part of solving this, testing in a systematic, scalable and reproducible way. This is the reason for setting up our first physical test lab, aimed at providing stability while moving forward with new products and features.

Testing today

Continuous integration

On each change proposal to our software we run tests. For the firmware in the device we build multiple different configuration and run unit-tests. For the Crazyflie client and the Python library we make sure we can build for Linux and Windows and that the code passes our style guidelines. If any test fail we go back and update the proposed changed and re-run the tests. This is our first level of defense against defects.

Release testing

For each release we follow a checklist of procedures and tests to make sure that quality does not degrade. We make sure all the examples in the Python library are working and that the Crazyflie can fly around as expected in our flight arena, using various positioning systems.

Limitations

The way we test today makes it very difficult to determine if we regress in, for example, flight capability or in radio communication quality. We either test different software packages in isolation, without hardware in the loop, or we test by having a human trying to estimate if any degradation has happened since the last release.

We run into scalability issues as our ecosystem grows, it is near impossible to test all the different combinations of products. And it is very hard for us to detect stability or quality regressions without having a more systematic approach to testing the software on relevant hardware.

Introducing the test lab

What we have done so far are two things:

  1. Created an infrastructure for setting up a site for testing our software on real devices
  2. Prepared a test lab in our printer room at our office in Malmö

Infrastructure: crazyflie-testing

This new Git repository contains the building blocks we need to setup our new test lab. You can check out the repository README.md file for information on how to run it.

It contains a beginning of an automated QA test suite and along with that a start of a set of requirements for those tests. The requirements are specified in TOML files which are parsed and accessed from the tests. They are also rendered to markdown, to be easier for human consumption.

The repository contains a way to specify a test site, which is a collection of devices to run the test suite against. You specify your site in a TOML file which contain information about each device, such as which decks are connected and the radio:// URI. This site specification is then used by the test framework when running the tests, making sure each test is run against all devices compatible with that test.

The infrastructure also has management scripts to perform tasks like flashing all devices in a site, or recovering them if they go into boot loader mode by accident. All aimed at being able to handle testing with the least amount of human intervention possible.

The most recent addition to the infrastructure is the ability to test swarms using the Crazyswarm project!

Crazylab Malmö

We define the site crazylab-malmö.toml which is the test lab we have setup in our printer room in Malmö, Sweden, it is currently defined as:

version = 1

[device.cf2_flow2_lighthouse]
radio = "radio://0/50/2M/E7E7E71706"
decks = ["bcFlow2", "bcLighthouse4"]
bootloader_radio = "radio://0/0/2M/B19CF77F05?safelink=0"

[device.cf2_flow2]
radio = "radio://0/50/2M/E7E7E71705"
decks = ["bcFlow2"]
bootloader_radio = "radio://0/0/2M/B16298A8A3?safelink=0"

[device.cf2_flow2_multiranger]
radio = "radio://0/50/2M/E7E7E71704"
decks = ["bcFlow2", "bcMultiranger"]
bootloader_radio = "radio://0/0/2M/B1CEE678C5?safelink=0"

[device.cf2_usddeck]
radio = "radio://0/50/2M/E7E7E71703"
decks = ["bcUSD"]
bootloader_radio = "radio://0/0/2M/B164C8AC96?safelink=0"

[device.cf2_lighthouse]
radio = "radio://0/50/2M/E7E7E71702"
bootloader_radio = "radio://0/0/2M/B1F519F77B?safelink=0"
decks = ["bcLighthouse4"]

[device.cf2_stock]
radio = "radio://0/50/2M/E7E7E71701"
bootloader_radio = "radio://0/0/2M/B177790F3A?safelink=0"

There are, for now, six devices with different combination of decks attached. Each night, after midnight, different GitHub actions starts working for us. One job builds the latest versions of the firmwares for the devices another job will upgrade all the devices in the lab and run the test suite, and a selection of the Python library examples. We will be notified if any of these jobs fail.

This acts as a safety net for us. We will quickly know if the communication performance degrades, or if we have messed up with our parameter or logging frameworks. And we can catch silly firmware bugs as early as possible.

Future

We want to keep adding test cases and other infrastructure to our testing framework. Going forward it would be really nice to be able to test positioning systems in some way. And of course, some type of test of flight, be it free flying tests or using some kind of harness.

It might be interesting to look into adding simulations (hardware in loop or not) to our testing setup. It is all a question of bandwidth, there are a lot of cool things to work on, and a limit on time and bodies.

You can help! You might even be able to help yourself while helping us! If you contribute tests that correspond to your use-case, then you can relax knowing that those tests will run each night, and that Bitcraze engineers will be notified the minute they fail.

Or you can define your own site and run the test-suite against all your devices to make sure nothing strange is going on.

Hopefully this infrastructure and lab will help all of us to do more cool stuff using the Bitcraze ecosystem!

We are happy to announce the availability of the 2021.06 release of the Crazyflie firmware and client! This release includes bugfixes related to flashing the Crazyflie and the Lighthouse deck and also the new concept of core parameters and logging variables, that we talked about in the blog post: Crazyflie logging and parameters interface.

There are new binaries for the Crazyflie (2021.06), the Crazyflie client (2021.6) as well as a new release of the Crazyflie python library (0.1.16). The firmware package can be downloaded from the Crazyflie release repository (2021.06) or can be flashed directly using the Crazyflie client.


Noteworthy features and fixes

Crazyflie STM firmware

* Added an app layer application of the wall following demo
* Fixed a bug where the initial position of the Kalman estimator was set to (1.5, 0, 0)
* Enable an estimator to be added by out of tree build
* Add and document core parameters and log variables
* Fix deck info mem implementation can return uninitialized buffer
  - Caused: Flashing errors with Flowdeck
* Rechecked INDI controller for mistakes/consistency

Crazyflie NRF firmware

* pm: Increase delay before powering system
  - Caused: Issues flashing Lighthouse deck

Crazyflie Python library (0.1.16)

* Add support for crazyflie-link-cpp
* Add get_value to parameter class
* Make tutorial more user friendly
* Improve reset_to[firmware|bootloader]

Crazyflie Client

* Do not allow upgrading firmware when connected to USB
* Console tab: new messages should always appear at the bottom
* Update install doc

Documentation

* Core logging groups and variables
* Core parameter groups and variables
* Improved motion commander tutorial

Help us out!

We always strive to release quality software and firmware, but we are not perfect! Please help us out by installing the new library, client and firmware and make sure that your applications, tools and algorithms still work as you expect! And if not please file an issue with us or contact us via the forum.

Happy hacking!

Background

In the past couple of weeks we have been busy trying to improve the development interface of the Crazyflie. We want to make developing with and for the platform a more pleasant experience.

We have started looking at the logging- and parameter framework and how to improve it for our users. The aim of this framework is to easily be able to log data from the Crazyflie and to set variables during runtime. Your application can use them to control the behavior of the platform or to receive data about what it is currently up to. As of today, in the firmware there are 227 parameters and 467 logging variables defined.

View from the cfclient of the different logging variables one could subscribe to

These logging variables and parameters have been added to the Crazyflie firmware over the course of years. Some are critical infrastructure, needed to be able to write proper applications that interface with the platform. Some are duplicates or were added as debug years ago. Others have in some way outlived their usefulness as the firmware and functionality has moved on. The problem is that we have no way of conveying this information to our users and this is what we are trying to rectify.

An attempt of stability

We are currently reviewing all of our logging variables and parameters in an attempt to make the situation clearer for our users … and ourselves. We are adding documentation to make the purpose of each individual parameters and logging variables more clear. And we are also dividing them up into two categories: core and non-core.

If a parameter or logging variable is marked as core in the firmware that constitutes a promise that we will try very hard to not remove, rename or in any other way change the behavior of it. The idea is that this variable or parameter can be used in applications without any fear or doubt about it going away.

If a variable or parameter is non-core it does not mean that it is marked for removal. But, it could mean that we need more time to make sure that it is the proper interface for the platform. It means that it could change in some way or in some cases be removed in later firmware releases.

The reason for doing this is twofold: we want to make the Crazyflie interface clearer for our users and we want something that we feel we can maintain and keep an up-to-date documentation of.

What is the result?

We have introduced a pair of new macros to the firmware, LOG_ADD_CORE and PARAM_ADD_CORE which can be used to mark a parameter or variable as core. When using these we also mandate that there should be a Doxygen comment attached to the macro.

Below is an example from the barometer log group, showing the style of documentation expected and how to mark a logging variable as core. Parameters gets treated in the same way.

/**
 * Log group for the barometer
 */
LOG_GROUP_START(baro)

/**
 * @brief Altitude above Sea Level [m]
 */
LOG_ADD_CORE(LOG_FLOAT, asl, &sensorData.baro.asl)

/**
 * @brief Temperature [degrees Celsius]
 */
LOG_ADD(LOG_FLOAT, temp, &sensorData.baro.temperature)

/**
 * @brief Air pressure [mbar]
 */
LOG_ADD_CORE(LOG_FLOAT, pressure, &sensorData.baro.pressure)

LOG_GROUP_STOP(baro)

We have also added a script In the firmware repository: elf_sanity.py. The script will return data about parameters and logging variables that it is included in a firmware elf. This can be used to count the number of core parameters. If we point it to a newly built Crazyflie elf, after we’ve done our initial review pass of the parameters and variables, we get the result below.

$ python3 tools/build/elf_sanity.py --core cf2.elf 
101 parameters and 78 log vars in elf

To produce a list of the parameters and variables you can add the --list-params and --list-logs options to the script.

What is the next step?

Once we have finished our review of the parameters and logging variables we will explore different ways of making the documentation of them available in a clear and accessible way. And we will come up with a scheme for making changes to the set of parameters and variables. Once this is all finished you can expect an update from us.

The end goal of our efforts is making developing for the Crazyflie a smoother process, and we would love to hear from you. What is confusing? What are your pain points? Let us know! So we can do better.

Hi all! I am Jonas Danielsson and I started at Bitcraze on the first of March. I am a software developer with experience of Linux-, open source- and embedded development. Sometimes all at once.

Jonas and Bosse out hiking

I have worked with embedded systems of different sizes since 2007 and look forward to getting to know the Crazyflie and all of its ecosystem.

I am interested in and care a lot about software- and product maintenance. At Bitcraze I hope to be involved both in developing new features, fixing bugs and finding ways of keeping the product and code in nice shape. To find ways of working that allow us to add a bunch of cool stuff as fast as we want to, without endangering the cool stuff we already have.

Also, I am interested in getting to know the Crazyflie community and to work together with you to create the best development experience we can achieve. Do not be afraid to reach out with ideas or suggestions on how we can improve!