Global semiconductor shortage and price updates

2021-04-26 | Marcus | Leave a comment

During the last year we, like many others, have had to deal with new challenges. Some of them have been personal while some have been work related. Although it’s been a challenge for us to distribute the team, we’ve tried to refocus our work in order to keep making progress with our products.

Our users have of course had challenges of their own. Having a large user-base in academia, we’ve seen users having restricted physical access to hardware and purchasing procedures becoming complex. Some users have solved this by moving the lab to their homes using the Lighthouse positioning or the Loco positioning systems. Others have been able to stay in their labs and classrooms although under different circumstances.

At Bitcraze we have been able to overcome the challenges we’ve faced so far, largely thanks to a strong and motivated team, but now it seems as if one of our biggest challenges might be ahead of us.

Semiconductor sourcing issues

Starting early this year lead-times for some components increased and there were indication that this might become a problem. This has been an issue for other parts like GPUs and CPUs for a while, but not for the semiconductors we use. Unfortunately sourcing of components now has become an issue for us as well. As far as we understand the problems have been caused by an unforeseen demand from the automotive industry combined with a few random events where the production capacity of certain parts has decreased. Together it has created a global semiconductor shortage with large effects on the supply chain.

For us it started with the LPS Node where the components suddenly cost twice as much as normally. The MCU and pressure sensor were mainly to blame, incurring huge price increases. Since these parts were out of stock in all the normal distributor channels, the only way to find them was on the open market. Here price is set by supply and demand, where supply is now low and demand very high. Prices fluctuate day by day, sometimes there’s very large swings and it’s very hard to predict what will happen. To mitigate this for the LPS Node we started looking for an alternative MCU, as the STM32F072 was responsible for most of the increase (600% price increase). Since stock of the LPS Node was getting low we needed a quick solution and found the pin-compatible STM32L422 instead, where supply and price was good through normal channels. The work with porting code started, but after a few weeks we got word that importing this part to China was blocked. So after a dead end we’re back to the original MCU, with a few weeks of lead-time lost and a very high production price.

Unfortunately this problem is not isolated to the LPS Node, the next issue we’re facing is the production of the Crazyflie 2.1 where the STM32F405, BMI088, BMP388 and nRF51822 are all affected with increases between 100 and 400 % in price. These components are central parts of the Crazyflie and they can not easily be switch to other alternatives. Even if they could, a re-design takes a long time and it’s not certain that the new parts are still available for a reasonable price at that time.

Aside from the huge price increases in the open market we’re also seeing price increases in official distributor channels. With all of this weighed together, we expecting this will be an issue for most of our products in the near future.

Planning ahead

An even bigger worry than the price increase is the risk of not being able to source these components at all for upcoming batches. Having no stock to sell would be really bad for Bitcraze as a business and of course also really bad for our customers that rely on our products for doing their research and classroom teaching.

To mitigate the risk of increasing price and not being able to source components in the near future, we’re now forced to stock up on parts. Currently we are securing these key components to cover production until early next year, hoping that this situation will have improved until then.

Updated product prices

Normally we keep a stable price for a product once it has been released. For example the LPS Node is the same price now as it was in 2016 when it was released, even though we’ve improved the functionality of the product a lot. We only adjust prices for hardware updates, like when the Crazyflie 2.0 was upgraded to Crazyflie 2.1. But to mitigate the current situation we will have to side-step this approach.

In order for us to be able to continue developing even better products and to support those of you that already use the Crazyflie ecosystem, we need to keep a reasonable margin. From the 1st of May we will be adjusting prices across our catalog, increasing them with 10-15%. Although this doesn’t reflect the changes we are seeing in production prices at the moment, we believe the most drastic increases are temporary while the more moderate ones will probably stick as we move forward.

Even though times are a bit turbulent now, we hope the situation will settle down soon and we think the actions we are taking now will allow us to focus on evolving our platform for the future.

Successful emulation

2021-04-12 | Max | Leave a comment

Approximately two month ago we wrote a blog post presenting our planned master thesis. Time flies and we have now reached a sufficient state where the results are presentable and possible to use. We have used the Renode framework and created a platform for the Crazyflie 2.1. In Bitcraze’s Github repository there now is a Renode fork with a custom Renode-infrastructure submodule. To get Renode up and running on your computer check the README found there.

On the Renode branch crazyflie there are two new REPL (Renode platform) files describing the platform. An example of the syntax is given below.

// I am a comment peripheralname: Namespace.ClassName @ parent 0x08000000 numericConstructorField: 0x100000 stringConstructorField: "template" Interrupt -> interrupthandler@3

In the cf2.repl all the external peripherals are connected while the stm32f405.repl contains the STM32F405 peripherals. Note that only the peripherals used by the current Crazyflie 2.1 have been added since they are the only ones that can be tested using the Crazyflie firmware.

When running Renode a RESC (Renode script) file is loaded. There are currently two RESC files for the Crazyflie, one that only loads the Crazyflie plattform and one that can be used for testing. The one for testing automatically starts the simulation and it also has a hook to exit Renode once the self test has passed.

As mentioned, Renode is usable both for automatically and interactively testing firmware. The current version of automatic testing is based on the firmware passing the self test. The plan is to incorporate this in the CI pipeline.

When used interactively it is possible to pause the emulation whenever the user wants to, either manually via a Renode command or by connecting to GDB. This allows reading (as well as writing to) memory addresses. Want to read the DMA status at a specific line of code or mess with the system by randomly flipping bits? Doable in the emulator without risking your Crazyflie crashing.

In the platform there also are our customized sensors to which data can be loaded. The data can be loaded either manually or via a file and then sent to the STM32. The scope of this master thesis however has been on firmware testing, not getting a simulated Crazyflie to fly in a virtual environment.

Emulation of hardware is not a trivial task, there are still improvements to be made and the everlasting question whether the emulation actually represents the real system.

One of the Crazyflie features that had to be simplified was the syslink and connection to the nRF microcontroller, which in the emulation simply sends messages back to the STM32. The most exciting part about it currently is how the STM32 receives a signal that no expansion decks are connected via the 1-wire when a scan is executed. Further improvements would be to emulate the radio, power management and support expansion decks, either via an external program or through Renode.

Of course there are other things to improve as well, there will always be someone who thinks of better ways to implement features and only time will tell how this emulator is going to evolve.

Josefine & Max

Event-Based Data Logging

2021-03-15 | Wolfgang Hoenig | Leave a comment

One crucial aspect of any research and development is to record and analyze data, which then can be used for quantifying performance, debugging strange behavior, or guide us in our decision making. We have been trying to improve the way that this is done to help all of you researcher out there with their work, so this blog post will explain an alternative, better, method to record whatever is happening on the Crazyflie in real-time.

Example data collection of received Lighthouse angles over time. The y-axis contains a unique measurement ID (16 in total for 4 sensors * 2 basestations * 2 sweeps/basestation). Thus, each dot represents the time when a certain kind of measurement was received. We do not receive all angles in fixed intervals, because of the interference between basestations.

Existing Logging Approaches

So far there have been two principle ways of recording data with the Crazyflie:

Logging: In the firmware, one can define global variables that can be streamed out at a fixed frequency, using logging configurations. Variables have a name and data type, and the list of all available variables can be queried from the firmware.
Debug Prints: It is possible to add DEBUG_PRINT(…) in any place that contains a string and possibly some variables. These are asynchronous, but not timestamped, so they are mostly useful to notify the user of some status change.

Both approaches have multiple backends. The logging variables can be streamed over USB, via the Crazyradio, or on a Micro-SD-card using a specialized deck. When using the radio or USB, the frequency is limited to 100 Hz and only a few variables can be streamed at this speed due to bandwidth limitations. The Micro-SD-card deck, on the other hand, allows to log a lot more logging variables at speeds up to 1 kHz, making it possible to access high-speed sensor information, e.g., the IMU data. The Debug Prints can be used over the radio or USB, or with a SEGGER J-Link and a Debug Adapter, over J-Link Real-Time Transfer (RTT). The former is very bandwidth limited, while the latter requires a physical wire connection, making it impossible to use while flying.

New Approach: Event-Based Logging

Event-based logging combines the advantages of the existing approaches and offers a third alternative to record data. Similar to debug prints, a user can trigger an event anywhere in the code and include some mandatory variables that describe this event (so-called payload). Similar to logging, these events are timestamped and have fewer bandwidth limitations. Currently, there is just one backend for the Micro-SD-card deck, but it would be possible to add support for radio and USB as well.

One of our first test cases of the event-based logging is to analyze the data we get from the Lighthouse deck. Here, we trigger an event whenever we received a raw sweep. This allows us to visually see the interference that happens between two LH2 basestations. We also use the event-based logging for time synchronization of the Crazyflie and a motion capture system: when we enable the IR LEDs on the active marker deck, we record an event that contains the Crazyflie timestamp. On the PC side, we record a PC timestamp when the motion capture system first detects the IR LEDs. Clock drifts can be computed by using the same logging mechanism when the IR LEDs are turned off.

Adding event-based logging had some other good side effects as well: the logging is now generally much faster, there is more user feedback about the correct buffer size usage, and the binary files are smaller. More details are in the documentation.

Future Work

We are working on adding event triggers to the state estimator. This will enable us to record all the sensor information during real flights with different positioning systems, so that we can tune and improve the Kalman state estimator. It will also be interesting to add support for events in CRTP, so that it can be used over USB and radio.

A new addition to the team: Jonas Danielsson

2021-03-08 | Jonas Danielsson | Leave a comment

Hi all! I am Jonas Danielsson and I started at Bitcraze on the first of March. I am a software developer with experience of Linux-, open source- and embedded development. Sometimes all at once.

I have worked with embedded systems of different sizes since 2007 and look forward to getting to know the Crazyflie and all of its ecosystem.

I am interested in and care a lot about software- and product maintenance. At Bitcraze I hope to be involved both in developing new features, fixing bugs and finding ways of keeping the product and code in nice shape. To find ways of working that allow us to add a bunch of cool stuff as fast as we want to, without endangering the cool stuff we already have.

Also, I am interested in getting to know the Crazyflie community and to work together with you to create the best development experience we can achieve. Do not be afraid to reach out with ideas or suggestions on how we can improve!

Communication Reliability

2021-02-22 | Wolfgang Hoenig | Leave a comment

Communicating with your Crazyflie is an important pillar for its operation. As more robots are controlled, the reliability of this communication link becomes more and more important, as the probability that there is no failure at any of the Crazyflies decreases exponentially with the number of robots. We have written about the low-level radio link before. Today, we focus on past, on-going, and future improvements to make the communication more reliable.

Reliability Challenges

As part of doing research with the Crazyswarm, I noticed several issues:

Sometimes commands do not seem to arrive at the Crazyflie, especially when using unicast (i.e., direct) communication with a specific Crazyflie.
Sending ‘too much’ data while the Crazyflie is flying can cause catastrophic crashes. One example is trying to upload a trajectory, while flying in a motion capture space. However, this occasionally even happens if just trying to update a parameter.

The radio link has a feature called Safelink, which essentially guarantees that packets are send and received in order and no packet gets dropped. This feature never worked reliably in the crazyflie_cpp implementation and is therefore not used in the Crazyswarm. However, it is the default mode for cflib. The Crazy RealTime Protocol (CRTP) also has the notion of different communication ports in order to prioritize important messages such as control commands over less important ones such as trajectory upload. However, this prioritization was never implemented in any of the clients.

Native Link Implementations

Another, non reliability related, issue always was that the performance of cflib is not stellar when connected to many Crazyflies. This is mostly because (C)Python has a Global Interpreter Lock (GIL), which prevents true multi-threaded operation. A common solution to allow true parallelism is multiprocessing or implementations in a language that compiles to machine code (native code).

Crazyflie-link-Cpp

The first native implementation is written in C++ and includes Python bindings using pybind11. The overall API is simple: A connection can be created given a URI, and data can be send and received. Internally, this library implements Safelink to guarantee packets are in order, uses priority queues to prioritize messages on important CRTP ports, and handles all multi-radio and multi-thread related synchronization issues. Unlike prior implementations, the bandwidth is shared uniformly between all connections, i.e., there is no race between different Crazyflies communicating over the same radio. We measured a 50% lower CPU utilization when connecting with the CF client, a 20% higher bandwidth, and a 20% lower latency compared to the pure Python implementation.

For now, this feature is experimental and needs to be enabled using an environment variable. If you want to give it a try, you can use

pip install cflinkcpp
USE_CFLINK=cpp cfclient

once everything is merged to master in the next couple of days.

Crazyflie-link-rs

The second native implementation is written in Rust. As it turns out, writing multi-threaded code in C++ is very difficult and error-prone, because it is up to the user to use the various synchronization primitives correctly. Rust, on the other hand, provides extensive compile-time checks to increase reliability by design. This implementation does not have quite as many features as the C++ version yet and requires a bit more work to be used with the client but it is going to be worked-on on Fridays. The goal is to reach functional parity with the other link implementations and be able to use the rust-implemented link with “USE_CFLINK=rs cfclient” in the future.

Future Work

As part of the development process, we also wrote many system tests that verify correctness and measure performance of the various link implementations. We already found and fixed several firmware bugs (681 and 688). The major open issue is that there is no flow control on the system link (the connection between the NRF51 and the STM32), which still causes packet loss in some cases.

Hardware emulation thesis

2021-02-15 | Max | Leave a comment

Hi!

Since the middle of January, Bitcraze has had two additional guests: us! We, Josefine & Max, are currently doing our master thesis at Bitcraze during our final semester at LTH.

Unfortunately, the pandemic means remote working which could have caused some difficulties with hardware and equipment accessibility. Fortunately, for us, our goal with the thesis is to emulate the Crazyflie 2.1 hardware in the open source software Renode. We can therefore do the work at home.

Since this is the first time either of us have tried to emulate hardware it is exciting to see if it will even be possible, especially as we do not know what limitations Renode might have. Thus far, four weeks in, there have been some hiccups and crashes but also progress and success. One example was when we got the USART up and running and it became possible to start printing debug messages and another was when the LED lights were connected and it was possible to see when they were turned on and off.

Example of output from partially implemented hardware.

If all goes well, the emulated hardware could be used as a part of the CI pipeline to automatically hunt for bugs. Academically, it would be used to further study testing methods of control firmware.

Getting a glimpse into the workings at Bitcraze and the lovely people working here has been most interesting and we are looking forward to the time ahead. Until next time, hopefully with a working emulation, Josefine & Max.

Experimental battery testing feature

2021-02-08 | Tobias | 1 Comment

The lithium polymer battery we use, as with basically all rechargeable batteries, suffers from degradation. That means that when using it, and as time goes by, its energy capacity as well as performance will degrade. The performance, which is very related to the batteries internal resistance, will result in that the Crazyflie will not be able to produce the same maximum thrust and it will not be able to carry as much payload. The loss of capacity is due to ageing and charge cycles, results in that the flight time will decrease. A common solution to monitor the degradation is to have a BMS, or Battery Management System, that constantly monitors the battery health. For the small type of battery that is used in the Crazyflie, this is not yet viable, but maybe there is something we can do to test part of the battery health anyway?

Theory

Since the internal resistance will result in a larger voltage drop during load we can exploit this property and measure it. We will however not only measure the batteries internal resistance but the resistance of the complete power path as a result of the components we have at hand on the Crazyflie.

So what we do is to activate the switch (mosfet) so the load (motor) will pull power from the battery. The power drawn will result in a voltage drop compared to a no-load situation, which we can measure and compare to a healthy setup. Since the measurement point is at the PCB traces, any of the components before that point can be causing the voltage drop, however the battery and connector are most likely of doing so as they are most prone to ageing.

Implementation

The load is achieved by, for a very short time, activate the motors at full thrust. We don’t want the Crazyflie to fly away as that would be a bit unhandy. Before activating the motors we measure the idle voltage and during load we measure the minimum voltage so we can calculate the voltage drop. This is pretty easy to do, the problem is to find a good level where we can distinguish a good battery from bad battery. Therefore this feature is pretty experimental. We tested many batteries and good batteries tend to yield a voltage drop between 0.60V – 0.85V while bad batteries go above 1.0V. Therefore the current threshold is set to 0.95V but it would be good to have more data so if you use this feature please give us feedback if the level is wrong. The testing was run on a “stock” setup with the standard battery, propeller and motors, and it is for these the level is set. A different setup will probably not work well and needs a different threshold. Also keep in mind that the connector can also be a “bad” guy as oxide can build up and result in a higher resistance. Often this can be solved with some e.g. WD-40 solvent or un-connecting/connecting the connector several times.

Usage

This is not in the 2021.01 release so one would have to run the latest on the master branch on both the crazyflie-firmware and the Crazyfie-clients-python. The simplest way to test this feature is to launch the cflient, connect to the Crazyflie, open the console tab and press the battery test button.

cfclient console tab after running battery test

When pressing the button the propellers will shortly spin and there will be an output in the console as highlighted in red. If the sag value is below 0.95V it will yield [OK] and if it is above it will say [FAIL].

A probably more useful use case is to test this automatically before taking of with e.g. a swarm. This can be done by setting the parameter health.startBatTest to 1 and after around 0.5s readout the result in the log variable health.batteryPass to check that it is set to 1. The health.batterySag log variable will contain the latest sag (voltage drop) measurement. Hopefully this experimental feature will be a good way of increasing reliability of flights.

Release 2021.01 available for download

2021-02-01 | Kristoffer Richardsson | 5 Comments

We’re happy to announce the availability of the 2021.01 release! The release includes the Crazyflie firmware (2021.01), the python library (0.1.13.1) and the python client (2021.1). The firmware package can be downloaded from the Crazyflie release repository (2021.01) or can be flashed directly using the client bootloader window.

Most of the improvements have been done in the Crazyflie firmware and include:

The App API in the Crazyflie firmware has been extended and improved to be able to handle a wider range of applications. The goal is to enable a majority of users to implement the functionality they need in an app instead of hacking into the firmware it self.

We have improved the Lighthouse support in the firmware and both V1 and V2 are now working well. Even-though everything is not finished yet, we have taken a good step towards official Lighthouse positioning.

A collision avoidance module has kindly been contributed by the Crazyswarm team.

A persistant storage module has been added to enable data to be persisted and available after the Crazyflie is power cycled. It will initially be used to store Lighthouse system information, but will be useful for many other tasks in the future.

Basic arming functionality has been added for platforms using brushless motors.

In the client the LPS tab now has a 3D visualization of the positioning system and a new tab has been added to show the python log output.

Unfortunately we have run into some problems for the Windows client build which is not available for this release.

Finally we have fixed bugs and worked to improve the general stability.

We hope you enjoy it!

Our new (temporary) team member: Wolfgang Hönig

2021-01-18 | Wolfgang Hoenig | 3 Comments

Hi, fellow Crayzflie’ers! I am Wolfgang Hönig and I am a visiting researcher at Bitcraze until early April. I am excited to be part of the team and will be working on making the Crazyflie (and its ecosystem) even more accessible, more robust, and versatile for other researchers and users.

I started working with the Crazyflie (1) back in 2014 as preparation for grad school. It was the only affordable, small, and open-source robot that I could find and helped me a lot to gain practical knowledge for controls, state estimation, and realtime systems. Now, 7 years later, it is still an exceptional product for the same reasons. During my PhD at the ACT Lab at the University of Southern California, I developed crazyflie_cpp, crazyflie_ros and, together with James A. Preiss, the Crazyswarm. The Crazyswarm in particular has helped me (& many other researchers) to validate robotics research (I worked mostly on multi-robot motion planning). After my PhD, I joined ARCL at Caltech as a postdoctoral researcher, where we used the Crazyflies for research in novel techniques that combine machine learning, motion planning, and controls (see GLAS and Neural-Swarm). Starting May 2021, I’ll be leading my own research group at TU Berlin.

At Bitcraze, I am looking forward to a close collaboration with the team. So far, we are planning to improve the low-level communication over the radio (joint work with Arnaud) and data collection and improvements to the state estimator by using the unique flight arena that has a motion capture, Lighthouse, and Loco Positioning system (joint work with Kimberly). If you are a researcher and have any other ideas, feel free to reach out via email or a comment!

New storage subsystem and new CI

2021-01-11 | Arnaud | Leave a comment

Storage is one of these very simple functionality that actually ends up being quite hard to implement properly. It is also one of these functionality that is never acutely needed, it is possible to hack around it, so it gets pushed to be implemented later. Later is now, we have now implemented a generic persistent storage subsystem in the Crazyflie.

In Crazyflie 1.0, we originally stored settings in a setting block in flash and required the bootloader to change the settings. When designing Crazyflie 2.0 we added an I2C EEPROM to make it easier to store settings, though until now we only stored a fix config block very similar to the one stored in the Crazyflie 1 and that only contained basic radio settings and tuning. This implementation is hard to evolve since the data structure is fixed in one point of the code.

What is now implemented is a generic key-buffer database stored in the I2C EEPROM. From the API user point of view, it is now possible to store, retrieve and delete a buffer using a string as a key. This allows any subsystem, or apps, in the Crazyflie to easily store and retrieve their own config blocks. There is 7KB of space available for storage in the EEPROM.

The first user of this new storage subsystem is the Lighthouse driver. The storage is used to store lighthouse basestation geometries and calibration data, this allows to configure a Crazyflie for a system/lab and have it running out of the box even after a restart.

A future use-case would be to implement stored-parameters: we have been thinking about implementing optional persistence for the parameters for a long time. This would allow to modify and then store new default values for any parameters already present in the Crazyflie. This would allow to very easily implement things like custom controller tuning in a quad made from a bolt for example.

At low level, we where hopping to be able to find a ready-to-use library or file system to store data in our small EEPROM, but unfortunately we did not find anything that would fit our needs. We then had to implement our own storage format.

The low level structure is documented in the Crazyflie firmware repos. Basically the data are stored as a table of “length-key-value” entries with a possibility for an entry to he a “hole”. When new buffers are added they are added at the end of the table and when they are deleted they are replaced by a hole. When the end of the table is reached, the table is de-fragmented by removing the holes and moving the data as much as possible to the beginning of the memory. This structure works very well for an EEPROM and could even be adapted to work well on FLASH.

New CI

When we started activating continuous integration/automatic build to our GitHub repos we did so using Travis CI for firmware builds and AppVeyor for windows builds. However, the GitHub CI offering, GitHub actions, has become quite complete lately and now supports Linux, Windows as well as MacOS builds.

We have now transitioned to GitHub actions for all our repos and we will also implement most of the release process using GitHub actions as well. This will hopefully streamline the release process and allow us to release new version of our projects more often.