Category: AI-deck

Last week we had the first ever Bitcraze DEV meeting! With about 10 participants, we covered a range of topics. The meeting was mostly focused around how to handle support and what the DEV meetings should be about. We also had a chance to get some feedback, and one of the points was sharing a bit more what we’re currently working on and what we might work on in the future. So in the light of that, this blogpost is about CPX (the Crazyflie Packet eXchange) protocol. We’ve mentioned CPX before (1, 2), but with this blogpost I want to share the current status and some thoughts on why we need something new.

As summer is approaching and things are winding down, I’m talking the opportunity to get back to the AI deck and CPX. The AI deck was officially released out of early access last month, but there’s still more work to be done with porting examples, adding some more functionality and increasing stability and performance.

For the AI deck we’re only supplying examples, there’s no functionality that will be used with the platform (except for the WiFi connection maybe). This is in contrast to for instance the Flow deck, where there’s a specified functionality the user can use and that should work. So in order to move forward I came up with a little demo that I want to get working during the summer. The goal is to make an application where I can fly around the Crazyflie with the keyboard and get a video stream back. To achieve this I’m using the Flow deck together with the AI deck and using WiFi for both CPX and CRTP (to send command and to get images and logging).

Why we need something new

I’ve written a post about CPX in the past (link) where I detailed the issues we are trying to solve. But in short we needed was a protocol that …

  • … could be routed though intermediaries to reach it’s destination
  • … could handle high transfer rates with large amounts of data as well as small messages
  • … could handle different memory budgets
  • … doesn’t drop data along the way if some parts of the system is loaded

As the Crazyflie echo system grows and becomes more complex we need new tools to work with it. When CRTP was implemented many years ago, the complexity we have today wasn’t something we could imagine. The Crazyflie had the only MCU and the hardware on the decks were used directly from it. Now we have multiple decks with more complex systems on them: AI deck (2 MCUs), Active marker deck (1 MCU) and the Lighthouse deck (1 FPGA). Looking forward these more complex decks might increase in the future. With more and more functionality in the Crazyflie and resources occupied, like DMA channels and pins, some functionality might need to move further out onto the decks.

For each deck new protocols are implemented and specific code is needed in the Crazyflie to handle it. Some things also become complex, like getting printouts from the different MCUs on the decks. So for the AI deck we wanted to test something new and more generic to see if it would be something we could use more in the future to talk directly to different MCUs in the system.

Will CPX replace CRTP? Probably not. We’re not sure what solution we will land in, but I think CPX is a good step in the right direction.

Current status

Back to my little demo. To reach the goal there’s a few things which needs to be fixed:

  • crazyflie-firmware/#1065: When starting to run CRTP over CPX (via WiFi) I’ve noticed that the UART2 driver was too slow, loading the system too heavily and creating problems down the line. So this is being worked on, and at the same time the old SYS-link over UART2 implementation is being moved to CPX instead.
  • aideck-esp-firmware/#12: We’ve had reports of intermittent performance issues for WiFi, which is also effecting.

Aside from the issues there’s also a few other features that are being added:

  • CRTP over CPX: Since I already have a connection for the images I also want to use this for controlling the Crazyflie. The latency is too high for controlling roll/pitch/yaw in real-time, but in my case I have the Flow deck for position control
  • CPX over CRTP: Although not part of the demo, this is interesting to look at for the future. One example is that right now we have an implementation where the Crazyflie firmware has a special implementation for the WiFi credentials. If we would like to set it from the ground we would first have to do CRTP to the Crazyflie, re-package it and then send it via CPX to the ESP32 on the AI deck. Instead I would like to send it via CPX directly from the ground, saving us extra work and complexity in the Crazyflie
  • Using Zeroconf/mDNS for finding AI decks: With this changes it will be possible to connect to the Crazyflie via the client, so we need a way to find the AI decks. For this Zeroconf/mDNS has been added, so AI decks will be automatically discovered on the local network.

The current status can be seen in the following draft PRs: crazyflie-firmware/#1068 and crazyflie-lib-python/#342. Note that until these are real PRs (not draft) they are not useful, so don’t try to use them yet.

CPX documentation

For more information on CPX and how it’s implemented, check out the documentation on our website we well as the specific documentation on using it from the GAP8.

We have worked hard last week to get a new fresh release out before the summer months are on our doorstep. Not only that we would like to make sure that important bugs are fixed before some of us go on our holiday, but also to be able to display our new AI deck features! Here is an overview of what has been changed

AI deck over air flashing

As you can probably see in the release notes of both the python libraries and the firmware, most of our changes are focused on making it possible to develop for the AI deck without using a programmer all the time. If the STM and NRF firmware of the Crazyflie is fully updated, and the ESP firmware on the AI deck, it should now be possible to flash an AI deck example binary with a Crazyradio! For older versions of the AI deck 1.X (Rev A to C) it is unfortunately still necessary to use the JTAG programmer one last time to flash a bootloader on the GAP8, but after that it should not be needed anymore.

Please check out the new update AI deck tutorial for setting up the AI deck for this new functionality.

Crazyflie Packet eXchange (CPX)

In the light of the work we have done for the AI deck, we also have started to implement a new, inter MCU protocol called the Crazyflie Packet Exchange. Since with the AI deck, we are adding 2 additional microprocessors to the Crazyflie architecture, it was crucial to handle the communication between all platforms and communication channels properly. Currently the functionality is mostly enabled to tailor Wifi streaming and console printouts for the AI deck, but it is meant to be a generic protocol which in the future, should be able to handle more combinations like for instance, command messages through wifi?

You can read about CPX in the crazyflie-firmware repository doc and we will be writing a more detailed blogpost about this later.

Controller Python bindings

For the last part of the Grand tour trip, we had a hackathon with the IMRC lab of TU Berlin and our close collaborator Wolfgang Hönig, in which we managed to convert the PID controller, Mellinger controller and the motor mixing into python bindings, which can be used in the experimental simulator of the Crazyflie.

There is no Pypi release of these, you will need to pull the latest crazyflie-firmware repo and build the bindings with ‘make bindings_python’

Additional fixes

We have some additional fixes to both the python libraries and firmware. For the STM we have updated the STD peripheral library and solved several build issues. For the cfclient, we fixed a lot of issues that were caused by either the latest version of python, as it was more stricter with type definitions, and some issues QT. Moreover, the LED ring headlight functionality has been restored, and the cfbridge.py script, used for the PX4 crazyflie 2.1 tutorial, is re-added, since it suddenly disappeared a few releases ago.

Update and Feedback

Make sure to update your cfclient with ‘pip install cfclient –upgrade’ and to reflash the new stable firmware. For AI deck users, try out our our new tutorial to try out both CPX, the over air flashing and the wifi example. The new AI deck functionalities has been subjected to some limited testing so if there is anything wrong or unclear, please let us know in the forum! The feedback will help the AI deck to become a more stable product for development, so we would be very grateful if you would be able to help out with that.

I know a lot of you will be too distracted by chocolate to read this post, so I will make it short.

I am, too, a little distracted by sugar

As I mentioned earlier, we’re a little under-staffed right now. Jonas left us for new adventures, and Arnaud is enjoying some time with his baby (here in Sweden parental leave is thankfully long for dads too). On top of that, Kimberly was away the last two weeks to visit various labs in Europe. She will talk to you about it once she’s back, I’m sure. But with just 4 people at the office, time is a valuable resource. So what are we doing with it?

Well, a lot of that has been dedicated to the AI deck, but that’s not the only thing we’ve been working on. Recently, we had the visit of one expert on dangerous goods shipment. During 2 days, we got to learn about how to properly send the batteries we have, the regulations that are involved and what we have to implement to ship them. It may sound boring… and honestly, it was not the most interesting. But we got a certification out of it, that now allows us to ship as many batteries as we want with your order ! The 2 batteries only restriction that we have on the shop should be lifted – but please be aware that if you exceed 2 batteries per Crazyflie, the shipping cost will be higher, because of the fee Fedex imposes on dangerous goods shipments.

And speaking of Fedex, there are some problems right now on their air routes. Avoiding Ukraine and dealing with some strikes for air traffic operators in Europe has not been easy on their infrastructure, and we have experienced some delays in deliveries unfortunately. It seems to go back to normal gradually, so let’s hope their usual speediness resumes soon.

We’re also working on the Mini BAMs, which is on the 18th of May and will talk about drones for aerial show. Our special guest speakers are from Collmot and Flapper Drones, make sure to answer this survey if you want to participate ! You will get more informations soon.

And if want to play around with the AI deck, you will have an interesting occasion in September. IMAV launched a competition, where the goal is to have the Crazyflie equipped with the AI deck perform vision-based obstacle avoidance at increasing speeds. Deadline for registering are Mid-May, you can find more informations here.

We are now enjoying a long Easter week-end, recharging our batteries with families (and chocolate!), hoping that the Swedish spring finally settles here. I hope you’re enjoying it too !

A lot has happened at Bitcraze over the last months, which left us quite short-staffed. Thankfully, Victor has joined us again for a while. He mainly works on finishing his thesis with us, and we all agree that having an extra person at the office feels nice – especially considering the exciting stuff he’s working on! But let’s hear it from him first:

“Hi! I’m Victor, 26 years old, and studying towards a bachelor’s degree in Computer Science and Computer Engineering at LTH. I worked at Bitcraze during the summers of 2019 and 2020 and I’m now doing my bachelor’s thesis here.
During this thesis I will make a prototype deck that combines multiple ToF solid state lidar’s (more specifically, the new VL53L5CX). While there exists the Multi-ranger deck today, this new sensor outputs a matrix of distances, which opens up new possibilities that the Multi-ranger can not. Onboard the deck, there will also be an ESP32-S3, which will collect the data from the sensors and then send it to the PC, either through the Crazyflie, or through WiFi. This is all super exciting stuff and has endless potential, so let’s see how far I will get!”

Meet Victor!

I’m sure you will hear more on his progress in the next months, so make sure to keep updated!

Stock issues

We’ve been dealing with the component shortage as good as we can, but production is still unpredictable. Sadly, it means the impact on our stock is too. . The AI deck, the Bolt and the battery chargers are unfortunately out of stock right now. We had to change slightly the Swarm bundles to adjust to the lack of chargers. We’re also low on Multi Rangers, which are expected to run out of stock next week.

All those products are expected back by mid-May, if luck is on our side. It depends on our manufacturer in China, where there is sadly a new Corona outbreak, so it’s not easy to say for sure if this estimation is accurate. We hope that production and delivery stay unimpacted. Just know that we are working on getting everything back on stock as soon as possible. If you want to stay updated on the status of one of our out-of-stock product, you can choose to be informed by mail in our webshop. Just go to the product’s page, and put your email there: you’ll be the first one to know when it’s back in stock !

During the last couple of months we’ve been working on getting the AI-deck out of early-access. One of the things needed for this to happen is an improved infrastructure for the AI-deck, like bootloading and how the deck fits into the rest of the eco-system. For this there’s two new repositories:

CPX (Crazyflie Packet eXchange)

With the addition of two new MCUs (ESP32 and GAP8) as well as the possibility to connect via the WiFi, we quickly run into the issue of how to communicate between the targets. Even more so since there’s no direct access between some of these (like the Crazyflie<->WiFi, Crazyflie<->GAP8, GAP8<->WiFi).

What we needed was a protocol that …

  • … could be routed though intermediaries to reach it’s destination
  • … could handle high transfer rates with large amounts of data as well as small messages
  • … could handle different memory budgets
  • … doesn’t drop data along the way if some parts of the system is loaded

We decided to design and implement a new protocol, which we’ve named CPX (Crazyflie Packet eXchange). The protocol solves the issues above by:

  • Each packet has a source and destination ID, so it can be routed to (and from) the target of the packet
  • Each link between targets can have it’s own MTU, which allows each target to optimize memory usage. In order to handle this, intermediaries are allowed to split packages along the way, so data can be transferred in smaller pieces.
  • Instead of dropping packages if targets become overloaded, congestion in created in the system, where the sender will not be able to send more data until the receiver has been able to handle it.

Currently the new protocol is used for the GAP8 bootloader, for setting up the WiFi on the ESP32 and in the WiFi streamer example. But we’re hoping to expand it in the future to include more functionality, like logging and other plumbing that could be used in user applications.

WiFi configuration changes

With CPX it’s now possible to set up the WiFi from either the Crazyflie, the GAP8 or from the ESP32 itself. For doing this from the Crazyflie we’ve added the option of configuring this using KConfig, where we’ve added the following options in the expansion decks menu for the AI-deck:

  • Do not set-up the WiFi: Should be used if another target is setting up the WiFi, like the GAP8
  • Act as an access point: This will make it possible to connect to the AI-deck as an access point
  • Connect to an access point: This will connect the AI-deck to an access point using SSID/PASSWD entered in the menuconfig

GAP 8 bootloader

To make things easier for the user we want to remove the requirement of using a JTAG dongle to program the GAP8. In order to achieve this we’ve implemented a bootloader for the GAP8 which uses CPX, which means it can be used either from the Crazyflie or over the WiFi. We still haven’t had time to implement the Crazyflie part, where this will fit nicely together with the cload and client deck firmware upgrade, but it’s currently working via the WiFi. So until the implementation is done via the Crazyflie Python library, this script can be used to bootload and start your custom GAP8 firmware. Note though, that you will first have to flash the GAP8 bootloader and set-up the WiFi.

What’s next?

We’re continuing working towards getting the AI-deck out of early access. For CPX and the GAP8 bootloader there’s still a few bugs to iron out and examples to be updated as well as improved support for building using our toolbelt.

Hello everyone, my name is Rik and I’m Bitcraze’s new intern. I took up this internship as part of my MSc studies at the faculty of Aerospace Engineering at the Delft University of Technology. Over the past year I have worked with the AI-deck as part of my thesis work at the MAVLab, and I’m looking to contribute to Bitcraze with my practical knowledge of the platform.

For my MSc thesis I have primarily worked on the design of very small optical flow CNNs, capable of running on the GAP8 SoC, using a neural architecture search (NAS). I have implemented a pipeline on the AI-deck, feeding a stream of camera images into the CNN to determine optical flow on-board. One of the last remaining goals of my thesis work is to design an application which utilizes the resulting dense optical flow. In the meantime, the NAS is ever running to find the best possible architecture.

With my practical knowledge I hope to contribute to making the AI-deck an easier platform to work with. Of course, working at Bitcraze is a great learning opportunity, too. I’m already learning a lot about embedded systems and programming. After several years of studying, it’s great to get my first relevant working experience. And maybe most important of all, so far it has been a lot of fun working at Bitcraze and I expect to have a lot more of it. And yes, the falafel is really good.

The AI-decks are back in stock! Also, last week we had our quarterly meeting, where we plan our focus for the next quarter. As it is also the start of the fiscal year, we also take this opportunity to update our 1 year and 3 year plans as well. We have a big plans coming up, but one of the important focuses that we will have this year, is to get the AI-deck out of early-access!

But what would be necessary for such a task? The AI-deck is one of the most complicated boards we have worked with, so do we have to evaluate its ‘out-of-readiness’ along the same standards than any of our other products.

Mega AIdeck Tutorial

So one of our idea is to be able to achieve a state of the AIdeck in order to write a mega AIdeck tutorial series. This implies that we are able to show how somebody could go from a datasets all the way to a flying aideck-crazyflie combo. Such a series could consist of the following topics:

  1. How to go from a dataset of images to a (Deep) Neural Network
  2. Testing the DNN on the computer with the Image WiFi examples
  3. Converting the neural network to Tensorflow Light (with basics on Edge AI and quantizing neural networks)
  4. AIdeck basics (How to access the camera, how to add the network to the cluster, how to send commands)
  5. Build and flashing the AIdeck and testing it out in the hand
  6. Attach the AIdeck to the crazyflie, make an app-layer application to fly and react on the image input.

From the first look of it, this sounds like it should be easy to do right? Actually, there are still much to be done in order to make this tutorial possible.

Replumbing the Communication

One of the more challenging aspects of the AIdeck as it now is, is that users need to buy a JTAG-enabled programmer in order to flash the GAP8 and the NINA module of the AI-deck. That is the reason why currently the AIdeck has these 2 x 10 pin jtag connectors attached, but ideally we would want to get rid of it completely. This means is that we need to have over air flashing of the GAP8’s binary and that the intercommunication of the NINA and AIdeck will become even more important.

Moreover, the communication protocol from the GAP8 to the STM32 of the Crazyflie is currently very basic, as of right now, it is only possible to send single characters. It might work in some situations, but what if you would like to send an array of values back to the Crazyflie, like the collision probability & steering angle like in PULP platform’s implementation of Dronet? And, would we like to keep on using two UART serial ports or perhaps just relay both NINA and GAP8 communication all through one? The later will make it easier for us to maintain the crazyflie-aideck communication but can perhaps introduce communication delays.

These are just a slice of the type of re-plumbing work for the AIdeck before we can even start our dream tutorial series, but at least it will give you an idea of what we are dealing with :)

Rik the Intern

From this week we have the honor of hosting Rik Bouwmeester for a couple of months. He is currently doing his Master Thesis at the MAVlab from the faculty of Aerospace Engineering of the TU Delft. Since he has experience of working the AIdeck before, he will be able to provide us with some user perspective and help us with the above mentioned issues. You can expect a blogpost from him soon!

Ever since the AI-deck 1.x was released in early access, we’ve been excited to see so many users diving in and experimenting with it. The product is still in early access, where the hardware is deemed ready but the software and documentation still needs work. Even so, we try to do our best to make the product as usable as possible. We’re happy to see some of our users doing great stuff, like the pulp-platforms latest paper “Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs“.

Firmware and Examples

The AI-deck consists of the GAP8 chip developed by Greenwaves Technologies. On their website there’s an explanation of development tools where you get a general understanding of what you can use. Also their GAP SDK documentation explains how to install and try out some of their examples as well, on both a GAP8 simulator on the computer or on the GAP8 chip on the AI-deck itself.

We also host an separate repository for some AIdeck related examples which runs with the GAP SDK.

Documentation and Support

Recently we also added the AIdeck documentation to the Bitcraze website, generated from the docuemtnation already available in the Github repository. There’s still improvements to make, so if you find any issues or any additions needed, don’t hesitate to help us improve it. On the bottom there is an ‘improve this page’ link where you can give the suggested change, or notify us by posting on the issue list of the AIdeck repository.

Also note that we have a separate AI-deck category on our forum where you can search for or add any AI-deck related questions. Remember that posting the issues that you are having will also help us to improve the platform and hopefully soon get it out of Early Access.

Workshop by PULP-platform

On the 16th of April we hosted a workshop given by PULP-platform featuring Greenwaves Technologies. In the workshop the an overview of the AI-deck and GAP8 was given as well as going through some basic hands-on exercises. About 70 people joined the workshop and we were happy it was so well received.

The workshop is a great source of information for anybody who is just getting started with the AI-deck, so have a look at the recordings on Youtube and the slides on the event page. Also make sure to check out their PULP training page for more tools that also can be used on the AI-deck. A big thanks to the PULP-platform and Greenwaves Technologies for taking part in the workshop!


Also we would like to ask if anybody who joined the workshop, to fill in this small questionnaire so that we can get some more feedback on how it went and how we can improve for the next one.

About a month ago we released the AIdeck 1.1, which has some slight upgrades and changes compared to the 1.0. Even though the AIdeck 1.1 is still in early access, we do see the number of support questions increase on our forum and in the issue list of the AIdeck example repo. Therefore we are planning to host an AI-deck getting-started workshop by the PULP lab on the 16th of April at 14:00 (Central European Summer time)!

PULP

The PULP lab has done many amazing research on the field of Edge ML and were one of the collaborators in the development of the AI-deck of which their work on the Pulp Shield was the main inspiration. For more information check out their guest blogpost and be sure to read their latest work on the AI-deck!

More over, they have been working on an opensource tools that also work on the GAP8chip which are must try-outs for any AI-deck users

All in all, since they clearly know what they are talking about, they are more than qualified to teach the rest of us how to work with all this! Also check out Luca Benini’s keynote at RISC-V or this week at the TinyML summit if you would like to learn more about PULP!

Date and content Workshop

The workshop will be tailored to those that have just started to to work with the AI-deck however, we think it will be interesting for regular users as well. Note that the tools mentioned above will not be handled this time.

These are the topics that will be discussed:

  • Hardware explanation (Gap8 specifics and AIdeck)
  • Software Preliminaries (GapSDK,, VM)
  • Hands-on examples

The workshop will take approximately 2 hours and will be on 16th of April in the afternoon, but the exact specifics will be given at a later date. So make sure to already block it in your calendar and to sign up for more information!

Sign up for more information

You can sign up to receive more information by giving your email address in this google form. We will also keep you up-to-date on our discord channel and the event page.

It did take much longer than anticipated, but we finally managed to get the AI-deck 1.1 back in stock! We had some difficulties with the post-production testing and Chinese new year was also a main contributor to the delay, but we are now very happy that we have a batch of AI-deck’s which are ready to ship!

There are only minor changes between the AI-deck 1.0 and 1.1 which can be read about in the “AI-deck product update” blog post. Since the AI-deck 1.1 is now using the gray-scale version of the camera module and the AI-deck 1.0 was using the Bayer RGB version, we now also offer the camera modules as stand-alone products for those that rather have the other variant.

For those of you that already have the previous AI-deck 1.0 and rather wished for the gray-scale camera module, please send us an email at contact@bitcraze.io, and we will work something out!

The AI-deck 1.1 is still a early access product, so remember to post any question or problems you have on our forum in the AIdeck group or to check out examples / documentation on the Github Repo. We also are planning to organize an online workshop / tutorial for the AI-deck in the coming months. We will hopefully have more information about that soon, so keep an eye out on the blogposts!