Category: AI-deck

Ever since the AI-deck 1.x was released in early access, we’ve been excited to see so many users diving in and experimenting with it. The product is still in early access, where the hardware is deemed ready but the software and documentation still needs work. Even so, we try to do our best to make the product as usable as possible. We’re happy to see some of our users doing great stuff, like the pulp-platforms latest paper “Fully Onboard AI-powered Human-Drone Pose Estimation on Ultra-low Power Autonomous Flying Nano-UAVs“.

Firmware and Examples

The AI-deck consists of the GAP8 chip developed by Greenwaves Technologies. On their website there’s an explanation of development tools where you get a general understanding of what you can use. Also their GAP SDK documentation explains how to install and try out some of their examples as well, on both a GAP8 simulator on the computer or on the GAP8 chip on the AI-deck itself.

We also host an separate repository for some AIdeck related examples which runs with the GAP SDK.

Documentation and Support

Recently we also added the AIdeck documentation to the Bitcraze website, generated from the docuemtnation already available in the Github repository. There’s still improvements to make, so if you find any issues or any additions needed, don’t hesitate to help us improve it. On the bottom there is an ‘improve this page’ link where you can give the suggested change, or notify us by posting on the issue list of the AIdeck repository.

Also note that we have a separate AI-deck category on our forum where you can search for or add any AI-deck related questions. Remember that posting the issues that you are having will also help us to improve the platform and hopefully soon get it out of Early Access.

Workshop by PULP-platform

On the 16th of April we hosted a workshop given by PULP-platform featuring Greenwaves Technologies. In the workshop the an overview of the AI-deck and GAP8 was given as well as going through some basic hands-on exercises. About 70 people joined the workshop and we were happy it was so well received.

The workshop is a great source of information for anybody who is just getting started with the AI-deck, so have a look at the recordings on Youtube and the slides on the event page. Also make sure to check out their PULP training page for more tools that also can be used on the AI-deck. A big thanks to the PULP-platform and Greenwaves Technologies for taking part in the workshop!


Also we would like to ask if anybody who joined the workshop, to fill in this small questionnaire so that we can get some more feedback on how it went and how we can improve for the next one.

About a month ago we released the AIdeck 1.1, which has some slight upgrades and changes compared to the 1.0. Even though the AIdeck 1.1 is still in early access, we do see the number of support questions increase on our forum and in the issue list of the AIdeck example repo. Therefore we are planning to host an AI-deck getting-started workshop by the PULP lab on the 16th of April at 14:00 (Central European Summer time)!

PULP

The PULP lab has done many amazing research on the field of Edge ML and were one of the collaborators in the development of the AI-deck of which their work on the Pulp Shield was the main inspiration. For more information check out their guest blogpost and be sure to read their latest work on the AI-deck!

More over, they have been working on an opensource tools that also work on the GAP8chip which are must try-outs for any AI-deck users

All in all, since they clearly know what they are talking about, they are more than qualified to teach the rest of us how to work with all this! Also check out Luca Benini’s keynote at RISC-V or this week at the TinyML summit if you would like to learn more about PULP!

Date and content Workshop

The workshop will be tailored to those that have just started to to work with the AI-deck however, we think it will be interesting for regular users as well. Note that the tools mentioned above will not be handled this time.

These are the topics that will be discussed:

  • Hardware explanation (Gap8 specifics and AIdeck)
  • Software Preliminaries (GapSDK,, VM)
  • Hands-on examples

The workshop will take approximately 2 hours and will be on 16th of April in the afternoon, but the exact specifics will be given at a later date. So make sure to already block it in your calendar and to sign up for more information!

Sign up for more information

You can sign up to receive more information by giving your email address in this google form. We will also keep you up-to-date on our discord channel and the event page.

It did take much longer than anticipated, but we finally managed to get the AI-deck 1.1 back in stock! We had some difficulties with the post-production testing and Chinese new year was also a main contributor to the delay, but we are now very happy that we have a batch of AI-deck’s which are ready to ship!

There are only minor changes between the AI-deck 1.0 and 1.1 which can be read about in the “AI-deck product update” blog post. Since the AI-deck 1.1 is now using the gray-scale version of the camera module and the AI-deck 1.0 was using the Bayer RGB version, we now also offer the camera modules as stand-alone products for those that rather have the other variant.

For those of you that already have the previous AI-deck 1.0 and rather wished for the gray-scale camera module, please send us an email at contact@bitcraze.io, and we will work something out!

The AI-deck 1.1 is still a early access product, so remember to post any question or problems you have on our forum in the AIdeck group or to check out examples / documentation on the Github Repo. We also are planning to organize an online workshop / tutorial for the AI-deck in the coming months. We will hopefully have more information about that soon, so keep an eye out on the blogposts!

It has been a while since we have done an update on the AI-deck! However, as announced in the New Year post, we do have some important changes coming up for the next batch of production. These changes include updated version of the GAP8 chip and switching to a gray-scale camera. We will call this version the AI-deck 1.1, but its firmware should be compatible with the previous version. Currently we don’t have a clear time-line of when we will receive the next batch but it will be soon.

Newer version of GAP8

Together with our collaborators at Greenwaves technologies, we decided to update the GAP8 chip to the latest version available. Since this required a microchip change on the product and to prevent any confusion, we decided to call this version the AIdeck 1.1.

The new version of the GAP8 chip includes several improvements, which resolves 1. conflict problem between uart and hyperbus and 2. Cluster DMA when doing the data transfer between L2 and L1 Memory. The last issue is considered a corner case will only occur if the AI-deck 1.0 is pushed to its limit with a large deep neural network. Although we expect that both issues will not affect the current users if you are experiencing any related problems with the previous version (AI-deck 1.0), please send us an email to contact@bitcraze.io.

Gray-scale Camera

In July we have given an update where we discussed the color camera and how to process it on the GAP8 chip. Even though at the first we were pretty optimistic on the possibilities with the camera, after discussing with the community, ETH zurich and Greenwaves T., we have decided to switch back to a gray-scale camera.

This is because most examples in the gap_sdk repo mostly assume a gray-scale camera. And even though the color camera would be good for image processing with anything of color, it is less ideal for neural networks on edge machine learning devices.

For those who committed themselves to the color camera module, do not worry! We are planning to still offer the color camera module as a separate product in our store! Also we will have a limited amount of grayscale cameras available for those who are unhappy with their AI-decks 1.0’s color camera.

Firmware, Docs and Examples

Since the AI-deck is still in early release, have all the code and documentation in this github repository. This contains all the start-up guides and keeps track of all the bugs and fixes. Some months ago, we managed to update all of this to the latest GAP SDK (3.8.1), which fixed the streamer issues we are having.

However, the quickest way on how to improve this Early Release product is to get feedback by its users. So if you are having any problems at all, do not hesitate to ask a question on to forum or to open up a issue in the repository!

With the raging pandemic in the world, 2021 will most likely not be an ordinary year. Not that any year in the Bitcraze universe has been boring and without excitement so far, but it is unusually hard to make predictions about 2021. Any how, we will try to outline what we see in the crystal ball for the coming year.

Products

What products are cooking in the Bitcraze pot and what tasty new gadgets can we look forward to this year?

Lighthouse

We did hope that we would be able to release the first official version of the Lighthouse system in 2020, but unfortunately we did not make it. It has turned out to be more complex than anticipated but we do think we are fairly close now and that it will be finished soonish, including support for lighthouse V2.

Once the official version has been achieved, we are planning to assemble an full lighthouse bundle, which includes everything you need to start flying in the lighthouse positioning system. This will also include the Basestations V2 as developed by Valve corporation, so stay tuned!

New platforms and improvements

We released the AI-deck last year in early release, but the AIdeck will be soon upgraded with the latest version of the GAP8 chip. For most users this will not change much but for those that really push the deep learning to the edge will be quite happy with this improvement. More over, we are planning to by standard equipped the gray-scale camera instead of the RGB Bayer filter version, due to feedback of the community. We are still planning to offer the color camera on the side as a separate product for those that do value the color information for their application.

Also we noticed the released of several upgraded versions of sensors for the decks that we already are offering today. Pixart and ST have released a new TOF and motion sensor so we will start experimenting with those soon which hopefully lead to a new Multiranger or Flowdeck. Also we are aware of the new DWM3000 chip which would be a nice upgrade to the LPS system, so we will start exploring that as well, however we are not sure if we will be able to release the new version of LPS in 2021 already.

One of the field that we have wanted to improve for a while but have not gotten to so far is the communication with the Crazyflie. The Crazyradio is using a quite old chip and the communication protocol has hardly been touched in years. There now exists a much more powerful nRF52 radio chip with USB port so it can give us the opportunity to make a new Crazyradio and, at the same time, rework the communication protocols to make them more reliable, easier to use and to expand.

People and Collaborations

Last year we have started several collaborations with show drone orientated business, which we are definitely moving forward with in 2021. For shows stability and performance is very important so with the feedback of those that work with that on a regular basis will be crucial for the further development and reliability of our products.

Moreover we would like to continue our close collaboration with researchers at institutes and universities, to help them out with achieving their goals and contribute their work to our opensource firmware and software. Here we want to encourage the community to make their contributions easy to use by others, therefore increasing the reproducibility of the implementations, which is a crucial aspect of research. Also we are planning to have more of our online tutorial like the one we had in November.

We also will be working with closely together with one of our very active community members, Wolfgang Hönig! He has done a lot of great work for the Crazyswarm project from his time at University of Southern California (USC) and has spend the last few years at Caltech. He will be working together with us for a couple of months in the spring so we will be very happy to have him. Moreover, we will also have 2 master students from LTH working with us on the topic hardware simulation in the spring. We are making sure that we can all work together in the current situation, either sparsely at the office or fully online.

In 2021, we will also keep our eyes open for new potential Bitcrazers! We believe that everybody can add her/his own unique addition to the team and therefore it is important for us to keep growing and get new/fresh ideas or approaches to our problem. Usually we would meet new people at conferences but we will try new virtual ways to get to know our community and hopefully will meet somebody that can enhance our crazy group.

Working from home

Due to the pandemic we are currently mainly working from home and from the looks of it, this will continue a while. Even though we think we have managed to find a way to work remotely that is fairly efficient, it is still not at the same level as meeting in real-life, so there is always room for improvement. Further more the lack of access to electronics lab, flight lab and other facilities when working from home, does not speed work up. We will try to do our best though under the current circumstances and are looking forward to an awesome 2021!

2020 was not an ordinary year, and through its different roller coaster motions, we’re proud of what we’ve accomplished. As 2021 grows nearer, it’s time to look back at this year and appreciate our accomplishments.

Community

The Crazyflie is still gaining a lot of interests in universities, and we’re always happy to discover what awesome things our users are doing with our products. One of our proudest moment was that a paper published in Science magazine used a Crazyflie in their research framework!

We actually made a montage of the different research videos where the Crazyflie, and we feel blessed to be part of those amazing feats. We were also quite surprised to see how the Crazyflie is implemented in different fields. Here is the video if you want to know more:

We’ve had stellar blogposts from guests this year, that you can read if you missed:

Of course, “community” this year means something different. We were sorry to not participate in the conventions and fairs where we usually meet a lot of interesting people, and had to find ways to do things differently. We now have a Discord server where people can join and discuss all things Bitcraze related!

Hardware

In the beginning of the year, we started right by releasing the Active Marker deck. Thanks to our collaboration with Qualisys, this new motion capture sensor can estimate the full body pose of the Crazyflie without unique marker positions or known starting positions.

A great deal of work was dedicated to the release of the AI deck in early access. Thanks to our newest deck, super-edge-computing is now possible on your Crazyflie !

Software

We improved our app API. We have seen it used more and more. Being able to add code to the Crazyflie without having to fork the code has proven to be quite useful and popular.

At the beginning of the year we finally managed to decode and use the Lighthouse V2 signals in order to get the Crazyflie to fly autonomously using lighthouse V2 basestations. Our focus for the end of year was to improve the Lighthouse V2, and thanks to some hard work, we managed to improve the calibration compensation for the newest version of Valve’s base station.

The client got a new facelift, as well as the python lib, and we released no less than 4 different new releases.

Documentation

We tried as much as we can to improve our documentation, and working from home proved how much it is needed. Kimberly wrote some step-by-step guides that help you discover more hands-on how to use our python library and the motion commander.

We also tried a new way to connect with you, while improving our existing tutorials, by having an online tutorial. It proved quite successful and it was a great way to talk about all the possibilities (and difficulties!) of a swarm.

The way we work

The biggest change was, as many of you, to learn to work from home. We managed quite well, learning to adapt our process. We now are experts in flying at home, and even though our flying arena is deserted, our kitchens are now more alive than ever.

I came in the company too, offloading the administration work, and we welcomed for some time our intern.

It has been about a month since the AI-deck became available in Early Access. Since then there are now quite a few of you that own an AI-deck yourself. A new development we would like to share: we thought before that we had selected a gray-scale image sensor. However, it came to our attention that the camera actually contains a color image sensor, which on second viewing of the video presented in this blogpost is pretty obvious in hindsight (thanks PULP project ETH Zurich for letting us know!).

A color image from the AI-deck

This came as a little surprise, but a color camera can also add some new possibilities, like making the Crazyflie follow a orange ball, or also train the CNNs incorporate color in their classification training as well. The only thing is that it will require an extra preprocessing task in order to retrieve the color image, which will be explained in the next section.

Demosaicing

Essentially all CMOS image sensors are gray-scale by definition. In order to retrieve color from a scene, manufacturers add a Bayer filter on top of the image sensor, so it filters out the red, green and blue on each pixels. This Color filter array does not need to be RGB, but all kinds of colors, but we will only talk about the Bayer filter. If the pattern of the filter is known, the pixels that related to a certain color will be interpolated with each-other in order to fill in the gaps in between. This process is called demosaicing and it creates the RGB channels that are converted to a color image.

Process of demosaicing with a Bayer filter

Currently we only implemented a simple nearest-neighbor interpolation scheme for demosaicing, which is fine for demonstration purposes, however is not the best technique out there. Such a simple interpolation is not very ‘edge and detail’ aware and can therefore cause artifacts, like these Moiré effects seen here below. Anyway, we are still experimenting how to get a better image and how to translate that to all the examples of the AI-deck example repository (see this issue if you would like to follow or take part in the discussion).

Moiré effect

So technically, once we have the color image, this can be converted to a gray-scale images which can be used for the examples as is. However, there is a reduction in quality since the full pixel resolution is not used for obtaining the full scale image. We are currently discussing if it would be useful to get the gray-scale version of this camera and make this available as well, so let us know if you would be interested!

Feedback and Early Access

Like we said before, there now quite a few of you out there that have an AI-deck in their procession. As it is in Early Access, the software part is still in full development. However, since we have not received any negative feedback of you, we believe that everything is fine and peachy!

Just kidding ;) we know that the AI-deck is quite a challenging deck to work with and we know for sure that many of you probably have questions or have something to say about working with it. Buying an Early Access product also comes with a little bit of responsibility. The more feedback we get from you guys, the more we can tailor the software and support to help you and others, thereby advancing the product forward and getting it out of the early access phase.

So please, let us know if you are having any trouble starting up by posting a thread on the forum (we have a special AI-deck group!). If there are any issues with the examples or the documentation of the AI-deck repo. We and also our collaborators at Greenwaves Technologies (from the GAP8 chip) are more than happy to help out. That is what we are here for :)

The AI-deck is now available in our online store! Super-edge-computing is now possible on your Crazyflie thanks to the GAP8 IoT application processor from GreenWaves Technologies. GAP8 delivers over 10 GOPS of compute power at exceptionally low power consumption enabling complex tasks such as path-finding and target following on the Crazyflie, consuming less than 0,1% of the total energy.

The AI-deck can host artificial intelligence-based workloads like Convolutional Neural Networks onboard. This will open up many research areas focusing on fully onboard autonomous navigation of tiny MAVs, like ETH Zurich’s PULP-Dronet. Moreover, there is also ESP32 WiFi connectivity with the possibility to stream the images to your personal computer.

We are happy that we managed to get everything ready so soon after our last update. Crazyflie AI-deck is in early access, which means the hardware design is now finalized and full support for building, running and debugging applications (including GreenWaves’ GAPflow tools for porting neural networks to GAP from TensorFlow) is available, however, limited examples of specific AI-deck applications have been developed so far. Read more about early access here. Even though there is still some work to be done, there are already some examples you can try out which we will explain in this blog post. Also, we aim to have all AI-decks pre-flashed with the WiFi streamer example so that you check out right away if your AI-deck is working.

Beware that you need an JTAG-enabled programmer/debugger in order to develop for the AI-deck!

Technical specifications

The AI-deck will come with two elongated male pin-headers, which enables the user to connect it to the Crazyflie with an additional deck. There are two 10 pin JTAG connectors soldered which enables connection with a JTAG-enabled programmer. This will be the main way to program the GAP8 chip and the ESP-based NINA module while it is still in early access.

Getting started

When you first receive your AI-deck, it should be flashed with a WiFi streamer example of the camera image stream. Once the AI-deck is powered up by the Crazyflie, it will automatically create a hotspot called ‘Bitcraze AI-deck Example’. In this repo in the folder named ‘NINA’ you will find a file called viewer.py. If you run this with python (preferably version 3), you will be able to see the camera image stream on your computer. This will confirm that your AI-deck is working.

Next step is to go to the docs folder of the AI-deck examples repository. Try out the WiFi demo and set up your development program with the getting-started guide. This guide contains links to the GAP-SDK documentation from GreenWaves Technologies. You can read more about the face detector example that we demonstrated in this blog post.

It has been a while since we have updated you all on the AI deck. The last full blogpost was in October, with some small updates here and there. It is not that we have not focused on it at all; on the contrary… this has been a high priority project for a while now. It is just quite a complex board with a lot of bells and whistles, which can be challenging to work with sometimes so early in development, something that our previous intern can definitely agree on. So therefore we rather wanted to wait until we were able to make sufficient progress before we gave you an update… and so we have!

A Crazyflie 2.1 with the AI deck

Together with Greenwaves technologies we have been trying to get the SDK of the GAP8 chip on the AI deck stable enough for an early release. The latest release of the SDK (version 3.4) has proved itself to work with relative ease on the AI deck after extensive testing. Currently it is possible to use OpenOCD for flashing and debugging, and it supports most commonly available debuggers with a jtag connector. In the upcoming weeks both of Bitcraze and Greenwaves will test and try out all examples of the SDK on the AI deck to make sure that everything is still compatible. Also the documentation will be extended as well. As there is so much to document, it might be difficult to catch all of it. However, if you notify us and Greenwaves on anything that is missing once the AIdeck is out, that will help us out to catch the knowledge gaps.

The AI deck also contains the ESP-based NINA module for establishing a WiFi connection. This enables the users to stream the video stream of the AI deck onto their computers, which will be quite an essential tool if they would like to generate their own image database for training the CNNs for the GAP8 (and it happens to also be quite practical for debugging by the way!). Currently it is required to set credentials of your local WiFi network and reflash the AI-deck to be able to connect and streaming the images, but we are working on turning the Nina into an access-point instead so no reflashing would be required. We hope that we will be able to implement this before the release.

Top view of the AI deck

We are also trying out to adjust applications to make suitable of the AI deck. For instance, we have adapted Greenwaves’ face-detector example to use the image streamer instead of the display available on the GAPuino boards. You can see a video of the result here underneath. Beware that this face-detector is not based on a CNN but on HOG descriptors, so it only works in good conditions where the face is well lit. However, it is possible to train a CNN to detect faces in Tensorflow and flash this on the AI deck with the GAPflow framework as developed by Greenwaves. At Bitcraze we haven’t managed to try that out ourselves ( we are close to that though!) but at least this example is a nice demonstration of the AI deck’s abilities together with the WiFi-streamer. This example and more testing code can be found in our experimental repo here. For examples of GAPflow, please check out the examples/NNtool section of the GAP8 SDK.

For some reason WordPress has difficulty embedding the video that was supposed to be here, so please check https://youtu.be/0sHh2V6Cq-Q

Seeing how the development has been progressing, we will be comfortable to say that the AI deck could be ready for early release somewhere in the next month, so please keep an eye out on our website! We will continue to test the GAP SDK’s stability and we are very thankful for Greenwaves Technologies with their help so far. We will also work on getting-started guides in order to get acquainted with the AI deck, supplementing the already existing documentation about the GAP8 chip.

Even-though the AI deck will soon be ready for early release, this piece of hardware is not for the faint-hearted and embedded programming experience is a must. But keep in mind that the possibilities with the AI deck are huge, as it will be mean that super-edge-computing on a 30 gram flying platform will be available for anyone. It will all be worth it when you have your Crazyflie flying autonomously while being able to recognize its surroundings :)

In this blog-post we wanted to give you guys an overview of our running projects and a general update of the status of things! We got settled in our home-labs and are working on many projects in parallel. There are a lot of development happening at the moment, but the general feeling is that we do miss working with each other at our office! With our daily slack Bitcraze sync meetings and virtual fikapause (Swedish for coffee breaks), we try to substitute what we can. In the mean time, we are going on a roll with finishing all our goals we have set at our latest quarterly meeting, so here you can read about those developments.

AI-deck

Crazyflie with AI-deck

The last time we gave an update about the AI-deck was in this blog post and in the final post of our intern Zhouxin. Building on his work, we are now refocusing on getting the AI-deck ready for early release. The last hurdle is mostly software wise on which we are considering several approaches together with the manufacturer of the Gap8 chip Greenwaves technologies. Currently we are preparing small testing functions as examples of the different elements of the AI-deck in our repo, which are all still in a very primarily phase.

Even though we still need some time to finalize the AI-deck’s early release, we will consider sending an early version of the AI-deck if you are willing to provide feedback while working with it. Please fill in the form and we will get back to you.

Lighthouse

We have made quite some progress on the development for the lighthouse V2. Kristoffer has been working hard from his homelab to get a seamless integration of both V1 and V2 in our firmware (check out this github issue for updates). Currently it is still very untested and very much in progress, however we do have a little preview for you to enjoy.

Crazyflie with LH basestation v2

Documentation

Right now, we are also doing a lot of revamping of the large web of documentation. Unfortunately this is a lot of work! As you noticed by now, we have added overview pages to guide the reader to the right information. We also have moved the tutorials to another part of the menu to avoid clutter on our website. In general we try to go through the repository docs to see if there is any information missing or outdated, however please let us know if you have encountered an error in any description or are missing crucial elements.

Our latest task is revamping the product pages as well, by putting all the necessary information about the hardware in just one place. Also, we are planning to make (video) tutorials soon about many elements of the Crazyflie and how to work with it. More about that later!

Production and Shipment

Production at our manufacturers in China are slowly starting up again. Although it is not yet back at full force, it does enable us to already start ordering to replenish our stock and to get started with finishing our test rigs. Moreover, we are also negotiating to resolve the propeller issue we mentioned earlier, but there is no update on that so far.

As mentioned in this blogpost, we are still shipping orders about twice a week. Both DHL and Fedex are functioning as normal, but we do notice that there is a delay of a few extra days on some deliveries. Please keep that in mind when ordering at our webshop.