# Category: Testing

You might, or might not have heard about a tool called Wireshark, it is quite popular in the software development world.

Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, communications protocol development and education. It makes analyzing what is going on with packet based protocols easier.

Most often Wireshark is used for network based protocols like TCP and UDP, to try to figure out what is happening with your networking code. But! Wireshark also allows you to write your own packet dissector plugin, this means that you can register some code to make Wireshark handle your custom packet based protocol.

For the latest release of the Crazyflie Python Library we added support for generating a log of the Crazy Real Time Protocol (CRTP) packets the library sends and receives. This is the (packet based) protocol that we use to communicate with the Crazyflie via radio and USB.

We generate this log in the special PCAP format that Wireshark expects. And we also created an initial version of a dissector plugin, written in the programming language LUA.

When we put this two things together it turns into a pretty cool way of debugging what goes on between your computer and the Crazyflie!

## What does it look like?

Wireshark gives you a graphical interface where you can view all the packets in a PCAP file. You will see the timestamps of when they arrived. Selecting a packet will give you the information that the dissector has managed to deduce as well as how the packet looked on the wire.

On top of that you get powerful filtering tools. In the below image we have set a filter to view only packets that are received or sent on the CRTP port 8, which is the port for the High level commander. This means that from a log file that contain 44393 packets we now only display 9. Which makes following what goes on with high level commands a bit easier.

The dissector knows about the different types of CRTP ports and channels and knows how to dissect an high level set-point, as seen by the image above.

## What can this be used for?

This functionality is, we think, most useful for when developing new functionality in the Crazyflie firmware, or in the library. You can easily inspect what the library receives or sends and make sure it matches what your code indented.

But it can also be useful when doing client type work! We recently located the source of a bug in the Crazyflie client with the use of this Wireshark plugin.

It was when updating the Parameter tab of the client to handle persistent parameters, and to use a sidebar for extra documentation and value control. As I was testing the code I noticed that every time I changed the value of ring.effect to a valid integer and then disconnected and reconnected, the value was set to 0. Regardless of the value I had set.

I recorded a session using the PCAP log functionality:

$CRTP_PCAP_LOG=ring.pcap cfclient  And the I fired up wireshark: $ wireshark ring.pcap


It was now possible for me to track what the library and firmware thought was going on with the ring.effect parameter, by tracking the crtp.parameter_varid field using Wireshark. Filtering down from from 3282 packets to 12 packets.

I had earlier figured out that the varid of the ring.effect variable was 183. This is a quasi-internal representation of a parameter that we do not expose in a good way. In the future we will try to make this Wireshark tracking work with the parameter name as well.

Looking at the write parameter packet from USB #3 to the Crazyflie I could see where I set the value of the parameter to 5, so far so good.

The surprising part however was seeing a write further down setting the parameter to 0! This mean that something in the client was actually setting this to zero!

After seeing this, locating the actual issue was trivial. I noticed that the Flight Control tab was setting the ring.effect parameter to the current index of the combo box in the UI. And when no LED-ring deck was attached, this amounted to always setting the value to zero.

But having confirmation that this was something happening on the client side, and not some kind of bug with the new persistent parameters was very helpful!

## How do you use this?

We have added documentation to the repository documentation for the library on how to generate the PCAP log and how install the Wireshark plugin.

But the quick-start guide is this:

• Copy the tools/crtp-dissector.lua script to the default Wireshark plugin folder
• Windows: %APPDATA%\Wireshark\plugin or WIRESHARK\plugins
• Linux: ~/.local/lib/wireshark/plugins
• Restart Wireshark or hit CTRL+SHIFT+L
• Set the environmental variable CRTP_PCAP_LOG to the filename of the PCAP log you want to generate
• Run Wireshark with the filename as an argument

And please report any issues you find!

Happy hacking!

We like to describe the Crazyflie as a versatile open source flying development platform. It is something that enables you to do cool stuff. It is not a finished, polished end product in its own.

The platform offers hardware extensions in form of decks, that you make your self or buy from Bitcraze. And the platform offers software extension, either by altering the firmware on the device or by writing programs that communicate with the platform.

This approach makes determining the expectations and requirements for the platform and the surrounding ecosystem a bit tricky. It is dependent on what you as a user plan to create using our products. And since the ecosystem is growing we need a, scalable, way to handle these fuzzy expectations during development and maintenance.

We think testing is big part of solving this, testing in a systematic, scalable and reproducible way. This is the reason for setting up our first physical test lab, aimed at providing stability while moving forward with new products and features.

## Testing today

### Continuous integration

On each change proposal to our software we run tests. For the firmware in the device we build multiple different configuration and run unit-tests. For the Crazyflie client and the Python library we make sure we can build for Linux and Windows and that the code passes our style guidelines. If any test fail we go back and update the proposed changed and re-run the tests. This is our first level of defense against defects.

### Release testing

For each release we follow a checklist of procedures and tests to make sure that quality does not degrade. We make sure all the examples in the Python library are working and that the Crazyflie can fly around as expected in our flight arena, using various positioning systems.

### Limitations

The way we test today makes it very difficult to determine if we regress in, for example, flight capability or in radio communication quality. We either test different software packages in isolation, without hardware in the loop, or we test by having a human trying to estimate if any degradation has happened since the last release.

We run into scalability issues as our ecosystem grows, it is near impossible to test all the different combinations of products. And it is very hard for us to detect stability or quality regressions without having a more systematic approach to testing the software on relevant hardware.

## Introducing the test lab

What we have done so far are two things:

1. Created an infrastructure for setting up a site for testing our software on real devices
2. Prepared a test lab in our printer room at our office in Malmö

### Infrastructure: crazyflie-testing

This new Git repository contains the building blocks we need to setup our new test lab. You can check out the repository README.md file for information on how to run it.

It contains a beginning of an automated QA test suite and along with that a start of a set of requirements for those tests. The requirements are specified in TOML files which are parsed and accessed from the tests. They are also rendered to markdown, to be easier for human consumption.

The repository contains a way to specify a test site, which is a collection of devices to run the test suite against. You specify your site in a TOML file which contain information about each device, such as which decks are connected and the radio:// URI. This site specification is then used by the test framework when running the tests, making sure each test is run against all devices compatible with that test.

The infrastructure also has management scripts to perform tasks like flashing all devices in a site, or recovering them if they go into boot loader mode by accident. All aimed at being able to handle testing with the least amount of human intervention possible.

The most recent addition to the infrastructure is the ability to test swarms using the Crazyswarm project!

### Crazylab Malmö

We define the site crazylab-malmö.toml which is the test lab we have setup in our printer room in Malmö, Sweden, it is currently defined as:

version = 1

[device.cf2_flow2_lighthouse]
decks = ["bcFlow2", "bcLighthouse4"]

[device.cf2_flow2]
decks = ["bcFlow2"]

[device.cf2_flow2_multiranger]
decks = ["bcFlow2", "bcMultiranger"]

[device.cf2_usddeck]
decks = ["bcUSD"]

[device.cf2_lighthouse]
decks = ["bcLighthouse4"]

[device.cf2_stock]
bootloader_radio = "radio://0/0/2M/B177790F3A?safelink=0"

There are, for now, six devices with different combination of decks attached. Each night, after midnight, different GitHub actions starts working for us. One job builds the latest versions of the firmwares for the devices another job will upgrade all the devices in the lab and run the test suite, and a selection of the Python library examples. We will be notified if any of these jobs fail.

This acts as a safety net for us. We will quickly know if the communication performance degrades, or if we have messed up with our parameter or logging frameworks. And we can catch silly firmware bugs as early as possible.

### Future

We want to keep adding test cases and other infrastructure to our testing framework. Going forward it would be really nice to be able to test positioning systems in some way. And of course, some type of test of flight, be it free flying tests or using some kind of harness.

It might be interesting to look into adding simulations (hardware in loop or not) to our testing setup. It is all a question of bandwidth, there are a lot of cool things to work on, and a limit on time and bodies.

You can help! You might even be able to help yourself while helping us! If you contribute tests that correspond to your use-case, then you can relax knowing that those tests will run each night, and that Bitcraze engineers will be notified the minute they fail.

Or you can define your own site and run the test-suite against all your devices to make sure nothing strange is going on.

Hopefully this infrastructure and lab will help all of us to do more cool stuff using the Bitcraze ecosystem!