While we've all been working to make GNU Radio better, it seemed about time to have some kind of central management tool to bring it all together. That's why I want to introduce the use of Hudson as our continuous integration sever:
Hudson provides a nice webpage interface to project builds, status, and reports. Its very configurable with a host of plugins for different purposes, so we can perform different tests and provide increased feedback and reports about the status of GNU Radio. My intention with this tool is to improve the quality of the code built from a few different perspectives. I'll go over some of the main features here.
Testing Multiple Builds
We can set up Hudson to test multiple builds and then build them periodically. Currently, three different builds are defined for the maint, master, and next branches of the main GNU Radio repository. While Hudson allows us to test multiple Git branches in one build, I prefer the separate reporting that comes out of doing them separately.
The builds are currently offloaded to a remote machine (another nice feature of Hudson) that builds all three branches on an Ubuntu 10.04 system. This is where the first real benefit of a CI system like Hudson can bring to the project. While it only tests the builds on a single operating system, we can extend the build process to test on different platform by adding more builds. The only caveat is that we have machines to perform each of these builds consistently available. I'm working on that.
We will eventually want to support periodic builds of GNU Radio on a number of system types. For an early list of possible installs, I'm thinking the following are the most necessary:
- Linux 64-bit (provided under Ubuntu)
- Linux 32-bit (Ubuntu again, probably)
- Fedora (various versions)
- Apple OSX
- Windows (cygwin or mingw or both)
I think that would cover a large number of issues and support the largest user base.
See, every time a build is performed, I get an email saying that it either a) passed or b) failed with some error(s). I can then see what the errors are and work to fix them. This will help keep an eye on the various platforms and what our code changes do to them.
Unit Test Results
We have a lot of unit tests defined for GNU Radio, but most people probably don't pay much attention to them. They are designed to ensure a level of quality in the code. Indeed, they are called "QA" code for precisely that reason. They should test the basic capabilities of each block in as many ways as possible to take care of any corner cases or other problems to arise in the code. Any new block or capability that is added to GNU Radio should come with some QA code. I admit, that I've been very bad about this in the past myself. But I'm hoping that Hudson will help change this attitude by clearly exposing the results of the QA code.
If you click on any of the builds, you should see a graph titled "Test Result Trend." This shows the number of tests completed and failed during the build (or more exactly, the "make check") process. If you look at master, you'll see that there are 437 tests run and no failures at the time of this post. That's good, but we need to do better. Using this visual feedback, we can not only see how many tests are run, but also click on the graph to find out more information on the tests.
Seriously, click on the graph. You'll see a table of "All Tests" that are broken down into "(root)" and "__main__". So not the most helpful of titles, but what you are really seeing is that the "(root)" results come from the CppUnit tests run on the C++ and the "__main__" set of results come from the Python unit tests. Clicking on this later will bring up a larger table that allows you to see what tests were run here and the time it took to run each test.
Using this interface, we can find out what tests are being run and how they are performing. With this kind of knowledge about what's going on in GNU Radio's QA process, I hope it serves as a good reminder and incentive to continue to improve the tests and write new tests to keep things moving.
Parsing the Output
Another good way to keep track of the quality of the code is to understand the errors and warnings being produced during the build process. Now, it's not likely that errors in the code are going to last long enough to get to the Git repo, but you never know. So I have Hudson using a plugin that can parse the output of "make" and search for specific strings. Specifically, it looks for any "error" and "warning" text that's generated. It then provides a very convenient interface for looking at the results. Click on the "master" project and click on the "Parsed Console Output" link on the left.
It's showing me that, dear god, we have 112 warnings generated during the build of the master branch. Most of these being signed and unsigned comparisons. Not a huge deal, but stuff that's often overlooked in the development process. But now here they are, plain as day and ripe for fixing. With this, we can now work on fixing these little warnings. Nothing critical, but useful information.
Full Output
That last thing I want to touch upon in this post is that, aside from the nice ways Hudson represents information about each of the builds, you can also access the full console output for each build. If you click on the "Console Output" link on the left menu bar under a particular build, you can see everything it did (you might have to click on the "full log" button on the top to see everything).
Final Thoughts
There is probably a lot more to Hudson that I don't know about and haven't talked about here. Specifically, there are hundreds of plugins for the tool, and I'm not sure what most of them do. So if anyone knows what else Hudson can do to help, please let me know.