Better Android Support!

One of the harder things about GNU Radio on a new system is building the dependencies. That was a large part of my task when getting GNU Radio for Android working. All of the required dependencies needed to build rebuilt using the Android toolchain, which meant that for each dependency, I had to figure out how to set up and call the NDK properly. That was hard, and good thing that there were quite a few hints on the Internet for various programs like Boost and LibUSB. For a while now, I've had information on how to build all of the dependencies for Android on our Android support page. This includes a page dedicated to building the basic libraries and dependencies as well as a page discussing how to build VOLK, GNU Radio, GRAnd, and gr-osomosdr.

It seemed like very few people could follow this through and successfully rebuild all of these packages themselves. And that's fair because I had only ever done this on my machines which are both running the same version of Ubuntu Linux (15.10) and everything else. It's hard to say how any deviations from these versions might affect things. So to help with this issue, I just published two new tools to help people get started.

Tarball of Dependencies

The first tool, which should be the go-to for anyone wanting to get up and running immediately, is a tarball of all of the pre-built dependencies as well as all of the GNU Radio support packages. You just download this, untar/uncompress it (into /opt/grandroid, generally, since that's where I tell everyone to look for this stuff), and now you're ready to start building Android apps and linking to GNU Radio. Bam. Done. The GNU Radio team also has a GPG key for signing, so you can get the public key, download the signature for the tarball and verify your download.

The tarball contains a MANIFEST.txt file that describes what's been built and packaged into this tarball. As of writing this, it looks like:

Boost: 1.58.0
FFTW: 3.3.4
OpenSSL: 1.0.2
Thrift: 0.9.3
ZeroMQ: 3.2.4
LibUSB: v1.0.19-and5
RTL-SDR: b829664e85ae67a35c06f09ffbc16207f4001324
UHD: 4e64eb4888ecbe159af290482615b518ca835682
VOLK: 9b3c79afe34ea6a9dbf20e5c25afa7b5038119bd
gr-omsosdr: 8c01b772ef20bb718902a4e613554bd705010131
GNU Radio: b7870d6cc1b984e44044fa60240e4ad77c8ccd45

For versioned packages, that version is given. For the other packages, these are generally projects that we had to hack to get to work under Android, so these are the git refs for my android branches of these projects.

Do-It-Yourself Script

Some people are (justifiably) paranoid about taking pre-built binaries, even if they are signed. To help them, I've also published the script that I used to build the tarball of binaries. This basically takes the instructions that I made on the wiki pages and puts it into a much better package and slightly more robust way of building things yourself.

Speaking of paranoia, the script will ask you to enter your admin password because it will run sudo to create the directory ($PREFIX). If you look at the script, you see these three lines:

sudo mkdir -p ${PREFIX}
sudo chown $USER:$USER -R ${PREFIX}
sudo -K # invalidates credentials for anyone paranoid

It creates the directory $PREFIX, assigns it over to the user that called this script, and then uses the -K to invalidate the sudo credentials so it won't be able to do anything else as the superuser after this.

The script is only tested on Ubuntu 15.10 (64-bit).

Available GNU Radio Components

When we build GNU Radio, we turn off a lot of extraneous components that aren't necessary or even useful on Android, like GRC or the QT and WX widgets. What is built includes:

• gnuradio-runtime
• gr-blocks
• gr-fft
• gr-filter
• gr-analog
• gr-digital
• gr-channels
• gr-zeromq
• gr-uhd
• gr-ctrlport (with Thrift support)
• Static libraries for each component

And also VOLK and gr-osmosdr separately. The GRAnd package provides audio sources and sinks for use under Android.

The one final GNU Radio component that I feel should be added here is gr-fec. However, right now, that requires GSL, which we have not yet built for Android.

GNU Radio Organizational Changes to Address Growth

I've been running the GNU Radio project for over five years. In this time, we've dramatically expanded its capabilities, prominence, and performance. We have attracted great developers and a fantastic user base. And we have built the highly successful and growing annual GNU
Radio conference.

I feel that I have accomplished a great deal of what I was hoping to when I took over this role. Part of my job (and much of the fun) was being hands-on with the code almost daily. But there's a lot more to do in GNU Radio, and it is time for a new vision to move it forward to where we see it going next.

While I was thinking about this role change, the opportunity arose for me to become a DARPA program manager, which is one of the most exciting opportunities I can imagine. I will still be working on radio, communications, and, of course, software radio, and I still hope to support the project and continue working with the community we have in whatever ways that I can. But this new position gives me an opportunity to pursue engineering, science, technology, and research in areas and ways that I haven't been able to do in my current role.

I am pleased to announce our new leadership team, who are already very familiar to many of you. Ben Hilburn will be taking over as Project Lead while Johnathan Corgan will become the Chief Architect in charge of the codebase. They have already been doing a great job of building up their team structure and their vision for the next stages of this project, which is exactly what we need to grow and move the project forward.
 

The transition is already happening, and we can all consider Ben and Johnathan to have taken up their roles today. I will still be actively involved in GNU Radio over the next couple of months until I start my new position at the end of May.

You'll be hearing more details from them soon. But I hope that everyone knows how proud I am of this project and what we're doing here as well as my confidence in Ben, Johnathan, and what is coming next.

Thank you all!

Performance Monitor Android App

With our continuing roll-out of support and features for GNU Radio on Android, we are looking to handle all command-and-control between the app and the flowgraph through ControlPort. While I've had the ControlPort server side working on GNU Radio apps for a while, we didn't have the mechanism in place for client-side ControlPort apps, which will be necessary for getting and setting parameters in the radio. I've now added two new projects to help us with this.

The first project is GrControlPort -- this is an Android library (not an application) that is meant to provide a simple abstraction layer for using ControlPort in Java. This is very similar to our Python-based RPCConnection class that we use to make getting and setting knobs easier in that language. You use this library to make the connection to the ControlPort app and then use the returned object to get and set the knobs of the radio. There's an abstracted class to make the user code agnostic to the backend service, like Thrift. The README file explains more about how to use this, and it includes a simple Java class that shows the basic modes of getting and setting knobs of a flowgraph.

https://github.com/trondeau/GrControlPort

The second project is GrPerfMon, which is an Android app that uses GrControlPort to manage the ControlPort connection. This app reads the "work time" Performance Counter and displays this info as a percent of the total time for all blocks in a flowgraph. This is very similar in nature to our gr-perf-monitorx application we use on a desktop. You give it the hostname/IP address and port number of the running flowgraph along with the update interval (defaults to 1 second) and connect. When connected, it will show a bar graph of the percentages for each block. The code is on Github here:

https://github.com/trondeau/GrPerfMon

Because ControlPort is handled over a TCP/IP connection, the flowgraph and the app don't have to run on the same device, although part of the motivation for this work is to provide the experience and knowledge to make this the default communications between an Android flowgraph and app layer. But more on that to come. For right now, the image below shows me profiling from a Nexus 7 tablet a flowgraph running on my desktop machine:

GNU Radio for Android

I have been slowly pushing out information here and there on my work on Android. I've been able to show it off at various conferences and workshops that I've attended over the past few months. It gets a lot of very positive feedback, and we know there are many, many exciting things we as a community of developers and wireless experts can do with this kind of a system.

It's time to start really pushing this project. I spent so much time getting things to work under the hood that I've completely lost track of any of the cool applications that I wanted to work on with it! But the more people that get access to this and know how to work with it, the more interesting the application space and possibilities will become. Because of that, I had been waiting for two things to finish before writing this post. First, I wanted to be able to use a bus-powered Ettus Research USRP with and Android device. And second, I wanted the process documented from the start of building the dependencies to the end of a working (if overly simple) app. Both of those are done.

To help get things started, I've created an Android support page on the gnuradio.org wiki:

• http://gnuradio.org/redmine/projects/gnuradio/wiki/Android

This page is designed to walk a developer through all of the stages from building the dependencies to GNU Radio itself, GRAnd, UHD, RTL-SDR, and gr-omsosdr. This should be pretty much copy-and-paste to get everything built and ready. The final part was finished just today: a walk-through for how to build a basic Android application that launches a GNU Radio flowgraph. This is done using Android Studio, which is the latest development environment for building Android apps. The project can be downloaded from it's Github page to provide a template to set up your own project and verify a working installation:

• https://github.com/trondeau/GrTemplate

Hopefully, we will continue to evolve the process to make setup and building apps easier in the future. The next step is walking through and providing the annoying boilerplate stuff for hardware access so you can start using USRPs and RTL-SDRs with your devices. But for right now, I want to make sure people who are excited about this stuff have a way to get in and start playing.

List of what we support on Android

So where do we stand with our GNU Radio support on Android?

• GNU Radio components:
  • Runtime, VOLK, Blocks, FFT, Filter, Analog, Digital, UHD
  • ControlPort
• Extras
  • gr-osmosdr including RTL-SDR and UHD support
    • HackRF support should be trivial but not done, yet
  • gr-grand: GRAnd (GNU Radio for Android) to support Android-specific I/O and capabilities such as:
    • audio input and output
    • shared buffers between app and flowgraph
    • sensors (e.g., accelerometer and luminosity)
• Tested hardware
  • RTL-SDRs
  • USRP B200 and B210 (the latter required external power on a Nexus 7)
• Android Version
  • Android Lollipop 5.0 or greater. Tested on 5.1.1
    • some complex support in previous versions is lacking in the ndk; we think we can get around it, just not out-of-the-box.

Latest GR Hackfest

We held our latest GNU Radio Hackfest at Ettus Research last week, and it was one of our most successful yet. It was also our largest gathering of one of these week-long events with something like 25 people at the most (plus or minus since a few were Ettus employees so had to come and go as they could). We had a huge amount of code worked on during the week with contributions from all over the field to make the project better. We fixed a number of bugs and issues that we've been meaning to deal with for a while, and we saw a good amount of other advances to the project happen. We haven't updated the hackfest wiki page yet with our projects, but you can certainly look forward to a lot of the results and features in the next release next week.

Updates and updates and updates

This blog has been rather silent lately, but that's not because there's not enough to talk about in the GNU Radio world. In fact, there's simply been too much! We've been too busy with events, adding new features, improving old features, and adding new website info that I keep forgetting to update here.

First, you can see for yourself the things that have been going on with the GNU Radio Project at the Events page. We just had hackfest at TU Delft that went quite well followed by another successful SDR Dev Room at FOSDEM. A Couple of the videos for that event are already online.

Meanwhile, coming up, we have:

• A hackfest hosted by Ettus Research.
• A GNU Radio Hacking Workshop before NEWSDR (details to come).
• A FOSS and GNU Radio session at the Wireless@VT Symposium.
• SRIF 2015 with Mobicom (details to come).
• GRCon15 (we're very close to announcing a date and location).

And fred harris and I are teaching another SDR class with the UCLA Extension program.

We have also announced our plans for the 3.7.7 release of GNU Radio, which will now include VOLK as a separate library.

And the GNU Radio community has been working hard on improving our websites. We are near to a release of our new CGRAN website, which now ties in nicely with PyBOMBS and will help us keep up-to-date with everything better. And we are working on a new front page for gnuradio.org. You know how you've hated being dumped into that Redmine wiki page? Well, that will still hang around as the developer's portal, but we want our main page to be more user friendly and to address the GNU Radio project as a whole, including GRCon and other events and where to find users and use cases of GNU Radio.

Along with all of this, we've been working on a number of features. I'll spend more time on each of these later, but wanted to get the word out there.

• Support for GNU Radio applications on Android. I'm calling this an alpha release since there's still lots to do to make things work smoothly here.
• Reviving ControlPort functionality by using the Apache Thrift project. We're close to having this merged into master, but the branch is up on my github page.
• Providing improved packet-based and burst digital signals. Again, a work-in-progress with the branch up on my github page.
• Adding sample rate and timing info on a per-item basis in a flowgraph. This will allow a block to ask for the exact time stamp of any item, and it provides a global understanding of the sample rate of each block. The code on my public branch works, but I have a better idea how to handle things when sample rates change during runtime.
• Improving our QA testing and continuous integration services. I've been rolling out a Jenkins server in my office (for now) that manages a handful of different nodes for different purposes. I have a box with the minimum dependencies we require for GNU Radio to make sure we're not moving past them as well as testing on various other things like OS X and 64/32-bit installs. And I even have a system testing our embedded SDK cross-compiler with on-board testing of the QA to see what breaks there. This will help ease my mind of missing something or overlooking some problems. I'll get weekly reports but can also kick off these tests whenever we need to get a state of the project.

I think that wraps up the main points of the project that we're working on. I'll update again with details of some of these projects when they've rolled out more fully, especially on the Android, ControlPort, the website, and GRCon items.

Release of 3.7.4

Well, I mentioned it here a few weeks ago and last week we did it. GNU Radio version 3.7.4 is now out and about. We're really happy with the release for a number of reasons. If you take a look at the release notes, you can see not only a lot of new features we've added on, but also the huge number of contributors. So not only are we feeling really good about the project itself, we have a great community of users that are doing really neat things and helping us improve our code. So a huge hand and big thanks to everyone!

I think that the release notes give you the basic idea of what's going on in this version, including ZeroMQ blocks for improved connections between networked flowgraphs over the UDP or TCP sources/sink, better QTGUI handling, and some nice improvements to the GRC interface. Also, as I mentioned in my last post, we now have our new FEC API concept included in gr-fec. Oh, and on that last note, we're very close to having some of the LDPC work done during last summer's GSoC integrated with the FEC API so we can all start to use it in our flowgraphs!

The good work is never done, though. We still have so much we can and want to do in this project, and so we've already started laying out some ideas and plans for release 3.7.5. There's some really fun work going on in GRC that I think everyone's going to love, and we're going to be one step closer to defaulting everything to QTGUI (don't worry! WXGUI will have a long lifetime as we migrate over!). We haven't made the full decision, yet, whether to switch the GRC defaults to QTGUI in this next release of wait until 3.7.6. I think we'll make that decision about midway through this release cycle to make sure we have enough time at get the development branch settled on that before the release.

Now, no guarantees, but I'm hoping we can get 3.7.5 out in time for GRCon14. That's a bit shorter than our normal time period between versions, but then again, we took a bit of extra time before 3.7.4. I think that it'd be nice going into the conference with a brand new version, plus a lot of features like some of the QTGUI and GRC updates are already done. We'll see what things look like by the end of August if it really makes sense to get 3.7.5 out the door by the conference.

Finally, I've said it elsewhere, and probably on other posts on this blog, but just a quick mention of 3.8. We have started collecting some ideas on what we're looking for with 3.8, but we haven't made any substantial efforts on it, yet. We're going to stick with 3.7 for a while longer because we feel like we're able to get a lot of work done with this API version without having to move on. In 3.8 we plan on updating our dependency list so we can be a little more free with features from our dependency libraries, and it'll help us fix a few holes we have in the runtime API that's causing some bugs that we've been tracking. But there's nothing really pushing us to a new API, yet, unlike our previous few API releases where we had some big changes. Instead, all of our big changes are all coming in our "minor" releases. But the only thing minor about it is our version naming scheme!

Update and 3.7.4

It's been a while since I've provided an update. Lots of travel, both work and otherwise, has kept me from letting everyone know what's been going on in GNU Radio. So let me see if I can start to fix that.

I think that our most interesting and fun development was the success of the ISEE-3 reboot team, including our own Balint Seeber (ok, so he's technically with Ettus, but for all of his work with GNU Radio, I'll claim him; at least partially) and John Malsbury. This was an incredibly cool and successful project, and Balint, John, and the rest of the reboot team have done and continue to do an amazing job. I'm not going to try and poorly rehash the work here. But there's been plenty of articles written on the effort and the links provided above here will give you a much better review of what happened and how.

In other news, we're close to releasing GNU Radio 3.7.4. I was really happy with our 3.7.3 release a few months ago, and this next release continues to strengthen the project. We have a lot more support and have debugged a few issues with the message passing infrastructure. I see the use of the message passing system as a growing trend in GNU Radio, especially for control and updating parameters of blocks.

Another development is a massive update and improvement to gr-atsc. While the original work here was great, it was based on GNU Radio ideas and capabilities from probably 10 years ago or so. Andrew Davis and Johnathan Corgan have done a wonderful job bringing this up to date and improving its speed and stability. This is part of a trend that we're working on (hopefully in the next version release) for more support of other digital TV standards, both transmitter and receiver code. Keep an eye out for gr-dtv in the near future.

A huge part of my work recently has been in the integration, development, documentation, and building examples for our new FEC API that is now a part of gr-fec. Historically, GNU Radio hasn't done a very good job with managing or allowing use of forward error correction (FEC) techniques. It's a difficult problem to solve to allow us to easily integrate different types of FECs in transmitters and receivers, either for streaming or packet-based communication links. Each FEC has different properties and conditions for use in the different types of comms links, and they can have different input/output data types that they are built to work with. The new FEC API that's been introduced allows us to handle these situations more seamlessly within a GNU Radio flowgraph.

The new gr-fec includes FEC encoders and decoders for a convolutional code (K=7, Rate=1/2, polynomials=[79, 109]) that has been highly optimized with SIMD (through VOLK), a repetition code that simply repeats each bit N times, and a dummy code that actually does no coding but provides a simple mechanism to use and study the API. We are also providing different blocks, called “deployments” in the FEC API, that allow us to make use of these encoders/decoders in streaming, tagged stream, and PDU message-passing flowgraphs. I was able to use the convolutional code FEC encoder and decoder in the gr-mac project for full network communications between multiple computers. This work can be found in the “fec_gmsk” branch of my fork of gr-mac.

The GNU Radio manual covers the API and deployments in a fair amount of depth. Obviously, you can always build the docs yourself with GNU Radio, but the main manual page on gnuradio.org will be updated to include this when 3.7.4 is released. Likewise, there are examples for the use of the codes and deployments that are distributed and installed with GNU Radio.

So that's just a brief review of what we've been up to the past few months. There are plenty of little fixes, tweaks, and additions that we've been making to the code base in this time that I've left out. But I'm really pleased with the progress and rate of progress we've been making lately. Lots of new, good stuff to help us fill a few gaps in capabilities we've had.

What's Happening in GNU Radio

There are lots of changes on the way in GNU Radio. But since I don't think that I can say it better, here is Johnathan Corgan's email to the GNU Radio mailing list from yesterday describing it:

Some of you may have noticed recent check-ins that have added new
top-level components (like gr-fft), or duplication of blocks (such as
into gr-digital.)  I'd like to explain the master plan Tom and I are
working from and what to expect over the next few weeks.

Essentially, we are making two things happen at the same time:

- libgnuradio-core is going away, and its blocks are being reorganized
into several new, smaller libraries

- The C++ API is changing to use pure virtual interface classes, C++
namespaces, and segregated header files

These two transitions will occur in such a way that as much work as
possible will be completed on the master branch but without requiring
any changes to your existing code, and the rest will happen on the
next branch.

The gnuradio-core component has accumulated hundreds of blocks over
the last 10 years.  In order to simplify and organize things, we are
creating several new top-level components to replace it.  The first
step of this was done when gr-digital collected much of the digital
modulation/demodulation code up into its own component.  The second
step was to create gr-wavelet and move the wavelet blocks into that.
When we are done, the following top-level components will hold all the
code that used to be in gnuradio-core:

* gr-blocks - "basic" blocks used widely across many flowgraphs, like
math operations, type conversions, stream/vector manipulation,
file/message/network sources/sinks, etc.

* gr-analog - analog waveform processing blocks such as signal
sources, AM/FM modulation/demodulation, AGC, squelch, power
measurement, etc.

* gr-digital - digital waveform processing blocks such as PSK and OFDM
modulation, scramblers, timing and carrier recovery, channel
equalization, bit packing/unpacking, framers, deframers, CRC, etc.

* gr-fec - Reed-Solomon and convolutional coding, future development

* gr-fft - blocks that wrap the external FFTW library

* gr-filter - FIR and IIR filters, filter design, resampling,
channelizers, channel models

* gr-vocoder - voice processing codecs

* gr-wavelet - wavelet processing blocks, future development

* gnuradio-runtime - top block and friends and runtime scheduler, very
small number of blocks for QA testing of runtime (vector source/sink,
null source/sink, etc.)

The process we are following to get this done is to create the new top
level components on the master branch and copy the existing blocks
into them, then remove the old blocks from gnuradio-core *on the next
branch only*.  In this way, developers can, if they wish, start
migrating their applications to use the new block hierarchy in their
applications as they become available, without having to do it all at
once.  Existing code, however, can also be left alone without any
impact as gnuradio-core itself will remain unchanged until the 3.7
release.

Part of this process is already done--gr-vocoder, gr-digital, and
gr-wavelet are part of the current release.  The new gr-fft was merged
into master recently.  Soon, gr-filter will be added in, and then we
will remove all fft and filter blocks out of gnuradio-core on the 3.7
branch only.  Today, several more blocks from gnuradio-core were
copied into gr-digital, and again, we'll soon delete those blocks out
of gnuradio-core on the 3.7 branch only.

Since we're touching so much code, we're taking the opportunity make a
change to the C++ API to implement pure virtual interface classes for
API visible code.  This was discussed on the mailing list March 12.
In addition, we're reorganizing the API to use C++ namespaces, which
shortens filenames, simplifies the Python wrapper generation, and
eliminates some redundancies in class naming.  You can see an example
of this in the new gr-fft directory on the master branch, or in the
gr-wavelet directory on the next branch.

The same process for dividing the work between the master branch and
next branch applies here.  Where possible, when we copy over blocks
from gnuradio-core to the new top-level components, we'll convert them
to use the new C++ coding style.  This won't affect any existing
user's code, as the old blocks will stay in gnuradio-core.  For all
the rest of GNU Radio outside of gnuradio-core, we'll be making the
C++ changes on the next branch, again so as to not disturb existing
user's code.

Finally, during all this, as we implement the new block structure,
we're going to pay special attention to:

- Reimplementing functionality using libvolk vector optimization where possible
- Updating or adding missing header file documentation
- Converting code where needed to conform to the GNU Radio coding style guide
- Adding QA test code if missing
- Creating GRC block wrappers if missing
- Removing obsolete code or other cruft

Few of these changes will impact Python or GRC users, except that most
blocks that once lived in the 'gr' namespace imported from gnuradio
will instead import from a different namespace under gnuradio, such as
is already done for gr-audio, gr-digital, gr-uhd, etc.

The net of all this is that we expect the 3.7 C++ API to be better
organized, easier to use, and have better documentation, but without
changing much of the functionality other than performance
improvements.

Tom and I will be working up documentation on the wiki detailing the
changes as we go along, and there will be a 3.6->3.7 conversion guide
for C++, Python, and GRC GNU Radio applications.

So, as they say, "pardon our dust".

Johnathan

New Server, New Service

We have recently updated the GNU Radio server to a new platform with updated versions of all of our services. There were a few hiccups along the way as proxies and other caching sites had to be refreshed for everyone to get to the new site properly, but I haven't heard any complaints in a while, so hopefully those initial jitters have gone away.

Among the new versions of our software is an updated Redmine. The new Redmine fixes some issues we had with registration, but it will also allow us to support multiple projects and other nice features like that. Keep an eye out for expanding information on the website as we integrate these new features.

Mostly, I hope this change hasn't effected anyone, and it should only serve to make the website more stable and nicer to use. The new server in the background has already saved us a ton of work.

One new change that has come about is an update from running Hudson to Jeknins. This will serve as our new continuous integration tool and I hope to start making good use of it in our project development cycle.

Initially, the most important change is that every time Jenkins builds GNU Radio, on a weekly basis, the resulting packaged tarball for the most recent master development branch will be published on our website. This means that people who don't want to or can't use git to access the software can get a new snapshot of the code weekly. The links to the tarball and the SHA1 sum of the tarball are found on the Downloads page of gnuradio.org.

Jenkins Interface: http://gnuradio.org/jenkins/

Download Page: http://gnuradio.org/redmine/projects/gnuradio/wiki/Download