Metadata File Format

One thing that I think GNU Radio is great for is science and experimentation. But one of the main drawbacks that we haven't addressed until now is the storage of information about data that we've collected. When making a collection, later on, we'd like to know a lot about the collection setup, properties of the radio, information related to the samples, and possibly many other things (like the date of collection, location, the weather, etc.). Part of the last GR Hackfest was spent figuring out the proper format for including and managing this kind of metadata in our data files.

Well, I'm happy to announce that we now have this capability natively in GNU Radio. We have a file sink to write metadata files and a file source to read these files back again. We have a few simple utilities that can read the metadata out of a file, too. Now that the mechanics for metadata are there, we'll be interested in using this concept more and more. For example, we'd like to update our utility uhd_rx_cfile that just captures raw samples from a UHD device to use metadata. This would very simply put in header data that includes the time stamp from the UHD device and the sample rate, frequency, and gain of the receiver. And then there are some plotting tools that we have to display files in different domains. We need to update these so that they understand the metadata and can adjust themselves accordingly (like setting the right time or frequency axis based on the metadata as well as knowing the right data type of the samples).

We'd also like help in knowing what other tools people would like to have for managing metadata files. Even more, we'd like help in building them! One tool that I can think of is a way to read a metadata file and split it up into multiple files where there's a single metadata header and the data segment for each file. We could then have smaller, more manageable files which we are capable of reconstructing due to the metadata. And I'm sure that there are plenty of other tools that we can create to make the handling of this kind of information more useful.

To start explaining the metadata concept, I'll summarize here by (mostly) copying what I wrote on the Hackfests wiki page (http://gnuradio.org/redmine/projects/gnuradio/wiki/Hackfest1211).

We now have Version 0 of the metadata files in 'master' and 'next', so they will make there way into the 3.6.3 release. There is a blocks::file_meta_sink and blocks::file_meta_source. The file sink creates a file and writes all data to it, along with headers of metadata. The file source is resourceful in that you can just give it the metadata file itself and it's capable of reading out everything it needs in order to pull the data and source it into a flowgraph. Meanwhile, other metadata gets transferred as stream tags so blocks down the line can also know what's happening with the data. Pretty good stuff!

For a more complete discussion of the metadata header format and how to use it, we've added a page to the Doxygen manual that explains it all. This manual page is not on the gnuradio.org's website version of the documentation since we have not made a release containing this capability (but it will be in 3.6.3, which will be released soon). For now, you can build the documentation locally (in either 'master' or 'next') and you'll have access to the page, which is located under "Related Pages" and "Metadata Information."