We recently fixed a bug in GNU Radio where the rectangular window actually didn't work. It would fall through to the Hamming window because of a missing 'break' statement. Embarrassing bug, but not all that detrimental since the rectangular window is so rarely used.
The one place it was being used was in the Hilbert transform block. We can build Hilbert transform filters using the filter.firdes.hilbert function. In this way, we've been able to set the window type to anything we'd like (Hamming, Hann, Blackman, etc.). But with the Hilbert transform block (that is, filter.hilbert_fc), we could only ever specify the number of taps to use in the transform and the window would default to a rectangular window -- except, as noted above, it would actually default to the Hamming window.
So we decided to add another argument when creating a Hilbert transform block that allows us to set whatever window we wanted and not force a decision on the users. To keep the API sane, though, I wanted to make sure there was a good default for this, which brought up the question: what is the right default window for a Hilbert transform?
That question depends a bit on who you ask. Johnathan Corgan mostly uses the Hilbert as a way to suppress sidebands while I tend to use it to convert real signals to analytic signals. These have slightly different properties, and the window used can help determine this (a few dB here and there). And while we have a decent understanding of these effects, I didn't really have a gut feel for which window was the right for any given course of action. So I played with it.
I created a GNU Radio flowgraph that runs a noisy (real) sine wave through Hilbert transforms with different windows and plotted the resulting PSD on the same graph. The first image you see below is the full two-sided spectrum. You can easily see how each of the windows has different attenuation in the negative frequencies. If we're trying to make an analytic signal, we want to remove as much of the negative frequency as possible to make the signal's real and imaginary parts as near orthogonal as we can. Looks like the Blackman-harris window is the best for this.
But for sideband suppression, we want to minimize signals near 0 Hz as much and as quickly as possible. Below are two zoomed-in looks at this graph. The first is right around 0 Hz and the second is offset a little so we can see the negative image of the 1 kHz sine wave.
From these images, the Hamming window produces the fastest overall roll-off, even if it doesn't get quite as low as the Blackman-harris window does for full negative frequency rejection. On the other hand, the Hann window looks like it's the best compromise solution.
I find these plots to be generally helpful in giving me a basic understanding for what I'm trading off with the different windows, and so I wanted to provide them here, too.
As for the right default? Well, we have always been using the Hamming window, and it provides the best sideband suppression results. So we're going to stick with it as the default window so we don't change the behavior of existing GNU Radio projects that use the Hilbert transform block. The upside is that now we can select the right window for our needs if the default isn't suitable.
Chris Sylvain found a bug with the Kaiser window. I fixed it and another bug in the Blackman window today. Here are some images of this program being run again after the fixes. Changes how they behave pretty dramatically.