Changing the sample rate

Digital audio comes at many sample rates: 44.1 kHz for CDs, 48 kHz for video, 8 or 16 kHz for phone calls and voice assistants, 96 or 192 kHz for studio recordings. Whenever these meet, one has to be converted to another, without going back to analogue. That’s sample-rate conversion, or resampling. Lowering the rate is called decimation (or downsampling), and raising it interpolation (or upsampling).

The obvious ways to do both are wrong, in ways the sampling page predicts. This page fixes them one at a time, then makes the result fast. It ends inside every modern audio converter, where resampling lets a stream of single bits carry CD-quality sound.

To halve the sample rate, the obvious thing is to keep every 2nd sample and throw the rest away. To divide it by 3, keep every 3rd. In general, keeping every $M$-th sample divides the rate by $M$.

But look at what that does. The kept samples are exactly what you’d have got by sampling the original signal at the lower rate in the first place. So it is sampling, again, and everything on the sampling page applies, with the new, lower rate. In particular, anything above the new Nyquist frequency (half the new rate) aliases: it folds back down and pretends to be a lower frequency.

The cure is the same as for an analogue-to-digital converter: an anti-aliasing filter. Before throwing samples away, remove everything that would alias, with a digital low-pass filter whose cutoff is the new Nyquist frequency, $f_s/2M$. This is exactly the kind of filter the FIR page designs. The filter must come first: once samples are thrown away, aliases are mixed in for good.

Tick low-pass filter first in the figure above. The tones above the new Nyquist frequency are removed before they can fold, and the sweep now rises, then fades away instead of turning round. The dashed curve on the top panel is the filter’s response.

So a decimator is a low-pass filter followed by “keep every $M$-th sample”. Notice the waste, though. The filter calculates an output for every sample at the high rate, and then $M - 1$ of every $M$ results are thrown away. Section 5 comes back to that.

Raising the rate by a factor $L$ means making $L - 1$ new samples between each pair of existing ones. What values should they have? The sampling page has the answer: if the signal was band-limited, the true values between the samples are completely determined, and a sum of sincs gives them exactly. The question is how to compute that efficiently, using the tools we have.

Two naive answers come to mind first. Repeat each sample $L$ times, which makes a staircase like a simple DAC’s output. Or join the dots with straight lines. Both are better than nothing, but neither gets the true values.

The right answer starts with something that looks even worse: insert zeros. Put $L - 1$ zeros after each sample. That adds no information at all, and changes nothing about the spectrum: the sampling page showed that sampled signals already contain copies of their spectrum at every multiple of the sample rate. At the old rate we only ever looked at the range up to the old Nyquist frequency. At the new, higher rate, the range is $L$ times wider, and it now includes copies that were there all along. They’re called images.

Then low-pass filter away everything above the old Nyquist frequency. That leaves only the original copy, which is the band-limited signal: this is the sum-of-sincs reconstruction from the sampling page, done digitally. One detail: the zeros make the signal $L$ times weaker on average, so the filter has a gain of $L$ to make up for it.

Most real conversions aren’t by a whole number. From 44.1 kHz (CD) to 48 kHz (video), the ratio is $48\,000/44\,100 = 160/147$. That’s done in three steps: up by $L = 160$, by inserting zeros, filter, then down by $M = 147$, by keeping every 147th sample. One low-pass filter serves both purposes. Its cutoff is the lower of the two Nyquist frequencies, so it removes the images from the upsampling and anything that would alias in the downsampling.

The trouble is the rate the filter runs at: $160 \times 44.1$ kHz $= 7.056$ MHz. At that rate, a filter that passes everything up to 20 kHz and blocks everything from 22.05 kHz is extremely sharp, relative to the rate, and needs over 22,000 taps. Running it directly costs over a hundred billion multiplications per second, for one channel of audio.

The direct method wastes most of its effort in two ways, and both can be skipped without changing the answer at all.

1. Outputs that are thrown away. Only every $M$-th filter output is kept, so only calculate those. An FIR filter’s outputs don’t depend on each other, so nothing is lost by skipping the rest.
2. Multiplying by zeros. Of the $L$ samples in each group at the high rate, $L - 1$ are inserted zeros. Each output is a sum of taps × samples, so most of its multiplications are by zero. Only every $L$-th tap meets a real sample.

Which taps meet real samples depends on where the output falls between the original samples. There are $L$ possibilities, so the filter splits into $L$ interleaved sub-filters: taps $0, L, 2L, \dots$, then taps $1, L + 1, 2L + 1, \dots$, and so on. Each output uses exactly one of them. These sub-filters are called phases, which gives the method its name: a polyphase filter.

Each sub-filter also has a neat meaning. Sub-filter $p$ is the low-pass filter’s impulse response, a sinc, sampled at an offset of $p/L$ of a sample. So it’s a recipe for estimating the signal a fraction $p/L$ of the way between samples, from the samples around it. Polyphase resampling just picks the right recipe for each output.

That view leads to arbitrary resampling. Store a large table of sub-filters, say 1000 per sample period, and any position between samples can be served by the nearest one (or by interpolating between two). The ratio no longer has to be a fraction of small whole numbers, and it can even change while running. That’s how a computer plays a 44.1 kHz file through a 48 kHz sound card, and how two devices whose clocks differ slightly stay in step.

The sampling page ended with two analogue filters around every converter, which have to be sharp and so are expensive. Resampling is the way out, and it’s how practically every modern audio converter works:

• An analogue-to-digital converter samples many times faster than needed, for example $64 \times 48$ kHz $= 3.072$ MHz. Nothing much exists up there, so a gentle, cheap analogue filter is enough to prevent aliasing. Then a digital decimator, as in sections 1–5, does the sharp filtering and brings the rate down to 48 kHz.
• A digital-to-analogue converter does the reverse. It interpolates up to a high rate digitally (section 3), so the images that the analogue filter must remove are pushed far above the audio band, and a gentle filter can remove them. The “digital filter” settings on many DAC chips are choices for this interpolation filter.

Digital filters are exact, repeatable and cheap, while sharp analogue ones are none of those, so moving the hard work into the digital domain is a clear win. And once a converter runs at 64 times the necessary rate, something surprising becomes possible.

Super Audio CD (SACD) stores audio in a format called DSD: a stream of single bits, $+1$ or $-1$, at 2.8224 MHz ($64 \times 44.1$ kHz). By the rule on the sampling page, one bit gives a signal-to-noise ratio of $6.02 + 1.76 \approx 8$ dB. A CD’s 16 bits give 98 dB. How can one bit possibly compete?

If the rounding error behaves like random noise, its total power is fixed by the size of the step, and it’s spread evenly over all frequencies from 0 to the Nyquist frequency. At 64 times the rate, it’s spread over a 64 times wider range, and only 1/64 of it lands in the audio band. That’s 18 dB less noise: 3 dB for every doubling of the rate, or half a bit.

That helps, but 8 + 18 dB is still nowhere near 98. And with only one bit, there’s a worse problem: the error isn’t random. With no room for dither, a one-bit sine wave is just a square wave, and its error is the square wave’s harmonics, however fast you sample. Select none in the figure below to see this.

The trick is feedback. Keep a running total of how far the output has fallen behind the input, and choose each output bit to catch up:

# A first-order sigma-delta modulator: one bit per sample.
total = 0
for n in range(num_samples):
    out[n] = +1 if total >= 0 else -1
    total += x[n] - out[n]      # how far the output is behind the input, so far

Add up this loop’s outputs over any stretch of time. They equal the sum of the inputs, minus the change in the running total. The total never gets far from zero (for inputs between −1 and +1, it stays within ±2), so over a long stretch, the average of the bits equals the average of the input. An input of 0.5 gets three $+1$s for every $-1$. The signal is carried by the density of $+1$s.

In frequency terms, the error is still large, but it’s been moved. The output is the input minus the change in the running total from one sample to the next, and a change from sample to sample is fast: slowly varying error cancels out, and what’s left is at high frequencies. The error has been shaped, pushed away from the low frequencies we care about and up into the megahertz range, where nobody listens. The name, sigma-delta (or delta-sigma), comes from the Greek letters for “sum” (the running total) and “difference”.

More elaborate feedback, with several running totals, pushes the noise out more steeply. A modulator of order $N$ shapes the noise with $N$ differences in a row. In theory, each doubling of the sample rate then buys about $6N + 3$ dB instead of 3, and measurements of the modulators in the figure below come close (within about 2 dB, except 1st order, which falls short). Higher orders need care: the loop can become unstable, so above 2nd order the feedback is tamed slightly (see the code comments if you’re curious).

The bitstream isn’t the end. To get ordinary 44.1 kHz audio, run a decimator (sections 1–5) dividing by 64. Its low-pass filter removes everything above 20 kHz, which is nearly all of the noise, and in doing so it averages many bits into each output sample. An average of many $\pm 1$s can take any value in between, so the output samples need many bits: the resolution comes from the averaging. The test that goes with this page measures 99.5 dB signal-to-noise for the decimated 5th-order bitstream at −6 dB.

Why bother with one bit at all? Because a one-bit converter is almost trivially simple: a switch between two voltages. A 16-bit converter needs 65,536 precisely spaced levels, and any error in the spacing is distortion. Two levels always lie on a straight line, so a one-bit converter is perfectly linear. All the difficult work moves into digital filters, which are exact.

One bit has real drawbacks too: it can’t be properly dithered, high-order loops overload on loud signals, and the noise above 20 kHz must be filtered before it reaches an amplifier. So most converter chips today use the same sigma-delta idea with a few bits instead of one, which keeps the advantages and avoids most of the drawbacks.

• Downsampling by $M$ (keep every $M$-th sample) is sampling again: anything above the new Nyquist frequency aliases. Low-pass filter first.
• Upsampling by $L$: insert zeros, which reveals copies of the spectrum (images), then low-pass filter them away. That’s sinc reconstruction, done digitally. Repeating samples or joining them with straight lines leaves images behind.
• A ratio $L/M$ (like 160/147 for 44.1 → 48 kHz) is up by $L$, one filter, then down by $M$.
• Polyphase filters compute only the kept outputs, and only the taps that meet real samples: the same answer, thousands of times faster for awkward ratios.
• Oversampling converters do the sharp filtering digitally. Taken to the extreme, with noise shaping (sigma-delta), a stream of single bits can carry more than CD quality.

• R. E. Crochiere and L. R. Rabiner, Multirate Digital Signal Processing (1983): the classic text on decimation, interpolation and polyphase filters.
• Julius O. Smith III, Digital Audio Resampling Home Page: arbitrary-ratio resampling with a table of sub-filters, as in section 5.
• R. Schreier and G. C. Temes, Understanding Delta-Sigma Data Converters (2005): the standard reference on sigma-delta modulation and noise shaping.
• S. P. Lipshitz and J. Vanderkooy, “Why 1-bit sigma-delta conversion is unsuitable for high-quality applications” (AES convention, 2001): the case against one bit, and for a few.