DSP Explorables

Low-bitrate voice coding

A digital phone call used to be sent at 64,000 bits per second. A mobile phone sends the same voice at about 8,000 or fewer, and it still sounds like you. The MP3 encoder saves bits with a model of the listener: what the ear can’t hear doesn’t need sending. Voice coders save them with a model of the talker. A voice is a buzz or a hiss, shaped by a tube (the throat, mouth and nose) whose shape changes a few times a second. Send the shape of the tube and a description of the buzz, and the other end can rebuild the voice.

This page builds that up from the simplest approach, one step at a time, and ends with a working coder of the kind inside most phones: CELP, code-excited linear prediction. Every coder here is real. It writes an actual stream of bits, and its decoder rebuilds the sound from those bits alone, so every bit rate quoted is counted, not estimated.

It draws on many earlier pages:

Every figure can use your own voice. Record a sentence or two here, at a normal speaking level, and choose your recording in any figure.

Nothing recorded yet.

Nothing is uploaded: the recording stays in this page, in your browser, and is gone when you leave it.

0 · Why speech is special

The bit rate of a digital sound is how many bits per second it takes: samples per second × bits per sample. CD audio is 44,100 samples a second at 16 bits each, for 705.6 kbit/s per channel.

A phone doesn’t need nearly that. Speech is perfectly understandable with nothing above about 3.4 kHz, so the telephone network samples at 8 kHz (its Nyquist frequency is 4 kHz). And it uses 8 bits per sample, for 64 kbit/s. That’s the starting point. The rest of the page is about getting to 8 kbit/s and below.

The synthetic speech (a made-up sequence of vowels and consonants, “sha-ee, t-oh, s-ah-oo, eh”, built from a buzz and formants so that its pitch and voicing are known exactly), at 44.1 kHz and at 8 kHz. Everything above 4 kHz is gone, including most of the hiss of the “s”. Listen: it sounds like a phone, but every sound is still clear. The sung phrase and your own recording behave the same way.

From here on, everything runs at 8 kHz. “The original” in each figure means the 8 kHz version.

1 · Fewer bits per sample

The obvious way to save bits is to use fewer for each sample, rounding every sample to one of $2^b$ levels (quantising it, as on the sampling page). Each bit removed halves the number of levels and doubles the rounding error, which costs about 6 dB of signal-to-noise ratio (SNR: the signal’s power over the error’s, in dB).

With evenly spaced levels (linear quantisation) there’s a catch. The rounding error is the same size whether the voice is loud or quiet, so quiet sounds suffer far more than loud ones. Speech has a huge range of loudness, between syllables, between people, and between a whisper and a shout. The telephone’s answer is μ-law (in North America and Japan; Europe uses the very similar A-law). Squash the samples with a logarithm before rounding, and expand them afterwards:

F(x) = \operatorname{sign}(x)\,\frac{\ln(1 + \mu|x|)}{\ln(1 + \mu)}, \qquad \mu = 255

The levels end up close together near zero and far apart near full scale. The rounding error then grows with the signal, and the SNR stays roughly constant over a wide range of loudness.

Top: the SNR of a sine wave at different levels, quantised with 8 bits. Linear quantisation gives 49.8 dB at full scale, but loses 10 dB with every 10 dB the signal drops: 9.3 dB at −40 dB. μ-law gives up some SNR at the top (37.4 dB), but stays within 3.5 dB of that down to −40 dB (34.0 dB). Bottom: where the levels are, between 0 and a tenth of full scale. μ-law packs them in near zero: its finest step is as fine as a 13.5-bit linear quantiser’s.

On the synthetic speech, 8-bit linear gives 32.8 dB and 8-bit μ-law 37.9 dB. At 4 bits, both fall apart: 8.3 dB linear and 13.1 dB μ-law.

Tick add dither to add triangular noise of ±1 step just before rounding, as on the sampling page. It costs some SNR: on the speech, 8-bit linear falls from 32.8 to 29.5 dB, and μ-law from 37.9 to 33.0 dB. What it buys is a different kind of error. Without dither, the rounding error follows the signal. For a quiet sine (40 dB below full scale), 69.7% of the 8-bit linear error is harmonics of the sine, and 22.3% of μ-law’s, heard as a gritty edge on quiet sounds. With dither, it’s under 1% either way: the error becomes a steady hiss, which the ear finds far less objectionable than distortion. Listen at 4 to 6 bits, with and without.

That’s why 8-bit μ-law often sounds better than its bit count suggests: quiet sounds get the fine steps, and loud sounds get coarse steps that their own loudness hides. Several computers of the late 1980s and 1990s stored their sound this way, the Acorn Archimedes with its 8-bit logarithmic samples among them. But at 4 bits a sample, even μ-law is clearly broken. Rounding samples more coarsely only goes so far.

2 · Prediction: send only the surprise

Speech changes smoothly from one sample to the next, so each sample is fairly predictable from the ones before it. If the decoder can make the same prediction, the encoder only needs to send the prediction error: the difference between the real sample and the prediction. That error is smaller than the sample itself, so it needs fewer bits for the same accuracy.

The prediction gain measures how much smaller: the signal’s power over the error’s, in dB. Each 6 dB of gain is worth about one bit per sample.

Top: 25 ms of the speech (grey) and the prediction (blue). Bottom: the prediction error, on the same scale.

Predicting each sample as the previous one gains 5.7 dB on the synthetic speech, worth about one bit.
Carrying on the slope of the previous two gains 8.1 dB.
The best 10 (section 3) gains 20.4 dB, more than three bits.

IMA ADPCM (adaptive differential PCM, 1992) is the simplest working version. It predicts from the previous sample, sends the difference with 4 bits, and adapts its step size: larger after large differences, smaller after small ones. The decoder can follow along, because it adapts from the codes alone. At 32 kbit/s, its segmental SNR (the readout; see below) is 18.8 dB. Plain 4-bit linear gets 8.1 dB and 4-bit μ-law 12.8 dB at the same rate.

Segmental SNR, used from here on, is the SNR of each 20 ms stretch, averaged over the stretches that aren’t near silence. Plain SNR is dominated by the loudest moments. Segmental SNR also shows how a coder does on the quieter ones, which matter just as much to the ear.

3 · Linear prediction

Instead of a fixed rule, predict each sample from the previous $p$ samples, with weights chosen to suit the sound:

\hat{x}[n] = -\left(a_1 x[n-1] + a_2 x[n-2] + \dots + a_p x[n-p]\right)

(The minus sign is a convention that makes the formulas below tidier.) Speech changes, so the weights are worked out afresh for each frame of 20 ms, from a 30 ms window around it. With $p = 10$ at 8 kHz, that’s linear prediction, the “LP” in CELP.

Finding the best weights. “Best” means the smallest average squared error. Setting the derivative of the error with respect to each weight to zero gives $p$ equations. The only things in them are the autocorrelation values $r[k]$ of the frame: how well it matches itself shifted by $k$ samples, as on the pitch page:

\sum_{i=1}^{p} a_i\, r[|k - i|] = -r[k], \qquad k = 1, \dots, p

Solving them: Levinson–Durbin. These equations have a special pattern (each diagonal of the matrix holds a single value), and the Levinson–Durbin recursion uses it. It finds the best predictor of order 1, then uses that to get the best of order 2, and so on, about $p^2$ operations in all instead of $p^3$ . Each step works out one new number, the reflection coefficient $k_m$ , and updates the earlier weights with it:

a_i^{(m)} = a_i^{(m-1)} + k_m\, a_{m-i}^{(m-1)}, \qquad a_m^{(m)} = k_m, \qquad E_m = E_{m-1}\,(1 - k_m^2)

$E_m$ is the prediction error’s power after step $m$ . Each step can only shrink it, and as long as every $|k_m| < 1$ , the filter in the next paragraph is guaranteed stable.

The payoff: the predictor is a model of the vocal tract. Write the error as a filter: $e[n] = x[n] + a_1 x[n-1] + \dots + a_p x[n-p]$ . Its transfer function is $A(z) = 1 + a_1 z^{-1} + \dots + a_p z^{-p}$ (as on the IIR page). Run it backwards, $1/A(z)$ , and the error turns back into speech. $1/A(z)$ is an all-pole filter whose frequency response is the smooth envelope of the frame’s spectrum, and its poles sit on the formants. The predictor has found the shape of the tube.

Top: one 30 ms frame’s spectrum (grey), the envelope $1/A$ of its predictor (blue), and, for comparison, the cepstral envelope from the pitch-shifting page (green, dashed). Middle: the prediction gain at each order, as Levinson–Durbin adds one coefficient at a time. Bottom: the poles of $1/A(z)$ , inside the unit circle. Angle is frequency: 0 Hz on the right, 4 kHz on the left. A pair above and below the line makes a peak; a lone pole on the line shapes the overall slope instead.

At 0.36 s, in the “ah”, the order-10 predictor’s poles put peaks at 742, 1104 and 2428 Hz. The vowel was made with formants at 730, 1090 and 2440 Hz. That frame’s prediction gain is 20.4 dB. (At 0.6 s, in the “ee”: 289, 2285 and 3046 Hz, made with 270, 2290 and 3010.) Drag order down from 10 and watch the envelope lose its peaks, one formant per pair of poles. Drag time through the sentence and watch the poles follow the vowels.

Two poles make one resonance (as on the IIR page), so 10 poles can describe 5 peaks: enough for the 3 or 4 formants below 4 kHz, plus the overall slope of the voice’s spectrum.

4 · The residual

Run the speech through $A(z)$ , and what comes out is the prediction error, called the residual. If the predictor has captured the tube, what’s left should be whatever went into the tube: the source.

Top: 40 ms of the speech. Middle: the residual over the same stretch, scaled up. In a vowel it’s a train of sharp pulses, one per vocal-fold cycle. Bottom: the average spectrum of each. The speech has a spectral flatness of 0.03, where 1 would be perfectly flat (white noise). The residual’s is 0.72: the formants have been taken out. Listen to the residual: buzzy where the speech is voiced, hissy in the “s” and “sh”, and much harder to understand.

That’s the source-filter model of speech, measured rather than assumed:

a source: a buzz of pulses from the vocal folds (voiced), or turbulent noise (unvoiced)
a filter: the vocal tract, $1/A(z)$ , which changes as the mouth moves.

The pitch-shifting page relied on the same split. So, to send speech, send the filter (10 numbers every 20 ms) and some description of the source. How good that description is decides everything else.

5 · The vocoder: send almost nothing

The cheapest description of the source: is it voiced, at what pitch, and how loud? The decoder then makes its own source, a train of pulses at the pitch period or white noise, and runs it through the filter. That’s an LPC vocoder. It’s roughly how LPC-10 worked, the US government’s 2.4 kbit/s standard from 1984.

Here each 20 ms frame sends:

the filter, as 34 bits (how, in section 9)
the pitch period from YIN, or “unvoiced”: 7 bits
the loudness: 5 bits.

That’s 46 bits, 2.3 kbit/s.

Top: what’s sent for pitch: one value per 20 ms frame (blue), or “unvoiced” (grey). The synthetic speech has 88 voiced frames out of 138. Bottom: the real residual (orange) and the pulses the decoder makes instead (blue). Listen: the vocoder is intelligible but robotic and buzzy, like an old speak-and-spell toy.

The vocoder makes no attempt to match the waveform, so SNR is meaningless here: it comes out negative. Its weakness is the source. A real voice’s pulses aren’t identical clicks: they vary in shape and spacing, carry some breath noise, and blur into each other at the edges of voiced sounds. A plain pulse train can’t do any of that. The rest of the page is about describing the residual properly, while still spending few bits on it.

6 · The pitch predictor

In a vowel, the residual is a train of pulses, roughly one per pitch period, and each pulse looks a lot like the one before. That’s another prediction: predict the residual from itself, one period earlier. Short-term prediction (section 3) used the last few samples. This long-term or pitch predictor uses samples from a whole period back, so it needs the period, which is what YIN finds.

Top: the residual (orange), and its prediction from one period earlier (blue). Bottom: what’s left once that’s taken away (green). The best lag (20 to 147 samples, 54 to 400 Hz) and gain are chosen for every 5 ms. On the voiced frames of the synthetic speech, this removes another 9.8 dB.

In the full CELP coder (section 7) it’s worth even more. Without it, the coder’s segmental SNR is 11.1 dB. With it, 19.7 dB. Listen to both.

In a CELP coder, the pitch predictor is called the adaptive codebook: a “book” of candidate excitations, one for each possible lag, made from the decoder’s own recent excitation. It adapts because it’s rewritten as the excitation goes by. One consequence: it can only repeat what has already been sent. Something else has to supply what’s genuinely new, such as the first pulse of a vowel.

7 · Analysis by synthesis

That something else is the fixed codebook: a set of candidate excitation patterns that the encoder chooses from. It sends only the chosen pattern’s number. The “CE” in CELP, code-excited, means exactly this. The question is how to choose.

The key idea of CELP is to choose by listening to the result. The encoder contains a complete copy of the decoder. For each 5 ms subframe it tries the candidates, runs each through the decoder’s filter, and keeps the one whose decoded output is closest to the real speech. This is analysis by synthesis: the encoder searches the decoder’s outputs, not the inputs.

Here, every 5 ms the encoder:

Works out the target: what the decoder still needs to produce, once the ringing left over from the last subframe is taken off.
Tries all 128 pitch lags of the adaptive codebook, each filtered, and keeps the best (7 bits) and its gain (4 bits).
Tries the fixed codebook on what’s left of the target. This is an algebraic codebook, as in G.729:
- four pulses of +1 or −1 in the 40-sample subframe
- each pulse limited to its own track of positions: 0, 5, 10, …; 1, 6, 11, …; 2, 7, 12, …; and the last pulse on either of the two remaining tracks.
That gives 8 × 8 × 8 × 16 = 8192 combinations of positions (13 bits) and 4 sign bits. Signs are fixed in advance, from the target. A full search of 8192 combinations is still quick, because only the pulses’ positions change from one candidate to the next.
Sends the fixed codebook’s gain (5 bits).

One 5 ms subframe at a time. Top: the target (grey, thick); what the adaptive codebook alone produces (blue, dashed); and what the adaptive codebook and the four pulses produce together (orange). Bottom: the excitation actually sent: the past excitation repeated at the pitch lag and scaled (blue), plus the four pulses (orange). The readout shows how much of the target’s energy each stage leaves as error.

Step through the subframes. In steady vowels the past does most of the work: on average it takes the error 12.7 dB below the target, and the pulses take it another 3.7 dB down. At the start of a vowel after silence there’s no useful past yet. In the first subframe of the “oh” and of the “eh” with any sound in it (at 865 and 2205 ms), the past removes nothing (0.0 dB), and the pulses take the error down by 9.5 and 10.0 dB.

The error being minimised in step 3 isn’t the plain difference between the speech and its copy, though. It’s filtered first, and that filter is the subject of the next section.

8 · Perceptual weighting

At 8 kbit/s, the decoded speech can’t be exact, so there will be noise. The useful question is where the noise goes. Noise at a formant, where the speech is loud, is masked by the speech, as on the MP3 page. Noise in the valleys between formants, where the speech is quiet, is easy to hear.

So CELP measures the error through a perceptual weighting filter before minimising it:

W(z) = \frac{A(z/\gamma_1)}{A(z/\gamma_2)}, \qquad \gamma_1 = 0.9,\ \gamma_2 = 0.6

Here $A(z/\gamma)$ means every coefficient $a_i$ multiplied by $\gamma^i$ , which pulls the poles in towards the centre and flattens the peaks. $W$ is large in the valleys and small at the formants. Errors in the valleys count for more, so the search avoids them, and the noise ends up where the speech hides it. It’s the same idea as the sigma-delta modulator’s noise shaping: the noise can’t be removed, but it can be moved to where it does less harm.

One loud 32 ms stretch: the speech’s spectrum (grey) and envelope (dashed), and the coding noise without weighting (orange) and with it (blue). The readouts cover the whole sound, comparing noise and speech at the formant peaks (within 10 dB of the envelope’s top) and in the valleys (more than 25 dB below it). On the synthetic speech, weighting leaves the noise at the peaks unchanged (−21.7 dB → −21.8 dB). It lowers the noise in the valleys from +0.3 dB to −5.2 dB, from as loud as the speech there to well below it. Plain SNR changes much less (19.1 → 20.4 dB). Listen to both: with weighting, the noise is less of a hiss.

Plain SNR can’t see this improvement, because SNR doesn’t know where noise is audible. That’s a general lesson for this page: SNR is a crude measure of how speech sounds. The readouts give it because it’s measurable, but listening is the real judge. Codec developers use listening tests, and standard perceptual measures that model hearing, which this page doesn’t have.

9 · Sending the filter: line spectral pairs

Every 20 ms, the decoder needs the filter: 10 coefficients. The obvious way to send them is to round each one to some number of bits. But the coefficients are fragile. A small error in one of them can move a pole outside the unit circle, and then the synthesis filter is unstable: its output grows without limit. And there’s no easy way to check a set of rounded coefficients without finding the poles.

Line spectral pairs (LSPs) are a different way to write the same filter. From $A(z)$ , make two polynomials, one by adding $A$ to a reversed copy of itself and one by subtracting:

P(z) = A(z) + z^{-(p+1)} A(z^{-1}), \qquad Q(z) = A(z) - z^{-(p+1)} A(z^{-1})

If $1/A(z)$ is stable, all the roots of $P$ and $Q$ lie exactly on the unit circle, and they alternate: one of $P$ ‘s, then one of $Q$ ‘s, and so on. So each is described by one number, its angle, which is a frequency: the line spectral frequencies (LSFs). And $A(z) = (P(z) + Q(z))/2$ gives the filter back. This makes them ideal to send:

Stability is easy to keep. Any set of LSFs that are in increasing order gives a stable filter. The decoder only has to keep them in order and a little apart.
Errors stay local. Each LSF affects the envelope mainly near its own frequency, so rounding errors don’t spread.
They blend smoothly. Between frames, the LSFs can be averaged to give in-between filters (the coder here does this for every 5 ms subframe), and an average of increasing lists is still increasing.

Here each LSF gets its own range and 3 or 4 bits, evenly spaced: 34 bits in all, as in FS-1016, a 4.8 kbit/s CELP standard from 1991.

Left: the poles of the original filter (grey) and of the filter rebuilt from what’s sent (blue; red if outside the circle), with the original’s LSFs on the circle (green for $P$ , orange for $Q$ , alternating). Right: the two envelopes. The readouts give this frame’s spectral distortion: the RMS difference between the envelopes, in dB, where about 1 dB is usually counted as inaudible. They also cover every loud frame of the sound.

Rounding each coefficient to 8 bits (80 bits a frame) makes 64 of the synthetic speech’s 109 loud frames unstable. At 10 bits (100 a frame), 10 still are. It takes 12 bits each, 120 bits a frame, to make none unstable.
The LSFs at 34 bits a frame make none unstable, with an average distortion of 1.70 dB.

Real coders go further. G.729 predicts each frame’s LSFs from the previous frames and sends only the correction, using vector quantisation (one index for a whole group of values, chosen from a trained table). That gets the filter into 18 bits per 10 ms.

10 · Putting it together

The coder on this page, per 20 ms frame of 160 samples:

What	Bits	How often	Bits per frame
The filter: 10 LSFs	34	each frame	34
Pitch lag (20 to 147 samples)	7	each 5 ms subframe	28
Adaptive-codebook gain	4	each subframe	16
Pulse positions (four tracks)	13	each subframe	52
Pulse signs	4	each subframe	16
Fixed-codebook gain	5	each subframe	20
Total			166 bits: 8.3 kbit/s

Every coder on the page, on the same sound: the spectrogram of what each decoder produces, and a count of the bits in each stream. On the synthetic speech:

Coder	Bit rate	Segmental SNR
16-bit PCM	128 kbit/s	50.0 dB (the limit of the measure)
μ-law, 8 bits	64 kbit/s	37.6 dB
IMA ADPCM	32 kbit/s	18.8 dB
Our CELP	8.33 kbit/s	19.7 dB
LPC vocoder	2.31 kbit/s	meaningless (−2.5 dB)

CELP matches ADPCM’s segmental SNR at a quarter of the bit rate, and its noise is shaped to hide under the formants, which ADPCM’s isn’t. Listen down the list, and then with your own voice. The CELP coder keeps your voice recognisably yours at an eighth of μ-law’s rate. The vocoder keeps the words but loses the person.

The coder here is a teaching version, simplified in places:

Whole-sample pitch lags. Real coders use a third of a sample or finer.
No postfilter. Real decoders add one that deepens the formant valleys a little more.
Plain scalar quantisation of the LSFs and gains, rather than predicted, vector-quantised ones.

Real coders also handle lost packets, background noise and silence.

Where it went next:

G.729 (1996) is an 8 kbit/s ACELP coder much like this one, and was a mainstay of internet telephony for years.
Mobile phones have used AMR (4.75 to 12.2 kbit/s, ACELP) since the early 2000s, and its 16 kHz wideband sibling AMR-WB for “HD voice”.
Opus (2012), used in video calls and web browsers, switches between a linear-prediction speech coder (SILK) and an MDCT music coder (CELT, a cousin of the MP3 page’s MDCT).
Most recently, neural codecs replace the hand-designed source and filter with networks trained on huge amounts of speech, and get natural-sounding voices at 3 kbit/s and below.

All of them still begin by predicting each sample from the ones before.

Summary

Phone speech starts at 64 kbit/s: 8 kHz × 8 bits, with μ-law keeping the SNR roughly constant across loud and quiet sounds.
Prediction sends only what the decoder can’t guess: ADPCM gets good speech at 32 kbit/s.
Linear prediction finds, every 20 ms, the 10 weights that best predict each sample (from the autocorrelation, by Levinson–Durbin). The predictor $1/A(z)$ is the vocal tract: its poles sit on the formants.
The residual is the source: pulses (voiced) or noise (unvoiced). An LPC vocoder sends only the filter, pitch and loudness (2.3 kbit/s): intelligible, but robotic.
CELP describes the residual properly: an adaptive codebook (the past, one pitch period back) and a fixed codebook (here four algebraic pulses), chosen by analysis by synthesis through a perceptual weighting filter that hides the noise under the formants.
The filter travels as line spectral frequencies, which keep it stable and quantise gracefully.