Speech synthesis: from a buzz to words

People have been building machines that talk for a long time. In the 1790s, Wolfgang von Kempelen made one of bellows, reeds and a leather mouth, played with the hands. At the 1939 New York World’s Fair, Bell Labs showed the Voder, an electronic voice played on keys and a pedal by trained operators. By the 1980s, Dennis Klatt’s rules could turn typed text into speech, and the voice Stephen Hawking used for decades came from that work. Today’s voices are learned from recordings by neural networks, and are hard to tell from people.

This page builds the classic kind, a formant synthesiser: it imitates what a voice does, step by step, and every step is something you can hear and change. Along the way it shows why a vowel is a vowel.

It leans on:

Low-bitrate voice coding, which first split a voice into a source and a filter, and found the filter by linear prediction. This page uses both.
Carving away harmonics, for the idea underneath: a bright source, shaped by filters. A formant synthesiser is subtractive synthesis, with the filters set to sound like a mouth.
Designing IIR filters, for resonators: the two-pole filters that make formants.
The first synthesis page’s instructions for playing the figures: the keyboards here set the voice’s pitch.

1 · A buzz and a filter

When you say “ah”, your vocal folds, two flaps in your throat, snap shut and spring open again a hundred or so times a second. Each snap is a click, and a click a hundred times a second is a buzz: a sound with a harmonic at every multiple of its pitch, each a little quieter than the last. On its own it sounds like nothing much.

Then the buzz goes through your throat, mouth and lips, a tube about 17 cm long whose shape you change as you talk. A tube like that resonates: some frequencies come through much louder than others. Those resonances are the formants. The buzz decides the pitch; the formants decide the vowel.

The buzz alone: a harmonic at every multiple of the pitch, falling steadily. Through a vowel’s filter: the same harmonics, but now some are much louder than others: the filter’s peaks (dashed), the formants, show through. Play the keyboard to hear it at any pitch: the vowel stays the same, because the filter does.

2 · One formant at a time

The synthesiser makes formants with resonators: two-pole filters, each with a frequency and a width, one after another in a chain. A voice has a formant every 1,000 Hz or so, more or less; this synthesiser uses five.

Add the formants one at a time. With only the first, every vowel is a muffled, nasal hum. With the first two, the vowel is there: “ee” and “ah” and “oo” are each clearly themselves. The third and the fourth make it sound like a person rather than a toy, and the fifth adds a little edge.

3 · The vowel chart

So two numbers, the first two formants, decide the vowel. Plot every vowel by them and you get the vowel chart. Phoneticians draw it with the second formant running right to left and the first top to bottom, because then it’s a map of the mouth: the first formant goes up as the jaw opens, and the second goes up as the tongue moves forward. “Ee” is high and front, “ah” low and back, “oo” high and back.

The vowels marked are from a classic study by Gordon Peterson and Harold Barney (1952): averages for 33 American men saying “heed”, “hid”, “head” and so on. Several of those “h_d” words are awkward or unfamiliar outside America, so the vowel lists on this page give everyday rhymes too: “hud” is the vowel in “cup”, “hood” the one in “book”. They’re American vowels, and an accent moves vowels around the chart: in Australian English, for one, several sit in quite different places, and the vowel in a word like “hod” isn’t the same vowel at all. So treat the labels as landmarks, not as your vowels: section 4 finds yours.

Drag the blue dot around the chart: the synthesiser follows, gliding from vowel to vowel as you go. Play the keyboard to sing whatever vowel the dot is on, or press Hold to keep it sounding. The third formant changes the colour a little (low, it makes an “r” sound). Your own vowels, once you’ve saved some, appear in orange.

4 · Your own vowels

Record a vowel, held steady for a couple of seconds, and the page finds its formants the way the voice coding page does. It records at 16,000 samples a second (up to 8 kHz: room for five formants), fits a filter of 18 poles to each 30 ms of the sound by linear prediction, finds where its poles sit (each sharp pair of poles is a formant), and takes the median over all the loud moments. Then it plays back the synthesiser’s copy of your vowel, to compare. The first three formants make the vowel; the fourth and fifth carry more of what makes a voice sound like you, so the copy uses them too, where they were found.

It works best on a low, steady voice. A voice’s spectrum is only its harmonics, so a high voice, whose harmonics are far apart, gives the fit less to go on, and the formants it finds get pulled towards the nearest harmonics. Tested on this synthesiser’s own vowels, it finds the first three formants within 20 Hz of the truth at a pitch of 110 Hz, but only within 61 Hz at 220 Hz; the fourth and fifth, within 24 Hz. Back vowels (“aw”, “oo”) are the hardest: their first two formants sit close together, and the fit can merge them into one, then report the third as the second. If a vowel’s copy sounds wrong, that’s the first thing to suspect: record it again.

Nothing is uploaded: your recordings stay in this browser.

Say a vowel, steadily (with the microphone, or open a recording). The spectrum is your recording’s; the dashed curve is the filter the formants found would make. Choose which vowel it was (each is listed with words that rhyme with it), and save it: it goes on the chart above, and the phrases below use it. Copy your vowels as numbers puts them on the clipboard; paste numbers back in to load them, on another device or after re-recording.

5 · Gliding: diphthongs

Some vowels don’t stay still. Say “aye”, slowly, and listen: it starts near “ah” and slides towards “ee”. It’s a diphthong: one vowel gliding into another. The “oh” in “go”, the “ow” in “how” and the “ay” in “day” are diphthongs too, which is why no single point on the chart sounds like them.

A diphthong, slowed down, as a spectrogram (bright: loud), with the synthesiser’s formants drawn over it (dashed). The first two formants slide from one vowel’s place to the other’s. Where each starts and ends is also where accents differ most: an Australian “day” starts with the mouth more open than an American one.

6 · Consonants

Consonants are mostly made by getting in the air’s way:

Hisses (fricatives) force air through a narrow gap: “s” behind the teeth, a hiss high up (around 6 kHz here); “sh” with the tongue further back, lower (around 3 kHz); “f” between lip and teeth, quiet and spread out. The synthesiser makes them from noise through a band of its own.
Stops block the air completely, then let it go: a moment of silence, then a burst. “P”, “t” and “k” add a puff of breath before the voice starts; “b”, “d” and “g” hum through the closure and start the voice straight away.
Nasals (“m”, “n”) send the voice out through the nose: a low, muffled hum.

The surprise is how you tell b, d and g apart. Their bursts differ a little, but mostly it’s the formant transitions: in the 50 ms after the release, the formants glide from where the closure put them into the vowel. From the lips (b), the second formant starts low; from behind the teeth (d), near 1,800 Hz; from the back of the mouth (g), the second and third start together. Researchers at Haskins Laboratories found this in the 1950s, by painting patterns onto film and playing them back as sound.

A stop and a vowel, slowed down, as a spectrogram, with the formants drawn over it. Watch where the second formant starts, and listen: play b, d, g, one after another, and the difference is in that short glide. Change the vowel, and “g“‘s start moves with it, while “b” and “d” stay put.

7 · Putting it together

String the sounds together, and give the pitch a shape. It drifts down a little through each phrase, starting a little lower again after each pause, and the last vowel carries the ending: it falls a little at the end of a statement, and rises at the end of a yes-or-no question. That shape (intonation) carries a lot: “are you okay” rises on “ay”, but “how are you” usually falls on “you”, like a statement, because a question that starts with a question word doesn’t need the rise to be heard as one.

Try:

Type sounds separated by spaces, from the chart’s vowels (ee i eh a ah aw u oo uh er), the diphthongs (ay eye oh ow oy) and the consonants (s sh f z h p t k b d g m n l r w y), with a comma for a pause, or try the examples. The first is the sentence every other talking page on this site uses, said in the site author’s own vowels, at 80 Hz. It’s a long way from a modern voice, and that’s the point: every part of it is a rule you can see.

The rules here are rough, and good formant synthesisers had hundreds more: how each sound changes its neighbours, how long each lasts in each place, the voice’s breathiness and tremble. Modern synthesisers skip the rules altogether. They learn from hours of recordings to predict the sound itself, sample by sample (WaveNet, in 2016, was the first to do it well), or a spectrogram that a second network turns into sound.

Summary

A voice is a source (the vocal folds’ buzz, or the hiss of air) and a filter (the throat and mouth).
The filter’s resonances are the formants. The first two decide the vowel; the rest make it sound human.
The vowel chart, F1 against F2, is a map of the mouth: the jaw opens F1, the tongue moves F2. Accents move vowels around it.
Linear prediction finds the formants of a recorded vowel, best for low voices.
Diphthongs glide from one vowel to another.
Consonants are hisses, stops and nasals; b, d and g are told apart mostly by how the formants glide into the next vowel.
Intonation (the pitch’s shape) turns a statement into a question.