Modulating with sound: FM synthesis

Additive synthesis needs dozens of sine waves, each with its own envelope, to sound natural. Subtractive synthesis gets a lot from a filter, but all the harmonics it leaves still rise and fall together, and it can’t make the sound of things that ring: bells, metal, glass, whose partials aren’t harmonics at all.

FM synthesis gets round both with almost nothing. Take one sine wave and wobble its frequency with another, very fast. That one trick makes a whole spectrum of new partials, harmonic or not; and changing how hard it wobbles changes them all at once, so a sound can brighten and mellow as it plays, as real instruments do. John Chowning found this at Stanford in the late 1960s, while experimenting with extreme vibrato. Yamaha licensed it, and in 1983 released the DX7, one of the best-selling synthesisers ever, whose electric piano and bells are all over the music of the 1980s. A few years later, cheap Yamaha FM chips put the same idea into PC sound cards, such as the AdLib, and into a generation of computer-game music.

It leans on:

Carving away harmonics, for LFOs and envelopes.
Modulation, part 1, which covers FM radio, sidebands and Bessel functions from the radio side. This page meets them again, as sounds.
Adding sine waves, for harmonics, and for playing the figures: click the keys, use the computer’s keys (A is C, Z and X change octave), or a MIDI keyboard, and Link knobs to a controller.

Two terms: the oscillator being wobbled is the carrier (it’s the one you hear), and the one doing the wobbling is the modulator.

1 · Vibrato, sped up

Start with vibrato: an LFO swinging a note’s pitch up and down a few times a second. Now turn the LFO’s rate up, and keep going.

A 440 Hz carrier, its frequency swung up and down by the swing, at the rate. Top: its frequency over a second. Bottom: its spectrum. Speed it up plays the swing at a rate climbing from 2 to 300 Hz over eight seconds.

At 5 Hz and ±30 Hz, the spectrum is a dense comb of lines 5 Hz apart, too close for the ear to separate, so it hears one note, wobbling. As the rate rises, the lines spread apart, at the modulator’s frequency either side of the carrier. Somewhere above about 20 to 40 Hz the wobble stops sounding like a wobble at all, and becomes a steady new tone colour: the lines are now heard as partials.

Those lines are sidebands, exactly as in radio. A carrier at fc, modulated at fm, has partials at fc, fc ± fm, fc ± 2fm, and so on.

2 · The modulation index

How strong each sideband is depends on one number, the modulation index: how far the frequency swings, divided by how fast it swings.

I = \frac{\text{swing}}{f_m}

The strengths follow the Bessel functions: the carrier is $J_0(I)$ strong, the first pair of sidebands $J_1(I)$ , the next pair $J_2(I)$ , and so on. You don’t need their formulas, only their shape: at a small index only the carrier and the first sidebands are there; as the index grows, energy spreads outwards to more and more sidebands, and each one rises and falls in turn.

A 1,000 Hz carrier, modulated at 100 Hz, so its sidebands are 100 Hz apart: each line’s strength, at the chosen index. The shading is Carson’s rule, the band that holds about 98% of the power. Turn the index up plays the index rising from 0 to 10 over six seconds.

At index 1, 7 lines are within 40 dB of the strongest; at 5, there are 19; at 10, 25. Carson’s band grows from 400 Hz to 1.2 kHz to 2.2 kHz. At an index of 2.405, the carrier itself vanishes ( $J_0$ crosses zero), and every bit of the power is in the sidebands. The sweep sounds like a filter opening, but with a strange, watery quality as each sideband swells and fades in turn.

That’s the key to FM as a synthesiser: one number controls the brightness, the way a filter’s cutoff does in subtractive synthesis, but without needing a bright wave to start from.

3 · Ratios: harmonic or clangorous

For a musical note, the carrier and modulator both follow the keyboard, at fixed ratios to the note’s frequency. The ratio decides where the sidebands land.

If the modulator is a whole-number multiple of the note (1 : 1, 1 : 2, 2 : 1 …), every sideband lands on a harmonic, and you hear a pitched note.
If the ratio isn’t a simple fraction (1 : 1.41, √2, say), the sidebands land between the harmonics. The sound has no clear pitch: it’s metallic, a bell, a gong, a struck piece of glass.

There’s one more twist. Sidebands below the carrier can go below 0 Hz. A negative frequency is the same spinning arrow turning the other way (see the Fourier transform), so it reappears as a positive frequency, upside down, and adds to (or cancels) whatever is already there.

A3’s partials (220 Hz), for the chosen carrier : modulator ratio and index. Orange lines have a noticeable share (more than 1% of the strongest line) folded back from below 0 Hz; blue ones don’t. The dashed lines are A3’s harmonics. Play the keyboard to hear each ratio.

At 1 : 1, the partials are every harmonic of the note, like a sawtooth. At 1 : 2, they’re only the odd ones (220, 660, 1,100 Hz …), like a square wave: hollow, clarinet-like. At 1 : 3.5 they’re harmonics of 110 Hz, an octave below the note, so the pitch drops. At 1 : 1.41 and 1 : 2.76 there’s no common fundamental, and the sound turns to metal.

4 · Brightness that moves

Here’s the idea that made FM a hit. Give the modulator its own envelope, separate from the carrier’s. Then the index changes through the note, so the brightness does too, with one envelope.

Chowning’s first convincing example was brass. Jean-Claude Risset had found that a trumpet gets brighter as it gets louder; so if the index follows the loudness, FM brass comes alive. A bell is the opposite: it starts bright, and its high partials die first, so it mellows as it rings.

A3’s spectrum over three seconds (the key is let go after 1.5 s), and the carrier’s and modulator’s envelopes. Pick a sound, and play it from the keyboard.

Brass’s index rises to 5 with the loudness, then settles to 3.5: the sound swells into brightness. “The same brass, its index fixed” holds it at 3.5 throughout: the same partials all through the note, which sounds flat and electronic, as on the first page’s “all fading together” note. The bell (1 : 1.4) starts at an index of 6, dies away in about two and a half seconds, and leaves the purer partials ringing. The electric piano’s index falls from 3 to 0.24 within half a second: a bright attack, then a mellow tone.

So FM solves the problem the first page ended on, cheaply: the whole spectrum changes through the note, from two oscillators and two envelopes.

5 · Operators and algorithms

Real FM synthesisers use more than two oscillators. Yamaha called each one an operator: a sine oscillator with its own envelope and its own ratio to the note. Each operator can be a carrier (heard) or a modulator (shaping another), and modulators can modulate modulators. The pattern they’re joined in is the algorithm.

The DX7 has six operators and 32 algorithms. This page uses four operators and eight algorithms, the layout of Yamaha’s later, cheaper four-operator instruments: enough to show every kind of connection.

The chosen algorithm (carriers on the bottom row, outlined; each modulator above what it modulates), and the harmonics it makes on A3. The operators are at 1, 1, 2 and 3 times the note, and the sliders set each one’s level: loudness for carriers, index for modulators. Play the keyboard to hear each.

A stack (algorithm 1: 4 → 3 → 2 → 1) is the richest: each modulator is itself modulated, so the partials multiply. At these levels it puts all 24 of the first 24 harmonics within 40 dB of the first. Two separate pairs (algorithm 5) give 13; one pair and two plain sines (7), 8; and all four heard (8, which is additive synthesis) only 3, the harmonics the four operators sit on.

Parallel carriers are how FM layers sounds: in a DX7 electric piano, one pair makes the warm body of the note, and another a short, bright “tine” at a high ratio, which dies away in a fraction of a second.

6 · Feedback

One more connection: an operator modulating itself, by feeding a little of its own output back into its phase. The DX7 allows it on one operator; so does this page’s operator 4.

Operator 4 alone, feeding back into itself. Top: its wave. Bottom: its harmonics (orange), against a sawtooth’s (grey dots, each at 1/n). Play the keyboard as you move the slider.

With no feedback, it’s a sine. At 0.5, the 2nd harmonic is 9.3 dB below the first, and the 10th 38.7 dB; at 1, they’re 7.0 and 24.1 dB down, close to a sawtooth’s 6.0 and 20.0. So a single operator with feedback can make a sawtooth-like wave for brass and strings. Push it past about 1.2 and the wave stops repeating cleanly: at 1.4 the power between the harmonics, noise, rises to 47 dB below the total, from 98 dB below at 1.

7 · FM on a sound card: the AdLib

In the late 1980s, PC games went from beeps to music largely thanks to one Yamaha chip, the OPL2 (YM3812). The AdLib card (1987) was built around it, and so was the hugely popular Sound Blaster (1989), which kept AdLib compatibility. A few years later its successor, the OPL3, came on later Sound Blaster cards; the Sega Mega Drive’s music came from another Yamaha FM chip.

The OPL2 is FM at its most economical:

Two operators per voice, a modulator and a carrier, and nine voices at once (or six, plus five drum sounds). That’s all the music in a game had to work with.
Feedback on the modulator, as in section 6.
Either FM (the modulator into the carrier), or both operators heard, side by side.
And its own trick: besides the sine, each operator can use three more waveforms, all made by switching off or flipping parts of the sine. A half-sine (only the top halves), a rectified sine (the bottom halves flipped up) and a quarter-sine (only the rising quarters). They cost the chip nothing, and they’re rich in harmonics, so two operators go a long way.

Top: the OPL2’s four waveforms (the ones in use marked). Middle and bottom: A3’s wave and harmonics, 0.1 s after the key goes down. Pick a sound in the style of 1990s game music, change anything, and play: like the chip, it plays at most nine notes at once.

Notice how much the waveforms change the sound with nothing else changed: the “plucked bass” modulates with a half-sine, and the “buzzy lead” uses a rectified sine into a half-sine carrier. That bright, slightly hollow, buzzy character is a big part of the sound of that era’s game music.

(This is a model in the OPL2’s style: its operators, waveforms and connections, but not the chip’s exact envelopes, levels and quirks.)

The next step for PC sound went another way: cards such as the Gravis UltraSound played back recordings of real instruments (samples) instead of synthesising them, which is a story for the next page in this series.

8 · Phase modulation in disguise

One more digital detail. The DX7 and the OPL chips don’t actually modulate the carrier’s frequency: they add the modulator’s output to the carrier’s phase. That’s phase modulation (PM). For a sine modulator the two give exactly the same partials (the frequency is the rate the phase changes, so one is the other, shifted a quarter of a cycle).

They differ when the modulator has an average above zero, as feedback can, and as the OPL’s half-sine does. True FM adds the modulator to the frequency, so an average above zero raises the pitch for as long as it plays. PM adds it to the phase, which only shifts the wave in time, and the pitch stays put.

A 440 Hz carrier, modulated at 110 Hz, by a sine or a half-sine: the spectrum of each way, overlaid. Play the plain carrier, then both, and listen to the pitch.

With a sine modulator, the two spectra match. (They only differ where sidebands fold back below 0 Hz, since the two start their modulators a quarter of a cycle apart, and folded partials then add up differently; at index 1 those are too weak to see.) With a half-sine (average 1/π), true FM’s pitch rises: at index 1, by 132.6 cents, more than a semitone, and at index 5 by 579.9 cents. Phase modulation’s stays at 440 Hz. A synthesiser whose notes slide out of tune as you change a modulator’s waveform or level would be useless, which is why digital “FM” synthesisers are really PM.

9 · The whole synth

Here’s a four-operator synth with everything on this page: eight algorithms, feedback on operator 4, each operator’s ratio, detune, level, waveform (the OPL2’s four), envelope and response to velocity. It plays up to eight notes at once.

The algorithm (left, or above on a narrow screen), and a scope and spectrum of the synth’s output as you play. Start from a preset and change anything. As on the subtractive page, you can save a sound in this browser, copy it as a code or a link, or download it as a file.

Some things to try:

Electric piano: two pairs (algorithm 5). Operators 1 and 2 make the body; operator 4, at 14 times the note, makes the bright tine at the start, and dies in a quarter of a second. Play hard and soft: the modulators respond more to velocity than the carriers, so harder playing is brighter, not just louder.
Tubular bells: ratios of 3.5 and 1.41, so the partials aren’t harmonics.
Brass: a stack with a little feedback; the modulators’ slower attacks make it swell into brightness.
Turn any modulator’s level up slowly while holding a note: that’s the index, and the brightness.

FM’s weakness is that it’s hard to program: the link between an operator’s settings and the sound is far from obvious, and many DX7 owners famously never went beyond its factory sounds. Sample playback, where the sound is a recording of the real thing, is the next big idea, and comes next in this series.

Summary

Modulating a carrier’s frequency with another oscillator makes sidebands at fc ± k·fm. Slow, it’s vibrato; above about 20 to 40 Hz, it’s a new tone colour.
The modulation index (swing ÷ modulator frequency) sets how strong each sideband is, following the Bessel functions: a bigger index spreads the power wider, so the sound is brighter.
Whole-number ratios make harmonic, pitched sounds (1 : 2 gives odd harmonics only); other ratios make inharmonic, bell-like ones. Sidebands below 0 Hz fold back.
Giving the modulator its own envelope makes the brightness move through each note, which is what makes FM brass, bells and electric pianos sound alive.
Operators join in algorithms: stacks for richness, parallel carriers for layers. Feedback turns one operator into a sawtooth-like wave, and then noise.
The AdLib’s OPL2 chip did FM with two operators, nine voices and four sine-based waveforms.
Digital “FM” synthesisers really use phase modulation, which keeps the pitch steady whatever the modulator does.