Pitch detection

A guitar tuner, a singing app that scores you, a pitch-correction plug-in, and a phone codec squeezing your voice into a few kilobits per second all need to answer the same question many times a second: what note is this? It sounds easy. The obvious answers all fail, in instructive ways, and the method most tools use today, YIN, is a sequence of small fixes to those failures. This page goes through them in order, and ends with a tuner that listens to your microphone.

0 · What pitch is

Many sounds are periodic: the waveform repeats, over and over, with a period of $T_0$ seconds. A sung vowel, a bowed or plucked string and a blown pipe are all like this, at least for a while. A periodic sound’s spectrum is made of harmonics, sinusoids at whole multiples of one frequency: $f_0$ , $2f_0$ , $3f_0$ , and so on (that’s Fourier’s claim, from the Fourier page). That lowest frequency, $f_0 = 1/T_0$ , is the fundamental frequency, and it’s almost always the pitch we hear.

Here’s the surprise. The fundamental doesn’t have to be there.

Top: a tone’s waveform, with a line at the start of each period. Bottom: its harmonics. Use remove to take away the lowest harmonics, starting with the fundamental itself. The waveform changes shape, but it still repeats every $T_0$ , because every remaining harmonic still fits a whole number of cycles into that time. Listen: with the fundamental gone, the tone sounds thinner, but its pitch is the same, and matches the pure sine at $f_0$ . Your ear hears the period, and the spacing of the harmonics, rather than looking for a component at $f_0$ .

This is the missing fundamental, and it’s not a lab curiosity. Small loudspeakers can’t reproduce low notes at all, and an old telephone line removed everything below about 300 Hz, but voices on it don’t sound higher-pitched. So a pitch detector must find the repetition rate. Looking for the lowest component, or the strongest, isn’t the same thing.

1 · Naive method 1: count zero crossings

A sine wave crosses zero going upwards exactly once per cycle. So count the upward crossings in a stretch of signal, and divide by the time it took.

A 200 Hz tone: the orange dots are where it crosses zero going upwards. With a pure sine there’s one per period, and the estimate is spot on. Turn up the harmonics: up to about 50%, they just change the wave’s shape. At 60%, the wave gains extra wiggles that cross zero too, and the estimate jumps to 3 times the true pitch. Or add noise: by −30 dB, the noise jitters the signal back and forth across zero near each crossing, and the count goes wrong.

Zero crossings work for clean, nearly sinusoidal sounds, like whistling, and they’re very cheap. But any sound rich in harmonics, which means almost any instrument or voice, can fool them. Since the zero line isn’t special, the method throws away most of what the waveform says.

2 · Naive method 2: the biggest peak in the spectrum

A smarter idea: take a spectrum, with an FFT, and report the frequency of its biggest peak.

The spectrum of one window of a test tone. The dotted lines mark the true harmonics. Bright tone at 110 Hz: the fundamental is the strongest harmonic, and the biggest peak is right. Now try strong 2nd harmonic: an octave too high. Sung “ah” is worse: the vocal tract’s resonances (the formants, here around 730 and 1090 Hz) boost the harmonics near them, and the biggest peak is the 7th harmonic, at 770 Hz. Missing fundamental: there’s nothing at 110 Hz to find. Then choose a 12 ms window: it’s too short to separate harmonics 110 Hz apart, so they merge into blurred humps and even the bright tone fails.

Two problems, then. The fundamental is often not the strongest harmonic, and can even be missing. And the window-length trade-off from the Fourier page bites: to separate harmonics $f_0$ apart, the window must last several periods. Low notes need long windows, and a long window can’t follow a pitch that changes quickly.

A partial fix: the harmonic product spectrum

The first problem has a neat partial fix. Take the spectrum, and squeeze copies of it horizontally by 2, 3, 4 and 5 (that’s decimating it). In the copy squeezed by 2, the 2nd harmonic lands on $f_0$ ; in the copy squeezed by 3, the 3rd harmonic does, and so on. Multiply all the copies together, and the only place where they’re all large is $f_0$ . This is the harmonic product spectrum.

Tick it in the figure above. It fixes the strong 2nd harmonic tone. But it fails on a pure sine (there are no harmonics to line up), it gives the sung “ah” an octave too high, and it can’t find a missing fundamental, because one of the things being multiplied, the spectrum at $f_0$ , is nearly zero.

The lesson of the last two sections: pitch is about repetition. So look for repetition directly.

3 · Does it repeat? Autocorrelation

If a signal repeats every $T_0$ , then a copy of it delayed by $T_0$ lines up with the original. The Fourier page gave us a way to measure how well two signals match: multiply them sample by sample, and add up. Do that between the signal and a copy of itself delayed by $\tau$ samples, called the lag:

r(\tau) = \sum_{j} x[j]\,x[j + \tau]

That’s the autocorrelation (“auto” because the signal is correlated with itself). At lag 0, the signal matches itself perfectly. At lag $T_0$ , it matches again, and $r$ has a peak. So: find the highest peak (other than lag 0), and that lag is the period.

Top: one frame of signal, 23 ms long (grey), and the same signal lag later (blue). Bottom: the autocorrelation at every lag, scaled so the match at lag 0 is 1. Drag the lag and watch the match peak where the blue curve lines up with the grey one: at $T_0$ , and again at $2T_0$ , $3T_0$ … For the bright tone at 220 Hz, the highest peak is $T_0$ : correct. But look at strong 2nd harmonic: a copy delayed by only half a period also lines up fairly well, and the highest peak is at $T_0/2$ , an octave too high. Untick the copy slides off the end and it goes wrong the other way.

Notice how the peaks get lower as the lag grows. That’s not the signal changing. It’s because the delayed copy slides off the end of the frame, so fewer and fewer products are added up (the red region in the top panel). This taper gives short lags an unfair advantage, which is how $T_0/2$ wins for the strong 2nd harmonic.

The obvious fix is to compare against samples beyond the frame, so every lag adds up the same number of products. Untick the box to do that. Now the peaks at $T_0$ , $2T_0$ , $3T_0$ are all equally high, because a signal that repeats every $T_0$ also repeats every $2T_0$ . Which one comes out highest is down to tiny details, and for the bright tone at 220 Hz it’s $2T_0$ : an octave too low.

These are octave errors, and they’re the central problem of pitch detection. A detector that’s an octave out is still technically finding a lag where the signal repeats, so neither choice of taper solves it. It needs a better rule for which match to pick.

4 · YIN: fixing autocorrelation, one step at a time

YIN (de Cheveigné and Kawahara, 2002) is named after yin and yang, for the interplay between autocorrelation and cancellation. It makes four small changes to autocorrelation, each fixing a specific problem. Follow them in order in the figure below.

Step 1: measure the difference, not the product. Instead of multiplying the signal by its delayed copy, subtract them, square, and add up:

d(\tau) = \sum_{j=0}^{W-1} \left(x[j] - x[j + \tau]\right)^2

If the signal repeats every $\tau$ samples, the difference is zero. So we look for dips, not peaks. That’s more direct (“how different is the signal from itself, delayed?”) and it’s unaffected by the signal’s level drifting up or down. But on its own, it makes the same mistake as untapered autocorrelation: the dips at $T_0$ , $2T_0$ , $3T_0$ are all nearly zero, and the deepest one is a toss-up.

Step 2: normalise. Divide each $d(\tau)$ by the average of all the values of $d$ at lags up to $\tau$ :

d'(\tau) = \frac{d(\tau)}{\frac{1}{\tau}\sum_{j=1}^{\tau} d(j)}

Now the values mean something. Around 1 means “no better a match than average”, and near 0 means “almost exact repetition”, whatever the signal’s loudness. Very small lags, where the signal hasn’t had time to change much, no longer look like good matches. On its own this doesn’t stop octave errors, but it makes the next step possible.

Step 3: take the first good dip, not the deepest. Choose a threshold, such as 0.1, and take the first dip that goes below it. Since $T_0$ comes before $2T_0$ and $3T_0$ , this prefers the true period. A normalised scale is what makes one fixed threshold work for quiet and loud sounds alike.

Step 4: find the dip between samples. The period is rarely a whole number of samples. A 220 Hz period at 44.1 kHz is 200.45 samples. Rounding it to 200 gives 220.5 Hz, which is about 4 cents sharp. A cent is a hundredth of a semitone (the step between neighbouring piano keys), so there are 1200 in an octave. Most people can hear a difference of 5 to 10 cents between two notes, and tuners aim for better than that. The fix: fit a parabola through the lowest point of the dip and its two neighbours, and take the bottom of the parabola. This is parabolic interpolation.

YIN’s steps, one at a time. Step 1: the difference $d$ , with dips at $T_0$ , $2T_0$ , … (dashed lines). For the default bright tone at 220 Hz, the deepest dip is $2T_0$ , an octave too low. Step 2: the normalised $d'$ : values near 1 mean “an average match”. The deepest dip is still $2T_0$ . Step 3: the first dip below the threshold (green): now $T_0$ , but 4 cents sharp, because the true period falls between two samples, as the close-up shows. Step 4: the parabola through the three lowest points finds the bottom of the dip between samples: 0.02 cents. Try sung “ah” at 147.3 Hz, which also fails steps 1 and 2, and add noise: $d'$ at the dip rises above zero, but YIN still finds the pitch.

The value of $d'$ at the chosen dip is useful in its own right. It says how periodic the sound is: near 0 for a clean, steady note, and near 1 for noise. That’s a ready-made answer to the question “is there a pitch here at all?”, which section 6 relies on.

5 · The cepstrum

An older, quite different idea comes from looking at the spectrum on a log (decibel) scale. A periodic sound’s harmonics, evenly spaced $f_0$ apart, make the log spectrum ripple regularly, like a wave. And a regular ripple is something a Fourier transform can measure. So take the spectrum of the log spectrum. A ripple repeating every $f_0$ hertz shows up as a peak at $1/f_0 = T_0$ : the period, again.

This is the cepstrum (Bogert, Healy and Tukey, 1963). They enjoyed reversing the first letters of familiar words for its features. Spectrum became cepstrum, frequency became quefrency (measured in seconds), filtering became liftering, and harmonics became rahmonics.

Top: the log spectrum of a sung “ah” at 110 Hz: a ripple of harmonics 110 Hz apart, riding on the slower rise and fall of the formants. Bottom: its cepstrum, with a clear peak at 9.09 ms, the period of 110 Hz. Tick show the envelope to keep only the low quefrencies (shaded) and transform back: what’s left is the smooth envelope of the spectrum (orange), the formants, without the harmonics. A pure sine has no ripple, so its cepstrum has no real peak, and the answer wanders as you change the pitch.

Why take the log? A voice is a buzzing source (the vocal folds, which give the harmonics) shaped by a filter (the vocal tract, which gives the formants). In the spectrum, the filter multiplies the source, and a logarithm turns multiplication into addition. In the cepstrum, source and filter therefore sit side by side instead of being tangled together: the formants at low quefrencies, the pitch further out. That separation is the basis of the formant-preserving pitch shifters to come, and of voice coders like CELP.

6 · Tracking pitch over time

Real sounds change. To follow a melody or a voice, run the detector on short overlapping frames, say 23 ms long, starting every 10 ms, and plot the answers. That raises two new problems.

Is there a pitch at all? Silence, breath and consonants like “s” and “sh” aren’t periodic. A detector will always return some number, so frames must also be classified as voiced (periodic, with a pitch) or unvoiced. YIN’s $d'$ at the dip is a natural measure: below a threshold means voiced.

Stray errors. Even good detectors make the occasional octave error, for a frame or two, when the voice changes or misbehaves. A pitch can’t really jump an octave for 30 ms and come straight back, so smooth the track with a median filter: replace each value by the middle value of it and its neighbours, sorted. Unlike an average, a median ignores a few wild values completely.

Top: a spectrogram (frequency against time, with colour showing loudness) with YIN’s pitch for each voiced frame (blue dots). For the synthetic phrase, the true pitch is dashed. It has steady notes, a glide, vibrato, an “s” (no pitch), and a low “oo” with bursts of creaky voice, when alternate cycles come out louder and softer, as voices often do at the end of a phrase. Bottom: $d'$ at YIN’s chosen dip, the voicing threshold, and the frames counted as voiced (green). Every sung frame is found, but in the creaky bursts the period really does double, and 15 frames come out an octave low. Tick median filter (7 frames): all 15 are fixed. Or lower the threshold to 0.1, and those frames count as unvoiced instead. Then record yourself singing, humming or whistling, and listen to the detected pitch played back as a sine.

Modern trackers go further. pYIN (Mauch and Dixon, 2014) keeps several candidate pitches per frame, with probabilities, and then finds the most likely path through them over time, using the Viterbi algorithm, which will reappear on the error-correcting codes page.

A tuner

Run YIN on the microphone many times a second, round the result to the nearest note, and show how many cents sharp or flat it is: that’s a tuner.

Start and sing, hum, whistle or play a note. The note name and cents come from YIN on the last 1024 samples (about 21 ms at 48 kHz), many times a second, searching from 40 Hz (just below a bass guitar’s low E) up to 1100 Hz. The trace shows your pitch over the last 10 seconds, on a scale of notes. Nothing is recorded or sent anywhere: it all happens in your browser.

7 · Where it’s used

Tuners, and apps that score your singing, are YIN or close relatives with a display attached.
Pitch correction detects the sung pitch, works out the nearest note in the song’s scale, and shifts the voice to it. Shifting is the subject of a future page.
Voice coders like CELP send the pitch period as one of their few parameters, so the decoder can rebuild the buzz of the voice.
Query by humming turns your humming into a pitch contour, and searches for melodies of the same shape.

Everything here assumes one note at a time. Finding the notes of a chord, or of a band, is much harder: the harmonics of different notes overlap and interfere, and the period of the whole sound may be long or not exist. That’s still an active research area, where machine learning now does most of the work.

Summary

Pitch is the repetition rate of a sound, $f_0 = 1/T_0$ . The harmonics are spaced $f_0$ apart, and the fundamental itself can be missing.
Zero crossings are fooled by harmonics and noise. The biggest spectral peak is fooled by strong harmonics, formants and missing fundamentals.
Autocorrelation looks for repetition directly, but suffers octave errors in both directions.
YIN fixes it with four steps: difference, normalise, first dip below a threshold, and a parabola for sub-sample accuracy (cents matter: 1200 to the octave).
The cepstrum finds the period as a peak in the spectrum of the log spectrum, and separates formants from pitch.
Over time: decide voiced or unvoiced, and smooth stray errors with a median filter.