The Shannon limit

How fast can you send bits through a noisy channel without mistakes?

The earlier pages gave two answers, and both seem wrong. The digital modulation page packed more bits into each symbol by using more points in the constellation, and there seemed no end to it: 4 points, 16, 1024. The error-correcting codes page went the other way. It fixed mistakes by sending extra bits, and the surest way to fix every mistake, repeating each bit over and over, sends almost nothing new. It looked as if perfect reliability had to cost the whole channel.

In 1948, Claude Shannon showed that neither is right. Every channel has a capacity: a rate, in bits per second, below which you can make the mistakes as rare as you like, and above which you can’t. It doesn’t depend on cleverness or computing power. It depends only on the channel: how much bandwidth it has, and how strong the signal is compared with the noise.

This page works up to that result from the simplest pieces, mostly by counting, and then measures the capacity of the room you’re sitting in.

A few terms first:

SNR (signal-to-noise ratio) is the signal’s power divided by the noise’s power. It’s often given in decibels: 10 dB is a power ratio of 10, 20 dB is 100, 30 dB is 1000.
A code turns groups of data bits into longer codewords, which are what’s sent. Its rate is data bits divided by bits sent.
log₂ asks how many times you have to double 1 to reach a number: log₂ 8 = 3, because 1 → 2 → 4 → 8. It counts bits: with 3 bits you can pick one of 8 possibilities. It works for numbers that aren’t powers of two too (log₂ 10 is about 3.32), and for fractions, where it’s negative (log₂ ½ = −1).

1 · How many levels can noise allow?

The obvious way to send more bits is to use more levels. Send one of 2 voltages, and each symbol carries 1 bit. One of 4 carries 2 bits, one of 8 carries 3, and one of M carries log₂ M. Keep the average power the same, and the levels just get closer together.

A carrier has two knobs, I and Q, so it can do this twice at once: some levels on I, and the same levels on Q. That’s exactly how QAM builds its constellations: 4 levels on each axis make the 4 × 4 grid of 16-QAM, with 4 bits per symbol. 8 levels make 64-QAM, and 32 make the 1024-QAM of the latest Wi-Fi.

Noise is what stops this. Each point arrives blurred into a cloud, and the receiver decides which point was sent by which one the received symbol lands nearest. On each axis, that means deciding which level it’s nearest: each level’s noise is a bell curve, and once the bells overlap, some guesses go wrong.

Top left: 3,000 symbols as they arrive on the I/Q plane, at the chosen SNR, around the points that were sent (rings). Orange symbols landed nearer the wrong point. Top right: the I axis alone: a bell curve of noise around each of its levels, at the same average power. The dashed lines are where the receiver switches from one level to the next. Bottom: the chance of misreading one axis, against SNR, for 2 to 32 levels per axis. A symbol is read correctly only if both its axes are.

At an SNR of 20 dB, 16-QAM (4 levels per axis) is almost never confused: each axis is misread 5.8 × 10⁻⁶ of the time. 64-QAM (8 levels) misreads an axis 2.55% of the time, and 5.03% of its symbols are wrong (143 of the 3,000 shown). 256-QAM (16 levels) misreads an axis 26.1% of the time. At 30 dB, 256-QAM misreads an axis 5.7 × 10⁻⁴ of the time, and 1024-QAM (32 levels) 8.41%.

In 1928, Ralph Hartley counted how many levels a channel could support. Space the levels by about the noise’s size, and the number that fit is roughly the received signal’s size divided by the noise’s size: $\sqrt{1 + \text{SNR}}$ , in terms of powers. At 20 dB that’s 10 levels on each axis, and at 30 dB, 31.6: about the 32 of 1024-QAM.

It’s a good rule of thumb, but the figure shows its weakness. The errors don’t stop at some number of levels. They creep up gradually: at 30 dB, 1024-QAM’s 32 levels per axis are still misread 8.41% of the time, and 16.1% of its symbols are wrong. There’s no clean answer to “how many levels” while every symbol is decided on its own. Shannon’s step was to ask a different question: how many bits get through after an error-correcting code has fixed the mistakes? Answering it needs a way to measure information.

2 · Measuring information

Information is what you didn’t know already. If a friend tells you the sun rose this morning, you’ve learnt almost nothing; if they tell you it snowed in Singapore yesterday, you’ve learnt a lot. The rarer the news, the more it tells you.

Shannon measured this with the surprise of an outcome: $\log_2(1/p)$ bits, for an outcome with probability $p$ . A fair coin’s heads has probability ½, so its surprise is $\log_2 2$ = 1 bit: exactly what one bit can say. An outcome with probability ¼ is worth 2 bits, like learning two coin flips at once. Every time something becomes half as likely, learning it is worth one more bit.

A source’s entropy is its average surprise: each outcome’s surprise, weighted by how often it happens. For a coin that lands heads with probability $p$ ,

H(p) = p \log_2 \frac{1}{p} + (1-p) \log_2 \frac{1}{1-p} \ \text{bits per flip.}

Top: 60 flips of a coin with the chosen chance of heads. Bottom: the entropy, in bits per flip, against the chance of heads. The readouts give each outcome’s surprise and their average.

A fair coin has an entropy of 1.00 bit per flip. A coin that lands heads 90% of the time has 0.47: heads tells you only 0.15 bits, because you expected it, and tails tells you 3.32 bits, but tails is rare. At 99% heads, the entropy is 0.08 bits per flip.

So a stream of flips from the 90% coin carries only 0.47 bits of information per flip, even though each flip is written down as a whole bit. That’s why such a stream can be compressed: a later page, on lossless compression, picks this up. Here, entropy is about to measure something else: the uncertainty that a noisy channel adds.

3 · The binary channel’s limit

Go back to the error-correcting codes page’s simplest noisy channel: each bit, independently, arrives flipped with probability $p$ . Think of the flips as a coin that lands “flip” with probability $p$ . The receiver doesn’t know how that coin landed for each bit, and that uncertainty is worth $H(p)$ bits per bit sent. Shannon showed it comes straight off the top: the channel’s capacity is

C = 1 - H(p) \ \text{data bits per bit sent.}

The capacity is a rate, like a code’s rate: the most data bits that each bit sent can carry. With bits flipping 10% of the time, $H(0.1)$ = 0.469, so the capacity is 0.531. Send 1,000 bits through that channel, and at most 531 of them can be your message; the rest have to go on protection. Put the other way round, each data bit needs at least 1 ÷ 0.531 ≈ 1.88 bits sent.

Shannon’s claim is that codes exist with any rate below 0.531 whose mistakes are as rare as you like. Not codes that make mistakes rare at rate 0.1, or 0.01: at any rate up to 0.531. And at any rate above it, no code can make them rare.

Why should 10% of flips cost 47% of the channel, not 10%? Because the expensive part isn’t the flips themselves, but not knowing which bits flipped:

If someone told the receiver which bits had flipped, it would flip them back and lose nothing: a capacity of 1.
If 10% of bits were simply lost, arriving marked “unknown” (an erasure), you’d lose just those bits: a capacity of 0.9.
With flips, every bit is suspect. A 1,000-bit block has about 100 flips, somewhere. There are about 2⁴⁶⁴ ways to place 100 flips among 1,000 bits, so saying where they are takes about 464 bits of information: on average, 1,000 × $H(0.1)$ ≈ 469. The code’s extra bits have to supply that, and it comes out of the data.

So $H(p)$ isn’t the number of bits that flip: it’s the information it takes to find them.

Top: the share of data bits still wrong after decoding, against each code’s rate: sending each bit once (the dot at rate 1), repeating it 3 to 41 times, and two Hamming codes. The green line is the capacity; the green region left of it is what Shannon says is possible. Middle: the capacity, 1 − H(p), against the chance each bit flips. Bottom: the same limit turned over: the fewest bits that must be sent for each data bit to arrive reliably, 1 ÷ (1 − H(p)), on a log scale. The dashed line is what repetition needs to get fewer than one in a million data bits wrong. All the error rates are exact (calculated, not simulated).

With bits flipping 10% of the time, the capacity is 0.531. Repetition reaches fewer than one in a million data bits wrong only by repeating each bit 23 times, a rate of 0.043. Hamming (7,4), at rate 0.571 (just above the capacity), still gets 6.69% of data bits wrong. With bits flipping 1% of the time, the capacity is 0.919, and repeating 7 times (rate 0.143) gets below one in a million.

In bits sent per data bit: with 1% flips, Shannon’s limit is 1.09 bits, against repetition’s 7. With 10% flips, 1.88 against 23. With 30% flips, 8.42 against 131.

Every code here trades rate for reliability, and to make errors vanish, the rate has to vanish too. The green region seems out of reach. Yet Shannon proved it isn’t, and the proof shows how.

4 · Why it works: long blocks

The trick is to code long blocks of bits at once. Flips are random, but over many bits, randomness becomes surprisingly predictable.

Toss a coin 10 times and you might get 2 heads or 8. Toss it 10,000 times, and 99 times in 100 you’ll get between 4,871 and 5,129 heads: within 2.6% of half. The same happens with flips. In a block of n bits, each flipping with probability p, the number that flip crowds ever more tightly around $np$ , as a share of the block.

That’s what makes counting possible. Say the block is n bits long and about $np$ of them flip. The number of different ways to choose which ones flip is huge, but it can be counted, and it comes to about $2^{nH(p)}$ : entropy appears again. Each codeword has to have room around it for all of those ways of being corrupted, without overlapping its neighbours’ room, or the decoder can’t tell which codeword was sent. There are only $2^n$ possible blocks in total. So the number of codewords that fit is at most

\frac{2^n}{2^{nH(p)}} = 2^{n(1 - H(p))},

and a code with that many codewords carries $n(1 - H(p))$ data bits in every n bits sent: a rate of $1 - H(p)$ . That’s the capacity, and it’s where it comes from.

Top: the chance of each number of flips in a block, as a share of the block. The green band holds 99% of blocks. Bottom: the room each codeword needs, per bit of the block: log₂ of the number of error patterns in the green band, divided by n. The dashed line is H(p).

With 100-bit blocks and 10% flips, 99% of blocks have 3 to 18 flips: between 3% and 18% of the block. There are 2⁶⁵ different patterns with that many flips, so each codeword needs 0.651 bits of room per bit, and the rate can be at most 0.349. At 1,000 bits, 99% of blocks have 76 to 125 flips, each codeword needs 0.539, and the rate can be at most 0.461. At 100,000 bits, the band is 9,756 to 10,245 flips (9.8% to 10.2%), each codeword needs 0.477, and the rate can reach 0.523, closing in on the capacity, 0.531.

In other words: a code has to budget its extra bits for the flips it will meet, and it has to budget for bad luck as well as the average. What it budgets for isn’t the number of flips but the number of ways they could fall, because its extra bits have to say where they are (as in section 3).

In a 100-bit block, 99% of blocks have anywhere from 3 to 18 flips. Planning for up to 18, almost double the average of 10, costs 0.651 bits of room for every bit sent, well above $H(0.1)$ = 0.469.
In a 100,000-bit block, 99% have 9,756 to 10,245 flips. Planning for the bad case now costs barely more than planning for the average: 0.477 bits of room per bit, close to $H(0.1)$ .

So longer blocks win twice. The room each codeword needs falls towards $H(p)$ , so the rate can rise towards the capacity. And the chance of a block having more flips than the code planned for keeps falling, so errors become rarer. You don’t trade one for the other: below the capacity, longer blocks are both closer to the limit and more reliable.

This counting shows the most that can fit. Shannon also showed that the most can actually be reached, and his argument for that is the strangest part of all.

5 · Random codes really do work

How do you find a code that packs codewords that tightly? Shannon’s answer was surprising: don’t try. To see what that means, it helps to look at codes in a slightly different way.

Every block code is, underneath, a lookup table: one codeword for each possible message. Sender and receiver both keep a copy.

Repeating 3 times is a table with 2 rows: the message 0 is sent as 000, and 1 as 111.
Hamming (7,4) is a table with 16 rows, one for each 4-bit message, each a 7-bit codeword. Of the 128 possible 7-bit patterns, only those 16 are ever sent.

Those codes fill in their tables by careful rules: repeat the bit, or add parity bits over overlapping circles. A random code fills in its table by tossing a coin for every bit of every codeword. Here’s a real one, with 2-bit messages and 8-bit codewords, made by the same code as the figure below:

Message	Codeword
`00`	`01110111`
`01`	`11010001`
`10`	`01011011`
`11`	`11111100`

It’s still an error-correcting code in exactly the ECC page’s sense. But there’s no group of check bits you can point to. The redundancy is in how few of the possible patterns are used: 8 bits can form 256 patterns, and this code uses 4 of them. Anything else that arrives means something went wrong, and the decoder picks whichever codeword is closest: the one differing in the fewest bits. That works if the codewords are far apart, as with the “bubbles” in the ECC page’s section on distance. Careful codes space their codewords out by design; a random code relies on luck. This little table isn’t very lucky: its codewords are only 3 to 5 bits apart, so two flips can already make a codeword look like another.

With long codewords, the luck becomes very reliable. The figure’s longest code has 16-bit messages (65,536 of them) and 64-bit codewords, a rate of ¼. 64 bits can form about 18 billion billion patterns, and the code uses only 65,536, so they land far apart. On average, two of its codewords differ in 32.0 of their 64 bits, and each codeword’s nearest neighbour is typically 15 bits away (12 to 17, in 200 that were checked). With 10% of bits flipping, a block gets about 6 flips on average: not nearly enough to carry it most of the way to another codeword.

Some coin tosses are still unlucky, and put two codewords close together. Shannon didn’t try to avoid that. Instead, he worked out the error rate averaged over every table the coins could produce, and showed that the average falls towards zero as the codewords get longer, at any rate below the capacity. If the average table is that good, then at least one particular table must be at least as good. So good codes must exist, even though the proof never says which table is one.

That can be tried directly. The figure builds random codes with blocks of 8 to 64 bits. Each block is sent through the binary channel, and decoded the most direct way: compare what arrived with every codeword in the table, and pick the closest.

The share of blocks decoded to the wrong codeword, against block length, for six flip probabilities. Solid lines: the rate is below the capacity. Dashed: above it. Each point sends 3,000 blocks; a point on the bottom edge means none of them went wrong. Measured as the figure loads.

At rate ¼, the capacity equals the rate when bits flip 21.5% of the time. Below that, longer blocks help: with 10% flips, 6.3% of 8-bit blocks go wrong, and 0.13% of 64-bit ones; with 5% flips, none of the 3,000 64-bit blocks do. Above it, longer blocks make things worse: with 25% flips, 29% of 8-bit blocks go wrong, and 56% of 64-bit ones; with 30% flips, 40% and 86%. With 20% flips, the capacity (0.28) is only just above the rate, and the error rate hardly moves.

That’s the theorem, happening. Below the capacity, every doubling of the block length cuts the errors; above it, nothing helps, and longer blocks only make the failure surer.

There’s a catch, and it explains why it took another 45 years to come close to the limit in practice. Decoding a random code means comparing with every codeword, and at rate ½ with 1,000-bit blocks there are $2^{500}$ of them, far more than there are atoms in the universe. Shannon proved good codes exist, but not how to decode them. Codes that can be decoded quickly have to have structure, and finding structure that’s nearly as good as randomness was the problem the next half-century worked on.

6 · From bit flips to noise: C = B·log₂(1 + SNR)

Real channels don’t flip bits: they add noise to voltages. The same counting works there, with one surprising fact about noise in many dimensions.

A block of n samples can be thought of as a single point in n dimensions, one coordinate per sample. Two samples make a point on a flat plane, three a point in space, and a thousand a point in a space we can’t picture, though the maths works just the same. The block’s length is the distance from the origin to that point: the square root of the sum of the squared samples, as in Pythagoras.

Now add noise to every sample. In one dimension, noise is unpredictable: it might be tiny or three times its typical size. In many dimensions, something strange happens. The noise’s length becomes almost exactly predictable.

Top: the lengths of 3,000 blocks of Gaussian noise (each sample of power 1), divided by √n so that every block length has the same typical value, 1. Bottom: where the middle 99% of those lengths fall, against the number of samples in the block.

With 1 sample, 99% of lengths fall anywhere between 0.008 and 2.741. With 100 samples, between 0.822 and 1.177 (within 18%). With 1,000 samples, between 0.944 and 1.061: within 6%.

So in many dimensions, noise doesn’t fill a ball around the point that was sent. It lands on a thin shell: almost always at the same distance, just in an unpredictable direction. That turns the counting into packing:

The received block (signal plus noise) has a typical length of $\sqrt{n(P + N)}$ , where $P$ is the signal’s power and $N$ the noise’s. Every received block lies in a ball of that radius.
Around each codeword, the noise puts the received block on a shell of radius $\sqrt{nN}$ . Codewords must be far enough apart for those noise balls not to overlap.
So the number of codewords is at most the number of small balls that fit inside the big one.

A sketch in two dimensions: every received block lies in the big ball, of radius √(n(P + N)). Each codeword (a dot) needs a ball of radius √(nN) to itself (dashed), for the noise. In many dimensions the small balls are thin shells, and the count is exact enough to give the capacity.

The volume of a ball in n dimensions grows as its radius to the power n. So the number of small balls that fit is about

\left(\frac{\sqrt{n(P+N)}}{\sqrt{nN}}\right)^{n} = \left(1 + \frac{P}{N}\right)^{n/2} = (1 + \text{SNR})^{n/2}.

That many codewords carry $\log_2$ of it in bits: $\tfrac{n}{2}\log_2(1 + \text{SNR})$ bits in n samples, or ½ log₂(1 + SNR) bits per sample. Notice the square root of $1 + \text{SNR}$ hiding in it: Hartley’s count of levels from section 1. Shannon’s formula is what Hartley’s becomes once the errors are counted properly.

The last step comes from the sampling theorem: a signal with bandwidth $B$ hertz is described exactly by $2B$ samples per second, no more and no less. So a channel of bandwidth $B$ carries

C = 2B \times \tfrac12 \log_2(1 + \text{SNR}) = B \log_2(1 + \text{SNR}) \ \text{bits per second.}

That’s the Shannon–Hartley theorem. A telephone line, about 3 kHz wide with an SNR of 30 dB (1000), can carry at most $3000 \times \log_2 1001$ ≈ 29,900 bits per second, in theory. Analogue dial-up modems topped out at 33.6 kbit/s, on lines a little wider and cleaner than that.

7 · Bandwidth against power

The formula has two ways to go faster: more bandwidth, or a stronger signal. They aren’t equal.

Doubling the power adds a fixed amount: at high SNR, about 1 bit per second for each hertz of bandwidth. Power buys bits only logarithmically. Bandwidth looks better: double it, and the capacity nearly doubles. But spreading a signal over more bandwidth lets in more noise, because noise has power in every hertz. With the signal’s power fixed, the SNR falls as the bandwidth grows, and the capacity levels off at a ceiling.

The noise’s power per hertz is written $N_0$ , so the noise in bandwidth $B$ is $N_0 B$ , and the SNR is $P/(N_0 B)$ .

Top: the capacity at a fixed signal power, against bandwidth. The dashed line is the ceiling, with unlimited bandwidth. Bottom: the same in terms of efficiency: bits per second per hertz (C/B), against the energy each bit gets, $E_b/N_0$ , at capacity. The red line at −1.59 dB is the floor: no code can work with less energy per bit. The dot is the chosen bandwidth.

With the signal’s power 40 dB above the noise’s power per hertz (a ratio of 10,000 Hz): in 100 Hz, the SNR is 20.0 dB and the capacity 666 bit/s. In 1 kHz, the SNR is 10.0 dB and the capacity 3.46 kbit/s. In 10 kHz, 0.0 dB and 10 kbit/s. In 100 kHz, −10.0 dB and 13.8 kbit/s. However wide the band, the capacity can’t exceed 14.4 kbit/s.

The ceiling isn’t measured: it falls straight out of the formula. Write the SNR out in full, and

C = B \log_2\!\left(1 + \frac{P}{N_0 B}\right).

As $B$ grows, the fraction inside the logarithm gets tiny, and two facts finish the job:

$\log_2 x = \ln x \div \ln 2$ , where $\ln$ is the natural logarithm (base $e$ ≈ 2.718). That’s just the rule for changing a logarithm’s base.
For small $x$ , $\ln(1 + x)$ is almost exactly $x$ : $\ln 1.01$ = 0.00995, and $\ln 1.001$ = 0.0009995.

Put together, $C \approx B \times \frac{P}{N_0 B} \div \ln 2$ , and the $B$ s cancel. However wide the band, the capacity creeps up towards

C_\infty = \frac{P}{N_0 \ln 2}

and never passes it. $\ln 2$ is about 0.693, so in the figure, $P/N_0$ = 10,000 Hz gives a ceiling of 10,000 ÷ 0.693 ≈ 14.4 kbit/s. The $\ln 2$ isn’t a property of the channel, though: it’s there because capacity is counted in bits. Counted in “nats”, the natural logarithm’s unit, the ceiling would be simply $P/N_0$ ; one bit is $\ln 2$ nats.

Put another way: the energy each bit gets is the power divided by the bit rate, $E_b = P/C$ . At the ceiling, that’s $E_b/N_0 = \ln 2$ , or −1.59 dB. This is the most fundamental number in communications: no system, however clever and however much bandwidth it uses, can send a bit reliably with less energy than 0.693 times the noise’s power per hertz.

At the other end, squeezing many bits into each hertz costs energy fast. To carry 2 bits per second per hertz takes at least 1.76 dB; 4 takes 5.74 dB; 6 takes 10.21 dB, and each extra bit per hertz costs more than 2 dB more. That’s the trade a deep-space probe and a cable modem make in opposite directions. The probe has a weak signal and plenty of bandwidth, so it works near −1.59 dB. The cable modem has a strong signal in a crowded cable, so it packs in many bits per hertz.

8 · How close real codes come

The bottom half of the last figure is a map. Every way of sending bits is a point on it: how many bits per second per hertz it carries, and how much energy per bit it needs. Shannon’s limit is a line across the map, and everything must sit to the right of it. How far right is the gap: the extra energy a real scheme needs compared with the best possible.

Here are the schemes from the earlier pages, each needing $E_b/N_0$ enough for 10⁻⁵ of its data bits to be wrong (a common benchmark). Without a code, QPSK, 8-PSK and 16-QAM carry 2, 3 and 4 bits per second per hertz (QPSK’s I and Q each carry one bit per hertz). With a code, QPSK’s bits per hertz are multiplied by the code’s rate: rate ½ gives 1.

Each scheme’s $E_b/N_0$ for 10⁻⁵ of data bits wrong, against its bits per second per hertz.

Calculated: QPSK, 8-PSK and 16-QAM without a code, from their formulas, and the two Hamming codes exactly, with hard decisions, as on the error-correcting codes page.
Measured: the convolutional codes, K = 3 and K = 7, with hard and soft decisions: 10 million bits at each of several Eb/N0 values, interpolated to 10⁻⁵.
Published: a turbo code and an LDPC code, as reported by their inventors.

The dashed line is the limit if every dimension may carry only +1 or −1, as BPSK and QPSK do.

QPSK without a code needs 9.59 dB, 7.83 dB more than the limit for 2 bits per hertz. 16-QAM’s gap is similar, 7.69 dB. Hamming (7,4) with hard decisions needs 9.17 dB: it saves less than half a decibel over no code, but carries only 1.14 bits per hertz, so it’s 8.93 dB from its limit. The K = 7 convolutional code, at 1 bit per hertz, needs 6.47 dB with hard decisions and 4.21 dB with soft ones, against a limit of 0.00 dB (0.19 dB with binary inputs). The 1993 turbo code needs 0.70 dB, and the 2001 LDPC code 0.23 dB, within 0.04 dB of the binary-input limit.

For forty years after Shannon, the best practical codes sat several decibels from the limit, like the K = 7 code here. Each decibel was worth a lot. NASA’s deep-space links gained a couple more by putting a Reed–Solomon code around the convolutional code, and every decibel saved meant a smaller antenna, or a probe heard from further away.

Then in 1993, Claude Berrou, Alain Glavieux and Punya Thitimajshima announced turbo codes, within 0.7 dB of the limit. Many experts didn’t believe the results until they reproduced them. Within a few years, David MacKay and others rediscovered low-density parity-check (LDPC) codes, which Robert Gallager had invented in 1962 but which had been too heavy to decode on the computers of his day. By 2001, an LDPC code with 10-million-bit blocks came within 0.04 dB of the limit for its kind of input. Both decode by passing messages back and forth between small parts of the code, and both are the subject of the next page. Wi-Fi, 4G, 5G, digital television and deep-space links now use them, along with polar codes (Erdal Arıkan, 2009), the first codes proved to reach capacity with a practical decoder.

9 · The capacity of your room

Shannon’s formula needs only a bandwidth and an SNR, and a room has both. Your loudspeaker and microphone make a channel: what’s the most it could carry?

A room’s SNR isn’t the same at every frequency. The loudspeaker is weaker at the bottom, echoes cancel some frequencies and reinforce others, and background noise is usually strongest at low frequencies. So the figure measures the SNR at each frequency separately, then applies Shannon’s formula frequency by frequency, the way OFDM splits a channel into many narrow carriers. Each slice of bandwidth $\Delta f$ with its own SNR carries $\Delta f \log_2(1 + \text{SNR})$ , and the capacity is their sum.

To measure SNR at each frequency, the figure plays a hiss made of every frequency from 100 Hz to 20 kHz (in steps of 11.7 Hz), which repeats exactly every 85 ms, 18 times over. Every repeat sent is identical, so every repeat received should be identical too, apart from the noise:

The average of the received repeats is the signal, at each frequency.
How much the repeats differ from their average is the noise.

The first 4 repeats are left out, while the room’s echoes build up. The loudspeaker and microphone may run on slightly different clocks (one may sit on a USB interface, say), which makes each repeat arrive a little earlier or later than the last. The figure measures that drift and corrects it, as the modem’s receivers did.

Top: the SNR at each frequency. Bottom: water-filling (see below): each frequency’s floor, the noise divided by how strongly the room passes that frequency, and the power poured over it. Before you measure anything, the figure uses the made-up echoey room from the digital modulation page, with noise 30 dB below the sound. Turn the loudspeaker down scales every SNR down, as if you’d played the hiss more quietly.

In the made-up room, the typical SNR is 32.6 dB, and the capacity from 100 Hz to 20 kHz is 211 kbit/s. In the OFDM modem’s band, 1.5 to 5 kHz, it’s 37.2 kbit/s, while the modem actually sends 2.46 kbit/s: 6.6% of the capacity. Turn the loudspeaker down by 30 dB: the capacity falls to 30.1 kbit/s, and water-filling raises it by 7.0%, leaving 14% of the frequencies silent. At 40 dB down, it’s 5.62 kbit/s, and water-filling adds 42.8%.

Use loudspeakers, not headphones, and turn off any processing on the microphone (see the modem page’s note). Nothing you record leaves your computer.

Why does the modem use only a few percent of the capacity? Partly by choice: it uses a narrow band, 1.5 to 5 kHz, and only QPSK, 2 bits per carrier, where the SNR could support far more. And partly by necessity: it sends everything twice, it spends time on a chirp, training symbols and cyclic prefixes, and it has to survive much worse rooms than this one. A modem built to approach capacity would measure the channel first, as this figure does, and then load each carrier with as many bits as its SNR allows. That’s exactly what DSL broadband does on telephone wires.

Water-filling is the other half of that idea. If some frequencies are noisier than others, should the transmitter still spread its power evenly? Shannon’s answer is no. Picture each frequency’s floor (its noise, divided by how well the room passes it) as the uneven bottom of a tank, and pour the power in like water. It settles to one level: deep where the floor is low, shallow where it’s high, and absent where the floor sticks out above the surface. That’s the best way to share a fixed amount of power. At high SNR it hardly matters, as the figure shows, but when the signal is weak, abandoning the worst frequencies and putting their power into the best ones makes a real difference.

10 · What the limit doesn’t promise

Shannon’s limit is exact, but it rests on assumptions, and it leaves things out.

Delay. Approaching capacity needs long blocks, as section 5 showed, and a receiver can’t decode a block until all of it has arrived. A phone call can’t wait seconds for each block. Real systems choose how close to the limit to get by choosing how long they can wait.
Effort. The proof uses random codes, which are hopeless to decode. Practical codes trade a little of the gap for decoders that work in time.
The channel. The formula assumes Gaussian noise, and a channel that stays the same and whose response the receiver knows. Real channels fade, drift and suffer interference. Their capacities can still be worked out, but they’re different numbers.
What the bits mean. Capacity counts bits, not meaning. A message with less entropy than its length, such as text, can be compressed first, and then sent in fewer bits. Shannon’s same paper founded that side of the subject too.

Summary

Using more levels sends more bits per symbol, but noise makes the levels blur, and the errors creep up gradually: there’s no clean limit while each symbol is decided on its own.
Information is measured in bits of surprise. A source’s entropy is its average surprise: 1 bit for a fair coin, 0.47 bits for a 90% coin.
A channel that flips bits with probability p has a capacity of 1 − H(p) data bits per bit sent. Below it, codes can make mistakes as rare as you like; above it, nothing can.
It works because in long blocks, the number of flips becomes predictable. Counting how many codewords fit gives the capacity, and even random codes reach it as blocks grow.
For noise, the same counting packs balls in many dimensions, where noise lands on a thin shell. The result is C = B·log₂(1 + SNR) bits per second.
With unlimited bandwidth, every bit needs at least −1.59 dB of energy against the noise.
Decades of codes closed the gap to the limit from about 8 dB (no code) to 4 dB (convolutional codes with soft decisions) to a fraction of a decibel (turbo and LDPC codes).
A room has a capacity too: measure the SNR at each frequency, and add up Shannon’s formula across them. Water-filling puts the power where the channel is best.