The model lab · Part 11
Why do neural networks need more than one hidden layer?
Meet Doodle-918. This is a model that can see loops and curves, and can interpret these into digits. You can draw a digit and watch how the different shapes become an answer.
Doodle-918
Model card- Type
- Three-layer feed-forward neural network with a skip connection
- Parameters
- 918: 640 pixel-to-primitive weights + 10 primitive biases + 80 primitive-to-shape weights + 8 shape biases + 80 shape-to-digit weights + 100 skip weights
- Layers
- 3 (64 pixels → 10 stroke primitives → 8 shape detectors → 10 digits)
- Input
- 64 numbers: the 8×8 grid, 1 where you drew ink and 0 where you didn't
- Output
- 10 probabilities, one per digit 0-9, summing to 100%
Doodle-525's job with one more layer in the middle. It reads every canonical digit at over 99% confidence where the two-layer model manages 94%, and it stays right when a stroke is smudged. That is the payoff for features that compose.
Draw a digit: 0-9
Click a pixel, or drag across the grid to draw a stroke at once. On a phone, draw with your finger. A three-layer network, 64 inputs, 10 stroke primitives, 8 shape detectors, 10 outputs, will read it.
Layer 1: 10 stroke primitives
The same seven-segment skeleton as the two-layer model, but each bar is cut into a left and a right half. Finer parts are what let the next layer tell the top of an 8 from the top of a 7. Colored cells are the pixels each neuron watches; everything else is weight zero.
Layer 2: 8 shape detectors
This is the layer Doodle-525 doesn't have. Each detector wants a specific combination of primitives, and refuses to fire if a stroke that would break the shape is present. A top loop needs the cap and both upper sides; an open curve needs the same cap but insists one side stay missing. That's how the network learns the difference between a circle and a corner.
A circle closed around the upper half: top bar plus both upper sides. 0, 8 and 9 have one.
The same circle in the lower half: both lower sides plus the bottom bar. 0, 6 and 8 have one; 9 famously does not.
A full bar across the middle, pinching the shape in two. Everything except 0, 1 and 7 has one.
A C-shaped arc down the right: top bar, upper-right line and waist, with the upper-left side missing. The top of a 2 or a 3.
The mirror image: an arc down the left with the upper-right side missing. The top of a 5 or a 6.
One unbroken vertical down the right edge with nothing on the left. The backbone of 1 and 7.
A full bar closing the shape off along the bottom. Seven digits rest on one; 1, 4 and 7 do not.
A full bar across the top. Present in every digit except 1 and 4, which start partway down.
Layer 3: combining shapes into digits
The output layer's 80 shape weights, one row per digit. Green means "this digit shows that shape", red means "that shape is evidence against it". Note that no two rows are identical: eight shapes are enough to give every digit its own fingerprint, which is why this layer never has to look at raw pixels.
| Digit | Top loop | Bottom loop | Waist | Open curve, left gap | Open curve, right gap | Straight right spine | Flat base | Flat cap |
|---|---|---|---|---|---|---|---|---|
| 0 | + | + | − | − | − | − | + | + |
| 1 | − | − | − | − | − | + | − | − |
| 2 | − | − | + | + | − | − | + | + |
| 3 | − | − | + | + | − | + | + | + |
| 4 | − | − | + | − | − | − | − | − |
| 5 | − | − | + | − | + | − | + | + |
| 6 | − | + | + | − | + | − | + | + |
| 7 | − | − | − | − | − | + | − | + |
| 8 | + | + | + | − | − | − | + | + |
| 9 | + | − | + | − | − | − | + | + |
Watch your drawing run through all three layers
Four columns, three hops: pixels wake up primitives, primitives assemble into shapes, shapes vote on digits.
Three hops instead of two. Your lit pixels wake up the stroke primitives they belong to; the primitives that fired combine into shapes, where green edges are strokes a shape needs and red edges are strokes that would break it; then the shapes vote on all ten digits. The middle column is the layer Doodle-525 doesn't have, and it's the reason this model can tell a loop from a corner instead of just counting strokes.
Prediction
Why the third layer earns its keep
Doodle-525 counts strokes. That works, but it means a digit is only a checklist, and the model has no idea that a 0 is round. Doodle-918 splits the job into two smaller questions, and it gets a vocabulary of shapes out of the split:
- Layer 1 finds parts. This layer holds ten primitives, and each one is a bar half or a short line. These parts are smaller than the ones the two-layer model uses, and that is deliberate, because you cannot compose a loop out of pieces that are already loop-sized.
- Layer 2 finds shapes. This layer holds eight detectors, and each one wants a specific combination of primitives while rejecting specific others. This is where "circular top", "circular bottom" and "straight spine" become ideas the network can represent on their own terms.
- Layer 3 names the digit. Every digit has a distinct signature over those eight shapes, so the final layer is a short vote. Each shape a digit shows adds a point, and each shape it lacks subtracts one.
You can load an 8, erase its lower-left line, and watch the bottom loop switch off. The model reads a 9, because a 9 is an 8 with the bottom loop opened up. If you erase the middle bar instead, the waist goes quiet and the model reads a 0. The two-layer model reaches the same answers by tallying strokes. This model gets there by noticing that the loop is gone, and that is the reason it stays confident when your drawing is smudged.
That is the whole argument for depth, and it is the same argument at every scale. A real vision model's early layers find edges, its middle layers find textures and parts, and its late layers find objects. Training discovers all of those middle features on its own. Here they were chosen by hand so that you can read them, all 918 parameters of them.
Meet Doodle-918
Doodle-918 reads the same 8×8 grid as Doodle-525 and answers the same question. The difference is entirely in the middle. Doodle-525 goes from pixels to strokes to digits. This model goes from pixels to stroke primitives to shapes to digits, and that one extra stop costs 393 and lifts its worst digit from 94% confidence to over 99%.
This is the third and last model in our vision lab, and the three of them together tell the whole story of depth in about a thousand parameters. Doodle-64 maps pixels straight to an answer. Doodle-525 puts one layer of parts in between. Doodle-918 puts two layers in between, and the second one is where the model stops counting and starts seeing.
What the two-layer model can't say
Doodle-525 works, and its stroke detectors are useful features. But its output layer only tallies which of the seven strokes are lit. To that model, a 0 is a checklist that says six particular strokes are present and one is absent. It has no way to represent the fact that those six strokes form a closed loop, because "closed loop" is not a stroke.
That matters the moment a drawing gets messy. A checklist degrades one item at a time. When you smudge a stroke, the tally drifts toward whatever digit has the next-most-similar checklist. A model that knows the loop is still closed does not care that the line making it is thin.
Layer 1: smaller parts than before
The first change is counterintuitive. Doodle-918's first layer finds smaller features than Doodle-525's. Each horizontal bar is cut into a left half and a right half, and that gives the model 10 primitives instead of 7.
The reason is composition. You cannot build a loop out of pieces that are already loop-sized. If a "top bar" is atomic, the network can never notice that the left end of it joins the upper-left line while the right end joins the upper-right one. Finer parts give the next layer something to combine. Real convolutional networks make the same trade. Their first layer finds tiny oriented edges, which are the least interesting features in the whole model, because everything else is assembled from them.
Layer 2: where shapes live
The second layer is the new one, and it holds eight shape detectors: a top loop, a bottom loop, a waist, two open curves, a straight right spine, a flat base and a flat cap. Each detector is an AND over primitives. It wants all of a specific set, and some detectors also refuse to fire if a particular primitive is present.
That veto is what separates a circle from a corner. The "top loop" detector and the "open curve, left gap" detector want the same cap and the same right-hand side. The only difference is that the loop requires the upper-left line and the curve requires its absence. One weight flips from positive to negative, and the model can tell a closed shape from an open one. This is exactly the pattern the issue behind this page asked for: circular tops in 8, 9 and 0, circular bottoms in 8, 6 and 0, open curves in 2, 5 and 3, straight lines in 1, 4 and 7.
These shapes are also shared. A top loop is reused by three digits, and a flat base is reused by seven. Reuse is the whole economic argument for a hidden layer. Eight shapes are cheaper than ten separate digit templates, and they generalize better, because a smudged loop is still a loop.
Layer 3: a very short argument
By the time the signal reaches the output layer, the hard work is done. Every digit has its own distinct fingerprint across those eight shapes, so the last layer is a simple vote. Each shape a digit shows adds a point, and each shape it lacks subtracts one. Eighty weights are enough to name all ten digits.
Those weights come from the model itself. It runs its own first two layers over the reference drawing of each digit and reads off which shapes light up. That is the closest a hand-built model gets to training. You show it the examples, and the answer falls out.
There is one honest wrinkle. A 0 is an 8 without the waist, and a 2 is a 3 leaning the other way, so a few pairs sit a single shape apart. Layer 3 therefore keeps a quiet direct line back to layer 1, and it consults the raw primitives at about a third of the volume it gives the shapes. Real vision models do the same thing, and they call it a skip connection. When an abstraction throws away something a later layer still needs, you let that layer look back.
What depth actually buys
You can load an 8, rub out its lower-left line, and watch the bottom loop switch off while the model reads a 9. That makes sense, because a 9 is an 8 with the bottom loop opened up. If you rub out the middle bar instead, the waist goes quiet and the model reads a 0. Doodle-525 reaches the same answers by tallying strokes. Doodle-918 gets there by noticing that a loop is gone, and that is why it stays confident when the drawing is imperfect and the two-layer model wavers.
That difference is the entire reason deep learning is deep. Every layer you add lets the model describe its input in a vocabulary the previous layer could not reach: first pixels, then edges, then shapes, then objects. A modern image model stacks dozens of these layers, and discovers all the middle vocabularies on its own. Ours were chosen by hand here so that you could read them.
If you have not yet, you can run the two-layer model on the same digits and compare. Both models get the right answer. The interesting comparison is how sure each of them is, and what it takes to confuse them.
Try it yourself
You can draw a digit, or load one with the example buttons, and then start vandalizing it. Erase half a stroke and watch a primitive dim without any shape switching off. Erase the whole stroke and watch a shape collapse, taking a digit's score down with it. Add a stroke that a shape forbids and watch that shape veto itself even though everything it needs is drawn.
Then press play on the network animation and watch all three hops in order. The pixels wake the primitives they belong to, the primitives assemble into shapes, and the shapes vote on the digits. The whole model is four columns, three layers, and 918 numbers, with no magic anywhere in it.
Put it to use
That is the end of the learning path
You have reached the last topic. You can go back to the full list of topics to revisit anything, or you can put what you have learned to use on the real models above.