What is gradient descent?

Nobody chooses the weights

Every model in this lab so far arrived with its weights already set. Doodle-64 got its 64 numbers from a rule we wrote by hand: positive where a 3 has ink, negative where an E does. That was honest teaching but dishonest engineering, because no real model is built that way. Nobody at OpenAI sat down and picked a value for weight number 4,000,000,001.

Real weights are found by training, and gradient descent is the basic procedure behind the models on this site. Doodle-64 makes that training loop small enough to inspect.

Start with a shrug

This training run starts by filling every weight with a small random number. The model has no information yet, so the values stay close to zero and its first predictions stay cautious. It begins with an honest "I have no idea" instead of a confident guess.

Doodle-64 could also learn from zero weights. Its pixels appear in different training examples, so they receive different updates even when they start equal. We use a random start because real training runs often begin that way, and because changing the seed lets you compare several downhill paths. A seed also makes any one path exactly reproducible.

One number for "how wrong are we?"

Before you can improve, you need a score. That score is the loss. Show the model a training drawing, let it produce a probability that the drawing is a 3, and compare that against the truth. A confident right answer scores near zero. A confident wrong answer scores enormously. Average that over every training example and you have one number summarizing the model's badness.

Now the whole of training has a shape: change the weights so the loss goes down. Nothing else. Every trick in modern machine learning is a variation on that one sentence.

The slope tells you which way is downhill

Picture the loss as a landscape. Each of the 64 weights is a direction you can walk, and your altitude is how wrong you are. You want the valley. The catch: you cannot see the landscape. You can only feel the ground under your feet.

That is enough. Calculus hands you the slope in each direction: if I nudge this one weight up, does the loss rise or fall, and how steeply? The collection of all 64 slopes is called the gradient. Take a small step against it, downhill in every direction at once, and your loss is a little lower than before. Do it again. That is gradient descent, all of it.

For our classifier the slope works out to something almost suspiciously simple: for each training image, take the model's prediction, subtract the true answer, and multiply by the pixel. Predicted 0.9 when the answer was 1? Barely wrong, tiny nudge. Predicted 0.9 when the answer was 0? Very wrong, big nudge, and every pixel that was lit takes the blame.

The learning rate is the size of your stride

The gradient says which way to go. It does not say how far. That is the learning rate, and it is the one dial that most reliably ruins a training run.

Set it too small and training crawls: correct direction, thousands of tiny steps, a bill to match. Set it too large and you leap clean over the valley and land higher up the far slope, then overcorrect back, bouncing wider and wider until the loss goes to infinity. Practitioners call that diverging, and it is spectacular to watch once and miserable to debug twice. The demo uses a middling rate, so the descent is quick but stays smooth.

Watch 64 weights find their values

The animation above records a real training run. Sixty-four weights begin as random scatter, while a bias starts at zero. The lines move quickly when the model is wrong, then flatten as its corrections shrink.

Generate a seed, train, and scrub through the result. The 8x8 heatmap resolves into useful evidence for the familiar examples in its training set. Then draw in the test panel and see those learned numbers make a prediction.

Valleys that are not the valley

Doodle-64 has 65 adjustable numbers: 64 pixel weights and one bias. Its logistic loss is convex, so it does not contain the local valleys shown in the 2D and 3D teaching views above. Those polynomial views make slopes, basins, and starting points visible.

Deep networks have non-convex loss surfaces in millions or billions of dimensions. No screen can show all of them at once, but a two-dimensional surface can still show what it means to follow a local slope.

A local minimum is a dip that is lower than everything immediately around it but not the lowest place on the map. Gradient descent only feels the ground beneath it, so at the bottom of a local dip the slope is flat and training stops improving. The weights work, just less well than they could have. A plateau is worse company: a wide flat stretch where the gradient is nearly zero and training seems to have stalled, sometimes for a very long time, even though better ground lies just ahead. Saddle points are flat in one direction and downhill in another, and in high dimensions they far outnumber true local minima.

The escapes are mostly nudges. Noise helps: real training computes the gradient on a small random batch of examples at a time, so the path jitters and can rattle out of shallow dips. Momentum helps: keep some velocity from the last step so you roll across flat ground instead of stopping on it. And multiple random restarts help, which is the deepest reason those starting weights are random. Different starting points explore different parts of the landscape.

This is how every model on this site was made

Scale is the only difference between the animation above and the training of a frontier model. Swap 64 weights for hundreds of billions. Swap ten pixel drawings for a large fraction of the public internet. Swap "is this a 3?" for "what token comes next?". Swap 120 epochs on your laptop for months on tens of thousands of GPUs.

The loop does not change: guess, measure how wrong you are, follow the slope downhill, repeat. When a lab says a model cost hundreds of millions of dollars to train, this is the thing that cost the money: an unfathomable number of very small steps down a hill nobody can see.