The model lab · Part 8
How do LLMs predict the next word?
Meet Parrot-43, a language model trained on nine sentences. Watch it predict the next word and see exactly where every prediction comes from.
Parrot-43
Model card- Type
- Bigram language model, the smallest honest next-word predictor
- Parameters
- 43: one count for every word pair it saw during training
- Layers
- 0: it's a lookup table with zero network layers (that's the punchline, keep reading)
- Input
- 1 word: the last word of the sentence so far
- Output
- A probability for every word that could come next, summing to 100%
ChatGPT and Claude do this exact job (predict the next token, append it, repeat) with billions of parameters and thousands of words of context instead of one.
The entire training data
the cat sat on the mat.
the cat chased a ball of yarn.
the dog chased the cat.
a dog sat on the mat.
the dog ate my homework.
my cat ate the fish.
the fish swam in the pond.
a bird sat on the fence.
the bird flew over the pond.
The sentence is empty, so Parrot-43 looks at how sentences start. Every green word opens a sentence in the training data.
Build a sentence
Open the first-word dropdown to see the model's ranked guesses, each with its probability and how many times it was seen. Pick one and predict again. Choose "." to end the sentence.
The smallest language model that could
Doodle-64 and Doodle-525 recognize things. Parrot-43 generates things, which makes it the closest cousin of ChatGPT and Claude in our lab. Its entire education is the nine sentences shown above, and its entire brain is 43 counts: how many times each word followed each other word in those sentences.
That's all "training" means here. Read the data, count the pairs. "the cat" appears three times, so after "the", the word "cat" gets 3 votes. Divide the votes by the total and you have probabilities. When you click a word in the demo, the highlighted corpus shows you exactly which sentences cast those votes. Nothing is hidden, because there is nothing else.
Generation is prediction in a loop
To write a sentence, Parrot-43 does what every LLM does: predict the next word, append it, and predict again from the new ending. Click through a few choices and you can splice its training sentences into one it was never taught, like "my cat sat on the fence". That is generation: new sentences out of old counts.
Now press "Always pick the favorite" and watch it get stuck chanting "the cat ate my cat ate my…". Always taking the single most likely word is a real failure mode in big models too. It's part of why LLMs add a dash of randomness, called , when they pick from their probabilities.
You'll also notice what it can never do: say a word it hasn't seen. Its 22-word vocabulary is the entire universe as far as it's concerned. And when you steer it onto a rare word like "flew", it has exactly one recorded continuation and marches straight down it. Tiny training data makes a tiny parrot.
What LLMs do differently
Three upgrades separate Parrot-43 from GPT or Claude. First, context: Parrot-43 looks at one previous word; an LLM weighs thousands of previous at once (that's what the 's is for). Second, generalization: an LLM doesn't store counts in a table. It compresses the patterns of its training data into billions of weights, the same multiply-and-add weights as Doodle-64, so it can handle sentences it has never seen. Third, scale: instead of nine sentences, most of the internet; instead of a 22-word vocabulary, ~200,000 tokens.
But the job description never changes. Every reply from a is next-token prediction repeated thousands of times: score every possible continuation, turn scores into probabilities, pick one, append, repeat. If you understand why Parrot-43 says "cat" after "the", you understand what a trillion-dollar industry is scaling up.
Where the hallucinations come from
Parrot-43 also demonstrates the famous LLM failure mode in miniature. Ask it to continue "my" and it says "cat" or "homework". Neither is true. They're just the likeliest continuations of its training data. Likely and true are different things. An LLM with billions of parameters blurs that line much more convincingly, but the gap never fully closes. That's a , and now you've watched one get built.