How do word embeddings predict the next word?

From a lookup table to a meaning map

Parrot-43 is a lookup table. It only remembers which word followed which. Show it a word from outside its training and it has nothing to say. Frontier language models work differently, and Parrot-2D shows you the first upgrade. Every word gets its own small list of numbers, called an embedding, and the model predicts from those numbers.

Parrot-2D gives every word just two numbers, and both numbers mean something you can read. The map above the demo places each word by those two numbers. Once a word is a point on a map, "what comes next" becomes a question about meaning, and you can watch the answer light up.

Two numbers that mean something

The first number is friendliness. Alice is friendly, so she sits on the right. Bob is unfriendly, so he sits on the left. The second number is role. Alice and Bob are people, so they sit low. "greets" and "ignores" are verbs, so they sit high. Two numbers put every word in its own corner of the map.

A frontier embedding gives a word thousands of numbers, and nobody labels what each one means. Parrot-2D shrinks that down to two numbers you can name. That is the trade the whole lesson rests on. You give up the scale of a frontier model and you get a picture you can look at.

Meaning drives the prediction

Now build a sentence and watch the map. When you land on Bob, the model rings him with gold and lights up the words it expects next. The bright one is "ignores". That is the unfriendly verb, and the training sentences taught the model that an unfriendly person is usually followed by an unfriendly verb. The friendly verb "greets" still glows a little, because a couple of training sentences broke the pattern on purpose.

This is why the model needs meaning. The words that light up after Bob are the words that sit near where an unfriendly verb should sit. You can read the prediction straight off the map, which is something a plain lookup table can never let you do.

The network behind the map

The map is a picture. Underneath it runs a small neural network, and now you can watch it work. It has three layers. On the left is one node for every word, and the current word switches on. In the middle sit two hidden nodes, and those two nodes are the two meanings you already know. The top node holds the friendliness of the current word, and the bottom node holds whether it is a verb. On the right is one node for every word that could come next, plus a period for ending the sentence.

Follow the signal through

Now pick a word on the left and follow the signal. The left wires copy the current word's two numbers into the two hidden nodes, so choosing Bob lights both hidden nodes at minus one, unfriendly and a person. From there the signal spreads to the outputs, and the longest bar is the word most likely to come next. Choose Bob and the top bar is "ignores", the same answer the map gives.

This is the shape of every model on this site, shrunk until you can see it. A word comes in, a hidden layer turns it into a handful of numbers about its meaning, and an output layer turns those numbers into a next-word guess. A frontier model has thousands of hidden numbers and billions of wires. Parrot-2D has two hidden numbers you can name, so the whole path fits on one screen.

Filling in the middle of the map

The four starter words sit in the four corners, which makes friendly and unfriendly look like the only two choices. Press "Add Charlie and sees" and two new words appear in the middle. Charlie is a person who is neither friendly nor unfriendly, so he sits between Alice and Bob. "sees" is a verb that is neither, so it sits between "greets" and "ignores".

The middle of the map is where meaning stops being a light switch and becomes a dial. Charlie can be followed by a friendly verb or an unfriendly one, because his sentences went both ways. Frontier embeddings are dials all the way down, which is how a model can tell "warm" from "hot" from "scalding" using nothing but numbers.

This is what frontier embeddings do

Every large language model starts by turning each word into an embedding, a long list of numbers that places the word in a space of meaning. Words that behave alike end up near each other, the same way Alice and Charlie sit near each other here. The model never sees the word "king". It sees the point where "king" lives, and it predicts from the neighborhood.

Parrot-2D has two dimensions you can read. A frontier model has thousands you cannot. The job is the same in both. Look up the current word as a point, find the points that tend to come next, and turn how close they are into probabilities. If you can read the prediction off this little map, you have seen the first thing every frontier model does with your words.

There is one thing two numbers cannot do. Parrot-2D sees only the single word in front of it, so it cannot tell "Bob" the speaker from "Bob" the one being spoken to. Fixing that needs a sense of where a word sits in the sentence, and a way to look back at the words before it. That idea is called attention, and it is the subject of the next lab model.