The model lab · Part 10
How do position and attention make language models grammatical?
Meet Finch-4. This is a model that adds a position signal and a small attention head to Parrot-2D, so it can build a subject, verb, object sentence.
Finch-4
Model card- Type
- Tiny position-aware language model with one attention head
- Parameters
- Four readable signals per step: friendliness, role, position, and a small attention weight
- Layers
- A word map, a position input, one attention head, and a next-word output
- Input
- The words so far plus each word's position in the sentence
- Output
- A probability for the next word or the period that ends the sentence
Frontier transformers repeat this pattern across many layers and heads. Finch-4 keeps one small head visible so you can follow its choice.
Parrot-2D needs a place for each word
Parrot-2D gives Bob the same two numbers every time. That works when Bob begins a sentence. It loses an important clue when Bob comes after a verb, because the model cannot see whether Bob is the subject or the object.
Finch-4 starts from the same friendly and role meanings. It adds one small number for the word's place in the sentence. This is enough to let the model treat Bob at the start differently from Bob near the end.
Position changes the input
The position signal is a label for the slot. A subject gets minus one, a verb gets zero, and an object gets plus one. The word and its slot travel into the model together.
Choose Bob in the visual. As a subject, Bob leads toward “ignores”. As an object, Bob leads toward the period. The same spelling now carries a different job in the sentence.
One word, two positions
Position signal -1. The friendly meaning points to the matching verb.
Position signal 1. The next token is the period.
A small attention head looks back
For the object slot in “Bob ignores”, the query gives more weight to Bob. That carries the subject role forward before the model picks an object.
key 1, value person
key 0.4, value verb
Generate a sentence
Bob ignores Alice.
Each word feeds back in as the input for the next one. Greedy chooses the top bar. Sample draws from the bars, and temperature spreads or concentrates those chances.
Attention carries context forward
Position says where each word sits. lets the object slot look back at the earlier words. The object query gives each earlier word a weight, then carries the useful pieces forward.
In the small head above, “Bob” receives the larger weight because it is the subject. A frontier uses many heads and many dimensions, yet this one visible set of weights shows the key move.
Generate a grammatical sentence
Generation is still a loop, and you can watch it run one word at a time. The model predicts a subject, then feeds that word back in to predict a verb, then feeds that in to predict an object, and finally a period. Every step reads the word that came just before it, so each choice depends on the ones already made. Position and attention keep those choices in their sentence roles.
Bob, Alice, greets, and ignores are the friendly and unfriendly words. Charlie and sees are the two words in the middle. Charlie is a person who leans neither way, so his sentence can go anywhere, and sees is a calm verb that anyone can use. Adding them lets the model build far more sentences than the four corners alone.
Greedy generation takes the largest chance every time. Sampling draws from the chances. A higher spreads the choices out, so you can see a less common sentence while the subject, verb, object shape stays in place.
Finch-4 continues the Parrot-2D embedding lab, where the two readable meaning signals begin.
Put it to use
Next up: Why do neural networks need more than one hidden layer?