Token by token
How a transformer reads a sentence
Before predicting the next word, the model lets every word look back at the words before it. That lookup is attention, and it's how the model works out that “it” means the box, not the robot.
1 · Attention: where each word looks
Tap any word. Arcs show how much it attends to the words before it. A thicker arc means more attention.
“it” leans hardest on “box”, so the model has resolved what “it” refers to.
2 · Prediction: what comes after “was”
The softmax turns the model's scores into probabilities over every word it knows. Because “it” attended to “box”, box-like words win.
Values as a table
| Next word | Probability |
|---|
Attention weights and probabilities are illustrative, chosen to show the mechanism, not taken from a real model run.