Why not just words?
A word-level vocabulary can never cover typos, new slang, or every language. Subword pieces let a fixed vocabulary of ~100k entries spell anything.
An interactive guide
How large language
models actually work
No math degree required. Follow a single sentence as it becomes numbers, flows through billions of parameters, and comes back out as the next word — one prediction at a time.
>
0
tokens in a typical vocabulary
0
transformer layers, stacked
0
parameters in frontier models
0
token predicted at a time
01 — The big picture
Everything is next-token prediction
An LLM does exactly one thing: given a sequence of text, it guesses what comes next. Everything else — reasoning, code, translation, poetry — is that one trick, repeated. Here is the full journey of your input.
→
→
→
→
→
STEP 01
02 — Tokenization
Models don't see words. They see tokens.
Before anything else, your text is chopped into tokens — common chunks of characters. Frequent words become one token; rare words shatter into pieces. Each token maps to an integer ID, and that integer is all the model ever receives.
Live tokenizer
type anything
0tokens
0characters
0chars / token
Approximation of byte-pair encoding for illustration. Real tokenizers learn their vocabulary from data — but the shape is the same: ~4 characters per token in English, far more per token in code or other alphabets.
Why you pay per token
Cost and context limits are measured in tokens, not words. "Strawberry" may be three tokens — which is also why models famously miscount its letters.
Whitespace matters
" the" and "the" are different tokens with different IDs. That leading space carries real information about word boundaries.
03 — Embeddings
Every token becomes a direction in space
Token ID 4917 means nothing on its own. The model looks it up in a giant table and gets a vector — a list of a few thousand numbers. Similar meanings end up pointing in similar directions, so geometry becomes semantics.
Embedding vector
click a token
king → [ 0.42, −1.09, 0.77, … ]
· 12 of 4096 dimensions shown
Meaning as arithmetic
Because directions encode meaning, you can do math on words. The classic result: king − man + woman ≈ queen. Nobody programmed that — it fell out of the training data.
Position is added in
Vectors alone are order-blind: "dog bites man" would look identical to "man bites dog." So a positional signal is mixed into each vector, telling the model where each token sits.
04 — Attention
Each token looks around and asks: who matters to me?
This is the breakthrough behind the transformer. Every token compares itself to every earlier token and pulls in information from the ones that are relevant. That's how a pronoun finds its noun, and how a closing bracket finds its opener.
Attention weights
hover a token to see what it attends to
Hover over it — watch it lock onto the right noun.
Illustrative weights from a single head. Real models run dozens of heads per layer in parallel, each learning a different relationship: syntax, coreference, topic, formatting.
Query, Key, Value
Each token emits a query ("what am I looking for?") and a key ("what do I offer?"). Matching pairs score high, and the winner's value gets mixed into the asker.
It only looks backwards
During generation, a mask blocks every token from seeing the future. The model must predict word 12 knowing only words 1–11 — otherwise training would be cheating.
Why context costs so much
Every token compares to every other, so work grows with the square of the length. Doubling the context roughly quadruples the attention math — the core reason long context is expensive.
05 — The stack
Repeat, dozens of times
One attention step isn't enough. A transformer block — attention, then a feed-forward network that does the "thinking" on each token — is stacked over and over. Early layers handle grammar; deeper layers handle meaning, intent, and abstraction.
Attention · tokens exchange information
Feed-forward · each token is refined alone
Residual + norm · keeps the signal stable
06 — Prediction & sampling
The output is a probability distribution
The final layer scores every token in the vocabulary. Those scores become probabilities, and one token is drawn. Temperature controls how adventurous that draw is — then the chosen token is appended to the input and the whole process runs again.
"The best way to learn programming is to ___"
drag the slider
Low (0.1–0.4)Focused and repetitive. Good for facts, code, extraction.
Medium (0.5–1.0)The usual default. Coherent with some variety.
High (1.2+)Creative, surprising, and increasingly incoherent.
07 — Training
From random noise to useful assistant
A fresh model's weights are random — its output is gibberish. Three stages turn that into something you'd want to talk to.
-
1
Pretraining months · thousands of GPUs
The model reads a huge slice of the internet, books, and code, endlessly predicting the next token. Every miss nudges billions of weights a tiny bit. This is where knowledge, grammar, and world structure are absorbed — and where nearly all the cost lives.
-
2
Instruction tuning days
Pretraining yields a document-completer, not an assistant — ask it a question and it might reply with more questions. Fine-tuning on curated instruction/response pairs teaches it to answer rather than continue.
-
3
Preference optimization ongoing
Humans (and other models) rank competing answers. Those rankings train a reward signal that shapes the model toward being helpful, honest, and safe. This stage sets the personality — not the knowledge.
08 — What follows from all this
Four consequences that explain most LLM behavior
It optimizes for plausible, not true
There is no fact database inside — only weights that make likely text. A confident, well-formed, wrong answer is a perfectly successful prediction by the model's own objective. Grounding it in retrieved sources or tools is the fix, not asking it to try harder.
Nothing persists between messages
Weights are frozen after training. A chat "remembers" only because the entire transcript is re-sent with every turn. Slide past the context window and the earliest part is genuinely gone.
Context steers the distribution
Your prompt doesn't instruct a program — it conditions a probability distribution. Examples, role, and format constraints all shift which continuations look likely, which is why showing beats telling.
Thinking out loud buys computation
Each token gets a fixed amount of compute. Letting the model write intermediate steps gives a hard problem more forward passes to work with — the mechanism behind chain-of-thought and reasoning models.
That's the whole machine.
Tokens in, vectors through a stack of attention, a probability distribution out — repeated once per word. Everything else is scale.
Back to the top ↑