0
Tokens
Build a conversation, then pull the covers back. The left side is what you read. The right side is what the model reads — the same conversation, serialized and numbered. It's numbers all the way down.
Bar height = attention weight: real attention math (query · key, scaled, softmax) run on this sequence — but with tiny untrained toy vectors, so the pattern is illustrative while the mechanism is genuine. Green bars are the tokens the focus position attends to; brass bars are special tokens; the plum bar is the focus attending to itself.
The loop, one token at a time1 · The focus position builds a query and scores it against every earlier token's key. Standard decoding is causal: it only looks back, never ahead. There's no draft-and-score of future text — the "prediction" is a single probability distribution over the whole vocabulary for the very next token.
2 · Those weights blend the earlier tokens' values into one context vector — "what matters for predicting what comes next, from where I stand."
3 · That vector (through many layers of this) becomes next-token probabilities. One id gets sampled, appended to the sequence — the numbers you watch arrive on the machine side — and the loop runs again with the sequence one token longer.
This page runs a real byte-pair-encoding tokenizer — same algorithm as GPT-2 and tiktoken — trained fresh on a small corpus so the whole thing fits in one file with no network calls. That means the token boundaries behave like the real thing (common words are one token, rare words shatter into pieces, the space rides on the front of the next word), but the ids won't match any production model's vocabulary.
The special tokens <|im_start|> and <|im_end|> are how the chat gets flattened into one string. Roles, structure, your prompt — the model receives all of it as a single sequence of integers, and everything it says back is just more integers appended to the end.
Thinking works the same way, because there is no other way. Between <think> and </think> the model is emitting ordinary tokens into the same sequence — and since every new token becomes context for the next prediction, the model is literally reading its own reasoning back as it writes it. There's no separate scratchpad or second brain: the "internal monologue" is the same predict-append-repeat loop, just wrapped in special tokens so the chat app can curtain it off. Agent loops extend the trick one more step — the model emits a tool call as tokens, something outside runs it and appends the result to the sequence, and prediction resumes. "Done" is nothing more than the model emitting the end token.