Tokens & Context Windows: The Two Ideas Behind Every AI Product
Every AI product you use is built on two ideas: tokens (what the model reads) and the context window (how much it can hold). Most explanations of them are wrong in the same way.
This one uses real numbers from a real tokenizer. Ten minutes, no prerequisites.
|
AI Dev Signals · Fundamentals Tokens & context windowsWhy the model can't count letters, and why it re-reads your entire chat every time you press enter. |
|
01 |
What the model actually reads |
A language model never sees your text. Before anything reaches it, a tokenizer chops your sentence into chunks and swaps each chunk for a number. The model only ever sees the numbers.
Here is a real sentence, tokenized with o200k_base — the tokenizer behind the current GPT models. Top row is what you typed. Bottom row is what the model receives.
|
Input — "The cat sat on the hamburger."
6 words → 7 tokens. The · marks a leading space — it belongs to the token. What the model receives is just this: [976, 9059, 10139, 402, 290, 139813, 13] |
Rule of thumb: ~4 characters per token in English, or ~0.75 words per token. Measured on real prose it runs closer to 4.9 characters — treat 4 as a safe overestimate when budgeting.
|
The space changes everything Most token demos quietly cheat by dropping the leading space. Same word, two very different results:
Words get split when they are rare in that position, not when they are long. Mid-sentence, common words are almost always one token. |
||||||||||
|
So why can't it count the r's in "strawberry"? The popular answer is that the model sees str + aw + berry and loses track. Tokenize the actual question and that story falls apart:
Mid-sentence, "·strawberry" is a single token. The model isn't juggling three fragments — it's holding one number that has no letters inside it at all. Asking it to count r's is like asking you to count the strokes in a word you only ever heard spoken. The real answer is worse than the myth, and more interesting. |
|
02 |
What the model actually remembers |
Nothing. That is not a joke and it is the whole lesson.
A model has no memory between turns. It is a pure function: tokens in, tokens out. When you send message #20, the app quietly re-sends all nineteen previous messages plus your new one, and the model reads the entire conversation from scratch, as if for the first time. The context window is simply the maximum size of that re-read.
|
A chat where each turn adds ~300 tokens
Ten turns of 300 tokens is 3,000 tokens of conversation — but you were billed for 16,500. Every turn pays for the whole history again. This is why long chats get slower and more expensive, and why prompt caching exists. |
|||||||||||||||||||||||||||
Simplified: real chats vary per turn, and cached input is discounted heavily. The shape is what matters.
|
The window is not empty when you start Before your first word, the context window already holds the system prompt, any tool and function definitions, retrieved documents, and files you attached. On a tool-heavy agent that overhead can run to tens of thousands of tokens. You are always renting the remainder. |
|
What actually happens when it fills up Here is where almost every explainer gets it wrong. Nothing is "quietly erased." The behavior depends entirely on which layer you are talking to:
So "the AI forgot what I told it" almost always means: a piece of software you never saw decided which parts of your conversation were worth re-sending. If you build on the API, that software is you. |
|
03 |
Where the numbers stand |
The context window arms race is over, and it ended in a tie. Every frontier lab has converged on 1M tokens — roughly 750,000 words, about ten novels. The interesting competition moved to cost and reasoning.
~750,000 words. Max output is the real differentiator: 128K on Opus 4.8 and GPT-5.6, ~64K on Gemini 3.1 Pro. |
~150,000 words — still a full-length book. Enough for the overwhelming majority of production tasks. |
~1,500 words. Its bar is empty because 2,048 tokens is 0.2% of 1M — too thin to draw honestly. Windows grew 488× in six years. |
|
A big window is a ceiling, not a promise Models reliably retrieve facts placed at the start and end of a long context, and degrade in the middle — the "lost in the middle" effect. A 1M window means the request won't error out. It does not mean the model used all of it well. Put what matters at the edges, and benchmark retrieval at the length you actually ship. |
|
Key takeaways
|
|
Token counts computed with o200k_base. Context windows verified against vendor documentation, July 2026. Model specs change monthly — check the source before you quote these. Marktechpost · AI Dev Signals |