What a Context Window Really Is
A model’s context window is the total amount of text it can look at in one request, including your system prompt, the conversation history, tool definitions, and the answer it is writing. Think of it as the working memory available for a single call, not the model’s permanent knowledge.
Tokens, Not Words
Context is measured in tokens. An English word is roughly 1.3 tokens, so a 128K context window holds about 90,000 to 100,000 words of text. Reading that much can cost a meaningful amount per request, so a large window is a capability and a bill.
What Fills It Up Fastest
- Conversation history in chat apps: every turn re-sends the whole transcript.
- Long documents pasted whole instead of being searched first.
- Tool definitions with verbose schemas repeated on each call.
- Large few-shot examples included in every request.
When It Runs Out
Older text gets truncated or dropped, and the model answers using only what remains. The failure is silent: the answer looks fine but forgets facts from the early part of the conversation. Symptoms include the model “forgetting” instructions you gave at the start or contradicting earlier statements.
Three Ways to Live Inside the Window
- Summarize history: condense older turns into a short running summary instead of resending raw text.
- Retrieve instead of paste: search the document and send only the relevant sections.
- Prune aggressively: drop tool definitions and few-shot examples that the current turn does not need.
Applications that manage context deliberately stay cheaper and more accurate than ones that simply buy a bigger window and paste everything.
No comments yet. Be the first to share your thoughts!