The Repetition Problem
Chat applications resend the system prompt, tool definitions, and conversation history on every turn. In a typical session that static prefix can be 80 percent of the tokens you pay for. Prompt caching lets the provider skip reprocessing that prefix and charge you a fraction of the usual input price.
How Caching Works
When you send a request, the provider checks whether the start of your input matches a recently cached prefix. If it does, the cached portion bills at the discounted cache-hit rate and only the new suffix is processed at the full rate. The longer your stable prefix, the larger the saving.
What Makes Cache Hits More Likely
- Keep your system prompt and tool definitions stable across turns.
- Append dynamic content at the end of the message, not the beginning.
- Send the full history in the same order every time instead of pruning from the front.
- Stay within the cache time-to-live by keeping sessions active, or re-use a session for follow-ups.
What Caching Does Not Fix
Caching lowers the input price per token; it does not change output pricing or reduce the number of tokens you generate. A model that produces rambling answers will still cost you, so caching pairs best with concise system prompts and clear output constraints.
Measuring the Difference
Most gateways report cache hit and miss tokens separately in the usage object. After you enable caching, compare a multi-turn session’s billed input against the same session before. Applications that keep a long, stable context typically see their input cost drop by more than half.
No comments yet. Be the first to share your thoughts!