prompt-caching

The mechanism by which anthropic‘s API caches a prompt prefix so subsequent requests sharing that prefix run cheaper and faster. claude-code‘s entire runtime layout is shaped around it (2026-04-27-claude-code-architecture-governance-engineering).

How it works

Caching is prefix-based. From the start of the request to each cache_control breakpoint, the content is eligible to be served from cache on subsequent requests with an identical prefix. Order matters:

Request layout (Claude Code):
1. System prompt        ← static, locked
2. Tool definitions     ← static, locked
3. Chat history         ← dynamic, follows
4. Current user input   ← last

Anything dynamic placed before something static invalidates everything after it.

Why Anthropic treats hit rate as an SLO

Per tw93, Anthropic internally pages SEV (severity incident) when cache hit rate drops — the economics of running an agent harness like claude-code depend on it, and so does rate-limit headroom for end users (2026-04-27-claude-code-architecture-governance-engineering).

Common cache-killers

• Timestamps in the system prompt (or anything else that varies per request).
• Tool-set churn mid-session — adding or removing a tool re-keys the cached prefix.
• Switching models mid-conversation. Cache is per-model. After 100K tokens with Opus, switching to Haiku for a “simple question” is more expensive than continuing on Opus, because the Haiku cache must be rebuilt from zero. If a switch is genuinely needed, hand the task to a Subagent running the other model with a small briefing message.

Workarounds Claude Code uses

• Dynamic info goes in the user message, not the system prompt. Claude Code emits <system-reminder> tags inside user messages for things like the current date — system prompt stays byte-stable.
• EnterPlanMode is a model-callable tool, not a tool-set switch. plan-mode would intuitively want a “read-only” tool roster, but that would invalidate the cache. Instead the model invokes EnterPlanMode as a regular tool call and the harness enforces read-only behavior elsewhere.
• Compaction runs as a forked summarization call against the cached prefix — it’s prefix-cached itself, costing roughly 1/10 of a fresh call.
• defer_loading: true for tools. Claude Code ships dozens of MCP tools; including all schemas every request would be expensive, but removing tools mid-session breaks cache. Solution: ship lightweight stubs (just the tool name) marked defer_loading: true. The model discovers them via a ToolSearch tool; full schema only loads when the model selects one. Cache prefix stays stable.

Note for future ingest: the exact defer_loading mechanism described here is from a single source (tw93) — corroborate with primary Anthropic documentation when found.

Cache-aware design generalizes beyond Claude Code

Reinforced from a different angle in 2026-04-27-agent-principles-architecture-engineering: prefix-cache hit rate demands stability, not just brevity. The reason “keep the resident layer short and stable” matters isn’t only token cost — it’s that any drift in the prefix breaks cache reuse for the rest of the session. This is why:

• Stable layers go first (system prompt, tool definitions); dynamic info (timestamps, user input, tool results) goes last.
• Skills lazy-loading helps cache because Skill bodies append after the stable prefix rather than mutating it.
• A large stable system prompt is sometimes cheaper than a small but mutating one — write cost is paid once, every subsequent call gets the ~90% read discount.
• Tool sets that churn mid-session (newly-added MCP tools, A/B-tested tool descriptions) silently destroy cache — see model-context-protocol.

The cross-system rule of thumb: anything that varies per-request goes in the user message; anything that varies across deployments goes at the very end of the resident layer.