| 1 | # Prompt-cache stability (the pinned prefix) |
| 2 | |
| 3 | Provider prompt caches (DeepSeek KV cache, Anthropic `cache_control`) only pay |
| 4 | off when the **byte prefix** of a request matches the previous one: the system |
| 5 | prompt, then the tool catalog, then `messages[0..n-1]`. Any change to those |
| 6 | bytes invalidates the cache for every token after the first difference. |
| 7 | |
| 8 | ## The invariant |
| 9 | |
| 10 | **After session start, the system prompt and tool catalog are frozen bytes. |
| 11 | History only grows. A cache miss is allowed only when we can name why.** |
| 12 | |
| 13 | Concretely: |
| 14 | |
| 15 | - The **header** (system prompt + tools) is composed once at session start and |
| 16 | re-composed **only** on an explicit, logged header-change op. The tool loop |
| 17 | performs **no** mid-loop system-prompt refresh, so an agent writing a file |
| 18 | (which changes the project-context pack, a directory listing, a skills scan) |
| 19 | cannot move the pinned prefix under the model's feet mid-turn. |
| 20 | - **History only grows.** Volatile facts the model must see (LSP diagnostics, |
| 21 | steer input, subagent completions, `<recommended_plugins>` on a matching |
| 22 | user turn) are appended to the message list, never spliced into the frozen |
| 23 | prefix. Workspace drift is delivered the same way: |
| 24 | at the start of each **new user turn** (never mid-tool-loop) the engine |
| 25 | recomposes the volatile contributors and, if anything differs from what the |
| 26 | model last saw, appends **one** `<context_update>` user-role message with a |
| 27 | bounded `+`/`-` line delta (new files in the project pack, edited AGENTS.md |
| 28 | lines, added skills, memory entries, goal text) *before* the user's message. |
| 29 | The header bytes stay pinned; the update is a normal append, so the prefix |
| 30 | still extends. The pinned system prompt tells the model once that updates |
| 31 | arrive this way. Each delta is delivered exactly once (`/cache stats` shows |
| 32 | `Context updates: N`). |
| 33 | - Every miss is **attributable**. `PrefixStabilityManager` (`prefix_cache.rs`) |
| 34 | records each change with a reason and reports it through `/cache stats`. |
| 35 | |
| 36 | ## What counts as a declared header change |
| 37 | |
| 38 | These re-pin the prefix under a logged `change:<what>` reason (an expected, |
| 39 | one-request miss): |
| 40 | |
| 41 | | Op | Reason | |
| 42 | | --- | --- | |
| 43 | | `/model` (SetModel) | `change:model` | |
| 44 | | Mode change (agent/plan/operate/yolo) | `change:mode` | |
| 45 | | Goal set / pause / resume / clear / status | `change:goal` | |
| 46 | | Mid-turn tool-surface change (deferred-tool admission/eviction, tool-search activation, runtime MCP tool arrival) | `change:tool_surface` | |
| 47 | | Session sync / restore (SyncSession) | `resume` | |
| 48 | | Session construction | `initial` | |
| 49 | |
| 50 | History resets that legitimately invalidate the tail (not the header) are |
| 51 | logged as `reset:<what>` — `reset:compaction`, `reset:clear`. |
| 52 | |
| 53 | Anything else that changes the header bytes with **no** declared reason is |
| 54 | **drift**: it is logged as `drift:<component>`, the original pin is **kept** |
| 55 | (so the same undeclared prefix keeps counting as a miss instead of quietly |
| 56 | becoming the new baseline), and `/cache stats` shows a `WARNING`. After the |
| 57 | mid-loop-refresh removal, drift should stay at zero in normal operation; a |
| 58 | non-zero drift count is a real bug to investigate. |
| 59 | |
| 60 | ## Attribution vs. the old behavior |
| 61 | |
| 62 | Two earlier behaviors are rejected, matching the DeepSeek Harness design: |
| 63 | |
| 64 | - **Detect-and-report + re-pin on drift.** The manager used to re-pin to the |
| 65 | new prefix on every change, so `/cache stats` looked "stable" again after one |
| 66 | bad step while the provider cache was already dead. It now keeps the original |
| 67 | pin on undeclared drift. |
| 68 | - **Recompose the system prompt from disk on every tool step.** The turn loop |
| 69 | used to call `refresh_system_prompt()` before every model request, including |
| 70 | mid-tool-loop. That is removed. Header refreshes happen only at the declared |
| 71 | edges above. |
| 72 | |
| 73 | Tool-result redaction (`prepare_model_bound_request`) is content-preserving |
| 74 | when no secret is configured (the common case), so it does not move the prefix. |
| 75 | When a configured secret appears in a tool result, redacting it is a security |
| 76 | requirement that correctly overrides cache stability for that one message. |
| 77 | |
| 78 | ## Verifying the fix |
| 79 | |
| 80 | `/cache stats` reports prefix stability, the pin reason, the last miss reason, |
| 81 | the undeclared-drift count, and the aggregate provider cache hit rate. In a |
| 82 | coding session, expect the first turn to be a write and every later step — |
| 83 | including steps after the agent writes files — to hit. |
| 84 | |
| 85 | ### Live end-to-end check (manual, key-gated) |
| 86 | |
| 87 | With a real `DEEPSEEK_API_KEY`, run a session that makes the agent take at |
| 88 | least three tool steps in one turn, then open `/cache inspect`. Every request |
| 89 | after the first should report `prompt_cache_hit_tokens > 0`; the base static |
| 90 | prefix hash and the tool-catalog hash must not move between steps. If the hit |
| 91 | drops mid-turn, the pin reason / drift count name the cause. |
| 92 | |
| 93 | ## KV-cache effect note (for contributors) |
| 94 | |
| 95 | Any new contributor to the session context must state its **KV-cache effect**: |
| 96 | does it belong in the frozen prefix (system + tools) or in append-only history? |
| 97 | Never splice a volatile fact (time, an instruction edit, a skill-catalog |
| 98 | change, a project-file change) into the prefix — append it as a user-role |
| 99 | message instead. A later request must be `previous ⊕ suffix` unless a logged |
| 100 | header change or a history reset explains the difference. |
| 101 | |
| 102 | ## Deferred: full reconstructability (Layer 3) |
| 103 | |
| 104 | DeepSeek Harness derives every request from an append-only session log via a |
| 105 | pure `deriveMessages()` projection, so prefix-extension is emergent rather than |
| 106 | managed. Codewhale now pins the header and delivers drift as `<context_update>` |
| 107 | appends; the remaining step is to make the session log the single source of |
| 108 | truth with a pure projection (and to persist the context-update baseline with |
| 109 | it). That is a follow-up lane, not part of this change. |
| 110 |