返回 last30days-skill
discovery-checkpoint-protocol-design-conventions.md
根目录 / docs / solutions / architecture-patterns / discovery-checkpoint-protocol-design-conventions.md
1 ---
2 title: "Checkpointed discovery protocol: five design conventions for host-side LLM judgment"
3 date: 2026-07-21
4 category: architecture-patterns
5 module: discovery-checkpoint-protocol
6 problem_type: architecture_pattern
7 component: tooling
8 severity: high
9 applies_when:
10 - "The product's primary consumer is a frontier reasoning model invoking the tool as an agent skill, not a traditional programmatic API client"
11 - "A pipeline stage needs semantic judgment (naming, classification, worthiness scoring) that only an LLM can supply"
12 - "Building a keyless or free-tier path where a silent heuristic fallback would degrade output quality without disclosing that an API key was assumed"
13 - "A CLI or script needs to persist state across multiple invocations while the host model performs judgment in between (checkpoint-and-resume design)"
14 - "An existing skill law or convention already establishes that host-side reasoning replaces engine-side API keys, and a new pipeline stage needs the same treatment"
15 symptoms:
16 - "v3.17.0 silently fell back to deterministic topic naming and junk heuristics for keyless users when the engine's own LLM judge was unavailable"
17 - "An API key was the de facto front door to real discovery judgment, contradicting the skill's keyless-path promise and its own LAW 7 host-is-the-reasoning-model precedent"
18 root_cause: wrong_api
19 resolution_type: code_fix
20 tags:
21 - "discovery-protocol"
22 - "host-judged-protocol"
23 - "checkpoint-files"
24 - "law-11"
25 - "keyless-path"
26 - "nominations-bundle"
27 - "provenance-enforcement"
28 - "bundle-id-ttl"
29 related_components:
30 - "skills/last30days/scripts/lib/discovery_handoff.py"
31 - "skills/last30days/scripts/last30days.py"
32 - "skills/last30days/SKILL.md"
33 - "tests/test_discover_handoff.py"
34 - "tests/test_discover_mode.py"
35 ---
36
37 # Checkpointed discovery protocol: five design conventions for host-side LLM judgment
38
39 ## Context
40
41 An Agent Skill's primary consumer is a frontier reasoning model: the engine
42 (`skills/last30days/scripts/last30days.py`) is invoked by Claude Code, Codex,
43 Gemini, or another agent runtime that read SKILL.md. v3.17.0 (PR #852) forgot
44 that and shipped judgment as an engine-side LLM pass: `lib/discovery_judge.py`
45 (since deleted by PR #856; the path is historical) resolved a reasoning
46 provider across Gemini/OpenAI/xAI/OpenRouter keys and,
47 per its own contract, never raised - "No provider, a failed call, or a
48 malformed payload logs a warning and returns None, and the caller falls back"
49 to deterministic heuristics. Every keyless user silently got the degraded
50 branch: heuristic topic names like "120k 1,600 ESP32s" and zero content
51 angles, with no signal that a better path existed.
52
53 PR #856 (v3.18.0) deleted the engine judge outright (CHANGELOG.md:18-20) and
54 replaced it with a three-command host-judged protocol, mandated by SKILL.md
55 LAW 11 "YOU ARE THE JUDGE" (skills/last30days/SKILL.md:233): the pipeline
56 pauses at its judgment points and persists versioned checkpoint files that
57 the hosting model judges between invocations. Leg 1 (`--discover
58 --nominate-only`) sweeps and writes the nominations bundle; the host writes a
59 judgments file; leg 2 (`--discover --judgments <file>`) resumes, deep-enriches,
60 and writes the pending report; the host writes an angles file; leg 3
61 (`--discover --finalize [--angles <file>]`) renders offline. The contracts
62 live in `skills/last30days/scripts/lib/discovery_handoff.py` (module
63 docstring, lines 1-18), the leg handlers in
64 `skills/last30days/scripts/last30days.py:1716-1986`, and the resumed pipeline
65 math in `skills/last30days/scripts/lib/pipeline.py:1525-1697`.
66
67 This doc records the five conventions that make a checkpoint protocol safe:
68 identity/TTL binding, the lossless-state-vs-capped-digest split, fail-closed
69 parsing of empty state, provenance enforcement across invocations, and
70 guarded writes plus stale-sibling invalidation.
71
72 ## Guidance
73
74 ### 1. Checkpoints are identity-bound and time-bound
75
76 Every host-authored file must echo the checkpoint's identity. Leg 1 mints a
77 random `bundle_id` (`discovery_handoff.py:270`), prints it in the digest, and
78 both host files (judgments, angles) must carry it back.
79 `_require_bundle_binding` (`discovery_handoff.py:638-673`) enforces the echo
80 and its error names BOTH ids and the cheap remedy:
81
82 ```python
83 raise HandoffContractError(
84 f"The {label} file is bound to bundle_id {file_bundle_id!r} but the "
85 f"{noun} is {bundle.bundle_id!r}. {location_label}:\n"
86 f"{_searched_lines(searched)}\n"
87 f"Correct the bundle_id field in your {label} file to "
88 f"{bundle.bundle_id!r} and re-run this same leg."
89 )
90 ```
91
92 WHY the remedy split matters: a mismatched echo means the host copied the
93 wrong id into an otherwise-current file, so the fix is edit-one-field and
94 retry THIS leg - never the expensive re-sweep (`_RESWEEP_REMEDY`,
95 `discovery_handoff.py:43`) or resume (`_RESUME_REMEDY`, lines 48-51)
96 remedies, which belong to missing/stale state. On the finalize leg the
97 message deliberately names the pending report, not the bundle, so the host's
98 retry is not misdirected (lines 653-664). `HandoffContractError` maps to
99 exit 2 in one place (`last30days.py:1984-1986`).
100
101 Time binding is a dedicated module constant with a deliberate non-reuse
102 comment (`discovery_handoff.py:32-36`):
103
104 ```python
105 # How long a nominations bundle stays valid. Deliberately a module constant
106 # and NOT the LAST30DAYS_REPORT_CACHE_TTL_SECONDS env knob: a user who
107 # lowered the report-cache TTL for drill freshness must not shrink the
108 # window a host has to author judgments.
109 DISCOVERY_HANDOFF_TTL_SECONDS = 3600.0
110 ```
111
112 Staleness is checked in the shared envelope validator via
113 `env.is_timestamp_fresh` (`discovery_handoff.py:430-436`, `env.py:121`), and
114 the pending report gets a FRESH TTL clock stamped at leg-2 write time
115 (`last30days.py:1854-1856`) because leg 2 started a new authoring window.
116 WHY: an unrelated cache knob silently shrinking the host's judging window is
117 exactly the class of cross-feature coupling a checkpoint file must not have.
118
119 ### 2. One checkpoint, two audiences, hard split: lossless resume state vs capped fenced digest
120
121 The nominations bundle serves the engine and the host, and the two halves
122 have opposite rules.
123
124 Engine half: the FULL judge pool with complete seed items, serialized
125 losslessly (`schema.py:848-852` states the contract;
126 `schema.nomination_to_dict`, `schema.py:917-932`, round-trips every item).
127 Leg 2's floor/velocity/entity math must score identically to a
128 single-process run: `_floor_survivor_records` is "Shared verbatim by
129 run_discover (one-shot) and run_discover_resume (protocol leg 2) so floor
130 semantics can never drift between the paths" (`pipeline.py:1199-1215`), and
131 velocity scores against the bundle's momentum window, never the resume-time
132 clock (`pipeline.py:1558-1561`). A capped bundle would silently starve
133 downgraded-topic scoring: host-junk rows, and heuristic-junk fallback rows
134 below the seed-source floor, never
135 get an enrichment pass, so their weak-signal velocity and the
136 seed-source-corroboration floor count are computed purely from bundle seed
137 items (`pipeline.py:1584-1593`, `rerank.py:110-136`) - truncate the items and
138 those rows under-count sources and engagement with no error anywhere. Parity
139 is test-pinned: `tests/test_discover_handoff.py:232`
140 (`test_parity_floor_and_velocity_inputs_survive_round_trip`) asserts
141 velocity, engagement totals, source sets, and entity-disambiguation inputs
142 (title + snippet) recompute identically after the round trip.
143
144 Host half: `build_host_digest` (`discovery_handoff.py:915-976`) is capped
145 (`_DIGEST_TITLE_MAX_CHARS`/`_DIGEST_SNIPPET_MAX_CHARS`/`_DIGEST_COMMENT_MAX_CHARS`,
146 lines 66-68) and its evidence lines ride inside the untrusted-content fence:
147
148 ```python
149 if evidence_lines:
150 lines.append("")
151 lines.append(rerank._fenced_untrusted_content("\n".join(evidence_lines)))
152 ```
153
154 That is the exact fence the rerank judge uses (`rerank.py:305-312`), and the
155 deleted engine judge fenced the same evidence the same way (v3.17.0's
156 `discovery_judge.py` imported `_fenced_untrusted_content` for both of its
157 prompts). Dropping the fencing during the rewrite was a caught regression;
158 the fence is now pinned by
159 `tests/test_discover_handoff.py:755`
160 (`test_digest_fences_untrusted_evidence_like_the_engine_judge`): scraped
161 titles/snippets/comments inside the fence, structural lines (ids, sources,
162 signal, bundle path) outside it. Host-supplied text going the other
163 direction is capped too - names at 96 chars, angles at 200
164 (`discovery_handoff.py:53-62`), "ported from the retired engine-judge pass"
165 because names become search queries and angles render verbatim on cards.
166
167 ### 3. Engine-written checkpoints parse strict-at-top, lenient-per-row, but FAIL CLOSED on structurally empty state
168
169 The readers are strict at the top level (readable, JSON object, right kind,
170 right schema version, bundle_id present, within TTL - all in
171 `_parse_handoff_envelope`, `discovery_handoff.py:381-437`) and lenient per
172 row: one corrupt nomination row is warned and skipped, never fatal
173 (`discovery_handoff.py:469-487`). But leniency has a floor. A non-list
174 `nominations` value raises (`discovery_handoff.py:460-466`), and zero valid
175 parsed rows raises too (`discovery_handoff.py:506-514`):
176
177 ```python
178 if not nominations:
179 # Leg 1 never writes an empty bundle (a zero-nomination sweep
180 # short-circuits with no bundle file), so an empty or all-invalid
181 # nominations array is corrupt state: fail closed, never hand the
182 # resume leg a silently empty pool.
183 raise HandoffContractError(...)
184 ```
185
186 WHY: without this, a corrupt bundle flows into leg 2 as an empty pool, which
187 floors to zero survivors and renders an authoritative-looking "Nothing solid
188 this window" brief - a green result manufactured from broken state. PR
189 #856's review caught this
190 empty-pool-renders-authoritative-empty-result failure; it is now pinned by
191 `tests/test_discover_handoff.py:439` (all rows malformed) and `:457` (empty
192 list). The invariant that makes fail-closed valid: leg 1 short-circuits a
193 zero-nomination sweep to the nothing-solid brief and writes NO bundle
194 (`last30days.py:1748-1750`), so an empty pool on disk is always corruption,
195 never a legitimate outcome.
196
197 ### 4. Cross-invocation state carries provenance, and the resume legs enforce it
198
199 Three kinds of provenance ride the checkpoint files:
200
201 Mock parity. Both checkpoints stamp `mock` at write time
202 (`discovery_handoff.py:307`, `last30days.py:1861`), and every resume leg
203 runs `_require_discover_mock_parity` (`last30days.py:1506-1532`): mock-born
204 state is rejected by a real run and real state by a `--mock` run, in both
205 cases exit 2 with a fix-the-flag or fresh-sweep remedy. WHY: "mock-born
206 state finalized by a real run would fake a real brief from fixture data, and
207 real state finalized by --mock would silently drop the round's queue write."
208
209 Sweep coverage. Leg 1 serializes the sweep's per-source outcome map into the
210 bundle (`discovery_handoff.py:308-311`), leg 2 restores it into its report
211 (`pipeline.py:1681-1692`), and leg 3 inherits it through the pending report,
212 so degraded coverage "survives the protocol instead of silently reading as
213 clean" (`discovery_handoff.py:115-118`). Every leg terminal - one-shot and
214 all three legs - exits through the ONE shared strict-exit helper,
215 `_discovery_strict_exit_code` (`last30days.py:1481-1503`, called at
216 1703, 1750, 1797, 1839, 1895, 1967), which turns `LAST30DAYS_STRICT_EXIT`
217 plus any non-clean source outcome into exit 3. The PR #856 review validated
218 this as a P1: before the fix, protocol legs silently exited 0 on degraded
219 sweeps because the status map was dropped between legs.
220
221 Store scoping. An explicit `--save-dir` is the SOLE handoff store.
222 `_search_paths` (`discovery_handoff.py:211-227`) returns "ONLY the save dir
223 when one was supplied, else the config dir", mirroring `_scoped_store_db`
224 (`last30days.py:432-437`): "a handoff file in the config dir must never
225 silently satisfy a save-dir run." The second validated P1 of the review:
226 with a fallback chain, a missing pending file in the save dir would let a
227 bare `--finalize` quietly consume the config-dir store's pending report and
228 finalize another store's run.
229
230 ### 5. Guard the write after the expensive work, and invalidate stale siblings on fresh rounds
231
232 Both engine checkpoint writes happen after minutes of paid-for work (a sweep;
233 a deep enrichment pass), so an OSError there is converted to the typed
234 contract error, never a traceback (`discovery_handoff.py:329-334` for the
235 bundle; `last30days.py:1869-1877` for the pending report):
236
237 ```python
238 except OSError as exc:
239 # A locked/read-only/full disk is the protocol's clean exit-2 path,
240 # never a traceback.
241 raise HandoffContractError(
242 f"Could not write nominations bundle {path}: {exc}"
243 ) from exc
244 ```
245
246 This is the repo's guarded-write convention (same shape as the discovery
247 queue's guarded end-of-run write) applied to checkpoints, and it is pinned by
248 `tests/test_discover_handoff.py:472`.
249
250 Fresh rounds invalidate stale siblings. A new leg-1 bundle starts a NEW
251 protocol round, so any pending report left by a prior round is deleted
252 alongside it (`last30days.py:1783-1788`); a leg 2 that ends nothing-solid
253 wrote no pending file this round, so it also unlinks any stale one
254 (`last30days.py:1830-1837`). WHY: without the unlinks, an unbound bare
255 `--finalize` inside the TTL could re-serve the PREVIOUS round's report as if
256 it belonged to the current sweep. The deliberate exception proves the rule:
257 a SUCCESSFUL finalize leaves the pending file in place
258 (`last30days.py:1905-1911`) so a retry with a corrected angles file keeps
259 working, and idempotency comes from replaying the leg-2 `run_ref` into the
260 queue (`last30days.py:1958-1964`) rather than from deleting state.
261
262 ## Why This Matters
263
264 The architecture smell this pattern removes: an external LLM API call inside
265 an engine whose invoker IS an LLM. That shape fails three ways at once. It
266 adds cost (a second metered model where a capable one is already in the
267 loop). It forks quality silently between keyed and keyless users - v3.17.0's
268 judge never raised on a missing provider, so keyless users got heuristic
269 names and no angles with zero indication anything was degraded, on the
270 skill's PRIMARY invocation path. And it produces a strictly worse judge: the
271 budget-priced engine-side model (flash-lite class, batched, no session
272 context) judged evidence the frontier host model could have judged directly.
273
274 The checkpoint protocol is the general remedy shape: pause the pipeline at
275 each judgment point, persist versioned, identity-bound, TTL-bound state, let
276 the host judge between invocations, and validate every resume so stale or
277 mismatched state becomes a clean exit 2 with a named remedy instead of
278 silent wrong output. LAW 11's framing (SKILL.md:233) is the contract in one
279 line: "You do not need an API key ... you ARE the reasoning model." The
280 one-shot path prints a loud note pointing at the protocol
281 (`pipeline.py:1421-1433`) precisely so a reasoning-model host can never
282 mistake heuristic output for a capability ceiling.
283
284 The five conventions are what make the pause safe. Splitting one pipeline
285 into three processes creates every classic distributed-state hazard in
286 miniature - stale state, cross-round state, cross-store state, fixture/real
287 crosses, silently-empty state, lost coverage warnings - and each convention
288 above closes one of them.
289
290 ## When to Apply
291
292 Apply this pattern when:
293
294 - A CLI or engine embedded in an Agent Skill needs semantic judgment
295 (naming, junk filtering, scoring, prose authoring) in the middle of an
296 otherwise deterministic pipeline - the host model is the judge; checkpoint
297 around the judgment points.
298 - You are about to add an LLM provider key, client, or "reasoning provider"
299 resolution to an engine whose invoker is already a reasoning model - that
300 is the smell; reach for the protocol instead.
301 - An existing engine-side LLM pass has a "silent heuristic fallback" - the
302 keyless majority is getting invisible degraded output today.
303
304 Do NOT apply it when:
305
306 - No reasoning model is in the loop. The one-shot cron/scripted path keeps
307 the single-process pipeline deliberately (`run_discover`,
308 `pipeline.py:1384`, and the degradation rule in SKILL.md:398): a
309 checkpoint pause with nobody to judge is just a hang.
310 - The judgment is expressible as a deterministic rule - the junk-shape
311 heuristics and the confidence floor stayed engine-side because they need
312 no model at all.
313
314 ## Examples
315
316 The three-command sequence as SKILL.md ships it (skills/last30days/SKILL.md:318-399),
317 one identical `--save-dir` threaded through all three legs:
318
319 ```bash
320 # Leg 1 - sweep and nominate (global trending; domain runs pass the domain
321 # phrase as the --discover argument on this leg only):
322 python3 scripts/last30days.py --discover --nominate-only \
323 --save-dir="$HOME/Documents/Last30Days"
324 # stdout: judging digest + bundle path + bundle_id. Host reads the bundle
325 # file, then writes judgments.json:
326 # {"bundle_id": "<echoed>", "judgments": [
327 # {"id": "n1", "name": "Gemma 4 chat templates", "junk": false, "worthiness": 85},
328 # {"id": "n2", "name": "Beginner asks how to deploy", "junk": true, "worthiness": 10}]}
329
330 # Leg 2 - resume with judgments; deep per-topic research (several minutes):
331 python3 scripts/last30days.py --discover --judgments judgments.json \
332 --save-dir="$HOME/Documents/Last30Days"
333 # stdout ends with angle inputs keyed by surviving id. Host writes
334 # angles.json: {"bundle_id": "<same>", "angles": [
335 # {"id": "n1", "podcast": "<hook>", "x_article": "<hook>"}]}
336
337 # Leg 3 - finalize offline: apply angles, render, record the topic queue.
338 python3 scripts/last30days.py --discover --finalize --angles angles.json \
339 --emit=compact --save-dir="$HOME/Documents/Last30Days"
340 ```
341
342 Failure-mode walkthrough (mismatched then stale checkpoint):
343
344 1. The host echoes a bundle_id from an earlier round into judgments.json and
345 runs leg 2. `_require_bundle_binding` raises; the CLI prints
346 `[last30days] The judgments file is bound to bundle_id 'aaaa...' but the
347 current nominations bundle is 'bbbb...'` plus the searched location, and
348 exits 2. Remedy as printed: correct the `bundle_id` field and re-run leg 2.
349 The expensive sweep is NOT redone - the bundle on disk is still current.
350 2. The host instead waits 90 minutes before judging. The envelope check
351 (`discovery_handoff.py:430-436`) finds `generated_at` outside the 3600s
352 TTL and exits 2: the bundle "is stale ... the momentum window it captured
353 has moved on. Run a fresh `--discover --nominate-only` re-sweep." Here the
354 expensive leg IS the remedy, because the state itself expired - the
355 protocol never asks for the expensive path when a cheap edit fixes the
356 problem, and never accepts cheap edits when the data has aged out.
357
358 The deterministic end-to-end twin of the whole sequence is pinned in CI:
359 `tests/test_discover_mode.py:2439`
360 (`test_discovery_cli_full_mock_protocol_three_legs_end_to_end`).
361
362 ## Related
363
364 - `docs/solutions/architecture-patterns/discovery-topic-queue-design-conventions.md` -
365 same feature family, the queue side: the persistent topic queue leg 3
366 writes into (idempotently, under the leg-2 `run_ref`), including the
367 guarded-write convention this protocol reuses.
368 - `docs/solutions/design-patterns/ranked-output-confidence-floor-honest-empty-state.md` -
369 the confidence-floor semantics the protocol preserves verbatim across the
370 process split (`_floor_survivor_records` shared by both paths), including
371 seed-source corroboration for junk shapes.
372 - `docs/solutions/logic-errors/non-daemon-executor-threads-defeat-wall-clock-budget.md` -
373 the enrichment wall-clock budget pattern leg 2's deep tier extends
374 (`RESUME_DEEP_ENRICH_BUDGET_SECONDS` 450s via
375 `LAST30DAYS_ENRICH_BUDGET_SECONDS`, `pipeline.py:1492-1508`; workers stay
376 daemon threads and never touch disk - the pending report is ONE post-loop
377 write from the main thread, `last30days.py:1867-1871`).
378 - PR #856 (protocol, engine-judge removal), PR #852 (the v3.17.0 engine
379 judge this replaced), CHANGELOG.md v3.18.0 / v3.17.0 entries.
380 - SKILL.md LAW 11 and the Step 1 DISCOVERY branch (skills/last30days/SKILL.md:233, 314-399).
381
381 lines MARKDOWN