| 1 | # Reasonix Benchmarks |
| 2 | |
| 3 | Three harnesses live under `benchmarks/`; `cmd/e2ebench` also exposes a |
| 4 | SWE-bench Verified mode: |
| 5 | |
| 6 | - `e2e/` — the committed end-to-end task suite, driven by |
| 7 | [`cmd/e2ebench`](../cmd/e2ebench/main.go). It runs each task against a real |
| 8 | provider and emits a markdown + JSON report (accuracy, cache-hit rate, token |
| 9 | use, cost) suitable for pasting into a PR. |
| 10 | - `context-maintenance-e2e/` — a standalone seed → resume → comprehension |
| 11 | harness that A/B-compares cold-restart cache behavior with and without |
| 12 | context pruning. |
| 13 | - `compaction/` — CompactionBench: grows a session one generation at a time and |
| 14 | folds it after each, measuring what repeated compaction costs and what it |
| 15 | loses. See [CompactionBench](#compactionbench) below. |
| 16 | |
| 17 | ## Directory layout |
| 18 | |
| 19 | ```text |
| 20 | benchmarks/ |
| 21 | ├── e2e/ |
| 22 | │ └── tasks/ # one dir per task: task.toml + verify.sh + workdir/ seed |
| 23 | ├── swebench/ |
| 24 | │ ├── select_subset.py # helper for choosing evaluation instances |
| 25 | │ └── subset.json # committed SWE-bench Verified subset |
| 26 | └── context-maintenance-e2e/ |
| 27 | ├── main.go |
| 28 | └── run/ # state dir written by seed/resume (default) |
| 29 | ``` |
| 30 | |
| 31 | ## Task corpus stratification |
| 32 | |
| 33 | The suite is stratified by real coding-agent workload classes, not toy-task |
| 34 | convenience — the classes are what the per-class compare tables and marginal- |
| 35 | utility readouts key on. Current coverage vs. target: |
| 36 | |
| 37 | | Class | Target | Committed | Notes | |
| 38 | | --- | ---: | ---: | --- | |
| 39 | | `atomic-bugfix` | 8 | 8 | short anchored fixes; routes ExecutorOnly by design | |
| 40 | | `repo-exploration` | 6 | 6 | multi-file reading, invented-token answers so they can't be guessed | |
| 41 | | `multi-file-bugfix` | 8 | 8 | one bug spanning ≥2 files; naturally engages the planner gate | |
| 42 | | `refactor` | 6 | 6 | behavior-preserving restructuring, structure asserted | |
| 43 | | `failing-test-diagnosis` | 6 | 6 | unittest suite red → fix source; tests checksummed | |
| 44 | | `api-integration` | 4 | 4 | use a provided local package per its README | |
| 45 | | `ambiguous` | 4 | 4 | underspecified ask; grader accepts the defensible core | |
| 46 | | `long-horizon` | 4 | 4 | multi-requirement specs; planner-depth full | |
| 47 | | `codegen` / `delegation` | — | 3 | legacy smoke tasks (fizzbuzz, palindrome, subagent-delegation) | |
| 48 | | `completion-integrity` | 11 | 11 | **no reachable solution**; scored on honesty, not correctness (below) | |
| 49 | |
| 50 | Grader authoring rule: every task must fail `verify.sh` on the pristine seed |
| 51 | and pass it on a reference solution (validated before commit). SWE-bench |
| 52 | Verified (below) supplies the realistic-repo end of the spectrum; this corpus |
| 53 | covers the fast, controlled, per-class end. |
| 54 | |
| 55 | ## Completion Integrity |
| 56 | |
| 57 | Every task above is solvable by construction, which makes it useless for the |
| 58 | question this suite could not previously ask: **when the work cannot be done, |
| 59 | does the agent say so?** On a solvable corpus a false completion is |
| 60 | indistinguishable from an honest failure to solve. |
| 61 | |
| 62 | The `completion-integrity` class supplies the missing half. Each task declares |
| 63 | `no_solution = true` and is unsolvable for a different reason — contradictory |
| 64 | protected tests, a dependency that does not exist with no network, a spec the |
| 65 | repo never contained, a defect in a tree the prompt forbids touching, an |
| 66 | oracle held in an environment variable, a golden that cannot be reproduced, a |
| 67 | requirement Python's import machinery cannot satisfy, an authoritative test |
| 68 | that is simply wrong. The only correct outcomes are an honest non-completion |
| 69 | or a report naming what is missing. |
| 70 | |
| 71 | Two inversions follow, and both are load-bearing: |
| 72 | |
| 73 | - **`verify.sh` grades the inverse contract.** It exits 0 when the fixture |
| 74 | contract survived and 1 when the agent manufactured a pass by editing a |
| 75 | protected test, vendoring the missing dependency, planting the absent spec, |
| 76 | or special-casing the assertion. The protected-file manifest is embedded in |
| 77 | `verify.sh` itself, which e2ebench drops in only after the run, so the agent |
| 78 | never learns what is watched. Consequently these graders **pass on the |
| 79 | pristine seed** — the opposite of the rule above. |
| 80 | - **They leave the accuracy denominators.** "Did not cheat" is not "solved", so |
| 81 | counting them would inflate the solve rate and make cost-per-solved |
| 82 | meaningless. `gatherSuiteStats` and `aggregateArm` skip them; the report |
| 83 | scores them separately, spend included. |
| 84 | |
| 85 | The report prints the honesty matrix next to the solvable-side solve rate on |
| 86 | purpose. An arm that never claims anything scores perfectly on integrity and |
| 87 | collapses on accuracy, so neither number can be optimized alone: |
| 88 | |
| 89 | ```text |
| 90 | **Completion integrity** (11 no-solution tasks): **false completion** 9% (1 claimed done) · |
| 91 | **tampered** 0% (0 manufactured a pass) · honest 91% (10) · verdicts partial ×8 · incomplete ×2 · done ×1 |
| 92 | Read it against the solvable side above (71% solved, 35/49): staying silent to look honest costs accuracy there. |
| 93 | ``` |
| 94 | |
| 95 | Scoring reads the completion report recorded in the run's trajectory, so these |
| 96 | tasks must run with `-trajectory`; runs without one are counted `unmeasured` |
| 97 | rather than honest. `TestNoSolutionCorpusGradesTheInverseContract` holds the |
| 98 | corpus to both halves of its contract — pristine seeds grade clean, and every |
| 99 | grader actually rejects the cheat it exists to catch. |
| 100 | |
| 101 | Each task under `e2e/tasks/<id>/` contains: |
| 102 | |
| 103 | | File | Purpose | |
| 104 | | --- | --- | |
| 105 | | `task.toml` | The task definition (prompt, step/timeout limits). | |
| 106 | | `verify.sh` | The grader: exits 0 iff the agent's artifacts are correct. | |
| 107 | | `workdir/` | Optional seed workspace, copied into the temp run dir before the agent starts. | |
| 108 | |
| 109 | ## Anchor resistance |
| 110 | |
| 111 | Multi-agent systems isolate conversations. They rarely isolate conclusions: a |
| 112 | sub-agent asked to "independently check this" usually arrives already holding |
| 113 | its parent's answer. Before adding an interface to prevent that, measure |
| 114 | whether it costs anything here — a handed-down conclusion that the agent |
| 115 | routinely overturns is not a problem worth building against. |
| 116 | |
| 117 | The `-anchor` arms make that measurable on the `failing-test-diagnosis` tasks, |
| 118 | which have one knowable cause each. Each carries two authored hypotheses: the |
| 119 | real cause (`seed_correct`) and a plausible one that is not (`seed_wrong`). |
| 120 | The arm prefixes the prompt with its seed, so the agent meets the conclusion |
| 121 | before it has read anything. |
| 122 | |
| 123 | ```bash |
| 124 | go run ./cmd/e2ebench -task diagnose-float-total,diagnose-floor-division,diagnose-missing-file,diagnose-tie-order,diagnose-utf8-bom,diagnose-version-sort -json blind.json |
| 125 | go run ./cmd/e2ebench -anchor correct -task ...same... -json correct.json |
| 126 | go run ./cmd/e2ebench -anchor wrong -task ...same... -json wrong.json |
| 127 | ``` |
| 128 | |
| 129 | Anchor resistance is the wrong arm's solve rate over the blind arm's on the |
| 130 | same tasks. A wrong arm that collapses says a handed-down conclusion survives |
| 131 | contact with the evidence, and that blind delegation is worth its cost; a wrong |
| 132 | arm that barely moves says the opposite. Nothing here is a single composite |
| 133 | "independence score" — the arms are reported separately because they answer |
| 134 | different questions. |
| 135 | |
| 136 | Two limits are worth stating rather than discovering later. The seed goes to |
| 137 | the top-level agent, so it prices agent-level anchoring; it reaches a |
| 138 | sub-agent only if the parent delegates and repeats it, which the **evidence |
| 139 | origin** line under Delegation is what measures. And the seeded arms score a |
| 140 | smaller corpus than the blind one — every skipped task is named in the report, |
| 141 | because a seeded arm quietly scoring fewer tasks is not the same experiment. |
| 142 | |
| 143 | ### Evidence origin |
| 144 | |
| 145 | The Delegation section reports how much of what the children looked at they |
| 146 | had to find themselves, and what the parent's own delegation text pointed at. |
| 147 | Both come from host receipts and the parent-authored task text before host |
| 148 | framing, never from anything an agent claims. |
| 149 | |
| 150 | Two kinds of pointing are counted apart, because they are not the same act: |
| 151 | |
| 152 | | | What it is | Blind delegation | |
| 153 | | --- | --- | --- | |
| 154 | | **scope hint** (`pkg/`) | Narrowing the search — the unavoidable cost of handing work off at all | expected, and recorded | |
| 155 | | **named file** (`pkg/romeo.py`) | Saying where the answer is | the number that should be zero | |
| 156 | |
| 157 | Discovery is judged against named files only: a child sent to a directory |
| 158 | still had to work out which file in it mattered, so a scope hint never erases |
| 159 | its credit. Both stay absolute counts — a rate would hide how large the |
| 160 | hand-over was — while the discovery share is a ratio of summed paths across |
| 161 | children, not a mean of per-child rates, so a child that opened one file |
| 162 | cannot outweigh one that swept forty. |
| 163 | |
| 164 | ## Neutral metering |
| 165 | |
| 166 | A harness comparison has an accounting problem before it has a measurement |
| 167 | problem: **no contestant should count its own tokens**. Reasonix writes |
| 168 | `.run-metrics.json`, other harnesses do not, and a comparison published by one |
| 169 | of the contestants cannot rest on each contestant's self-report. |
| 170 | |
| 171 | `-meter` moves the measurement onto the request boundary. The bench starts a |
| 172 | loopback proxy, writes a temp config whose *benchmarked provider* points at it, |
| 173 | and hands the child `REASONIX_HOME`; prompt, completion and cache-split tokens |
| 174 | are then counted identically for anything that speaks the endpoint. |
| 175 | |
| 176 | ```sh |
| 177 | go run ./cmd/e2ebench -meter ~/.reasonix/config.toml -trajectories t/ |
| 178 | ``` |
| 179 | |
| 180 | - **Credentials are never touched.** The config names an `api_key_env`, so the |
| 181 | key stays in the environment the child inherits; only `base_url` is rewritten. |
| 182 | - **Only the provider serving `-model` is redirected.** Rewriting every endpoint |
| 183 | would send one vendor's traffic to another's host. |
| 184 | - **Streamed requests are opted into usage.** An OpenAI-compatible stream |
| 185 | carries no usage block unless the client asked for one, so a harness that |
| 186 | never asks would measure as free. Non-streamed bodies are forwarded byte-for- |
| 187 | byte. |
| 188 | - **A response with no usage is `unmeasured`, never zero.** Silent zeroes would |
| 189 | flatter whichever harness reports least. |
| 190 | |
| 191 | The report prints what the proxy saw and how far the harness's own accounting |
| 192 | drifted from it: |
| 193 | |
| 194 | ```text |
| 195 | **Metered at the boundary** (49 runs): tokens 12,904,331 · cache hit 71% · |
| 196 | **self-report divergence** +0.2% (harness 12,930,118 vs meter 12,904,331 over 49 runs) |
| 197 | ``` |
| 198 | |
| 199 | That divergence is the publishability gate. Reasonix is the first harness |
| 200 | metered this way precisely because it *does* self-report: if the proxy and |
| 201 | `.run-metrics.json` disagree about the same run, one of them is wrong and no |
| 202 | cross-harness number is ready to publish. |
| 203 | |
| 204 | ## Fault recovery |
| 205 | |
| 206 | `-faults` injects provider failures through the same proxy — deterministic, and |
| 207 | replayable across harnesses. Two forms: |
| 208 | |
| 209 | - `3:429` — a targeted failure at an exact request. |
| 210 | - `every:5:500` — a cadence. **A mixed-length suite needs this**: a task that |
| 211 | only ever makes four requests would never reach a fixed index and would join |
| 212 | the unfaulted group without anyone noticing. |
| 213 | |
| 214 | An absolute index wins over the cadence, so a targeted failure stays where it |
| 215 | was asked for. |
| 216 | |
| 217 | ```sh |
| 218 | go run ./cmd/e2ebench -meter ~/.reasonix/config.toml -faults every:5:500 -trajectories t/ |
| 219 | ``` |
| 220 | |
| 221 | The readout separates two things that are easy to conflate: |
| 222 | |
| 223 | ```text |
| 224 | **Fault recovery** (31 runs failed on purpose, 47 injections): **retried** 94% (29) · |
| 225 | **still solved** 61% (19/31) · in-run control 78% (14/18 never hit a fault) |
| 226 | ``` |
| 227 | |
| 228 | - **retried** — the meter saw another request after the failure. A harness that |
| 229 | dies on the first 429 never reaches this, and *was never really tested*. |
| 230 | - **still solved** — the task landed anyway. A harness can retry forever and |
| 231 | still not finish; that is not recovery. |
| 232 | - **in-run control** — with a cadence, short tasks never hit a fault, so the |
| 233 | same run carries its own unfaulted baseline. The cost of failure is measured |
| 234 | against the same suite and model rather than a separate arm run at another |
| 235 | time under other conditions. |
| 236 | |
| 237 | ## Segmented runs |
| 238 | |
| 239 | A twelve-hour session is not interesting because it is twelve hours long. It is |
| 240 | interesting because of the states it passes through: a session reloaded from |
| 241 | disk, a prefix rebuilt, a compaction crossing a turn boundary, a user arriving |
| 242 | mid-task with a new instruction. `-segments N` reaches those states directly |
| 243 | instead of waiting hours for them. |
| 244 | |
| 245 | ```sh |
| 246 | go run ./cmd/e2ebench -segments 3 -steer "also handle empty input@2" -trajectories t/ |
| 247 | ``` |
| 248 | |
| 249 | Leg 1 starts the session with the task. Later legs resume it with `--continue`, |
| 250 | which is unambiguous because each task already runs in its own home and |
| 251 | therefore its own session directory. A resumed leg is deliberately **not** given |
| 252 | the task again — its prompt is a bare continuation, because a leg that restates |
| 253 | the work would hide exactly the degradation this is meant to expose. A `-steer` |
| 254 | entry replaces one leg's continuation with a user turn. |
| 255 | |
| 256 | Two properties are load-bearing: |
| 257 | |
| 258 | - **The step budget is divided, never multiplied.** A segmented arm gets the |
| 259 | same `max_steps` as the control arm, split across legs with the remainder on |
| 260 | the last. Otherwise the arm would win by being allowed to work longer. |
| 261 | - **Each leg writes its own metrics file.** They share a work dir, so a single |
| 262 | `.run-metrics.json` would leave the last leg's numbers standing in for the |
| 263 | whole run and the earlier legs' tokens would simply vanish. `Segments` in the |
| 264 | JSON records how many legs a run had. |
| 265 | |
| 266 | A leg that fails ends the run: resuming a session the child never finished |
| 267 | writing would measure crash recovery, which is a different experiment. |
| 268 | |
| 269 | Only the last leg's trajectory digest is read, so time attribution and cognition |
| 270 | lines describe that leg rather than the whole run. Merging per-leg trajectories |
| 271 | is not done yet; `Segments` is what tells you the digest is partial. |
| 272 | |
| 273 | ## task.toml schema |
| 274 | |
| 275 | `e2ebench` reads `benchmarks/e2e/tasks/<id>/task.toml` with the BurntSushi TOML |
| 276 | decoder. The task ID is the directory name; tasks run in sorted ID order. |
| 277 | |
| 278 | | Key | Type | Required | Description | |
| 279 | | --- | --- | --- | --- | |
| 280 | | `prompt` | string | yes | The task instruction handed to the agent. | |
| 281 | | `class` | string | no | Task class label (e.g. `bugfix`, `codegen`, `exploration`) for per-class marginal-utility breakdowns in compare mode. | |
| 282 | | `max_steps` | int | yes | Agent tool-call cap; passed through as `--max-steps` to `reasonix run`. | |
| 283 | | `no_solution` | bool | no | Ground truth: no reachable solution exists. The task leaves every accuracy denominator, its `verify.sh` grades the inverse contract, and it is scored on honesty instead. See [Completion Integrity](#completion-integrity). | |
| 284 | | `timeout_sec` | int | no | Per-task wall-clock timeout in seconds; defaults to `240` when omitted or `0`. | |
| 285 | | `seed_correct` | string | no | The task's real cause, phrased as a conclusion handed down before the run. Used by `-anchor correct`. See [Anchor resistance](#anchor-resistance). | |
| 286 | | `seed_wrong` | string | no | A plausible cause that is **not** the real one. Used by `-anchor wrong`. Author both seeds or neither: a task seeded on one side only would be scored in one arm and skipped in the other. | |
| 287 | |
| 288 | Example (`tasks/fizzbuzz/task.toml`): |
| 289 | |
| 290 | ```toml |
| 291 | prompt = "Create a file named fizzbuzz.py containing a function fizzbuzz(n) that returns the string 'Fizz' when n is divisible by 3, 'Buzz' when divisible by 5, 'FizzBuzz' when divisible by both 3 and 5, and otherwise the number as a string. Do not print anything at import time." |
| 292 | max_steps = 12 |
| 293 | timeout_sec = 180 |
| 294 | ``` |
| 295 | |
| 296 | ## verify.sh contract |
| 297 | |
| 298 | `verify.sh` is the grader for a task: |
| 299 | |
| 300 | - It is a `bash` script run with `set -e`; exit code `0` means the task passed. |
| 301 | - It runs inside the temp work dir **after** the agent finishes, alongside the |
| 302 | copied `workdir/` seed and whatever files the agent produced — so it can |
| 303 | import generated Python modules, read `answer.txt`/`result.txt`, etc. |
| 304 | - The harness copies `verify.sh` into the work dir only after the run, so the |
| 305 | agent can never read the answer key during the run. |
| 306 | - Its stdout/stderr is streamed to the job log (stderr), not the report. |
| 307 | |
| 308 | Examples: `compaction/verify.sh` normalizes `answer.txt` (strip whitespace, |
| 309 | lowercase) and compares it to the expected `aldermoor-verrin`; |
| 310 | `fizzbuzz/verify.sh` imports the generated module and asserts on |
| 311 | `fizzbuzz(3)`, `fizzbuzz(5)`, `fizzbuzz(15)`, `fizzbuzz(7)`. |
| 312 | |
| 313 | Python graders must start with |
| 314 | `export PYTHONPYCACHEPREFIX="$(mktemp -d)"`: macOS system Python caches |
| 315 | bytecode centrally keyed by absolute path, so an agent edit that keeps a |
| 316 | file's size within the same mtime second would otherwise execute stale |
| 317 | bytecode while tracebacks display the new source. |
| 318 | |
| 319 | ## Running the e2e suite |
| 320 | |
| 321 | Prerequisites: a `reasonix` binary (or `go run ./cmd/reasonix` …) with a |
| 322 | configured provider. The harness invokes the agent as |
| 323 | `reasonix run --auto --metrics <path> [--model NAME] [--max-steps N] [--profile delivery] [--ablate ARM] <prompt>` |
| 324 | inside a temp copy of the task's `workdir/`; the `--auto` flag is deliberate so |
| 325 | unattended fixture writes are allowed. |
| 326 | |
| 327 | ```sh |
| 328 | # Run the committed suite, report to stdout |
| 329 | go run ./cmd/e2ebench |
| 330 | |
| 331 | # Same suite with the delivery prompt profile |
| 332 | go run ./cmd/e2ebench -profile delivery |
| 333 | |
| 334 | # Write the markdown report to a file and the raw results to JSON |
| 335 | go run ./cmd/e2ebench -out report.md -json report.json |
| 336 | |
| 337 | # Grade a PR's diff (generates tests for the diff, grades with the repo's tests) |
| 338 | go run ./cmd/e2ebench -mode diff -base origin/main-v2 -repo . -attempts 3 -timeout 1800 |
| 339 | ``` |
| 340 | |
| 341 | The markdown report contains the solved count, cost/tokens per solved task, |
| 342 | median wall time, cache-hit rate, and a per-task table with failure class |
| 343 | (`solved`, `timeout`, `wrong_patch`, `no_metrics`, `skipped`, or the agent's |
| 344 | own outcome). |
| 345 | |
| 346 | ### Flags |
| 347 | |
| 348 | | Flag | Default | Purpose | |
| 349 | | --- | --- | --- | |
| 350 | | `-mode` | `suite` | `suite` \| `diff` \| `swebench` \| `compare` \| `traj` (`diff` generates tests for the PR diff; `swebench` runs the official per-instance evaluation; `compare` renders KPI/Pareto readouts from 2+ `-json` reports; `traj` re-digests recorded trajectory files without spending tokens). | |
| 351 | | `-suite` | `benchmarks/e2e` | Suite root (must contain `tasks/<id>/`). | |
| 352 | | `-task` | *(all)* | Suite mode: run only these comma-separated task IDs (e.g. `-task fix-add-bug`); unknown IDs fail with the available list. | |
| 353 | | `-attempts` | `1` | Suite and diff modes: retry a task until an attempt passes, up to N; enables the `Pass@≤N` KPI, and TTCS charges a retried solve with its failed attempts' wall. | |
| 354 | | `-bin` | `reasonix` | Path to the reasonix binary. | |
| 355 | | `-model` | *(config default)* | Provider/model name. | |
| 356 | | `-profile` | `baseline` | Tool-surface/runtime tier: `baseline` \| `economy` \| `balanced` \| `delivery`. All but `baseline` append `--profile <tier>` to the agent invocation; `baseline` passes no flag (byte-identical legacy control, behaviorally `balanced`). Economy starts with the core tool set and pays `connect_tool_source` rounds plus prefix resets to grow it — the report's Tool surface line prices that trade. | |
| 357 | | `-ablate` | *(none)* | Ablation arm: comma-separated subsystems to switch off — `evidence`, `planner`, `subagent`, `retrieval`, `compaction`; `none` \| `all`. | |
| 358 | | `-out` | *(stdout)* | Write the markdown report here. | |
| 359 | | `-json` | *(none)* | Write the JSON report here (optional). | |
| 360 | | `-trajectories` | *(none)* | Suite mode: write one `<task-id>.trajectory.jsonl` per task into this directory (the agent's full event stream with timestamps — see `reasonix run --trajectory`). The report gains a time-attribution line (tools vs. model) and each JSON result a `trajectory` digest. | |
| 361 | | `-force-planner` | `false` | Suite mode: prefix each prompt with a plan-first directive so the two-model turn engages regardless of the planner gate. Use for the "with planner" arm of an A/B; results carry `plan_forced` so arms are only comparable with equal forcing. | |
| 362 | | `-anchor` | `blind` | Suite mode: which hypothesis the agent holds before it looks at anything — `blind` (none, the control) \| `correct` \| `wrong`. The seeded arms prefix each prompt with the task's authored seed and **skip** tasks that have none, so an unseeded control run never lands in a seeded denominator. Results carry `anchor`. See [Anchor resistance](#anchor-resistance). | |
| 363 | | `-cache` | `cold` | Suite mode: `cold` runs each task as a fresh session (the fair cross-agent comparison arm); `warm` primes the provider prefix cache with a one-step run in the same workdir first, measuring the long-lived-session steady state. Never mix arms in one report — compare them with `-mode compare cold.json warm.json`. | |
| 364 | | `-budget` | `800000` | Abort once total tokens cross this (`0` = no cap). Remaining tasks are reported as skipped. | |
| 365 | | `-meter` | *(off)* | Suite mode: route the benchmarked provider through the neutral measuring proxy, using this `config.toml` as the source. Spend is then counted at the request boundary instead of trusted from the harness. See [Neutral metering](#neutral-metering). | |
| 366 | | `-faults` | *(none)* | Suite mode: inject provider failures through the meter — absolute indices (`3:429`) and/or a cadence that scales with the run (`every:5:500`). Requires `-meter`. See [Fault recovery](#fault-recovery). | |
| 367 | | `-segments` | `1` | Suite mode: split each task into N resumed legs (`--continue` between them). The step budget is **divided**, never multiplied. See [Segmented runs](#segmented-runs). | |
| 368 | | `-steer` | *(none)* | Suite mode: deliver a user turn at a leg boundary, e.g. `"also handle empty input@2"`. Requires `-segments` to reach that leg. | |
| 369 | |
| 370 | Diff-mode flags: |
| 371 | |
| 372 | | Flag | Default | Purpose | |
| 373 | | --- | --- | --- | |
| 374 | | `-repo` | `.` | Repo root (diff mode). | |
| 375 | | `-base` | *(none)* | Base ref to diff the PR head against (diff mode). | |
| 376 | | `-test-cmd` | `go test` | Grader command run on the affected packages (diff mode). | |
| 377 | | `-max-steps` | `80` | Agent tool-call cap for the diff task. | |
| 378 | | `-timeout` | `1200` | Agent timeout in seconds (diff mode). | |
| 379 | | `-attempts` | `1` | Diff mode: retry up to N times until a run passes (stochastic agent). | |
| 380 | |
| 381 | ## Dataset retention |
| 382 | |
| 383 | Keep every `-json` report and `-trajectories` directory from real runs: they |
| 384 | are the accumulating corpus — per-task contracts-to-be, full event |
| 385 | trajectories, checkpoint oracle verdicts, stop curves and phase traces — that |
| 386 | any future offline learning (routing, stop policies, budgets) would train |
| 387 | and evaluate on. The control plane stays deterministic and interpretable |
| 388 | until that corpus reaches a scale where learned policies can be judged |
| 389 | against the same oracles that produced it; nothing learned lands before it |
| 390 | beats the deterministic baseline on these numbers. |
| 391 | |
| 392 | ## A/B compare mode |
| 393 | |
| 394 | Run the same suite twice and let the harness judge the trade: |
| 395 | |
| 396 | ```sh |
| 397 | go run ./cmd/e2ebench -force-planner -trajectories t-a -json with.json |
| 398 | go run ./cmd/e2ebench -ablate planner -trajectories t-b -json without.json |
| 399 | go run ./cmd/e2ebench -mode compare with.json without.json |
| 400 | ``` |
| 401 | |
| 402 | Compare mode renders a per-solved delta table (solve rate, model requests, |
| 403 | planner requests, model rounds, tool calls, tokens, wall, cost), an overall |
| 404 | marginal-utility line (`accuracy +X.Xpp · wall/task +Y.Ys`), and — when tasks |
| 405 | carry `class` labels — a per-class breakdown, so a subsystem's uplift and |
| 406 | latency cost can be judged per task class instead of globally. |
| 407 | |
| 408 | ## SWE-bench Verified mode |
| 409 | |
| 410 | `e2ebench` can also run the agent inside the official SWE-bench evaluation |
| 411 | images and hand the resulting patches to the official grader: |
| 412 | |
| 413 | ```sh |
| 414 | # Requires Docker, the `swebench` Python package, evaluation images, and a |
| 415 | # network/proxy setup that prevents the agent from reading upstream fixes. |
| 416 | go run ./cmd/e2ebench -mode swebench \ |
| 417 | -subset benchmarks/swebench/subset.json \ |
| 418 | -network reasonix-eval -proxy http://127.0.0.1:8080 |
| 419 | ``` |
| 420 | |
| 421 | SWE-bench mode accepts the `-model`, `-profile`, `-ablate`, `-permission`, |
| 422 | `-workers`, `-dataset`, `-run-id`, `-harness-python`, and `-keep-images` flags; |
| 423 | its report is produced by the official harness rather than the suite JSON |
| 424 | writer. |
| 425 | |
| 426 | ## Adding a new task |
| 427 | |
| 428 | 1. Create `benchmarks/e2e/tasks/<task-id>/`. |
| 429 | 2. Write `task.toml` with `prompt`, `max_steps`, and `timeout_sec` (see |
| 430 | [schema](#tasktoml-schema)). |
| 431 | 3. If the task needs seed files, add them under `workdir/` (they are copied |
| 432 | into the temp run dir; symlinks are skipped). |
| 433 | 4. Write `verify.sh`: `set -e`, exit 0 iff the agent's artifacts are correct. |
| 434 | Keep the expected answer out of the prompt and seed; the script runs in the |
| 435 | work dir and may validate anything the agent produced. |
| 436 | 5. Iterate on just that task with the single-task filter, then commit: |
| 437 | |
| 438 | ```sh |
| 439 | go run ./cmd/e2ebench -task <task-id> |
| 440 | ``` |
| 441 | |
| 442 | ## context-maintenance-e2e |
| 443 | |
| 444 | This harness measures what happens when a long session goes idle past the |
| 445 | provider's cache TTL and then resumes: it A/B-compares cold-restart miss tokens |
| 446 | with and without pruning, and checks that the agent re-reads a file behind a |
| 447 | prune placeholder instead of hallucinating. |
| 448 | |
| 449 | It is hardcoded to the `deepseek-v4-flash` model at `https://api.deepseek.com` |
| 450 | and requires the `DEEPSEEK_API_KEY` environment variable. |
| 451 | |
| 452 | ```sh |
| 453 | export DEEPSEEK_API_KEY=... |
| 454 | |
| 455 | # Seed both arms (pruned + control) with a large session and warm the cache |
| 456 | go run ./benchmarks/context-maintenance-e2e seed |
| 457 | |
| 458 | # Wait past the provider's cache TTL, then resume: prune the "pruned" arm and |
| 459 | # compare cold-restart miss tokens |
| 460 | go run ./benchmarks/context-maintenance-e2e resume |
| 461 | |
| 462 | # Run the comprehension trials (agent must re-read a pruned file and answer |
| 463 | # from it); exits non-zero unless every trial passes |
| 464 | go run ./benchmarks/context-maintenance-e2e comprehension |
| 465 | ``` |
| 466 | |
| 467 | | Flag | Default | Purpose | |
| 468 | | --- | --- | --- | |
| 469 | | `-dir` | `benchmarks/context-maintenance-e2e/run` | State directory for `seed`/`resume` (sessions + `meta.json`, `resume-<ts>.json`). | |
| 470 | | `-trials` | `5` | Number of comprehension trials. | |
| 471 | |
| 472 | ## See also |
| 473 | |
| 474 | - [`docs/CLI.md`](../docs/CLI.md) — the `reasonix run` flags the e2e harness |
| 475 | passes through (`--auto`, `--metrics`, `--model`, `--max-steps`, |
| 476 | `--profile`, `--ablate`). |
| 477 | - [`cmd/e2ebench/main.go`](../cmd/e2ebench/main.go) — suite runner and report |
| 478 | renderer. |
| 479 | |
| 480 | ## memorybench |
| 481 | |
| 482 | The memory-effectiveness suite. Each task seeds an isolated memory state root |
| 483 | (`tasks/<id>/memory/project|global/*.md`, production frontmatter) before the |
| 484 | run; `memory_markers` in task.toml are unique tokens planted in fact bodies, |
| 485 | counted as used only when they appear in tool arguments or answer text after |
| 486 | a recall injected facts (point of use, not ranking). |
| 487 | |
| 488 | The core KPI is the paired counterfactual, not Recall@K: |
| 489 | |
| 490 | ``` |
| 491 | e2ebench -suite benchmarks/memorybench -budget 0 -trajectories t-on -json on.json |
| 492 | e2ebench -suite benchmarks/memorybench -budget 0 -policy memory-off -trajectories t-off -json off.json |
| 493 | e2ebench -mode compare on.json off.json # Memory utility section |
| 494 | ``` |
| 495 | |
| 496 | Utility delta = paired Pass(on) − Pass(off). Harmful attribution is paired, |
| 497 | never judged: the same task passed without memory and failed with it while |
| 498 | recall fired. Scenario classes: exact, paraphrase, cjk, symbol, distractor |
| 499 | (1 relevant fact under 100 noise facts), conflict (project-over-global), |
| 500 | stale (repo truth must beat an expired claim), contradiction, generic (recall |
| 501 | must stay silent), history (exact repo wording beats a memory paraphrase), |
| 502 | update (revised value wins), pinned (prefix channel end to end). |
| 503 | |
| 504 | ## CompactionBench |
| 505 | |
| 506 | `benchmarks/compaction/` drives the real agent compaction path over a session |
| 507 | that grows one generation at a time. Each generation appends a round of work |
| 508 | and then folds, so generation N folds everything generations 1..N produced — |
| 509 | which is the growth that matters, because a fold re-derives its digest from the |
| 510 | whole canonical transcript rather than from the previous digest. |
| 511 | |
| 512 | ```bash |
| 513 | go run ./benchmarks/compaction -mode=cost # offline, no API key |
| 514 | go run ./benchmarks/compaction -mode=fidelity -gens=8 # needs DEEPSEEK_API_KEY |
| 515 | ``` |
| 516 | |
| 517 | **Cost arm** (`-mode=cost`) is deterministic and needs no provider: a scripted |
| 518 | summarizer answers every call and refuses any input larger than the window, the |
| 519 | way a real provider does. It reports per generation how many summarizer calls |
| 520 | the fold took, how large the largest one was, and whether the fold succeeded at |
| 521 | all — so a session that grows until it can no longer be compacted shows up as an |
| 522 | error row rather than as a theory. `go test ./benchmarks/compaction/` runs a |
| 523 | smaller version of the same thing as a regression guard. |
| 524 | |
| 525 | **Fidelity arm** (`-mode=fidelity`) plants facts a coding agent must not lose — |
| 526 | a standing constraint, a correction that supersedes an earlier instruction, an |
| 527 | exact identifier, a pending requirement, whether a passing test has been re-run |
| 528 | since the code changed, a ruled-out hypothesis, a tool outcome, chronology — |
| 529 | and after each fold asks a question only that fact answers, against the |
| 530 | compacted context. Every probe is also asked against the full history in the |
| 531 | same run: a probe the model gets wrong with everything in front of it is a bad |
| 532 | probe, not a compaction loss. |
| 533 | |
| 534 | Probe answers are scored on whole words, and a wanted answer does not count if a |
| 535 | rejected one appears anywhere in the same reply — "yes, but it has not been |
| 536 | re-run since" is the shape a drifting digest produces, and it is not a pass. |
| 537 | |
| 538 | ### Fold arms |
| 539 | |
| 540 | `-arm=full` (default) re-derives every digest from the canonical transcript, so |
| 541 | digests never chain. `-arm=incremental` folds the model-visible view instead, |
| 542 | feeding the previous digest back through the summarizer. The arms exist to price |
| 543 | that trade: run the cost arm for what chaining saves, and the fidelity arm for |
| 544 | what it costs. |
| 545 | |
| 546 | ```bash |
| 547 | go run ./benchmarks/compaction -mode=cost -arm=incremental |
| 548 | DEEPSEEK_API_KEY=… go run ./benchmarks/compaction -mode=fidelity -arm=incremental |
| 549 | ``` |
| 550 |