| 1 | # `use_capability` replay evaluation |
| 2 | |
| 3 | This optional paired evaluation compares the cache-stable `use_capability` |
| 4 | proxy with a baseline that expands MCP tools into the provider-visible schema. |
| 5 | It is a diagnostic benchmark, not a Stable release gate. Live model runs are |
| 6 | optional and must use disposable Reasonix homes. |
| 7 | |
| 8 | ## What to measure |
| 9 | |
| 10 | For the same task set, run each task twice: |
| 11 | |
| 12 | 1. **Proxy (default):** `use_capability` only. Shared Host plus disk schema |
| 13 | cache. |
| 14 | 2. **Baseline:** a throwaway config that still expands MCP tools into the |
| 15 | provider request. Native Tool Search must stay off. |
| 16 | |
| 17 | Record `tools/list` count, first-token latency, and cache-hit tokens. Do not |
| 18 | upload prompts, secrets, tool arguments, or workspace paths. |
| 19 | |
| 20 | ## Procedure |
| 21 | |
| 22 | 1. Use disposable `REASONIX_HOME` and `REASONIX_CACHE_HOME` directories. |
| 23 | 2. Pick a representative task set that needs MCP discovery followed by a call. |
| 24 | 3. Run proxy and baseline for each task with the same model, effort, workspace, |
| 25 | skills, agents, and MCP configuration. |
| 26 | 4. Record content-free pairs matching |
| 27 | `internal/eval/replay/testdata/paired_runs.json`. |
| 28 | 5. Exercise the median helper: |
| 29 | |
| 30 | ```bash |
| 31 | go test ./internal/eval/replay/ -run TestMedianReportFivePairedRuns |
| 32 | ``` |
| 33 | |
| 34 | The repository fixture is synthetic and proves only the median helper. Teams |
| 35 | may replace its numbers with live observations for performance analysis, but no |
| 36 | paired-run dataset or threshold result is required for Stable publication. |
| 37 | |
| 38 | Native first-party Tool Search stays default-off and is not part of this eval. |
| 39 |