| 1 | # Search Quality Eval |
| 2 | |
| 3 | `skills/last30days/scripts/evaluate_search_quality.py` is an optional local evaluation step for retrieval quality. It is not part of the user-facing runtime and does not need to run in CI by default. |
| 4 | |
| 5 | What it does: |
| 6 | |
| 7 | - runs a baseline revision (default `origin/main`) against a candidate checkout |
| 8 | - evaluates the fixed 5 reviewer topics by default |
| 9 | - computes deterministic stability metrics: |
| 10 | - `Jaccard` overlap vs baseline |
| 11 | - retention vs baseline |
| 12 | - per-source counts and overlap |
| 13 | - optionally calls Gemini as a judge for graded relevance labels and then computes: |
| 14 | - `Precision@5` |
| 15 | - `nDCG@5` |
| 16 | - source-coverage recall across the judged union pool |
| 17 | |
| 18 | Recommended usage: |
| 19 | |
| 20 | ```bash |
| 21 | uv run python skills/last30days/scripts/evaluate_search_quality.py |
| 22 | ``` |
| 23 | |
| 24 | Useful flags: |
| 25 | |
| 26 | ```bash |
| 27 | uv run python skills/last30days/scripts/evaluate_search_quality.py \ |
| 28 | --baseline-rev origin/main \ |
| 29 | --candidate-rev HEAD \ |
| 30 | --no-default-topics \ |
| 31 | --topic "cursor IDE pricing" \ |
| 32 | --per-source-limit 5 |
| 33 | ``` |
| 34 | |
| 35 | Gemini configuration: |
| 36 | |
| 37 | - preferred on this workspace: set `GOOGLE_API_KEY` |
| 38 | - also accepted: `GEMINI_API_KEY` or `GOOGLE_GENAI_API_KEY` |
| 39 | - optional: set `GEMINI_MODEL` |
| 40 | - default model is `gemini-3-pro-preview` for the direct Gemini API |
| 41 | |
| 42 | Notes: |
| 43 | |
| 44 | - The script forces a clean env-based auth path when it shells out to `last30days.py`. |
| 45 | - It passes `XAI_API_KEY`, `OPENAI_API_KEY`, and `SCRAPECREATORS_API_KEY`, but intentionally does not pass browser-cookie X auth. That keeps evaluation runs on the popup-free path. |
| 46 | - It also strips `node` from the eval `PATH` and wraps `yt-dlp` with `--ignore-config`, so older revisions do not inherit local browser-cookie config either. |
| 47 | - `Jaccard` and retention are regression guards, not truth metrics. |
| 48 | - `Precision@5` and `nDCG@5` are only as good as the judged pool. They help compare revisions, but they are not a substitute for a larger labeled benchmark. |
| 49 |