| 1 | # Evaluating skill changes |
| 2 | |
| 3 | NVIDIA's [NeMo Skills](https://github.com/NVIDIA-NeMo/Skills) primarily supports |
| 4 | synthetic data generation, model training, and model evaluation. The closer |
| 5 | fit for Codewhale is NeMo Platform's |
| 6 | [Agent Optimizer](https://docs.nvidia.com/nemo-platform/latest/documentation/agents/optimize-agents/run-the-agent-optimizer): |
| 7 | its `nemo agents optimize-skills` flow evaluates a baseline, edits a configured |
| 8 | skills directory, reevaluates, and accepts improvements against the baseline. |
| 9 | It requires its platform services, agent/provider configuration, and evaluation |
| 10 | setup; it is not a drop-in Codewhale runtime. |
| 11 | |
| 12 | Use that experiment design around Codewhale's existing Engine. Keep provider, |
| 13 | model, tool definitions, permissions, fixtures, and task prompts fixed while |
| 14 | changing one skill. Start with `debug`, `handoff`, and `verify`, which affect |
| 15 | task completion and continuity directly. Preserve the baseline and candidate |
| 16 | skill hashes and all execution receipts. |
| 17 | |
| 18 | | Task family | Required behavior | Rejection condition | |
| 19 | | --- | --- | --- | |
| 20 | | Debug a failing command | Reproduce, locate the cause, make a bounded repair, verify the reported failure | Silently skips verification or reports an unrun check as passed | |
| 21 | | Continue a long task | Retain objective, corrections, authorization limits, working files, and running task handles | Repeats completed work, loses a constraint, or asks the user to manage automatic compaction | |
| 22 | | Review a plugin update | Inspect changed content and capabilities; use existing trust and enablement controls | Catalog text or a previous trust decision grants authority to new bytes | |
| 23 | | Verify a release candidate | Tie evidence to the tested source and binary, and distinguish local from provider/CI proof | Claims release readiness from stale or partial evidence | |
| 24 | |
| 25 | Measure task success first, then tool errors, unnecessary user interruptions, |
| 26 | input/output tokens, provider-reported cache hits and misses, elapsed time, |
| 27 | and cost at the recorded route price. Preserve the stable system/tool prefix |
| 28 | within a trial; never infer cache hits from prompt length or shared wording. |
| 29 | Reject every candidate that violates authorization, loses user data, or |
| 30 | regresses an existing acceptance case, regardless of token savings. |
| 31 | |
| 32 | Separate development tasks from held-out tasks. Select edits using only the |
| 33 | development set, then evaluate the frozen candidate against the untouched |
| 34 | holdout with repeated paired runs. Promote only an improvement that survives |
| 35 | those runs and the normal source gates. Keep the previous version for rollback. |
| 36 | |
| 37 | Codewhale's `crates/tui/src/eval.rs` provides deterministic tool-loop checks; |
| 38 | it does not measure model skill quality. Existing Engine, skill-discovery, |
| 39 | plugin-lifecycle, and compaction survival tests are prerequisite checks. A |
| 40 | real provider comparison is a separate, bounded-cost experiment. No model |
| 41 | quality or cost improvement is established merely by editing these files, |
| 42 | running offline tests, or reading NeMo's documentation. |
| 43 |