| 1 | # `crates/tui/tests/` |
| 2 | |
| 3 | Integration tests for the TUI binary. Per `CONTRIBUTING.md`, each crate's |
| 4 | integration tests live in its own `tests/` directory; the repository-root |
| 5 | `tests/` directory is unused. |
| 6 | |
| 7 | ## Harness consolidation (build-time lane #5247) |
| 8 | |
| 9 | `crates/tui/tests/` used to ship 26 root-level `*.rs` binaries, each linking |
| 10 | the full `codewhale-tui` graph plus `cucumber`/`wiremock`/`rio-vt`. That was |
| 11 | ~26 large link jobs per `cargo test -p codewhale-tui` and a major share of the |
| 12 | 30-minute suite in #4991. |
| 13 | |
| 14 | Since #5247 the remaining files are consolidated into **2 directory harnesses** (plus |
| 15 | the `codewhale-tui` bin unit tests): |
| 16 | |
| 17 | | harness | binary | what lives there | why it stays separate | |
| 18 | |---|---|---|---| |
| 19 | | `tests/integration/main.rs` | `integration` | 17 plain `#[test]`/`#[tokio::test]` suites: `adaptive_evidence_acceptance`, `cache_guard`, `coordination_acceptance`, `diagnostic_read_only`, `dotenv_authority`, `eval_harness`, `exec_persistent_service`, `exec_stream_drop_acceptance`, `exec_turn_usage`, `integration_mock_llm`, `palette_audit`, `protocol_recovery`, `reasoning_content_replayed_after_tool_call`, `skill_cli`, `telemetry_contract`, `verifiers_harness_contract`, `workflow_tool_stream_acceptance` | All are process-level but require no PTY or Gherkin runner; they share `wiremock`/`tempfile` and link the TUI once instead of 17 times. `crate::` for `eval`/`models`/`llm_client`/`network_policy`/`config`/`install` is satisfied by `integration/main.rs` re-exporting those `#[path]` modules at the harness crate root so `crate::config` etc. resolve. | |
| 20 | | `tests/cucumber/main.rs` | `cucumber` | 6 Gherkin runners: `core_session_command_extraction`, `directory_listing_acceptance`, `epic_acceptance_harness`, `eval_smoke_acceptance`, `plugin_e2e_acceptance`, `tool_lifecycle_acceptance` | Each defines a distinct `cucumber::World`; steps are registered per-World via inventory, so merging is safe and cuts 6 `cucumber` link jobs to 1. `plugin_e2e`’s PTY part is `#[cfg(all(unix, feature="long-running-tests"))]` and stays dormant in the default run. | |
| 21 | The former `tests/pty` harness was removed. It accumulated full-screen copy, |
| 22 | color, timing, and geometry assertions that were expensive to link and made the |
| 23 | implementation serve a simulated terminal. Visible UX is now accepted against |
| 24 | the actual binary in the terminal being changed. |
| 25 | |
| 26 | `ls crates/tui/tests/*.rs | wc -l` is **0** (all `*.rs` live under |
| 27 | `integration/` and `cucumber/`). The surviving binaries are the 2 directory |
| 28 | harnesses above. |
| 29 | |
| 30 | Filtering still works via the module path: |
| 31 | |
| 32 | ```sh |
| 33 | cargo test -p codewhale-tui --tests -- --list | grep adaptive_evidence |
| 34 | cargo test -p codewhale-tui --test integration adaptive_evidence_acceptance -- --nocapture |
| 35 | cargo test -p codewhale-tui --test cucumber tool_lifecycle -- --nocapture |
| 36 | ``` |
| 37 | |
| 38 | The shared helpers in `crates/tui/tests/support/` (`qa_harness`, `llm_client`) and fixtures in `crates/tui/tests/fixtures/` are untouched — harnesses reach them via `../support`. |
| 39 | |
| 40 | ## Mock LLM client (`integration::integration_mock_llm`) |
| 41 | |
| 42 | `crates/tui/src/llm_client/mock.rs` provides a `MockLlmClient` that implements |
| 43 | the `LlmClient` trait by replaying queue-driven canned responses and capturing |
| 44 | every outgoing `MessageRequest`. Tests mock at the **trait boundary** — never |
| 45 | at the `reqwest` HTTP layer — because the trait is the durable abstraction the |
| 46 | runtime is meant to depend on. |
| 47 | |
| 48 | Coverage today exercises the trait surface end-to-end: |
| 49 | |
| 50 | - streaming turn loop |
| 51 | - reasoning-content replay across tool-call rounds (V4 §5.1.1, the bug that |
| 52 | broke v0.4.9-v0.5.1) |
| 53 | - tool-call round-trip with chunked input JSON |
| 54 | - multi-tool-call ordering inside a single turn |
| 55 | - compaction-style non-streaming `create_message` |
| 56 | - sub-agent style independent parent/child mocks |
| 57 | - capacity-gate observation of a captured request before stream drain |
| 58 | |
| 59 | Full-engine journeys use `Engine::new_with_model_client` and the same mock. |
| 60 | A non-trivial model/protocol/user-visible behavior change needs one keyless |
| 61 | assembled journey at the nearest real entry path. Prefer semantic assertions |
| 62 | over large full-screen goldens; pin only the durable events, protocol text, |
| 63 | side effects, accounting, and next request whose exact shape is the feature. |
| 64 | Do not add a new snapshot framework, network call, provider key, timing sleep, |
| 65 | or platform shell merely to cover the journey. The Auto-Review guardian |
| 66 | journeys in `src/core/engine/tests.rs` are the first fixture (#5361); each |
| 67 | drives one mock-model tool turn through `Engine::run` and pins: |
| 68 | |
| 69 | - `auto_review_guardian_allow_executes_once_and_accounts_usage_without_prompt_leak` |
| 70 | — allow executes the tool exactly once, reviewer usage reaches `TurnUsage` |
| 71 | and `TurnComplete`, and the reviewer rationale/audit fields never appear in |
| 72 | the follow-up model request |
| 73 | - `auto_review_guardian_deny_returns_one_paired_failed_result` — deny yields |
| 74 | one paired `is_error` tool result carrying the rationale, no orphaned call, |
| 75 | no side effect |
| 76 | - `auto_review_guardian_parse_and_transport_failures_deny_closed` — reviewer |
| 77 | parse or transport failure denies fail-closed with an `Unavailable` receipt |
| 78 | - `auto_review_cancellation_promptly_drops_the_guardian_request` — cancel |
| 79 | drops the in-flight guardian future and interrupts the turn without a |
| 80 | follow-up model request |
| 81 | |
| 82 | ## `--record` mode for `deepseek eval` |
| 83 | |
| 84 | The offline `deepseek eval` harness now accepts `--record <DIR>`. When set, |
| 85 | each tool step appends one JSON Lines record to `<DIR>/<scenario>.jsonl` |
| 86 | (default scenario: `offline-tool-loop.jsonl`). Each line is a self-contained |
| 87 | JSON object with the schema: |
| 88 | |
| 89 | ```json |
| 90 | { "request": { "step": "list_dir", "kind": "List" }, |
| 91 | "response_events": [ { "type": "ok", "output": "…" } ] } |
| 92 | ``` |
| 93 | |
| 94 | The mock LLM client (`crate::llm_client::mock`) replays these fixtures by |
| 95 | mapping each `response_events` array onto a canned `Vec<StreamEvent>`. Drop |
| 96 | generated fixtures into `crates/tui/tests/fixtures/` so they ride the repo and |
| 97 | feed the mock in CI. |
| 98 | |
| 99 | Quick example: |
| 100 | |
| 101 | ```bash |
| 102 | cargo run --bin codewhale -- eval --record crates/tui/tests/fixtures |
| 103 | cat crates/tui/tests/fixtures/offline-tool-loop.jsonl | jq . |
| 104 | ``` |
| 105 | |
| 106 | The scenario name is sanitized to `[A-Za-z0-9_-]` before forming the filename, |
| 107 | so unusual scenario strings stay portable across platforms. |
| 108 |