| 1 | # codewhale Operations Runbook |
| 2 | |
| 3 | This runbook covers practical debugging and incident response for the local CLI/TUI runtime. |
| 4 | |
| 5 | ## Quick Triage |
| 6 | |
| 7 | 1. Confirm binary + config: |
| 8 | - `cargo run -- --version` |
| 9 | - `cat ~/.codewhale/config.toml` (or inspect configured profile) |
| 10 | 2. Enable verbose logs: |
| 11 | - `RUST_LOG=codewhale_tui=debug cargo run` |
| 12 | - For HTTP retries/reconnects: `RUST_LOG=codewhale_tui::client=debug cargo run` |
| 13 | 3. Capture current state: |
| 14 | - `ls ~/.codewhale/sessions` |
| 15 | - `ls ~/.codewhale/sessions/checkpoints` |
| 16 | - `ls ~/.codewhale/tasks` |
| 17 | |
| 18 | ## Incident: Turn Hangs or Stream Stops |
| 19 | |
| 20 | Symptoms: |
| 21 | - TUI remains in loading state |
| 22 | - partial assistant output with no completion |
| 23 | |
| 24 | Checks: |
| 25 | 1. Inspect retry/health logs (`codewhale_tui::client`) |
| 26 | 2. Verify endpoint connectivity: |
| 27 | - `curl -sS https://api.deepseek.com/beta/models -H "Authorization: Bearer $DEEPSEEK_API_KEY"` |
| 28 | 3. Confirm no local sandbox/permission deadlock in tool output |
| 29 | |
| 30 | Actions: |
| 31 | 1. If a foreground shell command is running, press `Ctrl+B` to move it to the background (the turn keeps running and the command becomes a background job under `/jobs`); use `Ctrl+C` instead if you want to cancel the turn. |
| 32 | 2. If the command was started in the background, ask the assistant to use `Bash` with `action: "cancel"` and the returned process id. |
| 33 | 3. Use `Esc` or `Ctrl+C` to interrupt the current turn when you want to stop the request itself. |
| 34 | 4. Retry prompt; if still failing, restart TUI. |
| 35 | 5. On restart, verify the previous queued/in-flight runtime turn is shown as interrupted rather than left in a running state. |
| 36 | |
| 37 | ## Incident: Network Outage / Offline Behavior |
| 38 | |
| 39 | Expected behavior: |
| 40 | - New prompts are queued while offline mode is active |
| 41 | - Queue state persists to `~/.codewhale/sessions/checkpoints/offline_queue.json` |
| 42 | |
| 43 | Checks: |
| 44 | 1. Open queue in TUI: `/queue list` |
| 45 | 2. Confirm persisted queue file exists and updates timestamp |
| 46 | |
| 47 | Actions: |
| 48 | 1. Restore connectivity |
| 49 | 2. Re-send queued entries (from `/queue edit <n>` + Enter, or normal input flow) |
| 50 | 3. Ensure queue file clears when queue is empty |
| 51 | |
| 52 | ## Incident: Crash Recovery Needed |
| 53 | |
| 54 | Expected behavior: |
| 55 | - Checkpoint stored at `~/.codewhale/sessions/checkpoints/latest.json` |
| 56 | - Startup begins a fresh session unless `--resume`/`--continue` is supplied |
| 57 | |
| 58 | Actions: |
| 59 | 1. Resume prior work explicitly via `codewhale --resume <id>` (alias |
| 60 | `codewhale resume <id>`; `codewhale --continue` recovers the newest |
| 61 | interrupted checkpoint for the workspace) or `Ctrl+R` in TUI |
| 62 | 2. If checkpoint inspection is needed, inspect `latest.json` for schema mismatch/details |
| 63 | 3. If schema is newer than binary supports, upgrade binary or remove stale checkpoint |
| 64 | |
| 65 | ## Incident: Persistent State Schema Errors |
| 66 | |
| 67 | Symptoms: |
| 68 | - Errors like `schema vX is newer than supported vY` |
| 69 | |
| 70 | Affected stores: |
| 71 | - sessions (`~/.codewhale/sessions/*.json`) |
| 72 | - runtime thread/turn/item records |
| 73 | - tasks (`~/.codewhale/tasks/tasks/*.json`) |
| 74 | |
| 75 | Actions: |
| 76 | 1. Confirm binary version and migration expectations |
| 77 | 2. Back up the state directory before editing |
| 78 | 3. Either: |
| 79 | - run with a newer compatible binary, or |
| 80 | - archive incompatible records and regenerate state |
| 81 | |
| 82 | ## Incident: MCP/Tool Execution Failures |
| 83 | |
| 84 | Checks: |
| 85 | 1. Validate `~/.codewhale/mcp.json` schema and server command paths |
| 86 | 2. Confirm server process can start manually |
| 87 | 3. Check sandbox denials in TUI history / logs |
| 88 | |
| 89 | Actions: |
| 90 | 1. Retry with required approvals (or YOLO only when appropriate) |
| 91 | 2. Temporarily disable failing MCP server and isolate issue |
| 92 | 3. Re-enable after verification with `/mcp` diagnostics |
| 93 | |
| 94 | ## Post-Incident Checklist |
| 95 | |
| 96 | 1. Preserve logs and relevant state files |
| 97 | 2. Record trigger, impact, and mitigation |
| 98 | 3. Add or update regression tests (retry/recovery/schema) |
| 99 | 4. Update this runbook and architecture docs if behavior changed |
| 100 |