| 1 | # Windows and Linux Desktop crash diagnostics runbook |
| 2 | |
| 3 | <a href="./DESKTOP_CRASH_DIAGNOSTICS_RUNBOOK.zh-CN.md">简体中文</a> |
| 4 | |
| 5 | This is the release, privacy, performance, and root-cause checklist for the |
| 6 | cross-platform Desktop diagnostics pipeline. Windows build `17763` remains a |
| 7 | priority experiment, not a code whitelist. A diagnostic release does not by |
| 8 | itself resolve a crash issue. |
| 9 | |
| 10 | ## Local transcript initialization failures |
| 11 | |
| 12 | When session creation, opening, or legacy migration fails to initialize the |
| 13 | transcript, search the Electron shell's `service.log` (and rotated |
| 14 | `service.log.1`) for `session transcript initialization failed`. The service |
| 15 | emits a structured `diagnostic` group to stderr, which the shell persists through |
| 16 | its existing log writer. This diagnostic has `version=1` and |
| 17 | `code=transcript_initialization_failed`; it does not change the migration ledger |
| 18 | format. The separate content-free online report is described below. |
| 19 | |
| 20 | | Field | Meaning | |
| 21 | | --- | --- | |
| 22 | | `diagnostic.session_key` | SHA-256 of the session ID; correlates failures for that runtime session | |
| 23 | | `diagnostic.covered_sequence` | Event sequence covered by the initial transcript | |
| 24 | | `diagnostic.baseline_message_count` | Input messages selected for initialization, at most 96 | |
| 25 | | `diagnostic.baseline_total_message_count` | Available candidate messages before tail selection; not necessarily the full history size | |
| 26 | | `diagnostic.baseline.record_count` | Transcript records produced from those input messages | |
| 27 | | `diagnostic.baseline.record_index` / `previous_record_index` | Zero-based failing record / first conflicting record in that selected baseline, when known | |
| 28 | | `diagnostic.baseline.role` | Allowlisted role, or `other` | |
| 29 | | `diagnostic.baseline.record_key` / `message_key` / `tool_call_key` | SHA-256 identity fingerprints; empty when the identity is absent | |
| 30 | |
| 31 | `diagnostic.baseline.code` distinguishes `duplicate_record_identity`, |
| 32 | `missing_record_identity`, `baseline_encode_failed`, and |
| 33 | `baseline_decode_failed`. Encoding/decoding failures have no record position. |
| 34 | Positions refer to transcript records, not global message offsets. |
| 35 | |
| 36 | Legacy migration also emits |
| 37 | `desktop session migration transcript initialization failed` with the same |
| 38 | diagnostic, `stage=legacy_import`, and `source_key`. Match `source_key` to the |
| 39 | existing `sourceKey` in `desktop/session-migration-v5.json` under the Reasonix |
| 40 | configuration directory. It identifies the same source across retries even |
| 41 | when a new attempt creates a different target session ID. Each failed attempt |
| 42 | emits one runtime diagnostic and, for this migration path, one migration |
| 43 | diagnostic. The failed source is retained and healthy migrations can continue. |
| 44 | |
| 45 | These new records exclude chat/reasoning bodies, tool arguments, original |
| 46 | paths, raw identifiers, and unrestricted error text. Fingerprints support |
| 47 | correlation; they do not by themselves prove which original data caused a |
| 48 | reported failure. Old `1.38.10` logs cannot retroactively supply these fields; |
| 49 | collect logs from a build containing this change after reproducing the failure. |
| 50 | |
| 51 | The same handled migration failure is queued as an `exception` report with |
| 52 | `source=desktop.session_migration`, `label=transcript.initialization`, and a |
| 53 | content-free classification in its fingerprint hint. It is delivered to |
| 54 | `https://crash.reasonix.io/v1/report` and appears on `/stats/diagnostics` under |
| 55 | the existing Desktop telemetry consent, retry, and per-version deduplication |
| 56 | rules. Session/source fingerprints and record identity fingerprints remain |
| 57 | local-only. Unrecovered panics from older builds continue through the existing |
| 58 | `go.runtime` / `go.fatal` crash path and are shown as high-severity crashes. |
| 59 | |
| 60 | ## Release order |
| 61 | |
| 62 | 1. Keep the Firebase project on Spark with no Cloud Billing account. Create |
| 63 | only a Realtime Database in `asia-southeast1`, deploy |
| 64 | `workers/crash-report/firebase/database.rules.json`, and confirm both client |
| 65 | reads and writes are denied. Do not enable Functions, Firestore, BigQuery, |
| 66 | Hosting, Storage, or Secret Manager. |
| 67 | 2. Configure the three repository secrets `FIREBASE_DATABASE_URL`, |
| 68 | `FIREBASE_CLIENT_EMAIL`, and `FIREBASE_PRIVATE_KEY`. The service account must |
| 69 | be dedicated to crash delivery and limited to Realtime Database. Never use |
| 70 | the Firebase Web configuration or ship Firebase SDK/configuration in a |
| 71 | Desktop artifact. |
| 72 | 3. Freeze one candidate SHA. Do not move or recreate a published tag. |
| 73 | 4. Back up D1 and run `npm run migrate:diagnostics-v2`. The command inspects the |
| 74 | complete schema and records a fresh Time Travel bookmark before writing. The |
| 75 | retired `metric_users` and `cli_metric_users` tables are intentionally not |
| 76 | required; any partial state among the active diagnostics tables fails closed. |
| 77 | 5. Run `npm run migrate:firebase-crash`. It records a D1 Time Travel bookmark, |
| 78 | applies phase 1 (`migrate-firebase-crash.sql`) and phase 2 |
| 79 | (`migrate-firebase-crash-capacity.sql`) in order, and fails closed on a |
| 80 | partially applied phase. Verify the outbox, receipts, compatibility lease |
| 81 | table, `firebase_crash_group_state`, and all delivery/lifecycle indexes. The |
| 82 | old lease table remains only for rolling-deployment compatibility. |
| 83 | 6. Verify `report_daily`, `report_installations`, |
| 84 | `report_event_dimensions`, `diagnostics_meta`, their fingerprint/date |
| 85 | indexes, and the ping window index. Confirm `installation_linked_since`. |
| 86 | 7. In **Actions > Deploy crash worker > Run workflow**, select `main-v2` and |
| 87 | choose `dry-run` for **Firebase crash history operation**. This uses the |
| 88 | existing repository secrets, runs behind the `canary` environment approval, |
| 89 | does not deploy the Worker, and must report at most 700 MiB reserved. After |
| 90 | reviewing the result, choose `apply` and enter the exact confirmation |
| 91 | `APPLY_FIREBASE_CRASH_DATA`; the job runs `--apply` followed immediately by |
| 92 | `--verify-only` on the same runner. Choose `verify-only` for later |
| 93 | independent audits. Authenticated operators may still run |
| 94 | `npm run migrate:firebase-data`, `npm run migrate:firebase-data -- --apply`, |
| 95 | and `npm run migrate:firebase-data -- --verify-only` locally. The default |
| 96 | checkpoint is `.firebase-crash-migration-state.json` (mode `0600`, |
| 97 | gitignored); use `--checkpoint=<path>` to relocate it and |
| 98 | `--reset-checkpoint` only to restart deliberately. Logs contain only counts, |
| 99 | fingerprint prefixes, and digests. |
| 100 | 8. Deploy the Worker in `dual` mode first. Smoke-test old Report/Ping/Metrics payloads, a |
| 101 | legacy `webview2` payload, and Windows/Linux `webRuntime` payloads using |
| 102 | `channel=test`. |
| 103 | 9. Compare D1 and Firebase for seven complete UTC days. Switch |
| 104 | `CRASH_STORAGE_MODE` from `dual` to `firebase` only after counts, |
| 105 | fingerprints, retained samples, and redaction match. In Firebase mode D1 |
| 106 | keeps aggregates and the bounded outbox but no new raw `reports` rows. |
| 107 | 10. Build signed Windows and Linux artifacts from the frozen SHA. Complete the |
| 108 | capability matrix and performance gates before a feature release. |
| 109 | 11. After seven more stable days, archive old D1 raw samples. Keep `d1`, `dual`, |
| 110 | and `firebase` as rollback modes; a Worker rollback does not require a client |
| 111 | update. |
| 112 | 12. Use the admin UI for the audited historical cleanup: ignore the synthetic |
| 113 | `[go panic] safe` / `v9.9.9` group; resolve `72daba81` in |
| 114 | `desktop-v1.19.3`; ignore the legacy `desktop.abnormal_exit` replay group. |
| 115 | |
| 116 | ## Spark capacity, lifecycle, and rollback |
| 117 | |
| 118 | The Worker enforces a fixed 700 MiB reservation budget: active groups reserve |
| 119 | 640 KiB, compacted groups 128 KiB, archiving groups 32 KiB, and archived groups |
| 120 | zero. At 80% the existing alert webhook and dashboard warn; a new group or |
| 121 | expansion that would cross the budget returns `503` before creating an outbox |
| 122 | row. Do not make the budget configurable. |
| 123 | |
| 124 | Only resolved/ignored groups are eligible. After 30 inactive days the latest |
| 125 | five samples become fenced markers and the retained-cycle first sample remains. |
| 126 | After 60 days all sample paths are tombstoned; 24 hours later the Firebase group |
| 127 | is conditionally deleted. D1 counts, status, notes, aggregates, and audit remain. |
| 128 | An archived fingerprint that reappears starts a new sample epoch without |
| 129 | resetting lifetime count/first-seen. Admin deletion uses the same tombstone |
| 130 | window while deleting its D1 group data atomically. |
| 131 | |
| 132 | Rollback is configuration-only: set `CRASH_STORAGE_MODE=d1` and redeploy. Do not |
| 133 | delete the outbox, receipts, group state, or Firebase data during rollback. Fix |
| 134 | the migration/capacity/ETag fault, rerun dry-run and `--verify-only`, then return |
| 135 | to `dual`. No Desktop or CLI update is required. |
| 136 | |
| 137 | ## Privacy and compatibility smoke |
| 138 | |
| 139 | Verify that old payloads may omit every new field and that legacy `webview2` |
| 140 | normalizes to `webRuntime`. Recovered and failed recovery events with the same |
| 141 | engine/kind/reason/exit code must share one fingerprint. Then verify: |
| 142 | |
| 143 | - raw install IDs are absent from report samples, rendered HTML, application |
| 144 | and audit logs, exports, and pending files; |
| 145 | - source modules are basenames; content, keys, accounts, hostnames, full paths, |
| 146 | GPU models, and driver versions are absent; |
| 147 | - a repeated event increments daily/install/event-dimension counts without |
| 148 | changing earlier event dimensions; |
| 149 | - deleting a test group deletes all three diagnostic aggregates; |
| 150 | - retention removes diagnostic facts, pings, and metric-user rows after 30 |
| 151 | days in bounded chunks; |
| 152 | - `channel=test` remains in the development namespace. |
| 153 | - duplicate `eventId` values return `202` without incrementing aggregates; |
| 154 | - Firebase timeout, 401, 429, or 5xx leaves a projected outbox row for the |
| 155 | six-hour retry, while a full outbox returns `503` so clients retain pending; |
| 156 | - automatic Desktop reports are sent once per version and dedup key, failed |
| 157 | sends do not enter the 512-entry/180-day ledger, and explicit Desktop/CLI |
| 158 | reports bypass local fingerprint suppression. |
| 159 | |
| 160 | ## Normal-experience gates |
| 161 | |
| 162 | The candidate must keep pre-shell work to one local configuration read, one |
| 163 | non-blocking ownership lock, and one small atomic lifecycle write. Runtime |
| 164 | discovery and all report/metric persistence run after shell startup or on the |
| 165 | bounded background consumers; COM and GTK callbacks only enqueue or increment |
| 166 | an atomic drop counter. Diagnostic failure remains fail-open. |
| 167 | |
| 168 | Compare the same SHA with diagnostics disabled. Diagnostic initialization p95 |
| 169 | must be at most 10 ms and p99 at most 25 ms. DOM-ready p95 may regress by no |
| 170 | more than `max(20 ms, 2%)`; shutdown p95 by no more than 20 ms; idle CPU by less |
| 171 | than 0.1 percentage points; and idle RSS by at most 2 MiB. During 30 minutes of |
| 172 | normal use there must be no diagnostic reload, polling timer, user-visible |
| 173 | prompt, or network request beyond existing ping/metrics traffic. |
| 174 | |
| 175 | ## Capability matrix |
| 176 | |
| 177 | Use the same candidate SHA throughout. Record Runtime and GPU/driver details in |
| 178 | the private lab worksheet; the client does not collect drivers. |
| 179 | |
| 180 | | Platform | Required coverage | |
| 181 | | --- | --- | |
| 182 | | Windows 10 LTSC 2019 `17763` | VM plus a physical GPU system; system and latest Evergreen WebView2; GPU on/off | |
| 183 | | Windows 10 `19045` | x64 control; system and Evergreen WebView2 | |
| 184 | | Windows 11 stable | x64 physical smoke; current stable WebView2 | |
| 185 | | Windows arm64 | release cross-build plus one device smoke | |
| 186 | | Ubuntu 22.04 | WebKitGTK 4.0 and X11 | |
| 187 | | Ubuntu 24.04 | WebKitGTK 4.1, X11, and Wayland | |
| 188 | | Debian 12, Fedora stable, Arch rolling | capability smoke, local session, representative Intel/AMD/NVIDIA coverage | |
| 189 | | Remote sessions | Windows RDP and Linux remote/xrdp | |
| 190 | |
| 191 | For every environment run 20 cold start/normal exits, 10 update restarts, a |
| 192 | 60-minute workload, 50 minimize/restores, sleep/resume, display/DPI changes, |
| 193 | and remote connect/disconnect where applicable. A test-only build may terminate |
| 194 | the renderer/web process to verify exactly one recovery. Collect WER and |
| 195 | Reliability Monitor on Windows, and journal/coredump metadata on Linux. Dumps |
| 196 | or cores require explicit consent, private transfer, and deletion after use. |
| 197 | |
| 198 | ## Root-cause gate and follow-up |
| 199 | |
| 200 | An environment association requires either two similar lab nodes reproducing |
| 201 | while controls do not, or three distinct online installations sharing a |
| 202 | fingerprint with at least 30 active installations and an impact rate three |
| 203 | times the control. A GPU workaround requires at least `2/20` GPU-on failures per |
| 204 | node, `0/40` GPU-off failures across two nodes, and two clean hours per node. |
| 205 | Scope any workaround by demonstrated capability/Runtime evidence, not distro |
| 206 | name or Windows build alone. |
| 207 | |
| 208 | Treat integrity failures as signing/injection/security-software investigations, |
| 209 | out-of-memory as memory/session resource investigations, and Runtime clustering |
| 210 | as evidence for a later minimum-version/update policy. A successful renderer |
| 211 | recovery is not an application crash. Lifecycle-only abnormal exits require a |
| 212 | matching WER, journal, dump, or core before closure. |
| 213 | |
| 214 | Observe production for seven complete UTC days: identity coverage should be at |
| 215 | least 95%; below 90% do not show an exact impact rate. Check legacy replay, |
| 216 | fatal/recovered/degraded coherence, recovery failures, platform/Runtime/GPU |
| 217 | rates, D1 growth, retention, and query latency daily. If evidence is |
| 218 | insufficient, keep the issue open and extend observation to 30 days. Ship a |
| 219 | later patch only for a demonstrated root cause and require `0/40` lab |
| 220 | reproductions after the fix. |
| 221 |