返回 CodeWhale
README.md
根目录 / telemetry-ingest / README.md
1 # `telemetry-ingest` — Codewhale's first-party telemetry endpoint
2
3 A Cloudflare Worker that accepts the batches described in
4 [`docs/TELEMETRY.md`](../docs/TELEMETRY.md) and writes them to Workers Analytics
5 Engine. One POST route. No response body on any path. The handler never reads
6 client-IP metadata and the closed payload schema accepts no address field.
7
8 It lives here and not in `web/` because the site is a separate deploy with its
9 own build (Next.js via OpenNext); this is a single 13 KiB script with no assets,
10 and coupling the two would mean a telemetry change rebuilding the marketing site.
11
12 **Deployed and live** at `https://telemetry.codewhale.net/v1/telemetry`, which
13 is the shipped default for `telemetry_endpoint`. workers.dev is disabled; that
14 hostname is the only way in.
15
16 The current source defaults usage analytics on under notice version5, with a
17 clear opt-out. Prior explicit declines remain off; a missing acceptance record
18 is not treated as consent. Schema3 carries `notice_version: 5` policy metadata;
19 legacy schema2 retains its original explicit `consent_version: 4` contract.
20 Schema1 clients are accepted only into first-party Analytics Engine. Local
21 schema3 validation is not proof this updated Worker has been deployed. A user who wants
22 to contact nobody sets `telemetry_endpoint = ""`, which writes batches to
23 `$CODEWHALE_HOME/telemetry/dryrun.jsonl` instead.
24
25 ---
26
27 ## Optional PostHog processor
28
29 The new sink is **unconfigured by default**. It uses no analytics SDK and does
30 not create a second collector. `POSTHOG_HOST` must be exactly
31 `https://us.i.posthog.com` or `https://eu.i.posthog.com`, and
32 `POSTHOG_PROJECT_TOKEN` must be supplied as a Worker secret. The separate
33 `POSTHOG_IP_SAFE_EGRESS_VERIFIED="true"` prerequisite requires the staging
34 receipt below. Never commit a real token. Missing any prerequisite or using
35 an untrusted host leaves the sink off.
36 Deployment, token configuration, retention/privacy settings, and activation
37 require separate operator approval; none is established by the local tests.
38
39 Validated schema3 batches carrying `notice_version: 5` and legacy schema2
40 batches carrying explicit `consent_version: 4` can reach the processor. Legacy v1 always remains first-party only. Capture uses
41 `/batch/`, fixed anonymous/no-geo controls, no incoming headers, no cookies,
42 no redirects, no response parsing, no retries, and a 1.5-second timeout.
43 PostHog failure is isolated from the successful Analytics Engine write.
44 See [the complete disclosure](../docs/TELEMETRY.md) and
45 [PostHog's capture contract](https://posthog.com/docs/api/capture).
46
47 **Egress activation remains unproved.** Cloudflare can add client-IP headers to
48 Worker subrequests destined for non-Cloudflare zones, even when application
49 code builds fresh headers. The guard is an operator prerequisite, not code
50 that strips platform headers. Before setting it, record a staging receipt for
51 the actual deployment and regional destination proving original client IPs
52 are absent from all received headers, including `CF-Connecting-IP`,
53 `X-Forwarded-For`, and `X-Real-IP`. Requalify if the egress path changes. Without
54 that receipt, leave the sink disabled; use a detached first-party delivery
55 context if the deployed network cannot satisfy the invariant. Local fetch
56 mocks do not prove edge behavior. See [Cloudflare's header behavior](https://developers.cloudflare.com/fundamentals/reference/http-headers/#cf-connecting-ip-in-worker-subrequests).
57
58 The cross-repository CWC contract is generated, not independently authored:
59 `node scripts/export-product-schema.mjs` (Node 22.18+) exports
60 `schema/cwc-product-v3.schema.json` from this validator's constants. The v2
61 artifact remains available for existing clients. It accepts
62 only `web-app` / `desktop`, one closed `product_usage` event, and the fixed
63 no-fingerprint browser envelope. Copies must be compared with this generated
64 artifact when changing the schema. `test/golden/browser-v3.json` is a complete
65 website wire example; CWC changes only its surface to `web-app` or `desktop`.
66 `operations_summary` is separate operator-consented service health: six `u32`
67 aggregates on `control-plane` with a new random install ID per batch, never
68 individual request records or end-user identifiers.
69
70 The website/app same-origin proxy, when configured, must validate and forward
71 only the closed batch body and reconstruct a content-type header. Do not pass
72 through the incoming request, cookies, user agent, authentication, or metadata.
73 The ingest's two-header invariant remains unchanged.
74
75 ## The one property that matters
76
77 `docs/TELEMETRY.md` publishes:
78
79 > Batches are **IP-stripped at ingest**. No IP is stored, logged, or joined to
80 > `install_id`.
81
82 This Worker is the whole of what makes that sentence true. There is no other
83 component. So:
84
85 - `src/index.ts` reads exactly two request headers — `content-type` and
86 `content-length` — and nothing else, ever.
87 - It never touches the `cf` property of the request, so country, colo, city,
88 region, ASN and coordinates are never in scope.
89 - `src/datapoint.ts` builds every stored row, and it cannot see the request at
90 all: its input type is the validated batch body.
91 - Nothing logs. `invocation_logs` is off in `wrangler.jsonc`, because Cloudflare
92 describes those as "enriched with information available to Cloudflare in the
93 context of the invocation" — exactly the class of automatic per-request record
94 this service promises not to keep.
95 - `test/no-ip.test.ts` reads the shipped source as text and fails the build if
96 any of those names appears, if the set of headers read grows past two, if a
97 `console.*` call is added, or if a `Response` is ever constructed with a body.
98 That file is the only place in this directory where the forbidden header names
99 are written down. A later edit cannot add one quietly.
100
101 Debugging without an IP is a solved problem: the schema carries `os`, `arch`,
102 `libc`, `surface`, `app_version` and `git_sha`, which is what crash triage
103 actually needs.
104
105 ## What it stores
106
107 Everything in [`docs/TELEMETRY.md`](../docs/TELEMETRY.md) and nothing else. The
108 validator in `src/schema.ts` is a **closed** field set: an unexpected key
109 anywhere in the batch — envelope, event, `counters`, `errors`, `turn_wall` —
110 rejects the whole batch with `400`. That is the point of the design. A future
111 client bug that starts attaching a path, a prompt, or a customer's provider
112 table name gets refused by the server rather than quietly stored.
113
114 `test/schema-doc.test.ts` parses the field names and enum spellings back out of
115 `docs/TELEMETRY.md` and asserts set equality against the validator, and
116 `test/ingest.test.ts` posts `crates/telemetry/tests/golden/v1.json` — the
117 client's own pinned v1 wire form — and asserts it is accepted byte for byte and
118 that deleting *any* key from it is rejected. The doc, the Rust client, and this
119 endpoint cannot drift apart without a red test.
120
121 ### Column layout
122
123 One Analytics Engine data point per event. A batch carries at most 200 events
124 (`BATCH_MAX_EVENTS`) and Analytics Engine allows 250 data points per invocation,
125 so a conforming batch never needs a second pass.
126
127 The layout is **positional and append-only** — Analytics Engine columns are
128 `blob1..blob20` / `double1..double20`, and the names below exist only in the SQL
129 you write. Renumbering silently rewrites every historical query. To add a field,
130 take the next free slot.
131
132 | column | contents |
133 |---|---|
134 | `index1` | `install_id` — random v4 UUID, client-rotated every 90 days. The only identifier in the schema. |
135 | `blob1` | `event` — `install_or_upgrade` \| `session_start` \| `session_end` \| `panic` \| `product_usage` \| `operations_summary` |
136 | `blob2` | `surface` |
137 | `blob3` | `os` |
138 | `blob4` | `arch` |
139 | `blob5` | `libc` |
140 | `blob6` | `app_version` |
141 | `blob7` | `git_sha`, `''` for every locally built binary |
142 | `blob8` | `tty` — `'true'` \| `'false'` |
143 | `blob9` | `install_kind` (`install_or_upgrade` only) |
144 | `blob10` | `previous_version` (`install_or_upgrade` only) |
145 | `blob11` | `session_source` (`session_start` only) |
146 | `blob12` | `duration_bucket` (`session_end` only) |
147 | `blob13` | `exit_class` (`session_end` only) |
148 | `blob14` | `cold_start_bucket` (`session_end` only; `''` on surfaces that do not measure it) |
149 | `blob15` | `providers`, comma-joined, already sorted and deduplicated |
150 | `blob16` | `panic_site` (`panic` only) — a `crates/…` path or the literal `<dep>` |
151 | `blob17` | `sent_at` — the *batch* timestamp. Events carry none. |
152 | `blob18` | `aggregate_counters` — closed JSON counts, for `product_usage` or `operations_summary` |
153 | `blob19` | `schema_version` |
154 | `blob20` | `privacy_version`: notice5 for v3, consent4 for v2, empty for v1; `blob19` keeps their meanings distinct |
155 | `double1..10` | `counters`: `turns`, `tool_calls`, `fleet_dispatch`, `workflow_run`, `subagent_spawn`, `mcp_server_connected`, `memory_search`, `approval_modal_shown`, `approval_auto_allowed`, `command_palette_open` |
156 | `double11..16` | `errors`: `auth_preflight_failed`, `provider_http_4xx`, `provider_http_5xx`, `tool_denied_by_policy`, `tool_timeout`, `network_error` |
157 | `double17..20` | `turn_wall`: `lt_5s`, `5_30s`, `30_120s`, `gte_120s` |
158
159 Columns not relevant to an event are `''` / `0`. `tty` is a blob because the 20
160 doubles are exactly used by the three numeric structs — Analytics Engine's
161 ceiling is 20.
162
163 ### What it structurally cannot store
164
165 Not "does not"; **cannot**, given the code as written:
166
167 - **The client IP, and anything derived from it** — never read. See above.
168 - **Any geo** — country, colo, city, region, ASN, coordinates, timezone.
169 - **Any key the schema does not name.** Unknown key ⇒ `400` for the entire
170 batch, so there is no path from an unexpected field to storage.
171 - **Any free-form string.** The published schema has no free-form string type and
172 no open-keyed map. Every field is an integer, a boolean, or a closed enum,
173 except `app_version`, `git_sha` and `panic_site` — each of which has a regex
174 here, so a path, a prompt, a URL, or a branch name fails the shape check.
175 - **A provider table name.** `providers` entries must be lowercase hyphenated ids
176 and the array must be sorted and deduplicated; `acme_internal_gateway` is
177 rejected. (See "known gap" below.)
178 - **A panic message.** Only `panic_site`, and only inside the `crates/`
179 allowlist or the literal `<dep>`.
180 - **Per-event timestamps.** There are none in the schema; only `sent_at`,
181 per batch.
182 - **Response content.** Every response is a bare status with a `null` body, so
183 the endpoint cannot echo back what it received or what it holds.
184
185 **Known gap, stated plainly.** `providers` is the one field whose *value* space
186 this endpoint cannot close. The authoritative list is
187 `codewhale_config::provider::all_providers()`, a Rust registry with no generated
188 artifact to read, and hard-coding a copy here would drift into silently dropping
189 a real user's route. The client closes it (`Event::is_bounded` →
190 `is_known_provider_id`) before the POST is made; the server enforces the shape a
191 closed `&'static str` enum can produce, plus the doc's sorted-and-deduplicated
192 rule. If a generated provider-id list ever lands in the repo, wire it in here.
193
194 ### Retention
195
196 Cloudflare stores Analytics Engine data for **three months**, and that is not
197 configurable — a ceiling rather than a policy, since no setting could make it
198 longer. `docs/TELEMETRY.md` states it.
199
200 ---
201
202 ## Deploy
203
204 Live. The commands below are the ones that produced the current deployment and
205 the ones that will produce the next one.
206
207 ```sh
208 cd telemetry-ingest
209 npm install
210 npm test # 109 tests, including the doc weld and the IP guard
211 npx wrangler deploy --dry-run --outdir=.wrangler/dry-run # no account touched
212 npx wrangler deploy # <- the only command that publishes anything
213 ```
214
215 The `routes` block in `wrangler.jsonc` binds it to `telemetry.codewhale.net` as
216 a custom domain, so the endpoint URL is
217 `https://telemetry.codewhale.net/v1/telemetry`. The workers.dev subdomain is
218 disabled: that hostname is the only way in.
219
220 Verified against the live endpoint before the client default was changed: the
221 client's golden batch returns `204` with a zero-byte body; an unknown key
222 returns `400`; `GET` returns `405`; a wrong content type returns `415`; a `POST`
223 to `/` returns `404`. Reading back from Analytics Engine returned exactly two
224 rows — `session_start` and `session_end`, carrying `install_id`, `surface=tui`,
225 `os=macos` — the documented shape and nothing else.
226
227 **Re-run the verification below after any deploy.** The client default now
228 points here, so a regression in this Worker is a regression in a promise
229 `docs/TELEMETRY.md` makes to users.
230
231 ### Analytics Engine dataset setup
232
233 There is none. The dataset named in `wrangler.jsonc`
234 (`codewhale_telemetry`) is created implicitly on the first successful
235 `writeDataPoint`, so there is nothing to provision ahead of the deploy. Confirm
236 it exists after the first batch:
237
238 ```sh
239 curl "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/analytics_engine/sql" \
240 --header "Authorization: Bearer $CF_API_TOKEN" \
241 --data "SHOW TABLES"
242 ```
243
244 The token needs **Account → Account Analytics → Read**. Querying is out of band
245 through this API; the Worker itself has no read path at all.
246
247 ### Rate limiting
248
249 `ratelimits` binding, 20 POSTs per 60s, **keyed on `install_id`** from the
250 validated batch body. Never on a network address — an IP-keyed limiter would
251 mean this Worker handles IPs, which is the whole thing it must not do. That is a
252 weaker limiter (an `install_id.json` can be rewritten between POSTs) and it is
253 the right trade: Cloudflare's edge already absorbs volumetric abuse, and the
254 client only flushes once per session anyway (one three-second attempt at
255 shutdown).
256
257 ### Size cap
258
259 `MAX_BODY_BYTES` is 72 KiB, computed rather than guessed. A conforming client
260 sends at most 200 events totalling at most 65536 bytes (`BATCH_MAX_EVENTS` and
261 `BATCH_MAX_BYTES` in `crates/telemetry/src/actor.rs`, both hard ceilings because
262 `parse_events` breaks *before* crossing them), plus 199 commas and ~375 bytes of
263 envelope keys and values — 66110 bytes worst case. 72 KiB is ~11% headroom.
264
265 The 512-record / 256 KiB rings in `crates/telemetry/src/buffer.rs` are the *disk*
266 cap, not the wire cap: a full ring drains as three batches, never as one POST.
267
268 `content-length` is checked first as a cheap reject, but it is client-supplied,
269 so the real bound is enforced while reading the body and the stream is cancelled
270 the moment it goes over.
271
272 ---
273
274 ## Owner reports and queries
275
276 The routine activity report is a checked-in command rather than SQL copied
277 from a chat:
278
279 ```sh
280 CF_ACCOUNT_ID=... CF_API_TOKEN=... npm run report:active-installs
281 CF_ACCOUNT_ID=... CF_API_TOKEN=... npm run report:active-installs -- --days 30 --json
282 ```
283
284 The metric is **observed active installs by UTC day**: distinct rotating
285 anonymous `install_id` values that produced a `session_start` event that day,
286 plus the weighted number of sessions started. It is never a count of people,
287 accounts, or total installs, and the report says so next to the numbers.
288 `npm run report:dau` still works — `scripts/report-dau.mjs` is a pure
289 re-export of the canonical `scripts/report-active-installs.mjs`, kept only so
290 the old name keeps working — but the canonical name matches what the number
291 actually is.
292
293 One command produces all four things the ledger asks for:
294
295 - the **daily series** (default `--days 15` — 14 complete UTC days plus the
296 partial current day, marked `*`);
297 - a **7-day trend** over complete UTC days only: the last 7 days' sum of
298 daily observed active installs against the previous 7, with the percentage
299 change. When the window cannot cover both sides it says so instead of
300 printing zeros;
301 - **event freshness**: the timestamp of the newest ingested event of any kind
302 and how stale it is, or a plain statement that nothing has been ingested;
303 - the **coverage caveats, printed with the numbers** in both text and
304 `--json` modes: clients older than the telemetry feature, opted-out
305 installs, and non-emitting environments (kill switches, fleet workers,
306 offline shutdowns, dropped flushes) are invisible, so every count is a
307 lower bound; and the 90-day id rotation means week-over-week comparisons
308 are not a retention metric.
309
310 The token needs **Account → Account Analytics → Read**. The report performs
311 two read-only SQL requests (the series and the freshness probe), does not
312 print credentials, and rejects windows beyond Analytics Engine's fixed 90-day
313 retention. Its read path is pinned by `test/report-active-installs.test.ts`:
314 the install id appears only inside `count(DISTINCT …)`, no payload column
315 (`blob2`+, `double*`) is ever selected, and the output wording can never label
316 the result as users, people, or accounts.
317
318 The underlying ad-hoc queries are one query each, which is what the column
319 layout was chosen for. Run them against the SQL API:
320
321 ```sh
322 query() {
323 curl -sS "https://api.cloudflare.com/client/v4/accounts/$CF_ACCOUNT_ID/analytics_engine/sql" \
324 --header "Authorization: Bearer $CF_API_TOKEN" --data "$1"
325 }
326 ```
327
328 `_sample_interval` says how many original rows a stored row represents;
329 Analytics Engine downsamples high-volume indexes, so every count is weighted by
330 it rather than using bare `count()`.
331
332 ### (a) How many installs and sessions
333
334 ```sql
335 SELECT
336 count(DISTINCT index1) AS installs,
337 sumIf(_sample_interval, blob1 = 'session_start') AS sessions_started,
338 sumIf(_sample_interval, blob1 = 'session_end') AS sessions_ended,
339 sumIf(_sample_interval, blob9 = 'install') AS first_installs,
340 sumIf(_sample_interval, blob9 = 'upgrade') AS upgrades,
341 sumIf(_sample_interval, blob13 = 'clean') AS clean_exits
342 FROM codewhale_telemetry
343 WHERE timestamp > NOW() - INTERVAL '7' DAY
344 ```
345
346 Read `installs` as what the published doc says it is and nothing more: the id
347 rotates every 90 days and is regenerated whenever the telemetry directory is
348 cleared, so **no count derived from `install_id` is a user count**. It is a lower
349 bound on distinct machine-installs seen in the window, and it undercounts a
350 returning user across a rotation.
351
352 Add `, blob6` to `SELECT` and `GROUP BY blob6` to cut by `app_version`; `blob3`
353 for OS, `blob2` for surface.
354
355 ### (b) Which error classes and panic sites dominate
356
357 ```sql
358 SELECT
359 blob16 AS panic_site,
360 sum(_sample_interval) AS rows,
361 sum(double11 * _sample_interval) AS auth_preflight_failed,
362 sum(double12 * _sample_interval) AS provider_http_4xx,
363 sum(double13 * _sample_interval) AS provider_http_5xx,
364 sum(double14 * _sample_interval) AS tool_denied_by_policy,
365 sum(double15 * _sample_interval) AS tool_timeout,
366 sum(double16 * _sample_interval) AS network_error
367 FROM codewhale_telemetry
368 WHERE timestamp > NOW() - INTERVAL '7' DAY
369 AND blob1 IN ('session_end', 'panic')
370 GROUP BY panic_site
371 ORDER BY rows DESC
372 ```
373
374 One query, two answers, because `panic_site` is `''` on every non-panic row:
375
376 - the **`panic_site = ''` row** carries the six error-class totals across all
377 `session_end` events in the window — that is the error ranking;
378 - every **other row** is one panic site, ranked by how often it fired.
379
380 Sessions that ended in a panic are visible either way: `blob13 = 'panic'` on the
381 `session_end` row, and the `panic` event carries the site.
382
383 ---
384
385 ## Verifying no IP is stored
386
387 Three checks, in increasing order of how convincing they are.
388
389 **1. The source cannot ask for it.** `npm test` runs `test/no-ip.test.ts`, which
390 greps the shipped source. Prove the guard is live by adding a
391 `request.headers.get("CF-Connecting-IP")` line to `src/index.ts` and re-running —
392 three tests go red — then revert.
393
394 **2. Every stored column is accounted for.** The schema is closed and the layout
395 above is exhaustive; there is no free slot an address could occupy. Confirm the
396 deployed dataset has exactly the columns you expect:
397
398 ```sh
399 query "SELECT * FROM codewhale_telemetry LIMIT 1 FORMAT JSON"
400 ```
401
402 The result carries `dataset`, `timestamp`, `_sample_interval`, `index1`, and the
403 `blob*`/`double*` columns. There is no address column, because Analytics Engine
404 has no implicit one — every column is written by `writeDataPoint`.
405
406 **3. Search the stored data for an address shape.** After the first real
407 batches, this returns zero rows:
408
409 ```sql
410 SELECT count() AS suspicious
411 FROM codewhale_telemetry
412 WHERE timestamp > NOW() - INTERVAL '7' DAY
413 AND (
414 match(index1, '\\d+\\.\\d+\\.\\d+\\.\\d+')
415 OR match(concat(blob1, blob2, blob3, blob4, blob5, blob6, blob7, blob8,
416 blob9, blob10, blob11, blob12, blob13, blob14, blob15,
417 blob16, blob17), '\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}|:[0-9a-f]{1,4}:')
418 )
419 ```
420
421 Also confirm nothing is being logged: with `invocation_logs` off and no
422 `console.*` call in the source, **Workers Logs → this Worker** should show no
423 per-request entries at all.
424
425 ---
426
427 ## Local development
428
429 ```sh
430 npm install
431 npm test # vitest, 109 tests
432 npm run typecheck # tsc --noEmit
433 npm run check # wrangler deploy --dry-run — touches no account
434 npm run dev # wrangler dev --local
435 ```
436
437 `wrangler dev --local` runs the real `workerd` with local Analytics Engine and
438 rate-limit bindings. Post the client's own golden batch at it:
439
440 ```sh
441 curl -i -X POST http://127.0.0.1:8787/v1/telemetry \
442 -H 'content-type: application/json' \
443 --data-binary @../crates/telemetry/tests/golden/v1.json
444 # HTTP/1.1 204 No Content, zero-byte body
445
446 curl -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8787/v1/telemetry
447 # 405, Allow: POST
448 ```
449
450 ### Responses
451
452 | status | when |
453 |---|---|
454 | `204` | accepted, zero-byte body, no headers |
455 | `400` | not JSON, or fails the published schema — including any unknown key |
456 | `404` | POST to a path other than `/v1/telemetry` |
457 | `405` | any method other than POST, on any path (`Allow: POST`) |
458 | `413` | body over `MAX_BODY_BYTES` |
459 | `415` | content type is not `application/json` |
460 | `429` | this `install_id` is over the rate limit |
461 | `500` | internal error; nothing was written |
462
463 Every one of them has an empty body. The client
464 (`crates/telemetry/src/client.rs`) reads only the status class and drops the
465 batch on anything that is not 2xx — no retry, no backoff, no re-queue — so a
466 rejection is invisible to the user by construction, and a 5xx here can never
467 become a client-visible error. That is what lets this endpoint fail closed:
468 when in doubt, refuse the batch.
469
469 lines MARKDOWN