| 1 | # Catalog refresh |
| 2 | |
| 3 | How Codewhale keeps model metadata current — what already auto-updates, what |
| 4 | is hand-maintained, and what a scheduled catalog job should (and should not) do. |
| 5 | |
| 6 | Related docs: [`PROVIDERS.md`](./PROVIDERS.md), RFC |
| 7 | [`rfcs/UNIFIED_PROVIDER_LOGIN.md`](./rfcs/UNIFIED_PROVIDER_LOGIN.md). |
| 8 | |
| 9 | --- |
| 10 | |
| 11 | ## Short answer |
| 12 | |
| 13 | | Question | Answer | |
| 14 | |---|---| |
| 15 | | Do users need a special model just to refresh models? | **No.** | |
| 16 | | Does Codewhale auto-update the public model catalog? | **Yes, at runtime**, from [Models.dev](https://models.dev/catalog.json), ~24 h TTL. | |
| 17 | | Is the offline bundled seed auto-committed in CI? | **Not yet.** Live cache covers running installs; the in-repo seed is still manual / PR-driven. | |
| 18 | | Should an LLM rewrite catalog JSON? | **No.** Ingest is deterministic public JSON. An LLM can *review* a PR, not own the source of truth. | |
| 19 | |
| 20 | --- |
| 21 | |
| 22 | ## Layers (lowest → highest priority) |
| 23 | |
| 24 | The shared catalog compiler applies these layers from lowest to highest: |
| 25 | |
| 26 | ``` |
| 27 | 0 bundled Models.dev |
| 28 | 5 bundled Codewhale facts |
| 29 | 10 live Models.dev |
| 30 | 15 verified cloud facts (optional, off by default) |
| 31 | 20 exact provider-owned live roster |
| 32 | 25 Codewhale account roster |
| 33 | 30 config.toml |
| 34 | 40 user overrides |
| 35 | policy DENY (final) |
| 36 | ``` |
| 37 | |
| 38 | Cloud facts use the existing compiler and provider lake, as described in |
| 39 | [`CLOUD_FACTS.md`](./CLOUD_FACTS.md). Capability provenance and price provenance |
| 40 | are separate: a capability patch cannot relabel inherited prices. Cloud price |
| 41 | patches replace the entire price block; unspecified token classes stay unknown. |
| 42 | |
| 43 | Route resolution also binds provider kind, configured identity and endpoint. |
| 44 | A fresh provider-owned roster is authoritative for its exact scope. Explicit |
| 45 | model selections remain explicit. Codex account observations/native cache and |
| 46 | Ollama endpoint tags keep their dedicated availability rules; a public catalog |
| 47 | row does not prove that an account can call that model. The installed Codex |
| 48 | `account/read` and `model/list` path is documented in |
| 49 | [`PROVIDERS.md`](./PROVIDERS.md). |
| 50 | |
| 51 | Legacy completion lists remain a last fallback where no applicable catalog |
| 52 | exists. Bundled seeds and static transport/billing rules remain release-owned; |
| 53 | refreshing catalog metadata does not introduce a new wire dialect or change |
| 54 | credential/billing ownership. |
| 55 | |
| 56 | Key code: |
| 57 | |
| 58 | | Piece | Path | Role | |
| 59 | |---|---|---| |
| 60 | | Live fetch + cache | `crates/tui/src/models_dev_live.rs` | Background refresh, TTL, atomic write, freshness status | |
| 61 | | Schema / parse | `crates/config/src/models_dev.rs` | Network-free Models.dev JSON shape | |
| 62 | | Compile + provenance | `crates/config/src/catalog.rs` | Ordered sources, independent price provenance, policy deny, id normalization | |
| 63 | | Provider lake merge | `crates/tui/src/provider_lake.rs` | Shared catalog projection with exact route-scoped provider authority | |
| 64 | | Offline seed asset | `crates/config/assets/models_dev.bundled.json` | Compact offline fallback only (`_meta.role` says so) | |
| 65 | | Validation script | `scripts/catalog_models_dev.py` | Secret-free fetch/validate dry-run (#4117) | |
| 66 | | Script tests | `scripts/catalog_models_dev_test.py` | Offline shape/scrub checks | |
| 67 | |
| 68 | --- |
| 69 | |
| 70 | ## What already auto-updates (runtime) |
| 71 | |
| 72 | When the TUI/runtime starts (and is not disabled): |
| 73 | |
| 74 | 1. Seed pickers from the **on-disk cache** if present (even if stale). |
| 75 | 2. If the cache is missing or older than **24 hours**, **background-fetch** |
| 76 | Models.dev (15 s timeout, explicit Codewhale user-agent, **no credentials**). |
| 77 | 3. On success: atomic write to |
| 78 | `~/.codewhale/catalog/models-dev-catalog.json` and publish rows into |
| 79 | ProviderLake as `CatalogSource::ModelsDevLive` — layer 10, carrying no |
| 80 | endpoint fingerprint. Models.dev is a public catalog describing a model, so |
| 81 | a refreshed row is treated exactly like the layer-0 seed it supersedes and |
| 82 | stays correctable by layer 15. `CatalogSource::Live` is reserved for a |
| 83 | provider's own credential-scoped `/models` answer at layer 20. |
| 84 | 4. On failure: keep prior cache or fall back to the **bundled** seed. Model |
| 85 | selection never hard-fails because Models.dev is down. |
| 86 | |
| 87 | ### Manual force refresh |
| 88 | |
| 89 | In the TUI: |
| 90 | |
| 91 | ```text |
| 92 | /model refresh |
| 93 | ``` |
| 94 | |
| 95 | That dispatches `AppAction::RefreshModelsDevCatalog` (async; does not block |
| 96 | the composer). When admitted cloud-facts settings are enabled, it also requests |
| 97 | a cloud refresh; hard-disable and trust-key checks still apply. Implementation lives under |
| 98 | `crates/tui/src/commands/groups/core/core.rs` and |
| 99 | `crates/tui/src/models_dev_live.rs`. |
| 100 | |
| 101 | ### Env knobs (tests / dogfood / offline) |
| 102 | |
| 103 | | Variable | Effect | |
| 104 | |---|---| |
| 105 | | `CODEWHALE_MODELS_DEV_URL` | Override base URL or full `*.json` catalog URL | |
| 106 | | `CODEWHALE_MODELS_DEV_PATH` | Load catalog from a local file; skip network | |
| 107 | | `CODEWHALE_DISABLE_MODELS_DEV_FETCH` | Truthy → never hit the network (`1` / `true` / `yes` / `on`) | |
| 108 | |
| 109 | Defaults: |
| 110 | |
| 111 | - Catalog URL: `https://models.dev/catalog.json` |
| 112 | - TTL: `24 * 60 * 60` seconds (`DEFAULT_MODELS_DEV_TTL_SECS`) |
| 113 | - Cache file name: `models-dev-catalog.json` under the Codewhale `catalog` |
| 114 | state dir |
| 115 | |
| 116 | Freshness values exposed for UI / status chips: `bundled` | `live` | `stale` | |
| 117 | `failed`. |
| 118 | |
| 119 | --- |
| 120 | |
| 121 | ## What does **not** auto-update (repo / release) |
| 122 | |
| 123 | These stay hand-maintained or release-lane work until a scheduled PR lands: |
| 124 | |
| 125 | | Surface | Why it drifts | |
| 126 | |---|---| |
| 127 | | `models_dev.bundled.json` | Offline seed; intentionally smaller than full Models.dev | |
| 128 | | `model_catalog.bundled.json` | Compact TUI seed | |
| 129 | | `provider_defaults.rs` / default model IDs | Product choice, not pure catalog dump | |
| 130 | | Static tables in `models.rs` | Fallback heuristics when catalog misses a row | |
| 131 | | Hand-curated `pricing.rs` rows | Vendor billing quirks; not always in Models.dev | |
| 132 | | New `ProviderKind` / wire dialect | Needs code, not only JSON | |
| 133 | |
| 134 | Runtime live refresh **does not** rewrite those files. Users on a recent |
| 135 | install with network still see new Models.dev rows; fresh clones offline, CI |
| 136 | hermetic runs, and first-boot without cache still depend on the seed. |
| 137 | |
| 138 | --- |
| 139 | |
| 140 | ## Maintainer tooling (no LLM) |
| 141 | |
| 142 | ### Validate / dry-run fetch |
| 143 | |
| 144 | ```bash |
| 145 | # Fetch Models.dev + print counts (never writes disk) |
| 146 | python3 scripts/catalog_models_dev.py refresh |
| 147 | |
| 148 | # Validate the committed offline seed still parses as Models.dev-shaped JSON |
| 149 | python3 scripts/catalog_models_dev.py snapshot --check \ |
| 150 | crates/config/assets/models_dev.bundled.json |
| 151 | |
| 152 | # OpenRouter public /models listing (no API key), dry-run only |
| 153 | python3 scripts/catalog_models_dev.py refresh --provider openrouter \ |
| 154 | --sort newest --limit 100 |
| 155 | ``` |
| 156 | |
| 157 | Design constraints of the script (intentional): |
| 158 | |
| 159 | - Public endpoints only — no `Authorization` headers, no API keys. |
| 160 | - Credential-shaped keys are scrubbed if present in remote JSON. |
| 161 | - **Disk writes are disabled** (`--write` / `--write-cache` fail closed). |
| 162 | Staging a new seed is a separate maintainer step so remote JSON is never |
| 163 | blindly committed by automation without review. |
| 164 | |
| 165 | ### Staging a new offline seed (manual) |
| 166 | |
| 167 | 1. Fetch Models.dev to a local file (curl / browser), or use |
| 168 | `CODEWHALE_MODELS_DEV_PATH` against a saved copy. |
| 169 | 2. Scrub to the allowlisted shape (`models`, `providers`, optional `_meta`). |
| 170 | Prefer the script’s public-document rules as the checklist. |
| 171 | 3. Keep seed **compact** — verified defaults for shipped providers, not a |
| 172 | full dump (see `_meta` on the existing asset). |
| 173 | 4. `python3 scripts/catalog_models_dev.py snapshot --check <path>`. |
| 174 | 5. Diff carefully: default wire IDs should stay aligned with |
| 175 | `DEFAULT_*_MODEL` offline. |
| 176 | 6. Open a normal PR. Do not force-push catalog history. |
| 177 | |
| 178 | Optional: use a cheap model **on the PR** to summarize “new / removed / |
| 179 | default-risk” — never as the author of the JSON. |
| 180 | |
| 181 | --- |
| 182 | |
| 183 | ## Recommended scheduled job (not shipped yet) |
| 184 | |
| 185 | Goal: keep the **in-repo offline seed** from rotting, without giving CI write |
| 186 | power over secrets or unsupervised LLM rewrites. |
| 187 | |
| 188 | ```text |
| 189 | cron (daily or weekly) |
| 190 | → fetch Models.dev (public, no keys) |
| 191 | → validate shape + scrub |
| 192 | → compare against crates/config/assets/models_dev.bundled.json |
| 193 | (and optionally report new ids vs provider defaults) |
| 194 | → if material change: open PR |
| 195 | title: chore(catalog): refresh Models.dev offline seed |
| 196 | → optional: agent comments a human-readable diff summary on the PR |
| 197 | ``` |
| 198 | |
| 199 | ### In scope for automation |
| 200 | |
| 201 | - Deterministic catalog ingest from Models.dev |
| 202 | - Secret-free PR diffs |
| 203 | - Drift reports (new model ids, missing defaults, pricing presence) |
| 204 | |
| 205 | ### Out of scope for automation |
| 206 | |
| 207 | - Claude Pro/Max / subscription OAuth “model discovery” (not a supported |
| 208 | third-party path; Anthropic expects API keys for third-party tools) |
| 209 | - LLM-authored edits to `models.rs` / `provider.rs` without review |
| 210 | - Force-pushing `main` or silent asset rewrites on the default branch |
| 211 | - Treating Models.dev as the only truth for OAuth-scoped routes (Codex |
| 212 | roster remains special-cased) |
| 213 | |
| 214 | ### Suggested workflow home |
| 215 | |
| 216 | `CodeWhale/.github/workflows/catalog-refresh.yml` (or similar), reusing |
| 217 | `scripts/catalog_models_dev.py` after a deliberate **write-safe** extension |
| 218 | that only runs in CI with a bot token for PR creation — still not on |
| 219 | `workflow_dispatch` without review if writes land in-repo. |
| 220 | |
| 221 | Nightly today (`/.github/workflows/nightly.yml`) builds release artifacts |
| 222 | only; it does **not** refresh catalogs. |
| 223 | |
| 224 | --- |
| 225 | |
| 226 | ## Do we need a “model dedicated to updating models”? |
| 227 | |
| 228 | **No for the core loop.** |
| 229 | |
| 230 | | Job | Right tool | |
| 231 | |---|---| |
| 232 | | Keep known models/windows/prices from Models.dev fresh for users | Runtime live fetch (already shipped) | |
| 233 | | Keep offline seed + release assets current in git | Scheduled CI → PR (to build) | |
| 234 | | Decide whether to bump a product default model | Human (or agent *review* on the PR) | |
| 235 | | Wire a brand-new provider kind / dialect | Human PR + tests | |
| 236 | |
| 237 | An LLM is optional **review** of a catalog PR. It is a poor **source of |
| 238 | truth** for catalog JSON. |
| 239 | |
| 240 | --- |
| 241 | |
| 242 | ## Auth note (Claude / Anthropic) |
| 243 | |
| 244 | Anthropic model **catalog** refresh does not require Claude Pro/Max OAuth. |
| 245 | Models.dev is public. Codewhale’s Anthropic route remains **API-key-based** |
| 246 | for inference (`ANTHROPIC_API_KEY`). Do not couple catalog automation to |
| 247 | subscription OAuth or Claude Code identity headers. |
| 248 | |
| 249 | --- |
| 250 | |
| 251 | ## Quick operator checklist |
| 252 | |
| 253 | - [ ] Running install: confirm network not blocked; optional |
| 254 | `/model refresh` after a big vendor launch. |
| 255 | - [ ] Offline / CI hermetic: set `CODEWHALE_DISABLE_MODELS_DEV_FETCH=1` or |
| 256 | point `CODEWHALE_MODELS_DEV_PATH` at a fixture. |
| 257 | - [ ] Before release: `snapshot --check` on the bundled seed; skim |
| 258 | `PROVIDERS.md` for known drift. |
| 259 | - [ ] After Models.dev adds a major family you ship by default: consider |
| 260 | seed PR + default-model decision separately. |
| 261 | - [ ] Never paste API keys into catalog assets or the automation script env |
| 262 | for Models.dev refresh. |
| 263 | |
| 264 | --- |
| 265 | |
| 266 | ## Issue / design anchors |
| 267 | |
| 268 | - Live Models.dev layer: #4187 |
| 269 | - Bundled seed demoted (not competing truth): #4188 |
| 270 | - Catalog automation script (validate / dry-run): #4117 |
| 271 | - Deeper metadata inventory and drift list: the `codewhale-ops` repo |
| 272 |