| 1 | # Configuration |
| 2 | |
| 3 | Everything you can tune in `/last30days` without editing the engine source. |
| 4 | Three layers, in order of how often you'll touch them: |
| 5 | |
| 6 | 1. **Per-run flags** - what you pass on the command line. |
| 7 | 2. **Environment variables and `.env`** - what's enabled across all runs. |
| 8 | 3. **Optional trend-monitoring stack** - SQLite store, watchlist, briefings. |
| 9 | |
| 10 | Per-client patterns and the experimental beta channel are at the bottom. |
| 11 | |
| 12 | > Skip ahead: [Where output is saved](#where-output-is-saved) - [API keys](#api-keys-env) - [Reasoning provider](#reasoning-provider-priority) - [Web search backend](#web-search-backend-priority) - [Trend monitoring](#trend-monitoring-store--watchlist--briefings) - [Per-client patterns](#per-client-patterns) - [Beta channel](#beta-channel) |
| 13 | |
| 14 | ## Why this document exists |
| 15 | |
| 16 | This is a focused **configuration reference** maintained alongside the engine. The runtime contract (the voice rules, the planner protocol, the LAWs the synthesizing model follows) lives in [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) - that file is authoritative when the two ever differ. This file's job is narrower: surface every knob a user or operator can turn, in one place, kept current with the code so client-facing setups stay reliable. New configuration knobs added to the engine should be reflected here in the same PR. |
| 17 | |
| 18 | --- |
| 19 | |
| 20 | ## Where output is saved |
| 21 | |
| 22 | | Platform | Default path | Override | |
| 23 | |---|---|---| |
| 24 | | Linux / macOS | `LAST30DAYS_MEMORY_DIR` defaults to `~/Documents/Last30Days/` | set `LAST30DAYS_MEMORY_DIR=/path` | |
| 25 | | Windows | `LAST30DAYS_MEMORY_DIR` defaults to `C:\Users\<you>\Documents\Last30Days\` | set `LAST30DAYS_MEMORY_DIR=C:\path` | |
| 26 | |
| 27 | Each run produces one file per topic, slug-named: |
| 28 | `<slug>-raw[-suffix].md`. Same topic + same suffix on the same day overwrites; same topic + same suffix on different days appends a date stamp. |
| 29 | |
| 30 | ### Recommended `.env` entry |
| 31 | |
| 32 | `.env` files don't travel between machines or harnesses, so set `LAST30DAYS_MEMORY_DIR` explicitly in `~/.config/last30days/.env` once per host. The `/last30days` slash command works without it (the SKILL.md wrapper has its own default), but **bare engine invocations** — `python3 scripts/last30days.py ...` from cron jobs, scripts, or agents that bypass the wrapper — silently no-op the file save unless the engine sees the env var. Mirrors the `LAST30DAYS_STORE` env-or-flag convention. |
| 33 | |
| 34 | ```bash |
| 35 | # ~/.config/last30days/.env (pick ONE — uncomment the line that matches your OS) |
| 36 | LAST30DAYS_MEMORY_DIR=~/Documents/Last30Days # POSIX — defaults to this path when unset |
| 37 | # LAST30DAYS_MEMORY_DIR=C:\Users\<user>\Documents\Last30Days # Windows |
| 38 | # LAST30DAYS_LIBRARY_OWNER=Your Name # Optional Atom feed author |
| 39 | # LAST30DAYS_LIBRARY_CONTEXT=off # Disable prior-run context (default: on) |
| 40 | ``` |
| 41 | |
| 42 | The engine's `.env` reader doesn't expand `$HOME` — only the tilde, via `Path().expanduser()` downstream. Use `~/...` or an absolute path; **don't** write the literal string `$HOME/...` into your `.env` (it gets stored verbatim and breaks path resolution). |
| 43 | |
| 44 | **Per-run overrides:** |
| 45 | |
| 46 | - `--save-dir <path>` - one-off output location. **Flag wins over env var.** If neither flag nor env var is set, the engine does not write a file (DB persistence is independent — see `LAST30DAYS_STORE` below). |
| 47 | - `--output <file>` - write the rendered output to an exact file path, using the format selected by `--emit`. |
| 48 | - `--json-profile {agent,raw}` - select the research JSON shape used with `--emit=json`. `agent` is the default, versioned workflow contract; `raw` preserves the full internal `Report` dump for debugging and power users. See the [JSON export reference](docs/reference/json-export.md). |
| 49 | - `--corpus <dir>` - add a local `.md`/`.txt` directory as a private ranked source; repeat the flag for multiple directories. PDFs are extracted only when `pdftotext` is on PATH and otherwise skip with a note. File modification time supplies recency, so the normal research window applies. |
| 50 | - `--corpus-all-time` - include relevant registered files whose modification time is older than the current research window. Without this flag, a 30-day run includes only files modified in those 30 days. |
| 51 | - `--register {default,exec,dev,creator,eli5}` - shape a standard single-topic Markdown or HTML research brief for its audience. `exec` is decisions-first with five core findings and numbers up top; `dev` gives GitHub, code, and technical signals more room; `creator` leads with hooks, Best Takes, community reactions, and virality metrics; `eli5` keeps the established evidence layout and asks the synthesizing agent for accessible language. Registers do not change retrieval, JSON exports, discovery, drill, library feed/search, or comparison output. |
| 52 | - `--discover [domain]` - trending discovery, two-stage: a river-listing sweep NOMINATES candidate topics, then each nomination gets a full research pass (Reddit with comments, X, YouTube, Techmeme, arXiv, HN, Polymarket, web) before ranking. Bare `--discover` (no domain) is **global trending**: every feed's own hot list (r/all rising/top-week, Hacker News front/best, Digg clusters when `digg-pp-cli` is on PATH) with no keyword gate; with a domain, the sweep is category-scoped and keyword-gated, and broad X activity joins when an X backend is authenticated. Every topic must clear a confidence floor (cross-source confirmation or a genuinely strong single-source spike); when nothing clears it the run reports "Nothing solid this window" instead of ranked noise. Run without a positional topic; it is mutually exclusive with `--drill`. `--emit=json` uses the separate versioned discovery contract (now with `outcome`, `weak_signal`, per-topic `top_comment` and `corroboration_count`) documented in the [JSON export reference](docs/reference/json-export.md). |
| 53 | - `--discover-shallow` - skip discovery's per-topic research passes and rank on listing evidence only. Faster and thinner; the confidence floor still applies. An explicit `--search` source list bounds both the sweep and the research passes. On a protocol run (below), adding it to the `--nominate-only` leg marks the bundle quick-tier so the resume leg uses the faster shallow research pass. |
| 54 | - `--nominate-only` - leg 1 of the three-command host-judged discovery protocol (agent hosts; SKILL.md drives it - one-shot `--discover` stays the scripting/cron form with deterministic topic names and no angles). With `--discover [domain]`: sweep the listings, write the nominations bundle (`discover-nominations.json` in the save dir, TTL one hour) for host judgment, print a judging digest, and stop - no enrichment, no queue writes. A zero-nomination sweep prints the nothing-solid brief directly. |
| 55 | - `--judgments <path>` - leg 2: resume from the nominations bundle, applying the host judgments file (`{"bundle_id": "...", "judgments": [{"id", "name", "junk", "worthiness"}, ...]}`, bound to the bundle by `bundle_id`). Runs the per-topic research passes (deep tier by default; budget tunable via `LAST30DAYS_ENRICH_BUDGET_SECONDS` below), writes the pending report (`discover-pending.json`), and prints per-topic angle inputs. Requires `--discover`. |
| 56 | - `--finalize` - leg 3: apply optional host angles to the pending report, render the final discovery brief, save artifacts, and record the topic queue (retries are idempotent - the pending file stays in place within its TTL). Offline; requires `--discover`. |
| 57 | - `--angles <path>` - optional host angles file for `--discover --finalize` (`{"bundle_id": "...", "angles": [{"id", "podcast", "x_article"}, ...]}`, sentences capped at 200 chars); omitting it ships the brief without angle lines. All three protocol legs must share one `--save-dir` (handoff files live there, else in `~/.config/last30days/`); contract failures (missing/stale/unbound handoff files) exit 2 with the remedy on stderr, and `--mock` protocol legs require `--save-dir` to stay side-effect-free. |
| 58 | - `--drill <target>` - deep follow-up over the fresh `~/.config/last30days/last-report.json` cache. Accepts a 1-based index (`--drill "cluster 3"` or `--drill "3"`) or a fuzzy cluster title/entity description. It re-fetches only sources that contributed to the matched cluster, enables their deep comment/transcript enrichment paths, merges/dedupes the evidence, and replaces the cache so drills can chain. Run it without a positional topic; if the cache is absent or expired, run a normal research pass first. |
| 59 | - `--verify-freshness` - opt into an act-time verification pass for conservatively extracted, source-grounded claims (Polymarket odds/end dates, GitHub stars, StockTwits sentiment ratios, and explicit status assertions). With a topic, verification runs after research; without a topic, it re-verifies the fresh `last-report.json` cache without repeating research. Verdicts are `current`, `stale`, `contradicted`, or `unsupported` and include evidence timestamps. Set `LAST30DAYS_VERIFY_FRESHNESS=on` in `.env` to make the pass default for normal research runs. |
| 60 | - `--save-suffix <name>` - distinguish runs of the same topic (e.g. per client: `--save-suffix=acme`). |
| 61 | - `--no-browser-cookies` - hard-disable browser-cookie extraction for this run, even when `FROM_BROWSER` is configured. MCP and folder-mode hosts use this for safe defaults. |
| 62 | - `--publish-html` - with `--emit=html`, publish the rendered HTML to `ht-ml.app` after local output/save-dir writes. This is explicit opt-in only; pages are public by default. |
| 63 | - `library feed` - scan `LAST30DAYS_MEMORY_DIR` plus `~/.local/share/last30days/briefs/`, then write a self-contained `index.html`, valid Atom `feed.xml`, and browser-ready pages under `briefs/`. The index is reverse-chronological and grouped by topic. For direct engine use: `python3 skills/last30days/scripts/last30days.py library feed`; use `--save-dir <path>` to scan and write another library directory. |
| 64 | - `library feed --publish` - publish each rendered brief and the HTML index through `ht-ml.app`. The generated `feed.xml` remains a first-class local artifact because this HTML host does not serve Atom with an XML content type. Host the output directory on any static host (for example, GitHub Pages) to make `feed.xml` subscribable. Publishing is explicit opt-in and pages are public by default; public pages may be crawled or indexed. |
| 65 | - `library search "<query>"` - incrementally sync `LAST30DAYS_MEMORY_DIR` and `~/.local/share/last30days/briefs/` through the shared library scanner, then run offline SQLite FTS5 across those briefs plus dated per-run sightings in `~/.local/share/last30days/research.db`. Results are grouped by topic run. The sibling search index lives at `~/.local/share/last30days/library.db`; hand edits, renames, and deletes are picked up on sync, and a corrupt index is rebuilt automatically. |
| 66 | - `LAST30DAYS_LIBRARY_OWNER=<name>` - optional feed-level Atom author. Defaults to `last30days research library`. |
| 67 | - `LAST30DAYS_LIBRARY_CONTEXT=on|off` - controls passive prior-run context on fresh research reports. It defaults to `on`; matching saved research appears in a short `From your library` section. Set `off` to skip the local index read and leave reports unchanged. Mock runs, eval replays, and internal fan-out subruns do not load library context, keeping fixtures deterministic. |
| 68 | - `--publish-password <password>` - optional shared password for `--publish-html` or `library feed --publish`. Prefer `LAST30DAYS_PUBLISH_PASSWORD=<password>` instead so the password is not visible in the process list or shell history. Use a unique non-personal password; never reuse the user's own password. The provider's update key is treated as secret and is not written to stdout, HTML, raw output, or `.publish.json` metadata. |
| 69 | - `--preflight` - optional permission inspector. It reports config source, project config trust/ignore state, browser-cookie plan, planned writes, optional commands, source availability, and endpoint overrides without reading browser cookies, writing setup/config/report files, or running research. First-run setup does not require it. Add `--emit=json` for the separate machine-readable preflight contract (`--json-profile` does not change it); use `--diagnose` when you need the full source diagnostic JSON. `doctor` is the health surface. |
| 70 | - `--welcome` - print the first-run welcome text (engine-owned; the skill relays it verbatim on first run). Safe: prints and exits, no reads or writes. |
| 71 | - `--record-fixtures <dir>` - developer-only, hidden flag that records scrubbed source responses for the offline research-quality eval harness. It writes `<dir>/http.json`; see the [eval reference](docs/reference/eval.md) before recording or committing fixtures. |
| 72 | - `setup --github-start` / `setup --github-poll` - the two-command ScrapeCreators GitHub device-auth split. `--github-start` submits the device flow, copies the code to the clipboard, opens the browser, and returns the code immediately (foreground); `--github-poll` waits for you to authorize and persists the key. `setup --github` still runs both in one shot for back-compat. |
| 73 | |
| 74 | The footer line `📎 Raw results saved to ${LAST30DAYS_MEMORY_DIR:-$HOME/Documents/Last30Days}/<slug>-raw.md` is the canonical pointer; if it shows backslashes on Windows update past v3.1.1. |
| 75 | |
| 76 | Every completed research pass writes a structured `last-report.json` cache beside `last-run.json`. HTML follow-up renders use it so `--emit=html --synthesis-file` can reuse report metadata/footer without fetching sources again; `--drill <target>` uses it as the grounded starting point for targeted re-research; bare `--verify-freshness` updates only the cached report's claim verdicts. Reuse is intentionally short-lived: `LAST30DAYS_REPORT_CACHE_TTL_SECONDS` defaults to `3600` (one hour). Set it to another integer number of seconds to tune the window, or `0` to disable report-cache reuse and post-run follow-ups. |
| 77 | |
| 78 | --- |
| 79 | |
| 80 | ## First-run onboarding |
| 81 | |
| 82 | On the very first `/last30days` run (no `~/.config/last30days/.env`, or `SETUP_COMPLETE` not set), the skill runs a consent-driven onboarding the model drives in chat. It takes one of three forms depending on the host: |
| 83 | |
| 84 | - **Claude Code Modal Flow** - the restored v3.0.0 guided NUX, used on hosts with `AskUserQuestion` (Claude Code). A welcome message, then modals for Auto/Manual/Skip setup, cookie consent, the ScrapeCreators signup offer, a TikTok/Instagram `INCLUDE_SOURCES` opt-in, and a first-topic picker. |
| 85 | - **Non-Modal Prose Flow** - the same work done conversationally on hosts without modals (OpenClaw, Codex, Cursor, Gemini CLI, Grok, raw CLI). |
| 86 | - **Grok Bot Prose Flow** - the prose flow on a Grok Bot host (`LAST30DAYS_HOST=grok-bot`, persisted to `.env` by this setup). It has no browser-session step: X is set up through the bot's X connector, with `X_BEARER_TOKEN` or `XAI_API_KEY` as backups (see [Grok Bot](#grok-bot) under Per-client patterns). |
| 87 | |
| 88 | The Modal and Non-Modal flows share the same consent points: |
| 89 | |
| 90 | 1. **Browser cookies** - the model asks before reading anything. On yes it runs `setup --allow-browser-cookies`, which extracts Firefox/Safari cookies (never Chrome unless `FROM_BROWSER=auto` or a named Chromium browser is explicitly configured) to unlock X/Twitter and other logged-in sources, and installs yt-dlp + the keyless Digg CLI. On no it runs setup without `--allow-browser-cookies` (or with `FROM_BROWSER=off`), which skips all cookie reads and still installs the tools. |
| 91 | 2. **Full Disk Access (macOS)** - if a cookie read is permission-denied, the model surfaces the System Settings > Privacy & Security > Full Disk Access fix and offers one retry. |
| 92 | 3. **ScrapeCreators GitHub signup** - offered on every first run (10,000 free calls). On consent it runs `setup --github`, which opens a browser for GitHub device-auth (or registers instantly via the `gh` CLI when installed) and, on success, **persists `SCRAPECREATORS_API_KEY` automatically** (0o600, masked in output) so TikTok, Instagram, empty-path Reddit search backup, and the YouTube transcript fallback activate on the next run. Decline anytime; you can run it later by asking to set up ScrapeCreators. The Step 5 opt-in has two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts AND top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything**, which also adds Threads + Pinterest. Comments are on by default; Threads and Pinterest are the only opt-in extras. |
| 93 | |
| 94 | Re-run onboarding by deleting `~/.config/last30days/.env`. The mechanical work lives in `scripts/lib/setup_wizard.py`; the consent conversation and both host flows are specified in `skills/last30days/SKILL.md` Step 0. The original v3.0.0 wizard is captured at `docs/reference/old-nux-wizard-v3.0.0.md`. |
| 95 | |
| 96 | --- |
| 97 | |
| 98 | ## MCP host controls |
| 99 | |
| 100 | These variables configure the local Go MCP server and are read from its process environment, not from the engine's `.env` files: |
| 101 | |
| 102 | | Env var | Default | Accepted values | Behavior and precedence | |
| 103 | |---|---|---|---| |
| 104 | | `LAST30DAYS_PYTHON` | unset | An executable name or path | Selects the Python 3.12+ interpreter used by the MCP server. A caller-supplied `RunOptions.PythonPath` remains the test/caller override; otherwise this variable must resolve to an executable. When it is unset, the server looks up `python3` on `PATH`. An empty or unresolvable value is an error rather than a fallback. | |
| 105 | | `LAST30DAYS_MCP_ALLOW_BROWSER_COOKIES` | unset (deny) | `1`, `true`, `yes`, or `on`, case-insensitive | A recognized truthy value removes the MCP layer's default `--no-browser-cookies` flag. Every other value keeps that denial. This switch grants no consent by itself: browser-cookie access still requires the engine's separately recorded consent and `FROM_BROWSER` configuration. | |
| 106 | |
| 107 | ## API keys (`.env`) |
| 108 | |
| 109 | The skill reads keys from a `.env` file. Two locations are supported: |
| 110 | |
| 111 | 1. **`~/.config/last30days/.env`** at the user level (global default) - loaded by default. |
| 112 | 2. **`.claude/last30days.env`** in the current project directory (project-scoped) - loaded only when trusted by setting `LAST30DAYS_TRUST_PROJECT_CONFIG=1` in the process environment or global config. |
| 113 | |
| 114 | Override the global location with `LAST30DAYS_CONFIG_DIR=/path` (or `LAST30DAYS_CONFIG_DIR=""` for no-config mode). File permissions should be `600` on POSIX hosts - the engine warns on every run if they aren't. |
| 115 | |
| 116 | **File syntax.** One `KEY=value` per line; whitespace around the key and value is trimmed. A line starting with `#` is a comment, and a `#` preceded by whitespace after an unquoted value starts a trailing comment (`RATE=1 # req/sec` stores `1`). A `#` glued to the value is literal (`TOKEN=abc#123` stores `abc#123`). Wrap a value in matching `"` or `'` to keep spaces or a `#` verbatim (`NAME="Jane # Doe"` stores `Jane # Doe`); no backslash escapes or `$VAR` expansion are processed. An empty value is ignored (the one exception is `LAST30DAYS_YT_PLAYER_CLIENT=`, where empty is a deliberate disable), so a secret can never be set to `""`. |
| 117 | |
| 118 | The project-scoped file is useful for **intentional per-client setups**: drop a `.claude/last30days.env` into each client folder (`SCRAPECREATORS_API_KEY`, `INCLUDE_SOURCES`, `LAST30DAYS_MEMORY_DIR`, `BSKY_HANDLE`, etc), then opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or `~/.config/last30days/.env`. Folder-mode hosts such as Codex desktop do not trust hidden project config by default, and discovery stops at the git root so unrelated parent folders cannot silently influence runs. An untrusted repo's `.claude/last30days.env` is not read. |
| 119 | |
| 120 | **`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract. |
| 121 | |
| 122 | **`BRIGHTDATA_API_KEY`** - optional, for the `amazon` source. The Bright Data CLI normally owns its own auth via `brightdata login`, so this is only needed if you would rather keep an explicit key in `.env` or the keychain. It is resolved through the standard config layering and passed to the CLI through the child process environment, never on the command line (where it would be readable from `/proc/<pid>/cmdline` by other local users on a shared host). |
| 123 | |
| 124 | **`LAST30DAYS_AMAZON_DOMAIN`** - optional, default `https://www.amazon.com`. The marketplace the `amazon` source searches; set it to `https://www.amazon.co.uk`, `https://www.amazon.de`, and so on. Product URLs are validated against this host, so records from other marketplaces are rejected. |
| 125 | |
| 126 | **`LAST30DAYS_META_ADS_COUNTRY`** - optional, default `US`. The Ad Library country the `meta_ads` source queries, as a two-letter code. The endpoint accepts exactly one country per call, so this selects it rather than widening it. There is deliberately no durable env form of the advertiser-page override: a page id belongs to one topic, and env values are inherited by every competitor sub-run in a comparison, which would render one brand's ads as every peer's. Use `--meta-ads-page` per run instead. |
| 127 | |
| 128 | ### Local corpus (your files) |
| 129 | |
| 130 | Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows): |
| 131 | |
| 132 | ```bash |
| 133 | # ~/.config/last30days/.env |
| 134 | LAST30DAYS_CORPUS_DIRS=~/notes:~/meeting-transcripts |
| 135 | # LAST30DAYS_CORPUS_IN_EXPORT=1 # explicit agent-JSON opt-in; off by default |
| 136 | ``` |
| 137 | |
| 138 | The slash-command experience remains primary: ask `/last30days` to include your registered notes. For direct engine scripting or development, the equivalent one-off invocation is: |
| 139 | |
| 140 | ```bash |
| 141 | python3 skills/last30days/scripts/last30days.py "MCP servers" \ |
| 142 | --corpus ~/notes --corpus ~/meeting-transcripts |
| 143 | ``` |
| 144 | |
| 145 | **Privacy:** corpus files are read locally, never sent through a source HTTP client, never forwarded to `LAST30DAYS_API_BASE`, never included in remote reranker/fun-scoring prompts, and do not consume network-source concurrency or retry budget. Matches appear in a badged **From your files** section. Corpus candidates are removed from `--publish-html`, `library feed --publish`, and the versioned agent JSON export by default, including corpus-derived cluster titles and source outcomes. Set `LAST30DAYS_CORPUS_IN_EXPORT=1` only when you intentionally want corpus results in the agent JSON written to local stdout/files. The unversioned `--json-profile=raw` debug dump remains a full local report and can contain corpus text; do not redirect it to an external system unless that is intentional. Extracted text is cached by file mtime in `~/.config/last30days/corpus-cache.json` with mode `0600`; a corpus-bearing `last-report.json` cache is also tightened to `0600`. Delete either cache at any time to clear it. |
| 146 | |
| 147 | **Source-by-source** - what each key unlocks: |
| 148 | |
| 149 | | Source | Key(s) | Required for | Free tier | |
| 150 | |---|---|---|---| |
| 151 | | Local corpus | `--corpus <dir>` or `LAST30DAYS_CORPUS_DIRS` | private `.md`/`.txt`; `.pdf` when `pdftotext` is on PATH | yes (offline) | |
| 152 | | Reddit (public) | none (default free keyless path). With `SCRAPECREATORS_API_KEY`: empty-only search backup by default; `LAST30DAYS_REDDIT_SC_MIN_ITEMS=<N>` backfills thin free runs; `LAST30DAYS_REDDIT_BACKEND=scrapecreators` pins SC primary with free fallback. `LAST30DAYS_REDDIT_KEYLESS_RATE` paces unauthenticated reddit.com requests (default `1` req/sec) | always on; SC knobs require `SCRAPECREATORS_API_KEY` | yes | |
| 153 | | Hacker News | none | always on | yes | |
| 154 | | Polymarket | none | always on | yes | |
| 155 | | StockTwits | none | auto-on for ticker/crypto topics only (gated by symbol detection); never registered for non-financial topics | yes (public API, ~200 req/hr per IP) | |
| 156 | | DripStack | none | opt-in only: per run with `--search dripstack`, or persistently with `INCLUDE_SOURCES=dripstack` in `.env`. Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active without the opt-in. | yes when opted in (public API, no auth) | |
| 157 | | GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes | |
| 158 | | YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds a server-side transcript fallback used only when yt-dlp fails (429 / bot-gate) | always on if `yt-dlp` present; SC transcript fallback default-on when key set (no credit spent unless yt-dlp fails) | yes | |
| 159 | | YouTube comments | `yt-dlp` CLI installed — **free and keyless, no API key and no opt-in needed**. Falls back to `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` containing `youtube_comments` only when yt-dlp is absent. Suppress with `EXCLUDE_SOURCES=youtube_comments`. | top comments (by likes) on the top ~3 videos by engagement | yes — free via yt-dlp (no credits spent) | |
| 160 | | TikTok comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `digg_count`) on the top ~3 TikTok posts | ~3 calls/run; 10K free calls | |
| 161 | | Instagram comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `comment_like_count`) on the top ~3 Instagram posts, via `/v2/instagram/post/comments` | ~3 calls/run; 10K free calls | |
| 162 | | Digg | `digg-pp-cli` on PATH (auto-installed during first-run setup via `npx -y @mvanhorn/printing-press-library@0.1.16 install digg --cli-only`; binary defaults to `$HOME/.local/bin` — Hermes/OpenClaw agent subprocesses must inherit that dir on PATH for Digg to activate; prior pp-digg installs use the same path) | always on if `digg-pp-cli` on PATH | yes (free, keyless, read-only) | |
| 163 | | arXiv | `arxiv-pp-cli` on PATH (auto-installed during first-run setup via `npx -y @mvanhorn/printing-press-library@0.1.16 install arxiv --cli-only`) | always on if `arxiv-pp-cli` on PATH; fires on research/technical topics and stays quiet otherwise (relevance + 365-day recency gating) | yes (free, keyless) | |
| 164 | | Techmeme | `techmeme-pp-cli` on PATH (auto-installed via `... install techmeme --cli-only`) | always on if `techmeme-pp-cli` on PATH; searches Techmeme's live archive and keeps only headlines dated within the research window (undated headlines flow through as low-confidence) | yes (free, keyless) | |
| 165 | | Trustpilot | `trustpilot-pp-cli` on PATH (NOT auto-installed; install on demand via `npx -y @mvanhorn/printing-press-library@0.1.16 install trustpilot --cli-only`) + (`INCLUDE_SOURCES` contains `trustpilot` **or** an explicit `--trustpilot-domain` / plan-level `trustpilot_domain`) | **opt-in, off by default**; `--trustpilot-domain=<domain>` (and per-entity `trustpilot_domain` in `--competitors-plan`) auto-activates the source for that run and bypasses the brand-shape gate. Persist with `INCLUDE_SOURCES=trustpilot` to skip per-run auto-enable. `EXCLUDE_SOURCES=trustpilot` still wins. Bare company names auto-resolve to the review-page domain via the CLI's search only when the source is already active. The session warms once before the search fan-out; a stale session does a ~10s headless-Chrome WAF-cookie harvest (set `LAST30DAYS_TRUSTPILOT_NO_BROWSER=1` to disable in cron/CI) | yes (no API key; cookie-replay after the one-time harvest) | |
| 166 | | Amazon | `brightdata` CLI on PATH **and logged in** (NOT auto-installed: `npm i -g @brightdata/cli` then `brightdata login`) + (`INCLUDE_SOURCES` contains `amazon` **or** `--search` includes `amazon`) | product records with live rating, rating count, and price, plus a capped sample of recent written reviews woven as buyer voice; the emoji footer shows each product's all-time-vs-last-30-days drift | **opt-in, off by default**. Free tier is 5,000 requests/month (~$7.50 equivalent); a typical run spends 4 (1 product search + up to 3 review pulls) regardless of how many reviews come back, since billing is per request. Past the free tier it bills the account balance at $1.50 per 1,000 records (~$0.32 for a default run). `--amazon-query=<keyword>` sets the product keyword when it differs from the topic; `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace. `EXCLUDE_SOURCES=amazon` wins. Never auto-fires: the model requests it per run or the user enables it durably | |
| 167 | | X / Twitter | one of: `X_BEARER_TOKEN` (official X API v2; opt-in outside Grok Bot: `LAST30DAYS_X_BACKEND=xapi`; covers recent posts, about the last week, unless your X developer project has full-archive access), a signed-in `grok` CLI (opt-in: `LAST30DAYS_X_BACKEND=grok`), `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, or `FROM_BROWSER` (cookie-jar auth). On a Grok Bot host the bot's X connector serves X first; see [Grok Bot](#grok-bot) | X items in results | X API bearer = your X developer project's credits; grok = Grok plan, opt-in only; cookie-jar / Bird = free; Xquik / xAI = key-based | |
| 168 | | TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls | |
| 169 | | Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network | |
| 170 | | Threads | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `threads` | Threads items | 10K free calls | |
| 171 | | Pinterest | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `pinterest` | Pinterest items | 10K free calls | |
| 172 | | LinkedIn | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `linkedin` | LinkedIn posts + articles (articles rank as high signal on person topics) | 10K free calls; power-user opt-in, not offered during first-run onboarding | |
| 173 | | Meta Ads | `SCRAPECREATORS_API_KEY` + (`INCLUDE_SOURCES` contains `meta_ads` **or** `--search` includes `meta_ads`) | **opt-in, off by default**; a brand's live Meta Ad Library creatives that *launched* inside the window, with ad copy, launch date, placements, CTA, landing product, any promo code, and spoken transcripts for the newest video ads. The 📣 footer names the advertiser page that was resolved, plus how much the brand is still running from before. Paid message only, never audience reaction: Meta publishes reach and spend for political ads alone, so commercial creatives carry no engagement. `--meta-ads-page=<page_id>` skips name-based resolution when it picks the wrong company or the brand advertises under product-line page names; `LAST30DAYS_META_ADS_COUNTRY` selects a non-US Ad Library (one country per call). `EXCLUDE_SOURCES=meta_ads` wins. Never auto-fires and never inferred from topic shape: on a non-brand topic, resolution returns unrelated advertisers | 10K free calls; a default run spends at most 7 (1 resolve, plus 1 more only when the first search finds no name match, + up to 2 creative pages + up to 3 transcripts), billed per request. Power-user opt-in, not offered during first-run onboarding | |
| 174 | | Telegram | `SCRAPECREATORS_API_KEY` + (`--telegram-sources=<handles>` **or** `TELEGRAM_SOURCES=<handles>` + `INCLUDE_SOURCES` contains `telegram`) | **opt-in, off by default**; public channel posts only (no keyword discovery). `--telegram-sources=aipost,durov` (or `TELEGRAM_SOURCES` env) auto-activates for that run like `--trustpilot-domain`. Accepts bare handle, `@handle`, `t.me/URL`, or `t.me/s/URL`; rejects joinchat links and numeric -100 IDs. `INCLUDE_SOURCES=telegram` or `--search telegram` without a channel list does not fetch. `EXCLUDE_SOURCES=telegram` wins. `TELEGRAM_MAX_PAGES` overrides page cap (quick=1, default=3, deep=6). Never on Recommended onboarding tier. | 1 credit per live posts page; 10K free calls | |
| 175 | | Xiaohongshu (RED) | logged-in x-mcp browser plugin or `xiaohongshu-mcp` service; optional `XIAOHONGSHU_API_BASE` for custom URLs | requested-only via `--search xhs` or `--search xiaohongshu`; auto-probes `http://localhost:18060` then `http://host.docker.internal:18060` | no last30days API key; depends on your local browser-session service | |
| 176 | | Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | Bluesky items | yes (app password at bsky.app) | |
| 177 | | TruthSocial | `TRUTHSOCIAL_TOKEN` | TruthSocial items | yes | |
| 178 | | Web search | one of: `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` | `--auto-resolve` and Step 2 supplements | Brave has a free tier; native WebSearch on Claude Code / Codex / Gemini works as a fallback | |
| 179 | | Perplexity Agent API / Search API / Deep Research | `PERPLEXITY_API_KEY` (preferred) or `OPENROUTER_API_KEY` (Sonar fallback) | `INCLUDE_SOURCES=perplexity`; `--deep-research` uses background Agent API with a direct key or synchronous Sonar through OpenRouter | no | |
| 180 | | Caption-free transcription | `GROQ_API_KEY` (free tier, preferred) or `OPENAI_API_KEY` (paid backstop); requires `ffmpeg` | Whisper transcription for audio/video without captions (groundwork: module shipped, not yet auto-invoked by the engine) | Groq free tier is generous; needs ffmpeg installed | |
| 181 | | Jobs / careers pages | none for public ATS pages; web backend improves fallback discovery | `--hiring-signals` and strong Hiring Signals in standard company reports | yes | |
| 182 | | Apify (alternate scraper) | `APIFY_API_TOKEN` | fallback for Reddit/TikTok/Instagram when ScrapeCreators is exhausted | yes (limited) | |
| 183 | |
| 184 | **Reddit keyless pacing.** Unauthenticated reddit.com requests (RSS, listing partials, shreddit) share one token bucket. The default is `1` request per second with a burst of 2, slow enough that engine fan-out does not trip HTTP 429 on a typical home IP. Set `LAST30DAYS_REDDIT_KEYLESS_RATE` to a float req/sec to trade wall-clock for coverage: higher finishes faster and loses more sub-requests to 429; lower is safer and slower. Invalid or non-positive values fall back to `1`. A 429'd RSS or listing sub-request is retried once after a short jittered pause, still through the limiter. Identical reddit.com requests within one command (subreddit listings, listing feeds, comment pages, which repeat across subqueries) are fetched once and memoized, so a typical four-subquery run issues roughly a quarter of the requests it used to. Comment enrichment covers 4 / 8 / 12 threads per subquery at quick / default / deep depth. This does not change ScrapeCreators routing (`LAST30DAYS_REDDIT_BACKEND` / `LAST30DAYS_REDDIT_SC_MIN_ITEMS`). |
| 185 | |
| 186 | **YouTube transcript tuning.** `LAST30DAYS_YT_SUB_LANGS` controls the comma-separated caption-language priority passed to yt-dlp and defaults to `en,es,pt`. `LAST30DAYS_YT_PLAYER_CLIENT` defaults to `android` so yt-dlp can pass YouTube's web bot-gate without cookies (search, transcripts, and comments); set it empty to disable. When `SCRAPECREATORS_API_KEY` is available, yt-dlp uses one fast attempt before the paid fallback; set `LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT` to the number of seconds allowed for that attempt when a throttled host needs longer than the 12-second default. A VTT completed before the timeout is reused rather than discarded. `LAST30DAYS_YT_SEARCH_TIMEOUT` sets the per-search yt-dlp deadline (default 120s). Comparison-mode fan-out also caps concurrent yt-dlp processes process-wide and caches identical searches within a run so redundant `ytsearch` calls do not self-throttle the same IP. |
| 187 | |
| 188 | **X backend priority (bird first).** The default X backend chain is bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key). Cookies beat `XAI_API_KEY` when both are present. A leftover grok login never steals the X lane; see below. `xapi` (the official X API v2 with `X_BEARER_TOKEN`) is opt-in on these hosts (`LAST30DAYS_X_BACKEND=xapi`), so an ambient bearer never spends X API credits when the free path comes back empty. **Grok Bot exception.** On a Grok Bot host (`LAST30DAYS_HOST=grok-bot`) the unpinned chain is the official chain instead: xapi (`X_BEARER_TOKEN`) → xai (`XAI_API_KEY`) → xurl (the X API through X's CLI). When the bot's X connector is in the session, its results come first and the chain is not called at all; see [Grok Bot](#grok-bot) under Per-client patterns. |
| 189 | |
| 190 | **Grok CLI (opt-in backup).** Install the Grok CLI (`curl -fsSL https://x.ai/cli/install.sh | bash`) and run `grok login`, and X can work with no X account, no browser cookies, and no `XAI_API_KEY`. However, grok is **opt-in only**: a leftover `~/.grok/auth.json` must never steal the X lane. Pin `LAST30DAYS_X_BACKEND=grok` to enable it. It is not "free" in the way the cookie path is: calls draw on your Grok plan, and depth costs several calls per run because the underlying tool caps each search at 10 posts. Results are validated before use — every returned post's ID is decoded to confirm it falls inside the requested date range, because the retrieval is performed by a language model and can otherwise return confident, well-formed posts that were never searched for. |
| 191 | |
| 192 | **X on cookie-less hosts.** Bird (the free X source) scrapes X using your logged-in browser cookies (`AUTH_TOKEN`/`CT0`), which agent hosts like OpenClaw, CI, or headless runs often can't supply — and scraping carries some account risk. On those, set `XQUIK_API_KEY` (or `XAI_API_KEY`) for full, ranked X coverage from a single API key: the same engagement-based ranking, first-party authorship, and handle (from/mentions) lanes the native X source gets. The official X API is the other keyed option: set `X_BEARER_TOKEN` and pin `LAST30DAYS_X_BACKEND=xapi`; it serves the same lanes but covers recent posts, about the last week, unless your X developer project has full-archive access. `--diagnose` reports whether the key is working (and flags an unpaid key as `payment-required`). |
| 193 | |
| 194 | **Extra bird cookie lookups on Linux and Mac mini.** On a MacBook the X cookie path is unchanged (Firefox/Safari/Chrome extract, gated by `FROM_BROWSER`). On **extra hosts** the engine adds two more ways to hand bird a complete `auth_token`+`ct0` pair, tried in order (first COMPLETE pair wins; no half-pair merge; nothing is ever written to the `.env` and cookie values are never printed): |
| 195 | |
| 196 | 1. an explicit env `AUTH_TOKEN`+`CT0` (never overwritten); |
| 197 | 2. the [`agentcookie`](https://github.com/) sidecar CLI — `agentcookie cookies --domain .x.com --json` — a soft dependency (absent = skipped; `AGENTCOOKIE=off` disables it) that delivers cookies on Linux, where the on-disk Chrome store can't be decrypted here; |
| 198 | 3. a live signed-in Chrome/Chromium session over the DevTools Protocol (`Network.getAllCookies`); |
| 199 | 4. the mainline browser extract, when `FROM_BROWSER` already lists a browser (on a Mac mini with a browser opted in, this native read runs *before* the CDP read). |
| 200 | |
| 201 | A host counts as an "extra host" when ANY of these hold: `AGENTCOOKIE=on` (explicit opt-in, any OS); the platform is Linux; a Darwin **Mac mini** (`sysctl -n hw.model` prefix `Macmini`); or a Darwin **agentcookie sink** role. The host is never inferred from the home directory, PATH, or Hermes/OpenClaw env — only those signals. A plain MacBook does no agentcookie spawn and opens no CDP socket unless `AGENTCOOKIE=on`. |
| 202 | |
| 203 | CDP endpoint resolution (extra hosts only, no port scan): `BROWSER_CDP_URL` if set, else port `18800` when it answers as Chrome, else `9222` + the X display number. Port `18800` is the last30days extras **NUX convention** — the agent launches a throwaway login Chrome with `SAND_CHROME_REMOTE_DEBUG_PORT=18800` (see SKILL.md's "X on Linux / Mac mini"), so it is not confused with a daily Chrome profile on `9222`+display (box-chrome's own built-in default). `18800` is tried first but falls through when it yields no complete pair, so a logged-out Chrome there never shadows a logged-in profile; pin `BROWSER_CDP_URL` if a stale session answers there. A Node `--inspect` endpoint is rejected; a Chrome page target is required. |
| 204 | |
| 205 | **Example `.env` skeleton** (placeholders only - replace with your own values): |
| 206 | |
| 207 | ```bash |
| 208 | # Reasoning + planning (one provider; see priority below) |
| 209 | GOOGLE_API_KEY=<your-gemini-key> |
| 210 | |
| 211 | # Web search backend (one is enough; Brave is the cheapest) |
| 212 | BRAVE_API_KEY=<your-brave-key> |
| 213 | |
| 214 | # Optional sources |
| 215 | SCRAPECREATORS_API_KEY=<your-scrapecreators-key> |
| 216 | INCLUDE_SOURCES=tiktok,instagram |
| 217 | # LAST30DAYS_REDDIT_KEYLESS_RATE=1 # keyless reddit.com req/sec; lower = fewer 429s, slower runs |
| 218 | # Xiaohongshu is requested-only: run with --search xhs after starting a local |
| 219 | # browser-session service. Defaults probe localhost, then host.docker.internal. |
| 220 | # XIAOHONGSHU_API_BASE=http://localhost:18060 |
| 221 | # Add perplexity to INCLUDE_SOURCES when you want the paid Perplexity source. |
| 222 | # PERPLEXITY_API_KEY=<your-perplexity-key> |
| 223 | # INCLUDE_SOURCES=tiktok,instagram,perplexity |
| 224 | # LAST30DAYS_PERPLEXITY_MODE=agent # agent | search | both; sonar is a legacy alias |
| 225 | # LAST30DAYS_PERPLEXITY_AGENT_MODEL=perplexity/sonar |
| 226 | # LAST30DAYS_PERPLEXITY_AGENT_MAX_STEPS=5 |
| 227 | # LAST30DAYS_PERPLEXITY_AGENT_MAX_OUTPUT_TOKENS=4096 # required for anthropic/* |
| 228 | |
| 229 | # X authentication (one option only) |
| 230 | AUTH_TOKEN=<your-auth-token> |
| 231 | CT0=<your-ct0-token> |
| 232 | # OR the official X API v2 bearer. Default on Grok Bot; elsewhere also pin |
| 233 | # LAST30DAYS_X_BACKEND=xapi. Covers recent posts, about the last week, unless |
| 234 | # your X developer project has full-archive access. |
| 235 | # X_BEARER_TOKEN=<your-x-api-bearer-token> |
| 236 | # OR xAI API key (paid) |
| 237 | # XAI_API_KEY=<your-xai-key> |
| 238 | # OR Xquik key-based X search |
| 239 | # XQUIK_API_KEY=<your-xquik-key> |
| 240 | # OR cookie-jar (free; logs in via your browser session). |
| 241 | # Unset = no browser-cookie reads. FROM_BROWSER=auto tries Firefox/Safari and |
| 242 | # the Chromium family (Chrome, Brave, Edge, Vivaldi, Opera, Arc, Chromium); it |
| 243 | # only prompts for macOS Keychain access on the browser that actually holds your |
| 244 | # X cookies. Or name a single browser, e.g. brave/edge. On Windows only Firefox |
| 245 | # is supported. |
| 246 | # FROM_BROWSER=firefox |
| 247 | |
| 248 | # Bluesky |
| 249 | BSKY_HANDLE=<your-handle>.bsky.social |
| 250 | BSKY_APP_PASSWORD=<your-app-password> |
| 251 | ``` |
| 252 | |
| 253 | After editing: `chmod 600 ~/.config/last30days/.env` (or `chmod 600 .claude/last30days.env` if using the project-scoped variant). |
| 254 | |
| 255 | **Troubleshooting:** if a source you expected to see isn't appearing in results, run `python3 scripts/last30days.py --preflight` for a human permission summary or `python3 scripts/last30days.py --diagnose` for full JSON diagnostics. Both are safe: they report source availability, config source, browser-cookie plan, external command availability, write destinations, and ignored untrusted project config without reading browser cookies or running live provider probes. |
| 256 | |
| 257 | ### Perplexity source modes |
| 258 | |
| 259 | Perplexity is a paid opt-in source. A direct `PERPLEXITY_API_KEY` enables the Agent API, Search API, and background Deep Research. Existing `OPENROUTER_API_KEY` installs remain compatible through synchronous Sonar: `perplexity/sonar-pro` for normal synthesis and `perplexity/sonar-deep-research` for `--deep-research`. Search API and Agent API features still require the direct key. |
| 260 | |
| 261 | `LAST30DAYS_PERPLEXITY_MODE` controls normal `perplexity` source runs: |
| 262 | |
| 263 | | Value | Behavior | Calls | |
| 264 | |---|---|---| |
| 265 | | `agent` (default) | Direct key: controlled Agent API synthesis with required `web_search`. OpenRouter-only: synchronous Sonar fallback. | at most one paid synthesis call per last30days run | |
| 266 | | `sonar` | Direct key: deprecated alias for `agent`. OpenRouter-only: synchronous Sonar fallback. | at most one paid synthesis call per last30days run | |
| 267 | | `search` | Direct key: raw ranked Search API rows. OpenRouter-only: falls back to synchronous Sonar. | at most one paid call per last30days run | |
| 268 | | `both` | Direct key: Agent synthesis plus Search rows. OpenRouter-only: falls back to synchronous Sonar. | direct: at most two paid calls; OpenRouter: at most one | |
| 269 | |
| 270 | With a direct key, normal `agent` mode uses the controlled `last30days-controlled-web-search/v1` profile: `perplexity/sonar`, a bounded `max_steps`, a local instruction, and only the configured `web_search` tool. It forces that tool for citation-critical grounding. It does not enable sandbox, file, finance, MCP, or function tools. OpenRouter fallback keeps the older OpenAI-compatible Sonar request and does not claim Agent API controls. |
| 271 | |
| 272 | The engine routes every normal Perplexity mode through one whole-topic planner subquery per command, including competitor fanout, and does not repeat it during thin-source retries. A generic source-fetch override cannot raise this paid-call cap. |
| 273 | |
| 274 | `LAST30DAYS_PERPLEXITY_AGENT_PRESET` is a separate explicit opt-in for a mutable Perplexity preset (`fast`, `low`, `medium`, or `high`). Presets can change their model, prompt, tools, cost, and output behavior. The engine still supplies its configured `web_search` tool so date, domain, location, result-count, and context constraints merge with the preset; other preset tools can remain enabled. Do not set this variable when you need the controlled profile. The engine never selects a preset automatically for normal runs. |
| 275 | |
| 276 | `--deep-research` requires a normal positional topic and ignores `LAST30DAYS_PERPLEXITY_MODE`. With a direct key it starts at most one Agent API background run with the explicit dynamic `high` preset. With only OpenRouter it preserves the older synchronous `perplexity/sonar-deep-research` fallback. It cannot be combined with discovery, drill, cached-only, competitor, or vs-mode. This is a separate paid action. The engine caps it at one planner subquery and does not repeat it during thin-source retries. Direct background runs merge the configured `web_search` constraints with the preset, but the provider controls its other tools and can change them. A local timeout stops waiting but does not stop a direct remote run. Direct artifacts retain the served model, response ID, provider status, incomplete reason, poll count, timeout, and safe error metadata; OpenRouter artifacts retain the served model, response ID, usage, and citation count. Neither stores request headers or raw tool traces. |
| 277 | |
| 278 | Perplexity-specific env vars: |
| 279 | |
| 280 | | Env var | Default | Applies to | Notes | |
| 281 | |---|---|---|---| |
| 282 | | `LAST30DAYS_PERPLEXITY_MODE` | `agent` | normal Perplexity source runs | `agent`, `search`, or `both`; `sonar` remains a deprecated alias for `agent`. | |
| 283 | | `LAST30DAYS_PERPLEXITY_AGENT_MODEL` | `perplexity/sonar` | controlled Agent profile | Explicit Agent model for normal synthesis. | |
| 284 | | `LAST30DAYS_PERPLEXITY_AGENT_MAX_STEPS` | `5` | controlled Agent profile | Clamped to the last30days safety range 1..15. | |
| 285 | | `LAST30DAYS_PERPLEXITY_AGENT_MAX_OUTPUT_TOKENS` | `4096` for `anthropic/*` models | controlled Agent profile | Required for explicit Anthropic models; clamped to the last30days safety range 1..32768. | |
| 286 | | `LAST30DAYS_PERPLEXITY_AGENT_TIMEOUT_SECONDS` | `120` | controlled Agent profile | Synchronous request timeout, clamped to 1..600 seconds. | |
| 287 | | `LAST30DAYS_PERPLEXITY_AGENT_PRESET` | unset | normal Agent runs | Explicit mutable preset only: `fast`, `low`, `medium`, or `high`. It replaces the controlled profile for that run. | |
| 288 | | `LAST30DAYS_PERPLEXITY_MAX_RESULTS` | `10` | Search API and all Agent `web_search` requests | Clamped to 1..20. | |
| 289 | | `LAST30DAYS_PERPLEXITY_SEARCH_CONTEXT_SIZE` | provider default | Search API and all Agent `web_search` requests | `low`, `medium`, or `high`; omitted unless set. | |
| 290 | | `LAST30DAYS_PERPLEXITY_DOMAIN_FILTER` | unset | Search API and all Agent `web_search` requests | Comma-separated domains, max 20. | |
| 291 | | `LAST30DAYS_PERPLEXITY_LANGUAGE_FILTER` | unset | Search API only | Comma-separated ISO 639-1 language codes. Agent API has no equivalent. | |
| 292 | | `LAST30DAYS_PERPLEXITY_COUNTRY` | unset | Search API and all Agent `web_search` requests | Two-letter country code such as `US`. | |
| 293 | | `LAST30DAYS_PERPLEXITY_RECENCY_FILTER` | unset | Search API and all Agent `web_search` requests | `hour`, `day`, `week`, `month`, or `year`; exact date filters take precedence. | |
| 294 | | `LAST30DAYS_PERPLEXITY_REASONING_EFFORT` | unset | controlled Agent profile | `minimal`, `low`, `medium`, or `high`. | |
| 295 | | `LAST30DAYS_PERPLEXITY_DEEP_TIMEOUT_SECONDS` | `600` | direct Agent API background Deep Research | Wall-clock polling deadline; remote work can continue after a local timeout. OpenRouter fallback is synchronous. | |
| 296 | | `LAST30DAYS_PERPLEXITY_MODEL` / `LAST30DAYS_PERPLEXITY_SEARCH_MODE` | unset | legacy Sonar config | Retained for config-file compatibility. They do not select an Agent API preset or search mode. | |
| 297 | |
| 298 | ### Encrypted credential sources (Keychain / pass) |
| 299 | |
| 300 | If you'd rather not keep keys in a plaintext `.env`, the loader has two |
| 301 | encrypted sources that decrypt secrets transiently at call time (never written |
| 302 | to disk, never logged). Both are **lowest-priority and additive** — an explicit |
| 303 | `.env` or process-env value always overrides them, so you can mix and match. The |
| 304 | `pass` source is only consulted for keys still missing after the higher-priority |
| 305 | sources, so a box that merely has `pass` installed pays no decrypt cost when |
| 306 | everything is already in `.env`. |
| 307 | |
| 308 | Effective credential priority is: process env > trusted project config |
| 309 | (`.claude/last30days.env`) > global config (`~/.config/last30days/.env`) > |
| 310 | macOS Keychain > `pass`(1). A Keychain-only setup still counts as configured: |
| 311 | the engine resolves `last30days-<KEY>` at runtime. First-run in SKILL.md |
| 312 | Step 0 must not treat a missing `.env` as unconfigured. |
| 313 | |
| 314 | | Platform | Source | Store keys with | Lookup convention | |
| 315 | |---|---|---|---| |
| 316 | | macOS | Keychain | `scripts/setup-keychain.sh` | service name `last30days-<KEY>` | |
| 317 | | Linux / Unix (anywhere `pass` exists, incl. macOS) | [`pass`(1)](https://www.passwordstore.org/) | `scripts/setup-pass.sh` | pass path `last30days/<KEY>` | |
| 318 | |
| 319 | ```bash |
| 320 | # macOS Keychain |
| 321 | ./scripts/setup-keychain.sh # interactive; --list / --delete KEY |
| 322 | |
| 323 | # pass(1) — Linux/Unix analog |
| 324 | ./scripts/setup-pass.sh # interactive; --list / --delete KEY |
| 325 | ./scripts/setup-pass.sh SCRAPECREATORS_API_KEY # just one key |
| 326 | ``` |
| 327 | |
| 328 | The `pass` source honors `PASSWORD_STORE_DIR`. If your store organizes secrets |
| 329 | under a different prefix, point the loader at it with `LAST30DAYS_PASS_PREFIX` |
| 330 | (works from your `.env` too, and must match where `setup-pass.sh` wrote them). |
| 331 | The prefix is used verbatim, so keep the trailing separator: |
| 332 | |
| 333 | ```bash |
| 334 | export LAST30DAYS_PASS_PREFIX="secrets/last30days/" # default: last30days/ |
| 335 | ``` |
| 336 | |
| 337 | Both sources cover the same key set as the `.env` skeleton above. |
| 338 | |
| 339 | #### Reusing existing macOS Keychain items |
| 340 | |
| 341 | If you already have keys stored under another Keychain naming convention, you |
| 342 | can reference them without copying the secret by setting non-secret alias |
| 343 | metadata in `LAST30DAYS_KEYCHAIN_ALIASES`. The loader still checks |
| 344 | `last30days-<KEY>` first; aliases are fallback lookups only. |
| 345 | |
| 346 | ```bash |
| 347 | # ~/.config/last30days/.env |
| 348 | LAST30DAYS_KEYCHAIN_ALIASES={"XAI_API_KEY":{"account":"keychain-user","service":"existing-xai-api-key"},"BRAVE_API_KEY":"existing-brave-api-key"} |
| 349 | ``` |
| 350 | |
| 351 | Each JSON key must be one of the supported env-var names (`XAI_API_KEY`, |
| 352 | `SCRAPECREATORS_API_KEY`, `BRAVE_API_KEY`, etc). A string value means "use this |
| 353 | service name with the current user account"; an object can specify both |
| 354 | `account` and `service`. Lists are allowed for fallback order: |
| 355 | |
| 356 | ```bash |
| 357 | LAST30DAYS_KEYCHAIN_ALIASES={"XAI_API_KEY":[{"account":"keychain-user","service":"existing-xai-api-key"},{"service":"last-resort-xai"}]} |
| 358 | ``` |
| 359 | |
| 360 | The alias value contains no secret material; it is safe to keep in `.env` as |
| 361 | configuration. The secret itself remains in its original Keychain item and is |
| 362 | read directly by the engine process. |
| 363 | |
| 364 | Write `LAST30DAYS_KEYCHAIN_ALIASES` as a single-line JSON value in `.env`. |
| 365 | Multiline JSON formatting is not supported because `.env` files are parsed |
| 366 | line-by-line. |
| 367 | |
| 368 | #### Disabling the Keychain source |
| 369 | |
| 370 | Set `LAST30DAYS_SKIP_KEYCHAIN=1` to switch the Keychain source off entirely, |
| 371 | making the loader a no-op on macOS as well: |
| 372 | |
| 373 | ```bash |
| 374 | LAST30DAYS_SKIP_KEYCHAIN=1 uv run pytest tests/test_footer_nudge_suppression.py |
| 375 | ``` |
| 376 | |
| 377 | Scope it to the tests that need a sealed Keychain rather than the whole suite: |
| 378 | the full run should keep exercising the positive-path Keychain tests. |
| 379 | |
| 380 | This exists mainly for tests and reproductions that assert on |
| 381 | "no credentials configured" behaviour. Clearing `os.environ` and pointing |
| 382 | `LAST30DAYS_CONFIG_DIR` at nothing is not sufficient on a machine with items |
| 383 | stored under `last30days-<KEY>`: Keychain is a third, independent source, so a |
| 384 | stored key can quietly satisfy a lookup the test expected to fail — and the |
| 385 | test then fails on a contributor's Mac while passing in Linux CI, where the |
| 386 | loader already no-ops. |
| 387 | |
| 388 | Unlike `LAST30DAYS_KEYCHAIN_ALIASES`, this switch is read from the process |
| 389 | environment only and never from a `.env` file. It gates a credential source |
| 390 | consulted *while* the config is being assembled, so a file-sourced value would |
| 391 | be read too late to take effect. |
| 392 | |
| 393 | ### Bluesky app-password format and search host |
| 394 | |
| 395 | `BSKY_APP_PASSWORD` should be a 19-char app password in `xxxx-xxxx-xxxx-xxxx` format (lowercase alphanumeric, three hyphens). Generate one at <https://bsky.app/settings/app-passwords>. The AT Protocol's `createSession` endpoint also accepts your main account login password, but that's bad hygiene — main passwords have no scope (an app password can be limited to non-DM access) and can't be revoked individually. |
| 396 | |
| 397 | The skill defaults to `api.bsky.app` for `searchPosts`, which is the canonical authenticated AppView. The previous default `public.api.bsky.app` is the unauthenticated public mirror and is currently blocked by BunnyCDN for `searchPosts` regardless of auth header (verified 2026-05-04). If Bluesky migrates infrastructure again, override the host without a code change by setting `BSKY_SEARCH_HOST` in your `.env`: |
| 398 | |
| 399 | ```bash |
| 400 | BSKY_SEARCH_HOST=api.bsky.app # default — change only if Bluesky moves |
| 401 | ``` |
| 402 | |
| 403 | ### Default source set (`LAST30DAYS_DEFAULT_SEARCH`) |
| 404 | |
| 405 | By default the engine decides the source set per query (everything available, minus `EXCLUDE_SOURCES`). To pin a **fixed** source set for every run without passing `--search` each time — and without patching `SKILL.md`, which a release would overwrite — set: |
| 406 | |
| 407 | ```bash |
| 408 | LAST30DAYS_DEFAULT_SEARCH=reddit,x,youtube,hn |
| 409 | ``` |
| 410 | |
| 411 | Accepts the same comma-separated names and aliases as `--search` (`web` → grounding, `hn` → hackernews, `bsky` → bluesky, `xhs` → xiaohongshu). Precedence: an explicit `--search` on the command line always wins; `LAST30DAYS_DEFAULT_SEARCH` applies only when the flag is omitted; when neither is set, per-query behavior is unchanged. `INCLUDE_SOURCES` / `EXCLUDE_SOURCES` keep their existing additive/subtractive roles on whichever set is selected. |
| 412 | |
| 413 | ### Audience register (`LAST30DAYS_REGISTER`) |
| 414 | |
| 415 | The default standard brief stays balanced and byte-compatible with prior releases. To keep a named audience preset across runs, set one of the supported values: |
| 416 | |
| 417 | ```bash |
| 418 | LAST30DAYS_REGISTER=exec # default | exec | dev | creator | eli5 |
| 419 | ``` |
| 420 | |
| 421 | An explicit `--register` wins over `LAST30DAYS_REGISTER`; the environment/config value defaults to `default`. Presets are intentionally named and bounded - arbitrary prompt or template files are not accepted. Existing `ELI5_MODE=true` configurations continue to resolve to the `eli5` register when no explicit register is selected, but new configuration should use `LAST30DAYS_REGISTER=eli5`. |
| 422 | |
| 423 | --- |
| 424 | |
| 425 | ## Reasoning provider priority |
| 426 | |
| 427 | `/last30days` needs one reasoning model for planning + reranking when you don't pass `--plan` yourself. Auto-detect priority (set `LAST30DAYS_REASONING_PROVIDER=<name>` to pin one): |
| 428 | |
| 429 | 1. **Gemini** - `GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY` |
| 430 | 2. **OpenAI** - `OPENAI_API_KEY` only. Codex ChatGPT auth at `~/.codex/auth.json` is intentionally not used as an OpenAI provider credential. |
| 431 | 3. **xAI** - `XAI_API_KEY` |
| 432 | 4. **OpenRouter** - `OPENROUTER_API_KEY` (reasoning provider, auto-resolve, and synchronous Sonar fallback for the Perplexity source) |
| 433 | 5. **Local / deterministic** - always available, lowest quality |
| 434 | |
| 435 | When you invoke `/last30days` from Claude Code, Codex, or Gemini, the host model **is** the reasoning provider for plan + synthesis - you don't need any of the keys above unless you also run the script headlessly (cron, CI, watchlist). |
| 436 | |
| 437 | --- |
| 438 | |
| 439 | ## Web search backend priority |
| 440 | |
| 441 | The search-source preference ladder, strict best-to-floor: |
| 442 | |
| 443 | 1. **Host web search** - whatever web-search capability the agent session already has: built-in search, a deferred web-search tool that must be loaded first, or an installed connector such as Brave, Firecrawl, Exa, Serper, or another provider. Best results; used automatically on hosts that have it. A failed lookup for one specific tool name is not fatal when another web-search capability is available. Signalled to the engine via `LAST30DAYS_NATIVE_SEARCH=1` (the skill sets this for you when your agent session has web search) so the engine does not run a worse search underneath it. |
| 444 | 2. **Paid engine backend** - one of `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY`, auto-detected in that order. Override per-run with `--web-backend=<name>`. |
| 445 | 3. **Explicit hosted MCP** - `--web-backend=parallel-mcp` opts this run into the anonymous `https://search.parallel.ai/mcp` server. Search objectives and queries reach Parallel; the option is never auto-selected. The free path needs no key, while an existing `PARALLEL_API_KEY` is sent as optional Bearer authentication for higher limits. |
| 446 | 4. **Keyless engine floor** - zero-key web search (DuckDuckGo, plus an optional SearXNG instance) and zero-key page fetch (Jina Reader). Runs only when the agent session has **no** host web search **and** no paid key is set, so headless/cron and hosts without a search tool still get general-web coverage. Force it explicitly with `--web-backend=keyless`. |
| 447 | |
| 448 | Relevant env vars: |
| 449 | |
| 450 | | Var | Effect | |
| 451 | | --- | --- | |
| 452 | | `LAST30DAYS_NATIVE_SEARCH=1` | Tells the engine your agent session has host-side web search; suppresses the keyless floor. Set automatically by the skill when web search is available. Leave unset when the agent has no web-search tool so the floor runs. | |
| 453 | | `LAST30DAYS_SEARXNG_URL=<base-url>` | Optional. A SearXNG instance used as the keyless-search fallback rung when DuckDuckGo returns nothing. | |
| 454 | | `LAST30DAYS_TRUSTPILOT_NO_BROWSER=1` | Optional. Truthy value disables the Trustpilot source's one-time headless-Chrome WAF-cookie harvest, so an automated/headless run (cron, CI, the eval harness) never spawns a browser. Trustpilot still degrades to empty gracefully. | |
| 455 | |
| 456 | Privacy note: the keyless floor sends the query (to DuckDuckGo / your SearXNG instance) and any fetched URL (to Jina Reader) to those third parties. It is intended for public-research use; results may be cached snapshots. It never runs when native search or a paid backend is in play. |
| 457 | |
| 458 | Visible quality difference between hosts with vs without native search or a configured backend. If your client setup produces thinner results than yours, this is usually why. |
| 459 | |
| 460 | --- |
| 461 | |
| 462 | ### `--hiring-signals` flag |
| 463 | |
| 464 | Use `--hiring-signals` for a focused company hiring-signal report: |
| 465 | |
| 466 | ```bash |
| 467 | python3 skills/last30days/scripts/last30days.py "Listen Labs" --hiring-signals |
| 468 | ``` |
| 469 | |
| 470 | The engine treats public jobs/careers postings as evidence of focus or priority shifts, not exact roadmap predictions. Standard company runs may include Hiring Signals automatically when multiple current roles support the same interpretation; weak or unavailable hiring evidence is omitted. |
| 471 | |
| 472 | ### `--x-posts` flag |
| 473 | |
| 474 | `--x-posts <path>` hands the engine an X result the hosting model fetched through its own X connector; it replaces the engine's X fetch for that run and works on any host. The value is a file path only (inline JSON exits `2`): a regular `.json` file in the `last30days-x-posts/1` shape, never read from inside the config dir or a credential store. |
| 475 | |
| 476 | | Field | Meaning | |
| 477 | | --- | --- | |
| 478 | | `schema`, `generated_at`, `topic`, `window {from, to}`, `provider`, `status` | Envelope header. `status` is `ok`, `partial`, or `error`; `error` is a short category (`credits`, `not-connected`, `unavailable`, `window-unsupported`), never raw tool output. `topic` must match the run topic and `generated_at` must be under 6 hours old, or the run fails closed with exit `2`. | |
| 479 | | `calls[]` | One entry per connector call: `lane` (`topic`, `from`, `mention`, `related`), `handles` (a subset of the run's `--x-handle` / `--x-related` handles), and `posts`. | |
| 480 | | `posts[]` | Flat rows with exactly eight fields: `id`, `author_handle`, `created_at`, `text`, `likes`, `reposts`, `replies`, `quotes`. Any other key is ignored and counted. | |
| 481 | |
| 482 | Limits: 8 MiB, strict UTF-8, at most 20 calls, 500 rows per call, 1,000 rows in total, 10,000 characters of text per row. Rows are rebuilt from validated parts: the citation is always `https://x.com/<handle>/status/<id>` (a row-supplied URL is never used), rows without an id or text, outside the window, or whose date disagrees with the id are dropped and counted, and an id sequence that looks generated rejects the whole file. The envelope is single-serve for the run. The hosted backend (`LAST30DAYS_API_BASE`) rejects the flag with exit `2`. Comparison runs take the per-entity `x_posts` field of `--competitors-plan` instead; a bare `--x-posts` on a comparison run exits `2`. |
| 483 | |
| 484 | ```bash |
| 485 | python3 skills/last30days/scripts/last30days.py "<topic>" --x-posts /tmp/x-posts.json |
| 486 | ``` |
| 487 | |
| 488 | ### `setup --store-key` |
| 489 | |
| 490 | `setup --store-key <NAME>` persists one credential to the global `.env` (mode `600`) from a single line on stdin, without echoing it: stdout shows `NAME=****` plus a JSON line `{"persisted": true, "key": "NAME"}`. `NAME` must be one of the credential names the engine loads from `.env` (for example `X_BEARER_TOKEN`, `XAI_API_KEY`, `SCRAPECREATORS_API_KEY`); an unknown name or an empty value exits `2`. Running it again with a new value replaces the stored one (rotating a rejected credential); other lines in the file are untouched. |
| 491 | |
| 492 | ```bash |
| 493 | printf '%s\n' "$TOKEN" | python3 skills/last30days/scripts/last30days.py setup --store-key X_BEARER_TOKEN |
| 494 | ``` |
| 495 | |
| 496 | --- |
| 497 | |
| 498 | ## Health check (`doctor`) |
| 499 | |
| 500 | One command answers "what could be on, what's turned on, what's working, and what isn't" — a four-state audit (WORKING / TURNED ON - UNVERIFIED / NOT WORKING / COULD BE ON), one line per source, with a CLI-health block for sources that need a downloaded binary, indented backup/comment sub-lanes, the backend the next run will use (for chained sources), and an exact fix on anything that isn't working: |
| 501 | |
| 502 | ```bash |
| 503 | python3 skills/last30days/scripts/last30days.py doctor # four-state audit (text) |
| 504 | python3 skills/last30days/scripts/last30days.py doctor --json # machine contract |
| 505 | python3 skills/last30days/scripts/last30days.py doctor --cached # serve the cached report while fresh |
| 506 | python3 skills/last30days/scripts/last30days.py doctor --postmortem # what actually broke on the last run |
| 507 | python3 skills/last30days/scripts/last30days.py doctor --probe # bounded live test (free/CLI sources) |
| 508 | ``` |
| 509 | |
| 510 | Slash-command form: `/last30days doctor`. Reporting problems is a successful run — the exit code is always 0, no browser cookies are read, and no secret values appear anywhere (key presence is booleans only). Backends within a chained source are probed sequentially with a 5-second budget per binary probe, so a chained source's worst-case check time is additive across its backends (only reached when several binaries hang at once). |
| 511 | |
| 512 | `doctor --postmortem` reads the last run's `last-report.json` (any age, labeled) and reports what actually happened per source — Failed / Partial / Succeeded / Skipped, with details and fix hints — so a run that returned less than expected can be diagnosed after the fact. It makes no network calls. |
| 513 | |
| 514 | **Network note:** plain `doctor` with a fresh run, `--cached`, and `--json` make **no** network calls. `doctor --probe` — and a plain `doctor` when there is **no** fresh run to learn from — run a **bounded** live test to verify WORKING instead of guessing. The probe is scoped to free HTTP endpoints (Reddit, Hacker News, Polymarket, GitHub) plus keyless CLIs; credit-gated sources (X, TikTok, Instagram, Threads, …) are never probed, so no ScrapeCreators credits are spent and no auth rate limits are tripped. Each source is probed concurrently under a per-source deadline so a slow source can never hang the command. |
| 515 | |
| 516 | Every live run writes its JSON result to `~/.config/last30days/doctor-cache.json` (beside `last-run.json`; honors `LAST30DAYS_CONFIG_DIR`). `doctor --cached` returns that stored report when it is younger than the TTL, and falls through to a live run — rewriting the cache — when it is stale, absent, or corrupt. The cache also self-invalidates on configuration change: the payload carries a schema stamp plus a fingerprint of non-secret config signals (which credentials are present as booleans, the `LAST30DAYS_X_BACKEND` / `LAST30DAYS_REDDIT_BACKEND` pin values, and `INCLUDE_SOURCES`), so adding or removing a key, changing a pin, or toggling an opt-in source makes the next `--cached` call run live — no raw secret ever enters the fingerprint or the file. Every report also carries `from_cache` (true/false) and `generated_at` (when the report was built), in the `--json` top level and as a final `generated: … (cached|live)` text line, so you can always tell how old a cached answer is. A failed cache write is never fatal — doctor prints a one-line stderr warning and continues. An explicit `doctor` without `--cached` always runs live and refreshes the cache. |
| 517 | |
| 518 | | Var | Effect | |
| 519 | | --- | --- | |
| 520 | | `LAST30DAYS_DOCTOR_TTL` | Freshness window for `doctor --cached`, in **seconds**. Defaults to `900` (15 minutes). `0` makes every `--cached` call run live. | |
| 521 | | `LAST30DAYS_DOCTOR_PROBE_TIMEOUT` | Per-source deadline (**seconds**) for `doctor --probe` live checks. Defaults to `10`. Caps each concurrent probe so a slow source cannot hang the command. | |
| 522 | | `LAST30DAYS_HOST` | Host self-identification. `grok-bot` switches X to the official chain (xapi → xai → xurl) and turns off browser-session discovery; any other value, or unset, leaves every host exactly as today. Persisted to `.env` by first-run setup on Grok Bot and exported per invocation by the skill; doctor prints the resolved value. The engine never infers the host any other way. | |
| 523 | | `X_BEARER_TOKEN` | App-only bearer for the official X API v2 (`xapi`). First rung of the chain on Grok Bot; opt-in elsewhere via `LAST30DAYS_X_BACKEND=xapi`. Full-archive search is tried first, then recent search, so coverage is recent posts, about the last week, unless your X developer project has full-archive access (the outcome detail says `window truncated to 7 days` when the fallback ran, and `search stopped at the lane deadline; results may be incomplete` when the shared 150s lane budget cut a search short). Exhausted credits (HTTP 402) report as `payment-required`. Doctor checks presence only, never the network. Loaded from `.env`, Keychain, or `pass` like the other keys. | |
| 524 | | `LAST30DAYS_X_HOST_LANE` | `1` declares that the hosting model's X connector is in this session, so `--diagnose` and planning list `x` as available and the run expects `--x-posts`. Read from the process environment only: a `.env` line is ignored (doctor says so), so a removed connector never leaves a stale declaration. A run with the signal but no `--x-posts` records X as `error` ("connector result not passed"). | |
| 525 | | `LAST30DAYS_X_BACKEND` | Pins the X backend (`bird` / `xai` / `xurl` / `xquik` / `grok` / `xapi`); doctor renders the pin and predicts "will use" accordingly. The unpinned auto chain is bird → xai → xurl → xquik (grok and xapi are opt-in only). Pin `grok` to enable it; a leftover `~/.grok/auth.json` is never auto-selected. Pin `xapi` to use `X_BEARER_TOKEN` on an ordinary host. On a Grok Bot host the unpinned chain is the official chain (xapi → xai → xurl) and this pin is the only way to select a backend outside it; the pin keeps its exclusive, no-failover meaning there, and doctor names the pinned backend. | |
| 526 | | `AGENTCOOKIE` | `on` opts any host (incl. a MacBook) into the extra bird cookie lookups (agentcookie sidecar + live Chrome CDP); `off` disables the agentcookie sidecar reader. Unset uses host detection (Linux / Mac mini / Darwin sink get the extras). See "Extra bird cookie lookups" above. | |
| 527 | | `BROWSER_CDP_URL` | Explicit Chrome DevTools endpoint (e.g. `http://127.0.0.1:18800`) for the extra-host CDP cookie lookup. Preferred over the `18800` / `9222`+`$DISPLAY` defaults. Extra hosts only. | |
| 528 | | `LAST30DAYS_REDDIT_BACKEND` | `scrapecreators` makes ScrapeCreators the primary Reddit backend; doctor renders Reddit's conditional routing with the pin applied. | |
| 529 | | `LAST30DAYS_REDDIT_SC_MIN_ITEMS` | Integer thinness floor for ScrapeCreators Reddit **search** backfill. Default `0` = empty-only (free path keeps any non-empty result; no credit spend). Set above `0` to backfill when free yield is below that count; merged results dedupe by post id. Requires `SCRAPECREATORS_API_KEY`. Ignored when `LAST30DAYS_REDDIT_BACKEND=scrapecreators` (SC is already primary). | |
| 530 | |
| 531 | Web search has **no** env pin — pin it per-run with `--web-backend=<name>` only (see [Web search backend priority](#web-search-backend-priority)). |
| 532 | |
| 533 | ### Strict exit for degraded runs |
| 534 | |
| 535 | By default a research run exits `0` even when a source failed mid-run (rate-limited, auth-failed, unreachable, timeout, schema-drift) — the report still renders, with the failure annotated in the per-source footer and a partial-coverage warning. Wrappers that need to distinguish degraded coverage from success (cron briefs, CI, downstream agents) can opt in: |
| 536 | |
| 537 | | Var | Effect | |
| 538 | | --- | --- | |
| 539 | | `LAST30DAYS_STRICT_EXIT` | Truthy (`1`/`true`/`yes`/`on`): the engine exits `3` when any source outcome is neither `ok`, `no-results`, nor `skipped-unconfigured` (so `partial`, `auth-failed`, `payment-required` for exhausted credits, `rate-limited`, and the other failure states all count as degraded). A one-line `strict-exit: degraded sources: ...` note goes to stderr. Default (unset): exit `0`, unchanged behavior. | |
| 540 | |
| 541 | Exit codes with the flag on: `0` clean run, `3` completed-but-degraded (report was produced), non-zero others unchanged (hard failures). Same hybrid pattern as `LAST30DAYS_DEBUG` — works shell-exported or in `.env`. |
| 542 | |
| 543 | --- |
| 544 | |
| 545 | ## Debug mode (`--debug`) |
| 546 | |
| 547 | Add `--debug` to any run to emit verbose `[DEBUG]` log lines to stderr from the source modules (X API, HTTP, etc.). Helpful for diagnosing API errors or unexpected behavior. |
| 548 | |
| 549 | **Always-on alternative:** set `LAST30DAYS_DEBUG=true` in your `.env` or export it from your shell. The flag still works as before; the env var is purely additive — works whether shell-exported or set in `.env`. |
| 550 | |
| 551 | --- |
| 552 | |
| 553 | ## Trend monitoring (`--store` + watchlist + briefings) |
| 554 | |
| 555 | The default behavior - one slug-named file per topic, overwritten on rerun - is the snapshot mode. For continuous monitoring, the repo ships three components most users miss: |
| 556 | |
| 557 | ### `--store` flag |
| 558 | |
| 559 | Adding `--store` to any run persists every finding to a SQLite database (default at `~/.local/share/last30days/research.db`). Findings dedupe on the `source_url` column (UNIQUE constraint), so the same URL across runs updates the existing row instead of creating a duplicate. The markdown file still saves; the SQLite is the time-series substrate. |
| 560 | |
| 561 | **Always-on alternative:** set `LAST30DAYS_STORE=1` in your `.env` instead of remembering `--store` on every invocation. The flag still works as before; the env var is purely additive. Same hybrid pattern as `LAST30DAYS_DEBUG` — works whether shell-exported or in `.env`. |
| 562 | |
| 563 | Relevant tables: `topics`, `research_runs`, `findings`, `settings`. Schema: [`scripts/store.py`](skills/last30days/scripts/store.py). |
| 564 | |
| 565 | ### Discovery topic queue (`LAST30DAYS_DISCOVERY_QUEUE`) |
| 566 | |
| 567 | `--discover` runs remember what they surfaced (table `discovery_topics` in the same research.db). Re-surfaced topics get a `**Pipeline:**` line on their card ("surfaced 2nd time", "marked covered") so the discovery brief doubles as a podcast / X-article content pipeline. On by default for real runs; `--mock` runs never write. With `--save-dir`, queue rows land in that directory's scoped `research.db`, never the global one. |
| 568 | |
| 569 | | Var | Effect | |
| 570 | | --- | --- | |
| 571 | | `LAST30DAYS_DISCOVERY_QUEUE` | Set to `off` to disable queue writes and card annotations. Any other value (or unset) keeps the queue on. Works shell-exported or in `.env`. | |
| 572 | | `LAST30DAYS_ENRICH_BUDGET_SECONDS` | Wall-clock budget (seconds) for the deep-tier per-topic research batch on the discovery resume leg (`--discover --judgments <file>`). Default `450`; unset/invalid/non-positive values fall back to it. The one-shot `--discover` path keeps its fixed quick-tier 240s budget regardless. Works shell-exported or in `.env`. | |
| 573 | |
| 574 | Manage the queue from the engine CLI: |
| 575 | |
| 576 | ```bash |
| 577 | # Uncovered surfaced topics (name, domain, surface_count, last_surfaced, status) |
| 578 | python3 skills/last30days/scripts/last30days.py queue list |
| 579 | |
| 580 | # Mark a topic done after you record the episode / publish the article. |
| 581 | # Requires the exact topic name; unknown names exit 2 instead of no-opping. |
| 582 | python3 skills/last30days/scripts/last30days.py queue cover "Gemma 4 chat templates" |
| 583 | ``` |
| 584 | |
| 585 | Both respect `--save-dir` scoping. |
| 586 | |
| 587 | ### `watchlist.py` - recurring topics |
| 588 | |
| 589 | [`scripts/watchlist.py`](skills/last30days/scripts/watchlist.py) manages topics that should be researched on a schedule. Subcommands: `add`, `remove`, `list`, `run-one`, `run-all`, `config`. Built-in delivery to Slack incoming webhooks (`hooks.slack.com/...`) or any HTTPS endpoint, fired only when new findings appear. |
| 590 | |
| 591 | Two-step flow (the watchlist holds the topic; an external scheduler invokes the run): |
| 592 | |
| 593 | ```bash |
| 594 | # 1. Add the topic to the watchlist |
| 595 | # Default schedule daily 8am; --weekly switches to Mondays 8am |
| 596 | python3 scripts/watchlist.py add "british airways middle east" --weekly |
| 597 | |
| 598 | # 2. Configure delivery and budget (optional) |
| 599 | python3 scripts/watchlist.py config delivery "https://hooks.slack.com/services/..." |
| 600 | python3 scripts/watchlist.py config budget 5.00 |
| 601 | |
| 602 | # 3. Trigger via cron / Task Scheduler / GitHub Actions |
| 603 | python3 scripts/watchlist.py run-one "british airways middle east" |
| 604 | # or run every enabled topic, gated by daily_budget |
| 605 | python3 scripts/watchlist.py run-all |
| 606 | ``` |
| 607 | |
| 608 | The schedule field stored on each topic is metadata - the actual cron / Task Scheduler invocation is your responsibility. Watchlist runs hardcode `--quick` and `--lookback-days 90` when spawning the underlying engine. |
| 609 | |
| 610 | ### `briefing.py` - daily / weekly digests |
| 611 | |
| 612 | [`scripts/briefing.py`](skills/last30days/scripts/briefing.py) reads the SQLite store and emits structured data the agent then synthesizes into prose. Modes: `generate` (daily), `generate --weekly`, `show [--date DATE]` (display a saved briefing). Briefs save to `~/.local/share/last30days/briefs/`. |
| 613 | |
| 614 | ### Recommended cadence pattern |
| 615 | |
| 616 | | Step | Cadence | Command | |
| 617 | |---|---|---| |
| 618 | | Baseline | one-time per topic | `/last30days "<topic>" --days=30 --store` | |
| 619 | | Add to watchlist | one-time per topic | `python3 scripts/watchlist.py add "<topic>" --weekly` | |
| 620 | | Recurring run | daily or weekly (external scheduler) | `python3 scripts/watchlist.py run-all` | |
| 621 | | Digest | weekly | `python3 scripts/briefing.py generate --weekly` | |
| 622 | |
| 623 | --- |
| 624 | |
| 625 | ## Per-client patterns |
| 626 | |
| 627 | The skill is built to flex around different client environments. Four patterns that compose well: |
| 628 | |
| 629 | **Codex note:** the repository includes `.codex-plugin/plugin.json` so Codex can treat the existing |
| 630 | `skills/last30days/SKILL.md` tree as plugin metadata without maintaining a separate Codex copy. |
| 631 | The Codex marketplace catalog points at the repository root URL: Codex clones the repo, reads the |
| 632 | root `.codex-plugin/plugin.json`, and loads skills from `./skills/`. The Agent Skills install |
| 633 | command documented in the README remains the broadest cross-host path. |
| 634 | |
| 635 | **Grok note:** the repository includes `.grok-plugin/plugin.json` and `.grok-plugin/marketplace.json` |
| 636 | so xAI's Grok Build CLI (`grok`) can install last30days as a native plugin. Grok also reads the |
| 637 | Claude Code manifests for compatibility; the native pair is the first-class lane. The Grok |
| 638 | marketplace catalog uses a bare Git URL source (no commit pin) so `grok plugin marketplace add |
| 639 | mvanhorn/last30days-skill` tracks HEAD — the same pattern as the Codex catalog. `npx skills add` |
| 640 | remains a valid cross-host fallback. |
| 641 | |
| 642 | ### 1. Trusted per-client `.claude/last30days.env` |
| 643 | |
| 644 | When each client has its own working directory, drop a `.claude/last30days.env` into the client folder and opt in with `LAST30DAYS_TRUST_PROJECT_CONFIG=1` from your shell or global `~/.config/last30days/.env`. The skill loads the project file only after that trust signal. Typical contents: |
| 645 | |
| 646 | ```bash |
| 647 | LAST30DAYS_MEMORY_DIR=C:\Users\<you>\Clients\acme\Research\Last30Days |
| 648 | SCRAPECREATORS_API_KEY=<acme-scoped-key-or-shared> |
| 649 | INCLUDE_SOURCES=tiktok,instagram |
| 650 | BSKY_HANDLE=<acme-bluesky-handle>.bsky.social |
| 651 | ``` |
| 652 | |
| 653 | `cd` into the client folder, run `/last30days <topic>` as normal, no wrappers. Combine with `--save-suffix=<client-slug>` per run if you also need to differentiate filenames within that folder. |
| 654 | |
| 655 | ### 2. Per-client save dir + suffix wrapper |
| 656 | |
| 657 | For workflows where you don't `cd` into a client folder (running from anywhere, scripted batches), a tiny shell function isolates each client's research without engine changes. |
| 658 | |
| 659 | PowerShell example: |
| 660 | |
| 661 | ```powershell |
| 662 | function Run-L30D-Client { |
| 663 | param([string]$ClientSlug, [Parameter(ValueFromRemainingArguments=$true)]$Args) |
| 664 | $env:LAST30DAYS_MEMORY_DIR = "C:\Users\$env:USERNAME\Clients\$ClientSlug\Research\Last30Days" |
| 665 | /last30days @Args --save-suffix=$ClientSlug |
| 666 | } |
| 667 | # Usage: Run-L30D-Client acme "british airways middle east" |
| 668 | ``` |
| 669 | |
| 670 | Bash example: |
| 671 | |
| 672 | ```bash |
| 673 | l30d-client() { |
| 674 | local client=$1; shift |
| 675 | LAST30DAYS_MEMORY_DIR="$HOME/Clients/$client/Research/Last30Days" \ |
| 676 | /last30days "$@" --save-suffix="$client" |
| 677 | } |
| 678 | # Usage: l30d-client acme "british airways middle east" |
| 679 | ``` |
| 680 | |
| 681 | ### 3. Custom category-peer subreddits |
| 682 | |
| 683 | [`scripts/lib/categories.py`](skills/last30days/scripts/lib/categories.py) holds a table of `(category_id, trigger_keywords, peer_subreddits)`. If a client lives in a vertical that isn't covered (legal-tech, real-estate-tech, B2B HR SaaS), add a row. Pure data, no logic. |
| 684 | |
| 685 | Section 2a of `SKILL.md` documents the merging rule the skill applies when your topic matches a category. |
| 686 | |
| 687 | ### 4. Pre-built `--competitors-plan` JSON |
| 688 | |
| 689 | For competitor-vs-comparisons that recur, a pre-written JSON skeleton per client industry saves real time: |
| 690 | |
| 691 | ```json |
| 692 | { |
| 693 | "Competitor B": { |
| 694 | "x_handle": "competitor_b_handle", |
| 695 | "subreddits": ["sub1", "sub2"], |
| 696 | "github_user": "competitor-b-org", |
| 697 | "context": "Founded 2019, focused on ..." |
| 698 | }, |
| 699 | "Competitor C": { ... } |
| 700 | } |
| 701 | ``` |
| 702 | |
| 703 | Pass as `--competitors-plan @client/competitors-plan.json` (or as a string). See `SKILL.md` section "If QUERY_TYPE = COMPARISON" for the full schema. |
| 704 | |
| 705 | ### Grok Bot |
| 706 | |
| 707 | On a Grok Bot host the bot exports `LAST30DAYS_HOST=grok-bot` on every engine call (first-run setup also persists it to `.env`), and X search runs through official channels only, in this order: |
| 708 | |
| 709 | 1. **X connector (primary).** Add the "X for Grok Bot" plugin and connect your X account in Grok Bot settings (it provisions an X developer account for you; paid Grok Bot plans include X API credits). When one of the plugin's post-search tools is in the session, the bot exports `LAST30DAYS_X_HOST_LANE=1`, fetches the posts itself, writes them to a `last30days-x-posts/1` file, and passes it with `--x-posts <path>` (see [`--x-posts` flag](#--x-posts-flag)). Connector calls draw on the credits included with Grok Bot and cover the full research window. `--diagnose` lists `x` as available whenever the lane signal is set; a run with the signal but no `--x-posts` file records X as `error` ("connector result not passed"). |
| 710 | 2. **`X_BEARER_TOKEN` (backup).** An app-only bearer from the X developer console, funded by your own X developer project. The engine tries full-archive search first and falls back to recent search, so coverage is recent posts, about the last week, unless your X developer project has full-archive access; the outcome detail says `window truncated to 7 days` when the fallback ran. Exhausted credits (HTTP 402) report as `payment-required`, which `LAST30DAYS_STRICT_EXIT` treats as degraded. |
| 711 | 3. **`XAI_API_KEY` (backup).** xAI's licensed X search from console.x.ai: full window, topic search only (no from/mention handle lanes). |
| 712 | |
| 713 | Persist either key without echoing it: |
| 714 | |
| 715 | ```bash |
| 716 | printf '%s\n' "$TOKEN" | python3 skills/last30days/scripts/last30days.py setup --store-key X_BEARER_TOKEN |
| 717 | ``` |
| 718 | |
| 719 | Browser sessions are not read on this host, and no login window is opened; `setup` still installs the free CLIs. Doctor prints the resolved host value, so a missing `LAST30DAYS_HOST` export is visible at a glance. |
| 720 | |
| 721 | --- |
| 722 | |
| 723 | ## Beta channel |
| 724 | |
| 725 | Experimental customizations live on a private companion repo (`mvanhorn/last30days-skill-private`) installed as `/last30days-beta`. Never ship beta-only changes to the public marketplace without a review PR against the public repo. Workflow guide: `BETA.md` in the private repo. |
| 726 | |
| 727 | This is the right home for client-specific changes you don't intend to upstream - custom category rows, internal subreddit lists, per-vertical plan templates. |
| 728 | |
| 729 | --- |
| 730 | |
| 731 | ## Cross-references |
| 732 | |
| 733 | - The CLI flag surface: `python3 scripts/last30days.py --help` |
| 734 | - The skill contract (voice, LAWs, pre-flight protocol): [`skills/last30days/SKILL.md`](skills/last30days/SKILL.md) |
| 735 | - Shared package vocabulary and engine/harness terminology: [`CONCEPTS.md`](CONCEPTS.md) |
| 736 | - Contributor guidance: [`CONTRIBUTORS.md`](CONTRIBUTORS.md) |
| 737 |