| 1 | # How Reddit & X Search Work in last30days |
| 2 | |
| 3 | ## Architecture Overview |
| 4 | |
| 5 | ``` |
| 6 | User: /last30days "kanye west" |
| 7 | ↓ |
| 8 | ┌─────┴─────┐ |
| 9 | ↓ ↓ (concurrent via ThreadPoolExecutor) |
| 10 | [REDDIT] [X/TWITTER] |
| 11 | ↓ ↓ |
| 12 | OpenAI Bundled Bird or |
| 13 | API xAI API |
| 14 | ↓ ↓ |
| 15 | Parse Parse |
| 16 | ↓ ↓ |
| 17 | Enrich ───┘ |
| 18 | (fetch ↓ |
| 19 | actual [MERGE] |
| 20 | upvotes) ↓ |
| 21 | ↓ [NORMALIZE → FILTER → SCORE → DEDUPE] |
| 22 | └───────────↓ |
| 23 | [OUTPUT to SKILL.md agent] |
| 24 | ``` |
| 25 | |
| 26 | Both searches run **in parallel** using Python's `ThreadPoolExecutor(max_workers=2)`. |
| 27 | |
| 28 | --- |
| 29 | |
| 30 | ## Reddit Search |
| 31 | |
| 32 | ### How it works |
| 33 | |
| 34 | Reddit search uses the **OpenAI Responses API** with the `web_search` tool, domain-filtered to `reddit.com` only. |
| 35 | |
| 36 | **API Call:** |
| 37 | ``` |
| 38 | POST https://api.openai.com/v1/responses |
| 39 | Authorization: Bearer {OPENAI_API_KEY} |
| 40 | ``` |
| 41 | |
| 42 | **Payload:** |
| 43 | ```json |
| 44 | { |
| 45 | "model": "gpt-5.2", |
| 46 | "tools": [{ |
| 47 | "type": "web_search", |
| 48 | "filters": { "allowed_domains": ["reddit.com"] } |
| 49 | }], |
| 50 | "input": "Search Reddit for threads about {topic}..." |
| 51 | } |
| 52 | ``` |
| 53 | |
| 54 | The prompt asks the model to: |
| 55 | 1. Extract core subject (strip noise words like "best", "tips", "top") |
| 56 | 2. Search 3 patterns: `"{topic} site:reddit.com"`, `"reddit {topic}"`, `"{topic} reddit"` |
| 57 | 3. Return JSON with `title`, `url`, `subreddit`, `date`, `relevance` |
| 58 | 4. URLs must contain `/r/` AND `/comments/` (real threads only) |
| 59 | |
| 60 | **Model fallback chain:** `gpt-5.2 → gpt-5.1 → gpt-5 → gpt-4.1 → gpt-4o → gpt-4o-mini` |
| 61 | Triggers on HTTP 400/403 with access error keywords. |
| 62 | |
| 63 | ### Enrichment (the secret sauce) |
| 64 | |
| 65 | After search, each thread gets **enriched** by hitting Reddit's free JSON API: |
| 66 | |
| 67 | ``` |
| 68 | GET https://reddit.com/r/{sub}/comments/{id}/{slug}/.json |
| 69 | ``` |
| 70 | |
| 71 | No API key needed. This returns the actual thread data: |
| 72 | |
| 73 | | Data Point | Source | |
| 74 | |---|---| |
| 75 | | Upvotes (score) | Reddit JSON API | |
| 76 | | Comment count | Reddit JSON API | |
| 77 | | Upvote ratio | Reddit JSON API | |
| 78 | | Top 10 comments (text + score) | Reddit JSON API | |
| 79 | | 7 key comment insights | Extracted via heuristics | |
| 80 | | Actual post date | `created_utc` timestamp | |
| 81 | |
| 82 | **This is why Reddit results have real engagement metrics** — the enrichment step fetches actual upvote/comment data, not AI estimates. |
| 83 | |
| 84 | ### Depth settings |
| 85 | |
| 86 | | Depth | Threads requested | Timeout | |
| 87 | |---|---|---| |
| 88 | | `--quick` | 15-25 | 90s | |
| 89 | | default | 30-50 | 120s | |
| 90 | | `--deep` | 70-100 | 180s | |
| 91 | |
| 92 | --- |
| 93 | |
| 94 | ## X/Twitter Search |
| 95 | |
| 96 | X search has **two backends** — the skill auto-detects which to use. |
| 97 | |
| 98 | ### Priority: Bundled Bird (env auth) → xAI API (paid) |
| 99 | |
| 100 | ```python |
| 101 | if node_available and AUTH_TOKEN and CT0: |
| 102 | use bundled Bird # Free, popup-free, env-authenticated |
| 103 | elif XAI_API_KEY: |
| 104 | use xAI API # Paid, uses grok-4-1-fast |
| 105 | else: |
| 106 | skip X entirely # No X results |
| 107 | ``` |
| 108 | |
| 109 | ### Backend 1: xAI API |
| 110 | |
| 111 | **API Call:** |
| 112 | ``` |
| 113 | POST https://api.x.ai/v1/responses |
| 114 | Authorization: Bearer {XAI_API_KEY} |
| 115 | ``` |
| 116 | |
| 117 | **Payload:** |
| 118 | ```json |
| 119 | { |
| 120 | "model": "grok-4-1-fast", |
| 121 | "tools": [{ "type": "x_search" }], |
| 122 | "input": "Search X for posts about {topic} from {from_date} to {to_date}..." |
| 123 | } |
| 124 | ``` |
| 125 | |
| 126 | The prompt asks grok to return JSON with: |
| 127 | - `text`, `url`, `author_handle`, `date` |
| 128 | - `engagement`: `{ likes, reposts, replies, quotes }` |
| 129 | - `why_relevant`, `relevance` score |
| 130 | |
| 131 | **Engagement data comes from grok's x_search tool** - it has direct access to X's data. |
| 132 | |
| 133 | ### Backend 2: Bundled Bird client (free alternative) |
| 134 | |
| 135 | The repo vendors a search-only subset of Bird's Twitter GraphQL client and shells out to it with Node.js. No global `bird` install is required. The Python wrapper passes `AUTH_TOKEN` and `CT0` via env, which keeps normal local runs headless and avoids browser-cookie prompts. |
| 136 | |
| 137 | **Bundled Bird returns raw X API data** - likes, reposts, replies are real engagement metrics from X's API, not estimates. |
| 138 | |
| 139 | | Metric | Bundled Bird | xAI API | |
| 140 | |---|---|---| |
| 141 | | Post text | Real | Real | |
| 142 | | Likes/reposts | Real (X API) | Real (x_search tool) | |
| 143 | | Replies/quotes | Real | Real | |
| 144 | | Author handle | Real | Real | |
| 145 | | Relevance score | Default 0.7 (re-ranked by relevance.py) | AI-assessed 0.0-1.0 | |
| 146 | |
| 147 | ### Depth settings |
| 148 | |
| 149 | | Depth | xAI posts | Bundled Bird results | xAI timeout | Bird timeout | |
| 150 | |---|---|---|---|---| |
| 151 | | `--quick` | 8-12 | 12 | 90s | 30s | |
| 152 | | default | 20-30 | 30 | 120s | 45s | |
| 153 | | `--deep` | 40-60 | 60 | 180s | 60s | |
| 154 | |
| 155 | --- |
| 156 | |
| 157 | ## Post-Processing (both sources) |
| 158 | |
| 159 | After both searches complete: |
| 160 | |
| 161 | 1. **Normalize** — consistent formatting, timezone handling |
| 162 | 2. **Date filter** — hard filter to requested date range |
| 163 | 3. **Score** — relevance scoring (engagement-weighted) |
| 164 | 4. **Sort** — highest scores first |
| 165 | 5. **Deduplicate** — remove duplicate URLs |
| 166 | 6. **Fallback** — if all items filtered out, keep top 3 by relevance |
| 167 | |
| 168 | --- |
| 169 | |
| 170 | ## Error Handling |
| 171 | |
| 172 | | Layer | Strategy | |
| 173 | |---|---| |
| 174 | | HTTP requests | 3 retries with exponential backoff (1s → 2s → 3s) | |
| 175 | | Model access errors | Automatic fallback to next model in chain | |
| 176 | | Reddit enrichment | Per-item try/catch; keeps unenriched item on failure | |
| 177 | | X source detection | Silent fallback from Bird → xAI → skip | |
| 178 | | Overall pipeline | Errors stored as `reddit_error`/`x_error`, shown to user | |
| 179 | |
| 180 | --- |
| 181 | |
| 182 | ## Key Files |
| 183 | |
| 184 | | File | Purpose | |
| 185 | |---|---| |
| 186 | | `skills/last30days/scripts/last30days.py` | Main CLI entry point | |
| 187 | | `skills/last30days/scripts/lib/pipeline.py` | Multi-source retrieval orchestration | |
| 188 | | `skills/last30days/scripts/lib/reddit_public.py` | Reddit public JSON search | |
| 189 | | `skills/last30days/scripts/lib/reddit_enrich.py` | Fetch real engagement data from Reddit JSON API | |
| 190 | | `skills/last30days/scripts/lib/xai_x.py` | X search via xAI API | |
| 191 | | `skills/last30days/scripts/lib/bird_x.py` | X search via bundled Bird client (free) | |
| 192 | | `skills/last30days/scripts/lib/providers.py` | Reasoning provider and model selection | |
| 193 | | `skills/last30days/scripts/lib/env.py` | API key loading, source detection | |
| 194 | | `skills/last30days/scripts/lib/http.py` | HTTP transport with retries | |
| 195 | | `skills/last30days/scripts/lib/relevance.py` | Query matching and relevance scoring | |
| 196 | | `skills/last30days/scripts/lib/dedupe.py` | URL-based deduplication | |
| 197 |