| 1 | --- |
| 2 | title: Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items |
| 3 | date: 2026-06-09 |
| 4 | category: docs/solutions/logic-errors |
| 5 | module: lib/rerank |
| 6 | problem_type: logic_error |
| 7 | component: search_ranking |
| 8 | severity: high |
| 9 | symptoms: |
| 10 | - on-entity, high-engagement items that name the brand but omit the trailing descriptor of a multi-word query are demoted in keyless/fallback rerank results |
| 11 | - observed case is a 323-point HN thread about Stripe scoring 0 on a "Stripe payments" query |
| 12 | - the entity-miss demotion lands twice (ENTITY_MISS_PENALTY on rerank_score plus a secondary final_score penalty), so a false miss guarantees burial regardless of engagement |
| 13 | - reddit keyless comment-enrichment slot selection skips the same on-entity threads via an independently duplicated full-phrase check in _slot_priority |
| 14 | root_cause: logic_error |
| 15 | resolution_type: code_fix |
| 16 | related_components: |
| 17 | - reddit_keyless |
| 18 | - comment_enrichment |
| 19 | tags: |
| 20 | - entity-grounding |
| 21 | - rerank |
| 22 | - keyless-fallback |
| 23 | - multi-word-entity |
| 24 | - substring-match |
| 25 | - false-demotion |
| 26 | - reddit-keyless |
| 27 | - duplicated-logic |
| 28 | --- |
| 29 | |
| 30 | # Keyless rerank entity grounding required full multi-word phrase, falsely demoting on-entity items |
| 31 | |
| 32 | ## Problem |
| 33 | |
| 34 | The keyless/fallback rerank path's entity-grounding demotion required the FULL multi-word primary-entity phrase as a contiguous substring of the candidate's text (`primary_entity.lower() not in haystack`), so on-entity items that omitted a trailing search descriptor were falsely flagged as entity misses and buried by a deliberately decisive double penalty. |
| 35 | |
| 36 | ## Symptoms |
| 37 | |
| 38 | - On a "Stripe payments" query, a 323-point HN thread titled "Stripe is friendly to 'friendly fraud'" was demoted to score 0 — purely because its text never contained the literal phrase "stripe payments" (the trailing word "payments" was missing). |
| 39 | - The burial is guaranteed by design, not incidental: a flagged entity miss takes −25 `ENTITY_MISS_PENALTY` on `rerank_score` in `_fallback_tuple`, PLUS `ENTITY_MISS_FINAL_PENALTY` applied directly in `_final_score` (added 2026-04-19 after engagement + freshness drowned the diluted penalty). A false positive on the check means confirmed-good signal cannot recover. |
| 40 | - The same over-strict check had been independently re-implemented in `reddit_keyless._slot_priority` (keyless Reddit comment-enrichment slot selection), so scarce comment slots were also steered away from head-token-only posts. |
| 41 | |
| 42 | ## What Didn't Work |
| 43 | |
| 44 | - **Naively relaxing the check** — the full-phrase check existed for a real reason: on 2026-04-19 an off-topic video with zero brand mentions ranked #2 on a Hermes query (documented in the `ENTITY_MISS_FINAL_PENALTY` comment in `skills/last30days/scripts/lib/rerank.py`). Any fix had to keep that demotion firing. |
| 45 | - **Word-boundary matching** — rejected; it re-introduces over-demotion on plurals/possessives/compounds ("stripes", "Stripe's"). |
| 46 | - **Graded penalty** (full-phrase = 0, head-only = half, none = full) — rejected; it half-punishes items that are 100% about the entity. Lexical coverage is not topical degree. |
| 47 | - **Any-token grounding** — rejected; "payments" alone would ground completely generic posts. |
| 48 | - **Distinctiveness gate for generic heads** — rejected as complexity to patch a failure mode that is already a safe no-op (see Why This Works). |
| 49 | - **Trusting the docstring** — `reddit_keyless._slot_priority`'s docstring claimed to "mirror rerank's demotion signal," but its inline reimplementation (`entity in _post_text(post).lower()`) had silently drifted from being a mirror into being a second copy of the bug. It was found only by a code-reuse review, not by tests. |
| 50 | |
| 51 | ## Solution |
| 52 | |
| 53 | Ground on the **head token** of the primary entity instead of the full phrase, via one shared helper used by both paths. |
| 54 | |
| 55 | **Site 1 — new helper in `skills/last30days/scripts/lib/rerank.py`:** |
| 56 | |
| 57 | ```python |
| 58 | def _entity_grounded(haystack: str, primary_entity: str) -> bool: |
| 59 | tokens = primary_entity.lower().split() |
| 60 | if not tokens: |
| 61 | return True |
| 62 | return tokens[0] in haystack |
| 63 | ``` |
| 64 | |
| 65 | `_fallback_tuple` switches from the inline phrase check to the helper: |
| 66 | |
| 67 | ```python |
| 68 | # before |
| 69 | if haystack.strip() and primary_entity.lower() not in haystack: |
| 70 | # after |
| 71 | if haystack.strip() and not _entity_grounded(haystack, primary_entity): |
| 72 | ``` |
| 73 | |
| 74 | **Site 2 — secondary penalty in `_final_score`: no code change needed.** It keys off the explanation string set by site 1, so it inherits the fix automatically: |
| 75 | |
| 76 | ```python |
| 77 | if candidate.explanation and "entity-miss" in candidate.explanation: |
| 78 | base = max(0.0, base - ENTITY_MISS_FINAL_PENALTY) |
| 79 | ``` |
| 80 | |
| 81 | **Site 3 — `skills/last30days/scripts/lib/reddit_keyless.py` `_slot_priority`:** replace the drifted reimplementation with a call to the shared helper: |
| 82 | |
| 83 | ```python |
| 84 | # before |
| 85 | return entity in _post_text(post).lower() |
| 86 | # after |
| 87 | return rerank._entity_grounded(_post_text(post).lower(), entity) |
| 88 | ``` |
| 89 | |
| 90 | Tests: `tests/test_rerank_v3.py` gained `test_fallback_grounds_on_head_token_not_full_phrase` (the Stripe regression) and `test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` (guards the 2026-04-19 behavior). `tests/test_reddit_keyless.py`'s two old-contract tests were rewritten as `test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match`. |
| 91 | |
| 92 | ## Why This Works |
| 93 | |
| 94 | - **Root cause:** trailing tokens of a multi-word query ("payments" in "Stripe payments") are usually category descriptors the user/planner appended for search, not part of the entity name. Requiring the whole phrase conflates "doesn't repeat my search phrasing" with "isn't about my entity." The brand head token alone is sufficient grounding; items that never name the brand at all still miss the head token and stay demoted — so the original 2026-04-19 fix keeps firing. |
| 95 | - **Asymmetry argument:** the demotion is engineered to be decisive (double penalty across `rerank_score` and `final_score`), so a false entity-miss is fatal-by-design, while a false grounding merely defers the item to normal relevance/freshness/quality ranking. When the punishment is capital, the conviction standard should be conservative. |
| 96 | - **Substring (not word-boundary) is deliberate:** it catches plurals/possessives/compounds ("stripes", "Stripe's"). Degenerate short heads ("X", "Go", "C") make the check vacuously true, which merely **disables** the penalty — reverting to the pre-grounding baseline — rather than burying good items. Every failure mode of this rule degrades toward "no penalty," never toward "bury good signal." |
| 97 | - **Accepted, bounded limitation:** head-collision with a different famous entity ("Hermes Agent" → a "Hermes Birkin" thread now escapes demotion). This is lexically unfixable — any token rule strong enough to kill the collision re-kills the Stripe case; the discriminator is semantic. The LLM rerank path (which receives the full phrase as prompt guidance and judges semantically) covers this when API keys exist; the keyless path accepts the bounded risk. |
| 98 | |
| 99 | ## Prevention |
| 100 | |
| 101 | - **Shared helper as single source of truth:** when one module's behavior must "mirror" another's signal, it must *call* the same function, not re-implement the check. The `reddit_keyless._slot_priority` drift happened precisely because the mirror was a copy. The fix wires it to `rerank._entity_grounded`, and the docstring now states this explicitly: "keying on the same head token keeps the two paths from diverging." |
| 102 | - **Docstrings record deliberate trade-offs:** `_entity_grounded`'s docstring documents WHY head-token (not phrase), why substring (not word-boundary), and the safe-failure direction. Future readers see the rejected alternatives were considered, not overlooked — and won't "tighten" the check into a regression. |
| 103 | - **Both directions pinned by named tests:** |
| 104 | - `tests/test_rerank_v3.py::test_fallback_grounds_on_head_token_not_full_phrase` — false-demotion regression (the Stripe HN thread must not be flagged). |
| 105 | - `tests/test_rerank_v3.py::test_fallback_still_demotes_when_head_token_absent_on_multiword_topic` — the fix must not neuter the demotion (guards the 2026-04-19 off-topic-video incident). |
| 106 | - `tests/test_reddit_keyless.py::test_slot_priority_grounds_on_head_token_not_full_phrase` and `test_intent_modifier_topic_prioritizes_head_token_match` — the mirrored path asserts the same contract. |
| 107 | - **Audit tests when changing a contract:** tests that encode the old behavior as correct must be rewritten to the new contract, not worked around — the two old `test_reddit_keyless.py` tests would have silently re-blessed the bug. |
| 108 | - **For decisive penalties, route through one flag:** the `_final_score` backstop keys off `"entity-miss" in candidate.explanation` rather than re-running the check — so there was exactly one site to fix and the second penalty inherited it for free. Prefer this signal-propagation pattern over duplicating predicate logic at each penalty site. |
| 109 | |
| 110 | ## Related Issues |
| 111 | |
| 112 | - [PR #484](https://github.com/mvanhorn/last30days-skill/pull/484) — "fix(reddit): relevance-aware comment-enrichment slot selection in keyless path" — introduced the `_slot_priority` mirror this fix reroutes through the shared helper. |
| 113 | - [PR #457](https://github.com/mvanhorn/last30days-skill/pull/457) — "fix(reddit): restore free path via keyless RSS + shreddit scrape" — established the keyless Reddit path. |
| 114 | - [PR #488](https://github.com/mvanhorn/last30days-skill/pull/488) (open) — "fix(reddit): relevance floor + relevance-first ranking" — external PR touching the same ranking surface; coordinate before merging both. |
| 115 | - [Issue #468](https://github.com/mvanhorn/last30days-skill/issues/468) (open) — relevance scoring over-pruning on-topic YouTube items; same symptom family in a different source. |
| 116 | - [../architecture/search-quality-eval-manual-by-default-2026-05-10.md](../architecture/search-quality-eval-manual-by-default-2026-05-10.md) — how to validate ranking/grounding changes like this one (manual eval, not CI-gated). |
| 117 | - [../workflow-issues/release-consistency-test-cascade-2026-05-16.md](../workflow-issues/release-consistency-test-cascade-2026-05-16.md) — sibling prevention pattern: lockstep artifacts drift unless mechanically unified. |
| 118 |