The sift was feeding Opus a magnitude-blind, under-trusted, truncated shortlist — only 68% of the
genuinely-best jobs reached the curator. Fixes:
- embed.py: embed the FULL description (was desc[:600]) + role_category + industry; richer profile
text (current_role + experience summary + seniority). The best semantic signal, fed real content.
- sift.py: magnitude-preserving min-max fusion (was rank-position, which flattened cosine 0.95 vs
0.72) and embedding-LED weights (W_VIBE 0.6 / W_WHITEBOX 0.4). Embed runs FIRST so the white-box
f_semantic reuses the real cosine (threaded as job["_vibe_cosine"]) instead of token overlap.
- config.py: SIFT_TOP_K 18 → 28 (the gate was cutting good jobs before Opus).
- curate.py: richer Opus briefs — full responsibilities (was desc[:500]) + role_category/industry/
seniority/pay, so even the shortlist is fully described.
Regression (120 pairs): sift membership recall 0.68 → 0.92, score MAE 24.7 → 18.4. 31 tests pass.
Phase 0 — measurement gate:
- tests/test_contracts.py: BOARDS↔NORMALIZERS alignment, funnel-shape guards, overlap() None
contract, and the frontend↔backend EXACT-STRING vocab check (scout.ts options must resolve in
coerce.py, else silent keyword-fold).
- tests/run_regression.py + fixtures/regression_set.json (+ _seed_regression.py): a frozen 120-pair
set (tech + non-tech) Opus-judged for recruiter-fit targets. Two metrics: score MAE and SIFT TOP-K
MEMBERSHIP RECALL (do the best jobs reach Opus?). Baseline: MAE 24.7, recall 0.68 — i.e. ~32% of
the genuinely-best jobs are cut before the curator ever sees them (worse for non-tech).
Phase 1 — coverage (India non-tech):
- Enable Indeed (misceres) + enrich indeed_to_scoutjob with details.description + location_mode
(jobType is employment type, not skills → required_skills now []).
- Add WorkIndia (shahidirfan) — India blue/grey-collar non-tech: build_workindia_input +
workindia_to_scoutjob (real skills + description; per-job apply URL from job_id since source_url
is generic and would dedup-collapse the deck).
- Drop Wellfound from the stack (US-startup-heavy + 400s); kept registered-but-off.
- BOARDS_ENABLED = naukri,foundit,linkedin,indeed,workindia. Pool ~90 → ~140 jobs/search.
Regression scores unchanged (engine scoring untouched). 31 tests pass.
