Engine Phase 2: fix the pre-Opus funnel (lifts ALL matches)

The sift was feeding Opus a magnitude-blind, under-trusted, truncated shortlist — only 68% of the
genuinely-best jobs reached the curator. Fixes:
- embed.py: embed the FULL description (was desc[:600]) + role_category + industry; richer profile
  text (current_role + experience summary + seniority). The best semantic signal, fed real content.
- sift.py: magnitude-preserving min-max fusion (was rank-position, which flattened cosine 0.95 vs
  0.72) and embedding-LED weights (W_VIBE 0.6 / W_WHITEBOX 0.4). Embed runs FIRST so the white-box
  f_semantic reuses the real cosine (threaded as job["_vibe_cosine"]) instead of token overlap.
- config.py: SIFT_TOP_K 18 → 28 (the gate was cutting good jobs before Opus).
- curate.py: richer Opus briefs — full responsibilities (was desc[:500]) + role_category/industry/
  seniority/pay, so even the shortlist is fully described.

Regression (120 pairs): sift membership recall 0.68 → 0.92, score MAE 24.7 → 18.4. 31 tests pass.
This commit is contained in:
raulgupta
2026-06-25 15:50:14 +05:30
parent 04e6fb64bb
commit eb6ded5b3d
7 changed files with 72 additions and 28 deletions

View File

@@ -11,24 +11,32 @@ from app.engine.llm import embed_client
def _job_text(job: dict) -> str:
# The embedding is the best semantic matcher — feed it the FULL signal, not a 600-char stub.
# role_category + industry anchor non-tech JDs (narrative prose, few repeated title tokens).
d = job.get("details") or {}
parts = [
job.get("title", ""),
job.get("organization", ""),
" ".join(d.get("skills") or []),
(d.get("description") or "")[:600],
d.get("role_category") or "",
d.get("industry") or "",
" ".join(d.get("skills") or job.get("required_skills") or []),
(d.get("description") or "")[:2000],
]
return " · ".join(p for p in parts if p)
def _profile_text(prefs: dict, ctx: dict | None) -> str:
ctx = ctx or {}
# Embed the candidate richly too — current_role + experience summary + seniority, not just title+skills.
exp = ctx.get("experience_summary") or ctx.get("summary") or ""
parts = [
prefs.get("title", ""),
" ".join(prefs.get("role") or []),
" ".join(ctx.get("skills") or []),
ctx.get("current_role", "") or "",
" ".join(ctx.get("skills") or []),
" ".join(prefs.get("industry") or []),
" ".join(prefs.get("experience") or []),
str(exp)[:1200],
]
return " · ".join(p for p in parts if p)