Engine v2: embeddings sift + Opus curator + Postgres feed persistence
- A1: fire the hard filter (Apify is precise); embedding vibe-ranker (embed.py) blends with the floor-free white-box (rank.py) to sift ~90 -> top 18 - A2: Opus curator (curate.py) reads the 18 -> honest <=count + Gemini-voiced report cards; graceful fallback to the white-box + templated cards - LLM via opencode.ai/zen gateway (llm.py): Opus chat + direct-OpenAI embeddings - Run LLM work off the event loop (asyncio.to_thread) so the Redis response publishes - C1: dedicated Postgres (app/db/) persists the per-user feed; get_scout_feed replays it so matches survive navigation/refresh - match contract + report-card fields (schema.py); skills.py; tests/
This commit is contained in:
raulgupta
@@ -0,0 +1,171 @@
|
||||
# Engine Build Analysis — from 90 raw cards to a ranked, explained shortlist
|
||||
|
||||
> Written 2026-06-19. Companion to `ENGINE_DESIGN.md` (the locked design) + `SIGNAL_AUDIT_V2.md`
|
||||
> (the signal inventory). This doc is the **build-readiness analysis**: what we have *today*, the gap,
|
||||
> and the smallest engine that turns the live multi-board sweep into "only the recommended ones, with
|
||||
> aligned stats." It translates the design into a phased, buildable plan against the real data surface.
|
||||
|
||||
---
|
||||
|
||||
## 1. The gap (current vs target)
|
||||
|
||||
**Today (Slice 1a):** fetch ~90 live jobs → `search.run_sweep` assigns a **placeholder** score
|
||||
(`matchScore = max(55, 95-i)`, i.e. rank = fetch order) → **all 90** shown. Critically:
|
||||
- `user_context` (the user's resume/skills/experience) is **fetched by the orchestrator but ignored**
|
||||
by `handle_run_search` — so nothing is actually matched *to the user*.
|
||||
- The card's "stats" are **fabricated in the frontend**: `scoutJobToMatchRole()` builds the QX
|
||||
`breakdown` from `required_skills[:3]` + fixed deltas `[10,-7,-32]`, derives `growth` from that, sets
|
||||
`reason = "Live from Naukri"`, and `proofReady = matchScore≥84`. **None of it reflects the user.**
|
||||
|
||||
**Target:** fetch ~90 → **engine** (filter + rank *against the user* + select) → **top ~20–30** →
|
||||
each card carries a **real** match %, a real "why picked", a real skill breakdown, and a FIT/STRETCH tag.
|
||||
|
||||
Two asks map to two engine outputs:
|
||||
- *"only show the ones recommended"* → **hard filter + score + top-N selection**.
|
||||
- *"stats more aligned"* → **move the breakdown/why/growth from fabricated-frontend to real-backend**,
|
||||
and have the card render what the engine emits.
|
||||
|
||||
---
|
||||
|
||||
## 2. The input surface we actually have NOW
|
||||
|
||||
### Job side — per fetched `ScoutJob` (already extracted by the normalizers)
|
||||
`title` · `required_skills` (full list in `details.skills`) · `seniority_level` · `location_city` +
|
||||
`location_mode` · `salary_lpa` / `payLabel` · `industry` · `role_category` · `employment_type` ·
|
||||
`description` (clean text) · `note` (board) · `offsite_apply` · freshness (**all on-demand = fresh by
|
||||
construction**; we already drop `isExpired`).
|
||||
|
||||
### User side
|
||||
- **Configure prefs (`ScoutPrefs`) — always present:** `title`, `role[]`, `location[]`, `workMode[]`,
|
||||
`experience`/seniority, `industry[]`, `years`, `targetComp`, `companyStage`, `availability`,
|
||||
`dealBreakers[]`, `priorities[]` (rank), `stretch` (safe/balanced/reach).
|
||||
- **`user_context` (orchestrator-assembled, currently ignored) — present when the user has a profile:**
|
||||
`skills[]` (resume + live LinkedIn, deduped), `current_role`, `experience_history`, `education`,
|
||||
`linkedin_headline/summary/experience`. Plus the **QScore quotient vector** (qscore-service).
|
||||
|
||||
**Coverage reality:** Configure prefs are guaranteed; `user_context` may be sparse/empty (the "No
|
||||
resume" state). The engine **must** degrade gracefully (design §3.3 coverage-renormalization) — rank on
|
||||
what's present, never fabricate, never collapse to a neutral 60 (the v1 bug).
|
||||
|
||||
---
|
||||
|
||||
## 3. The design (recap) vs what's buildable first
|
||||
|
||||
`ENGINE_DESIGN.md` §3 cascade: ① normalize → ② hard-filter → ③ white-box utility → ④ rank-fusion →
|
||||
⑤ LLM rerank+explain → ⑥ output → (log → learned ranker).
|
||||
|
||||
The **full** vision needs ESCO taxonomy + depth-2 KG, contrastive skill embeddings, GPT-5.4 rerank, and
|
||||
a learned pairwise ranker. **None of that is needed to ship the core value.** Stages ② + ③ + ⑥ —
|
||||
deterministic, white-box, no LLM, no taxonomy service — already deliver *filter + rank + select +
|
||||
explain* against the current data. That is the first engine; the rest are precision upgrades.
|
||||
|
||||
---
|
||||
|
||||
## 4. Phased build
|
||||
|
||||
### Phase 0 — Deterministic white-box engine (`app/engine/rank.py`), no LLM, no `user_context` required
|
||||
The shippable core. Filters 90 → top-N, scores, emits real stats. Works on Configure prefs alone, gets
|
||||
sharper when `user_context` lands (Phase 1).
|
||||
|
||||
**② Hard filter (deterministic gate — the design's point: embeddings can't do this):**
|
||||
- **Location / work-mode fit** — port `_score_location_fit` + `_location_filter_blocks_match` from the
|
||||
*old* `matchmaking/app/engine/core.py` (the audit says it did this better): remote→100 (0 if seeker
|
||||
onsite-only); else city/country substring vs preferred locations; unknown→neutral 60; **gate** blocks
|
||||
no-fit unless prefs are "Anywhere".
|
||||
- **Seniority band** — drop jobs far outside the user's level (e.g. a 2-yr seeker vs a Lead role),
|
||||
governed by `stretch` (reach allows +1 level).
|
||||
- **Deal-breakers** — exclude on stated breakers where detectable (on-call, on-site, etc.).
|
||||
- **Freshness** — already fresh; drop any `isExpired`/dead.
|
||||
|
||||
**③ White-box utility, coverage-renormalized** — `U = Σ_present c_f·w_f·φ_f / Σ_present c_f·w_f`:
|
||||
| factor | prior w | φ (0–1) from |
|
||||
|---|---|---|
|
||||
| skill_match | .35 | overlap(user.skills, job.required_skills), synonym-normalized; required>preferred |
|
||||
| experience | .25 | user `years`/seniority vs job `seniority_level` band |
|
||||
| location | .15 | ported location-fit score |
|
||||
| salary | .10 | job `salary_lpa` vs `targetComp` band — **present-only** (most are "Not disclosed" → weight redistributes) |
|
||||
| industry/role | .05+.05 | job `industry`/`role_category` vs prefs |
|
||||
| semantic | .10 | cheap title/description keyword overlap (NOT embeddings in v0) |
|
||||
|
||||
Sparse profile (no `skills`) → skill weight redistributes to location/experience/recency. Never penalize absence.
|
||||
|
||||
**⑤′ Selection (the "only recommended" lever)** — sort by U, cut by `stretch`:
|
||||
`safe`→ U≥0.85 · `balanced`→ U≥0.70 (+ a few stretch) · `reach`→ U≥0.55. **Floor of ~8–10 so the deck
|
||||
is never empty** (relax threshold if too few pass); **cap ~25–30** (never the full 90).
|
||||
|
||||
**FIT vs STRETCH (§3.5):** FIT = skill-gap≈0 (score current overlap, mild over-qual penalty);
|
||||
STRETCH = one level up / adjacent, scored by **reachability** (fraction of required skills the user has
|
||||
or is one hop from). Tag each card; the `stretch` pref sets how many STRETCH cards ride along.
|
||||
|
||||
**⑥ Aligned stats (emitted by the engine, rendered by the card):**
|
||||
- `match` = round(U×100) — **real**, not fetch-order.
|
||||
- `breakdown[]` = the real top dims (matched skills Strong / partial Solid / missing Light) — **from
|
||||
user↔job skill overlap**, replacing the fabricated `[10,-7,-32]`.
|
||||
- `growth` = the weakest *reachable* dim + the lift it buys.
|
||||
- `reason` = a real templated "why" ("Matches 6/8 must-have skills · Senior fit · fresh from Naukri").
|
||||
- `proofReady` = user is Strong on the job's key skill (real).
|
||||
- `fit | stretch` tag.
|
||||
|
||||
### Phase 1 — Wire `user_context` (the accuracy unlock)
|
||||
`handle_run_search` reads `params["user_context"]` (already delivered by the worker) → feeds
|
||||
`skills/current_role/experience_history/education` + QScore quotient into ③. QScore/assessments are
|
||||
**boosters** (present-only, coverage-penalized), never a penalty when absent. This is the user-arm the
|
||||
whole product is about — Phase 0 ranks on prefs; Phase 1 ranks on *who the user actually is*.
|
||||
|
||||
### Phase 2 — LLM rerank + explanation (GPT-5.4 on top ~20)
|
||||
Calibrated 0–1 fit + a genuine 1-line "why" + growth nudge; replaces the templated reason on the top
|
||||
slice. Log soft-scores as cold-start supervision. **Only the top ~20, cached per (role, location)** —
|
||||
never LLM all 90 (cost/latency).
|
||||
|
||||
### Phase 3 — Learned ranker
|
||||
Accrue SAVE/DISMISS/APPLY labels (the `record_feedback` action) → pairwise/listwise ranker → shift
|
||||
fusion weight off the expert priors.
|
||||
|
||||
---
|
||||
|
||||
## 5. The stats-alignment refactor (a contract change)
|
||||
|
||||
Right now the card's "intelligence" is **invented in `scoutJobToMatchRole`**. For stats to be honest,
|
||||
the engine must own them and the ScoutJob must carry a `match` block:
|
||||
|
||||
```
|
||||
match: {
|
||||
score: number, // U×100
|
||||
fit: "fit" | "stretch",
|
||||
reason: string, // real why-picked
|
||||
breakdown: [{name, score, level}], // real user↔job skill dims
|
||||
growth?: {text, from, to},
|
||||
proofReady: boolean,
|
||||
}
|
||||
```
|
||||
The frontend `scoutJobToMatchRole` then **reads `job.match.*`** instead of fabricating it. Net: delete
|
||||
the fake-breakdown logic; the card becomes a pure renderer of engine output. (Same modal/stats UI — it
|
||||
just gets real numbers.)
|
||||
|
||||
---
|
||||
|
||||
## 6. Key decisions & risks
|
||||
- **Skill normalization without ESCO (v0):** a **seed synonym map** (k8s→Kubernetes, React.js→React,
|
||||
ML→Machine Learning, JS→JavaScript…) + India tech terms + lowercase/punctuation fold. Beats exact
|
||||
match; upgrade to ESCO+KG+contrastive later. This is the single biggest accuracy lever (§3.2).
|
||||
- **Sparse profiles are the norm, not the exception** — coverage-renormalization is load-bearing, not
|
||||
a nicety. Test the "No resume" path first.
|
||||
- **Never an empty deck** — selection has a floor; if few pass, lower the bar and label them honestly
|
||||
("a few reaches").
|
||||
- **Don't double-count QScore** (already digests interview/roleplay) — §2 of the audit.
|
||||
- **LLM only on the top slice, cached** — the §3 "retrieve cheap, rerank expensive" discipline.
|
||||
- **Freshness is free here** — on-demand fetch means every shown listing is seconds old (§6.1).
|
||||
|
||||
---
|
||||
|
||||
## 7. Recommendation (smallest first step)
|
||||
|
||||
Build **Phase 0 + Phase 1 together** as `app/engine/rank.py` — a deterministic, coverage-aware,
|
||||
white-box ranker that consumes `user_context`, filters 90 → top ~25, and emits the real `match` block.
|
||||
No LLM, no ESCO service, no new infra; it ships *both* asks ("only recommended" + "aligned stats").
|
||||
Then add Phase 2 (LLM rerank/explain) once the deterministic core + the frontend `match`-contract are
|
||||
in. Phase 3 (learned) follows the live save/apply loop.
|
||||
|
||||
**Open scoping questions for the build:** (a) wire `user_context` now or rank on Configure prefs first?
|
||||
(b) how aggressive should default selection be (target deck size)? (c) include the GPT-5.4 rerank in v1
|
||||
or ship deterministic-only first? (d) seed-synonym map now vs minimal exact+fold to start?
|
||||
Reference in New Issue
Block a user