matchmaking-v2/research/docs/ENGINE_DESIGN.md

Matchmaking v2 (Scout) — Engine Design & Locked Plan

Canonical design doc. Captures the engine research, the locked decisions, and the open items. Written 2026-06-12. Revised 2026-06-16 with live-evidence findings. Supersedes the dead matchmaking service.

Companion docs (gathered evidence — read alongside):

• SIGNAL_AUDIT_V2.md — evidence-based signal inventory; supersedes §2/§5 below.
• AUTOAPPLY_STATE.md — validated browser-use auto-apply engine; supersedes/extends §6.
• DECOMMISSION_PLAN.md — how v2 slots in over the old matchmaking (port :8006).

2026-06-16 deltas in one line: the engine is richer mostly by wiring (interview/roleplay/ assessment are LIVE data, not "gaps"); a new high-trust signal (course watch-behavior) was found; auto-apply is now a validated board-agnostic vision agent (not just hosted-submit/ handoff); v2 reuses port :8006 (not :8011); Pathways is cut.

---

0. Why v2 exists

The old matchmaking service is being decommissioned. Diagnosis (verified June 11–12):

• Wrong shape: it was a nightly job aggregator — scraped ~13.8k jobs into its own Postgres and scored 21 users against the stale pile. 986,770 match-score rows for 21 users.
• Dead scheduling: not cron — a self-rescheduling RQ chain in Redis that went empty (likely an ElastiCache eviction) and silently stopped. Ingestion dead since May 27.
• Weak engine: ranked on the lowest-trust signals (self-declared prefs + claimed skills
  • an often-absent QScore that, at 30% weight, collapsed everyone to a neutral 60).

v2 is a fresh, purely on-demand service. No aggregation. Fetch live listings when the user searches, score only those.

---

1. Locked architecture (the shape)

• On-demand only. Per user search, fetch ~25–200 live listings from LinkedIn + Naukri + Indeed via Apify, score only those. No corpus, no nightly batch.
  • Sync fetch is viable: Apify run-sync-get-dataset-items (300s cap), cold start ~1.2s.
  • 3-board search ≈ $0.15–0.25 uncached; shared cache keyed role+location, TTL 1–6h.
  • Vendor picks — VALIDATED 2026-06-16 (live probe, poc/browser-apply/apify_probe.py):
    • Direct-ATS bovi/greenhouse-lever-ashby-job-scraper = the autonomous backbone — public JSON endpoints (no auth), returns clean ATS apply_url, 100% Tier-1 offsite in-sample, ~$1.50/1k. Company/preset-driven (not free-text). This is the new, highest-yield inbound.
    • LinkedIn harvestapi/linkedin-job-search — 60% offsite (applyMethod.companyApplyUrl), real ATS diversity. Needs a one-time approvePermissions (full-account-access actor).
    • Indeed misceres/indeed-scraper — ~35% external, but ~0% clean-ATS for India/ML (breadth, low autonomous yield).
    • Naukri → use muhammetakkurtt/naukri-job-scraper (11k users, no permission gate), NOT memo23 or the fishy infinity_and_beyond. Its applyRedirectUrl is the true external link, but ~93% point to bespoke small-company career pages, not clean ATS.
    • Measured combined yield: ~43% have an offsite link, ~14% immediately Tier-1 autonomous on a free-text search (more if you lead with direct-ATS). Full data → apify-inbound memory.
• Surface: ONE responsive web app, no extension, no native app download.
  • The cross-origin limitation is now solved a third way (validated): a server-side/cloud AI browser agent (browser-use) fills any board's form by vision — no extension, no ATS creds needed. See §6 + AUTOAPPLY_STATE.md. (Hosted-submit/prepared-handoff remain fallbacks.)
• Slots in over the OLD matchmaking — port :8006, not a new :8011 (revised; see DECOMMISSION_PLAN.md). v2 inherits the existing api.growqr.ai/matchmaking/* route + the frontend's matchmaking wiring. Still FastAPI in the course-service mold: Redis-Streams worker, registered in orchestrator AGENT_URLS, reached via the orchestrator WS, frontend talks to it through the existing useAgentSession(...) hook. The old matchmaking's own Postgres (corpus) is dropped — v2 is on-demand, no corpus.

---

1.1 Apify call contracts (verified live 2026-06-17)

Probed each actor's real input schema (poc/browser-apply/apify_probe.py, key loaded). Field-level contract + the ScoutPrefs→params coverage matrix live in ENGINE_INPUTS.md. Headlines:

• MVQ (Minimum Viable Query) = title + a resolvable location. No MVQ → do not call Apify (engine returns {needs:[...]}; frontend also gates "Find my matches"). Cache key = title+geo, TTL 1–6h.
• Free-text lane: harvestapi/linkedin-job-search (jobTitles[],locations[]), misceres/indeed-scraper (position,location,country), muhammetakkurtt/naukri-job-scraper (keyword,cities[],maxJobs*).
• Curated lane (separate): bovi/greenhouse-lever-ashby is company-list-driven; user inputs only filter within it (titleKeyword,locationKeyword,remoteOnly,keywords[]).
• Naukri is the richest target for India — uniquely exposes department, industry, ₹ salaryRange, numeric experience. Lead with it. Gotcha: Naukri industry/department/cities take numeric codes, not labels (labels are display-only).

1.2 Input coercion layer (the board adapters)

Nothing from the UI goes straight to an actor — a per-board adapter parses/maps/converts each ScoutPrefs field to that actor's exact schema. Code: engine/board_adapters/ (coerce.py + build_{naukri,linkedin,indeed,ats}_input(prefs)).

6 coercion dimensions: ① parse/split ("City · Country"→{city,country,remote}) · ② enum-map (label→board enum) · ③ unit-convert (₹→lakhs bucket; seniority⇄years) · ④ id-lookup (name→code: LinkedIn geoIds/industryIds, Naukri numeric codes) · ⑤ cardinality (multi-select→array where supported, else fan-out N searches + dedup — Indeed/bovi take single strings) · ⑥ presence (Any→omit, Remote→flag, Anywhere→geo-only).

Verified lookup maps (built into coerce.py):

Map	Examples
COUNTRY_ISO2 (Indeed)	India→IN, US→US, UK→GB
WORKMODE_*	Remote→remote, On-site→office, Any→omit
SENIORITY_YEARS (Naukri experience)	Fresher 0 · Junior 2 · Mid 5 · Senior 9 · Lead 13
SENIORITY_LINKEDIN (experienceLevel)	Junior→associate, Mid→mid-senior, Senior→director
COMP_NAUKRI_SALARY (salaryRange)	₹16–22L→15to25, ₹22–30L→25to50
ROLE_NAUKRI_DEPT (codes)	Product→10 · Engineering→5 · Design→15 · Data→3 · Sales→14 · Ops→2
INDUSTRY_NAUKRI (codes)	Fintech→114 · SaaS→110 · E-commerce→108 · EdTech→133 · HealthTech→131 · Consulting→126
CITY_NAUKRI (codes, 1→N)	Bengaluru→[97] · Mumbai→[134] · Delhi NCR→[6,73,220,350] · Hyderabad→[17] · Pune→[139]
INDUSTRY_LINKEDIN_ID	best-effort — TODO verify vs LinkedIn taxonomy; falls back to keyword fold

System-set (not user): maxJobs/maxItems (25–200 budget) · freshness (freshness=7/postedLimit=week/ recentWindowDays=30 — the liveness gate) · sortBy=date · fetchDetails/followApplyRedirects=true (real offsite apply URL) · Naukri country implicit IN.

Composite-keyword fallback for boards lacking department/industry (LinkedIn/Indeed/bovi): keyword = [seniorityPrefix] + title + [industryTerm] (e.g. Indeed position="Senior Payments PM fintech"), while Naukri stays clean (keyword="Payments PM" + structured department/industry/salaryRange).

Hard rule — degrade, don't break: any value that doesn't resolve to a board enum/code is never sent as an invalid enum. It falls back to the keyword fold or post-fetch filter, and is logged. A missing lookup loses precision, never the call.

Engine-side-only inputs (no board param on any actor — used for ranking/floors/exclusions/ordering): priorities, stretch, intent, availability, dealBreakers, sort. See ENGINE_INPUTS.md §6.

---

2. Signals — keep / discard (the input audit)

➜ SIGNAL_AUDIT_V2.md supersedes this section (evidence-based, from live-DB sampling 2026-06-16). Deltas: (a) course watch-behavior is a NEW high-trust booster — revealed preference ("completes React content → step toward frontend") beats stated; (b) interview/ roleplay/assessment are LIVE data, not gaps (§5) but are already digested into QScore, so v1 uses the QScore quotient as the competence proxy to avoid double-counting; (c) Pathways is CUT (feature and its questionnaire-as-signal — weak link: broken skill-gap, impractical roles). The trust⟂coverage discipline below is unchanged and is what keeps "more signals" from re-causing the v1 collapse.

Governing tension: trust ⟂ coverage. High-trust signals have low coverage; high-coverage signals have low trust. Design rule: a signal earns SCORE WEIGHT only if its coverage is high enough that absence isn't the norm; otherwise it is an ADDITIVE BOOSTER (bonus when present, never a penalty when absent). The old engine broke this rule — that's the v1 collapse bug.

KEEP

Tier A — backbone (high coverage; filter + light score)

• years_experience (date-derived, not the self-declared band)
• current_role / title (prefer social-branding LinkedIn-verified)
• preferred_locations + work mode (hard filter)
• target_roles / target_industries (filter, not score — it's want not is)
• skills (normalized — see §3.2)

Tier B — verified enrichers (medium coverage; real weight when present)

• verified title / company / industry (social-branding; industry only via BrightData path)
• degree / field_of_study / institution
• certifications
• assessment scores (passed / percentage — user-service, shipped Jun 11; earned, grows)
• QScore quotient vector (10-D; enricher/tiebreaker — not additive-only, coverage-penalized)

Tier C — premium boosters (low coverage; ADDITIVE ONLY, never penalize absence)

• interview 8-D rubric (~5–10% coverage)
• roleplay 6-D rubric (<3%)
• improvement velocity / coachability
• video on-camera score (<1%)
• (Doubles as an adoption flywheel: "do a mock interview → sharper matches.")

DISCARD (noise for matching)

• ATS overall_score ⚠️ measures resume quality/formatting, not job fit — the trap.
• parser confidence, resume_check signals (QA, not the person)
• follower/connection counts, post engagement (vanity)
• brand_score, pathway_archetype (rare/external/unpopulated)
• self-declared salary_range (zeros/guesses — use the listing's market salary)
• engagement streak/trends (engagement ≠ fit)
• names, avatar, email; cover-letter analysis & coaching narratives (for humans, not the scorer)
• institution_tier — discarded deliberately; sidesteps the worst India pedigree-bias vector.

---

3. THE ENGINE (the v2 redesign)

3.0 Core verdict (research-backed)

• Embedding/cosine similarity must NOT be the ranker [proven, 3-0]. It rewards topical word-overlap over qualification fit, can't enforce hard constraints (seniority, certs), and collapses on sparse profiles — exactly the v1 failure mode. Embedding = one input signal.
• Winner: a deterministic white-box utility score, fused with other rankers by WEIGHTED-AVERAGE RANK [proven: WAvg rank +31.9% nDCG@10; beats RRF +5.8% and every single ranker]. "Retrieve cheap, rerank expensive" still applies — the boards did retrieval; we spend compute reranking the 25–200 fetched listings.

3.1 The cascade (per search)

① NORMALIZE     skills → ESCO canonical; titles → canonical role
② HARD FILTER   location/work-mode, must-have certs, seniority floor   (deterministic gate —
                                                                         embeddings CAN'T do this)
③ WHITE-BOX     U = Σ_{f∈present} c_f·w_f·φ_f  /  Σ_{f∈present} c_f·w_f
   UTILITY      factors: skill_match, experience, location, salary, semantic_sim (ONE input),
                industry/company, + QScore-quotient / assessment boosters
④ FUSION        weighted-average RANK over: utility ‖ embedding-sim ‖ LLM soft-score
⑤ LLM RERANK    GPT-5.4 on top ~20: calibrated 0–1 fit vs a 6-level rubric;
   + EXPLAIN     ALSO logged as cold-start supervision; emits "why" + growth nudge
⑥ OUTPUT        ranked list + breakdown + explanation
   ↓
   LOG soft-scores + user save/apply → accrue labels → swap ③④ priors for a learned
   pairwise/listwise ranker (margin loss / CMMD — NOT pointwise MSE)

3.2 Skill matching (the core accuracy lever) [proven method]

Exact taxonomy lookup is a dead end (only 41/202 soft skills exact-match ESCO; real-world span-match F1 caps ~27%). The stack:

1. Synonym canonicalization to ESCO (k8s→Kubernetes, ML→Machine Learning) — seed from ESCO + v1 role synonyms + India tech terms.
2. Depth-2 knowledge-graph traversal of RELATED_TO edges for implied/transferable skills (Kubernetes → "container orchestration"). Powers STRETCH.
3. Contrastive embedding alignment to the taxonomy to beat the ~27% ceiling. Validate locally — the F1@5 0.72 result is on Chinese ads. v1 ships synonym+KG+cosine-to-ESCO; upgrade to a trained contrastive aligner later.
4. Required vs preferred weighting: required near-gating, preferred additive.

• Taxonomy = ESCO. EU-origin — sanity-check India role/skill coverage early; extend, don't swap.

3.3 Coverage-aware combination [our design — no validated formula exists]

Renormalize over present signals only, each weighted by confidence c_f (coverage × trust from the audit): U = Σ_{present} c_f·w_f·φ_f / Σ_{present} c_f·w_f. A missing signal redistributes its weight to what we know — never penalizes, never flattens. This makes the v1 "QScore absent → everyone 60" bug impossible by construction. Validate c_f offline (confirm it doesn't over-reward sparse profiles).

3.4 Cold-start weighting (no labels at launch) [proven]

• Day 1: expert-prior weights — starting point skill .35 / experience .25 / location .15 / salary .10 / semantic .10 / company .05. Priors to calibrate, not optima.
• From day 1, in parallel: GPT-5.4 as teacher — calibrated 0–1 soft scores on a 6-level rubric, logged on every match. Synthesizes supervision (study: discriminative specificity 3.5× — 0.058 → 0.208) instead of waiting for outcomes (<0.05% dense).
• As save/apply accrues: train a pairwise/listwise ranker + contrastive/paraphrase augmentation; shift fusion weight off the priors.

3.5 FIT vs STRETCH [our design — research left open]

• FIT: skill-gap ≈ 0; score current-state overlap; mild over-qualification penalty.
• STRETCH: target = one level up / adjacent; score by reachability = fraction of the job's required skills the user has OR is within depth-2 KG hops of. "Reach but reachable" = small, bridgeable gap → this is the prototype's growth nudge.
• Maps onto the Scout intent picker: current-state → "more like this"; aspiration → "step up / new".

---

4. Evaluation & model roles — no human in the loop

• Runtime is fully autonomous. No human ever reviews, approves, or scores a user's matches.
• Eval grading is automated and cross-model to avoid self-grading inflation:

  Role	Model
  Match generation, cold-start soft-score teacher, explanations/growth nudges	GPT-5.4
  Independent accuracy grading (LLM-as-judge)	Opus 4.8

  • Cross-family grading kills self-preference bias (no model marks its own homework).
  • Residual generic LLM-judge bias (position/verbosity) → prompt hygiene: randomize order, pairwise, strip length cues.
  • Un-gameable ground truth = real save/apply rate once live. Opus is the pre-launch proxy.
• Metrics: nDCG@10, MRR, P@5, calibration MAE on a synthetic + dogfood gold set (~50–100 India-representative pairs; no hired annotators). Online: interleaving + A/B once live.
• Note: Opus-as-judge is a deliberate, fine exception to "always GPT-5.4" (it's Anthropic, not an OpenAI call). Eval pipeline needs an Anthropic API key alongside OpenAI.

---

5. Transport readiness (do the pipes exist?)

Backbone exists: orchestrator context_resolver builds user_context from resume-builder + social-branding /api/state/{clerk_id}; matchmaking calls qscore-service directly. REVISED 2026-06-16 — the "gaps" were assumptions; the data is LIVE. Verified by sampling the prod DBs over SSH (see SIGNAL_AUDIT_V2.md §3):

• ✅ READY: all Tier A + QScore quotient vector.
• ✅ LIVE (was "partial"): verified LinkedIn title/company/industry — social-branding does a live LinkedIn fetch; LinkedIn education; certifications.
• ✅ LIVE (was "❌ GAP"): assessment scores (user-service), interview rubrics (growqr_interview, 210 reviews — 8-D rubric + video presence + confidence-ratio), roleplay rubrics (growqr_roleplay, Gemini video analysis), improvement velocity (trend_data/historical_comparison). None are a build — they're a query. Caveat: filter degenerate sessions (no-show/low-talk poison scores).
• Transport already exists: interview & roleplay services already receive résumé/LinkedIn context via the orchestrator context_resolver. matchmaking-v2 reuses that exact pattern — no new transport to build. v1 wiring = consume the assembled user_context + one extra read of course watch-behavior (growqr_course).
• Still: QScore is the competence proxy for v1 (it already consumes interview/roleplay); read raw rubrics later only if they add lift beyond the aggregate.

---

6. Auto-apply (the heart) — "one tap at the end"

➜ AUTOAPPLY_STATE.md supersedes/extends this section (validated POC, 2026-06-16). Headline update: the original "full auto-submit = NO-GO" is REVISED — a server-side AI browser agent (browser-use + gpt-4.1 vision) is now a validated, board-agnostic "master key":

• Proven: a real application was submitted to OpenAI on AshbyHQ (on-screen success confirmed); the generic adapter then filled Greenhouse forms it had never seen (4/5) and India-native ATS (Zoho Recruit, Keka) with no tuning, honestly bouncing login-walled / no-form pages.
• Architecture: one generic engine + a thin adapter registry (Ashby adapter carries earned quirks; generic adapter handles the long tail by vision). Verification-first: screenshot-as- ground-truth + a hard accuracy gate (never trust the agent's self-report).
• Tiered reality: autonomous for public ATS (Ashby/Greenhouse/Lever) + India-native ATS; login-gated boards (LinkedIn Easy Apply, Naukri dashboard, Workday/Oracle) stay human/Cloud-assisted.
• Economics (measured): gpt-4.1 vision-off ~$0.13/app (clean ATS), ~$0.21–0.27 (India ATS), ~3–6 min/app; gpt-4o-mini is a false economy (failed + not cheaper). Latency needs parallelism.
• Anti-bot is the make-or-break risk: Ashby spam-flagged the fast LinkedIn-routed submit; the fix was direct-to-ATS + human-like pacing + verified upload. reCAPTCHA/hCaptcha remain a wall (simple text CAPTCHAs are vision-solvable). New POC code: poc/browser-apply/ (the old poc/auto-apply/ mock is retired).

By the time the user taps, all the work is done (found, tailored, filled). The tap ends an application, it doesn't start one. In the no-install web app, "one tap" resolves two ways:

• 🛰️ Hosted submit — backend POSTs the application to an ATS API (Greenhouse / Lever / Ashby). True one-tap-done, any device, zero install. Gated by employer creds → coverage = the integrations we provision; grows over time.
• 🔗 Prepared handoff — one tap opens the board's own page (LinkedIn/Naukri) with résumé + cover letter + answers staged; user taps submit there. Zero install, any device, zero ban risk.
• Phase 4 full auto-submit: NO-GO for now (unattended submit ~46% reliable + ToS-exposed; the survivor pattern is user-device extension autofill + human submit). Opt-in only when revisited.

POC validates the hosted-submit path end-to-end → see poc/auto-apply/. Hard rule: never fire test applications at real employers' live postings. POC-0 = mock ATS; POC-1 = our own ATS sandbox/test job.

6.1 LISTING FRESHNESS / LIVENESS — first-class gate (learned from the POC, June 12)

The personalized demo tapped a real LinkedIn posting that was already "no longer accepting applications." Then we empirically checked ~7 recent AI-Architect postings: essentially all were dead (closed / HTTP 404 / redirect-to-search = LinkedIn's expired-job behavior). Findings:

• A perfect match for a dead job is worse than no match — it's actively undelightful.
• Permalinks rot in days. Never hand a user a cached posting URL — it's probably dead.
• Tap-time server-side liveness re-check is unreliable: LinkedIn serves bot-walled / guest content (our httpx check false-returned "open" for a job we knew was closed), and expired URLs 302→search or 404. You cannot trust a scrape to tell you a posting is open. Requirements for v2:

1. Freshness at the DATA layer: the on-demand Apify fetch returns posting date / isExpired / applicant count — filter & rank by recency, drop/flag stale before presenting. The whole point of on-demand: every shown listing was fetched seconds ago, so it's fresh by construction.
2. Honest age signals in the UI ("posted 3d ago", "closes soon") — derive from posting age (no scraper returns a true deadline field).
3. Apply handoff targets a live view, never a stored permalink: the freshly-fetched listing or the board's live search for the role (always current). Recover gracefully, never dead-end.

---

7. Proven vs. validate-locally (honesty ledger)

Locked & proven (adopt method)	Our design — must validate locally
Hybrid fusion > embedding-only ranker	Exact fusion weights (60/25/15 was tuned on a tiny set)
Weighted-avg rank > RRF	The coverage-renormalization formula + c_f values
Synonym + KG-expand + contrastive skills > exact match	FIT/STRETCH reachability scoring
LLM-as-teacher for cold-start; pairwise > pointwise	India pedigree/fairness mitigations
Embedding-only collapses on sparse profiles	ESCO coverage for India roles

Headline accuracy numbers do NOT transfer (single-paper, small-benchmark, domain-mismatched — gig matching, Chinese job ads, offline corpora). Adopt the patterns, re-tune on GrowQR data.

8. Open items

Resolved 2026-06-16:

• ✅ Apify inbound validated (Naukri → muhammetakkurtt; direct-ATS = autonomous backbone; offsite- yield measured). ✅ Auto-apply POC validated end-to-end (master key, real submission, India ATS).
• ✅ Signal audit refreshed against live evidence (SIGNAL_AUDIT_V2.md). ✅ Agent capability pass (interview/roleplay strong, Pathways cut). ✅ Slot-in/decommission plan (DECOMMISSION_PLAN.md).

Still open:

• Build the engine itself — wire the v2 input set (Backbone + verified LinkedIn + QScore + course watch-behavior) via the reused context_resolver; implement the §3 cascade.
• Latency benchmark of the actors from ap-south-1; parallelize apply (3–6 min/app is too slow serial).
• Submit-time anti-bot lane (the make-or-break): Cloud stealth / CAPTCHA / human-final-tap.
• Build the ~50–100 pair India gold set; validate Opus-judge agreement. ESCO India coverage check.
• Label volume for the learned ranker (old corpus was 21 users — accrue via the live save/apply loop).

UI — in progress (2026-06-17, branch job-ui-final):

• ✅ Scout UI built in-place in dashboard-ui/scout (not a separate index.html POC) around the locked feature set; screen-by-screen decisions + as-shipped divergences tracked in SCOUT_UI_SPEC.md ("BUILD PROGRESS" section). Re-verified honest against this doc — all cuts/improvements, no unbacked claims. Notable: cascading Country→City location (feeds Tier-A preferred_locations + ported location-fit); a no-resume "Any/open" state that is the UI face of §3.3 (sparse profile → coverage renormalizes, never fabricates); loader + scope made honest (≤120, on-demand framing).
• 🔧 UI still waiting on the engine for: apply-result statuses (proof screenshots + auto/failed→manual split), per-job resume-tailoring spinner, and the real signal/match-breakdown wiring behind the cards.

Key sources (engine research, 2026)

• Synapse (arXiv 2604.02539) — hybrid fusion > embedding-only; WAvg rank fusion +31.9% nDCG@10.
• JobMatchAI (arXiv 2603.14558) — white-box utility weights; ESCO synonym + depth-2 KG expansion.
• ConFit / ConFit v2 (arXiv 2401.16349) — contrastive + paraphrase augmentation; label sparsity <0.05%.
• Person-Job Fit distillation (arXiv 2601.10321) — LLM-as-teacher soft scores; specificity 0.058→0.208; CMMD.
• JobSkape (arXiv 2402.03242) — ESCO span-match F1 ~26–27% ceiling.
• CareerBERT (arXiv 2503.02056) — embedding degradation on short/sparse profiles (coverage collapse).
• Springer 12651-025-00409-x — only 41/202 soft skills exact-match ESCO.