Root cause: the VPS ran two Compose projects from the same file — api under
-p matchmaking-v2-staging, postgres/redis under -p matchmaking-v2 — creating
two disjoint <project>_default bridges. Every dependency hostname (postgres,
redis) was NXDOMAIN from the api container, so feed store init, pool_load,
save_feed, and RedisStreamWorker all failed with [Errno -2].
Fix: standardize on a single project (matchmaking-v2) for all three services
with an external shared bridge (matchmaking-v2_default) and inline DB/Redis
host pinning. The deploy script (sync-staging.sh) now brings up all three
services, pre-creates the external network, and has a public_health_url case.
Apify: moved the token from query params (4 sites) to an Authorization header
so no request URL can leak it. All raise_for_status() paths now raise
token-free _ApifyHTTPError. The live 403 (platform-feature-disabled: monthly
usage cap exceeded) is an external blocker — no code fix.
Tests: 15 focused regression tests covering network topology, volume config,
deploy-script alignment, and Apify token redaction (with call-counters).
Grow event emitter fixes (wrong route / silent failure cannot recur):
- URL: /events/ingest -> /events/ingest/service (canonical service ingest path)
- Add x-growqr-source: matchmaking-service header
- Call resp.raise_for_status() so a 401 is caught and logged, not silently
swallowed (401 is now observable; a wrong route surfaces immediately)
- Action-specific dedupe keys so retries collapse but later distinct user
actions are NOT dropped:
matches.generated -> matchmaking:generated:{user}:{sig}
feed.viewed -> matchmaking:feed:{user}:{updated_at|none}
match.viewed -> matchmaking:viewed:{user}:{opp_id}
match.applied -> matchmaking:applied:{user}:{opp_id} (already present)
All four emit handlers fire the event AFTER the persistent mutation
(set_opportunity_state / save_feed), so no event leaks when a mutation fails.
Tests (tests/test_grow_events.py, 12 cases, red->green TDD):
- URL normalizes to /events/ingest/service (with/without trailing slash)
- x-growqr-source header present
- 401 response raises_for_status -> logged warning; does not escape _post
- emit-after-mutation ordering; no emit when mutation raises
- dedupe keys stable for same evidence, distinct for different evidence
across generated/feed/viewed/applied event types
The LLM (Opus score stage / Haiku cards stage) can return None or empty
content, causing _salvage_objects to crash with
'NoneType' object has no attribute 'find' — which killed the curation
pipeline and returned zero matches to the user.
- Guard _salvage_objects and _salvage_kept against None/empty input
- Add explicit empty-content checks in both stages with logging
- Log content[:200] on unparseable score stage for QA diagnostics
RCA of a prod "Naukri returned 0" on an AI-Architect search: NOT a cursor issue (cursors removed;
naukri-feed is a date feed with no page param), NOT a city-name or keyword bug — the EXACT same input
flaked 0 then returned 15 on retry. The scrape-based actors (Naukri-feed, curious_coder LinkedIn)
intermittently return 0 under the site's anti-scrape rate-limiting, then recover.
Fix: _fetch_board retries a board ONCE (fresh) when it returns 0. One guard covers BOTH the Naukri and
LinkedIn intermittency (cleaner than per-board fallbacks). A genuinely-empty query just returns 0 again
(cheap). 67 tests pass (3 new: recovers transient 0, no retry on success, no infinite loop).
All sibling services on the box manage schema with Alembic (async env, `alembic upgrade head` in the
compose command). matchmaking-v2 was the outlier (Base.metadata.create_all + no migration for the new
user_feed.exhausted_at column). Bring it onto the pattern:
- alembic.ini + alembic/env.py (async, URL from settings.DATABASE_URL, target = Base.metadata) +
script.py.mako — copied/adapted from user-service.
- versions/0001_baseline: the current prod schema (user_feed WITHOUT exhausted_at + opportunity_state).
- versions/0002_pool_and_exhausted: CREATE user_job_pool + ADD user_feed.exhausted_at (additive).
- Dockerfile: COPY alembic.ini + alembic/. requirements: alembic>=1.13.0.
Verified both paths: fresh DB → upgrade head builds all 3 tables; prod-like (existing tables+data) →
stamp 0001 → upgrade runs ONLY 0002, existing rows survive. 64 tests pass.
PROD ADOPTION (one-time): `alembic stamp 0001_baseline` on RDS before the first deploy, then the
compose's `alembic upgrade head` applies 0002.
A search-quality + cost epic. Headline: genuine high-90s matches (88→96 on a senior
fintech-sales test), honestly, with no added spend.
SCORING — honest, computed, calibrated:
- rubric.py (NEW): the published rubric — 6 weighted dimensions + anchors. The overall is COMPUTED
(rubric.aggregate), never model-emitted. A genuinely-aligned match arithmetically reaches the 90s.
- curate.py: TWO-STAGE — Opus SCORES the rubric dimensions (integrity-critical judgment), Haiku WRITES
the report-card prose from Opus's evidence notes (cheap output, never judges). ~25% cheaper Opus +
tighter calibration + better latency. Robust salvage parse; growth parse tolerant.
- rubric.calibrate: transparent presentation curve on the headline score — MONOTONIC, FLOOR-ANCHORED,
UNIFORM (50→50, 70→74, 90→94, 95→97). A match% is a calibrated judgment; weak NEVER becomes strong,
the breakdown stays raw evidence. Gated by CALIBRATION_ENABLED/GAMMA.
- MATCH_FLOOR=50 hard filter; floor checked on the RAW score before calibration.
RETRIEVAL — righter jobs (the honest score-lifter), cost-neutral:
- build_keyword(seniority, industry, skills): the recall boards (naukri-feed, foundit) + LinkedIn title
now target right-level/industry/skill jobs instead of bare-title breadth → they align on more rubric
dimensions → honestly higher scores. Verified live: no over-narrowing (138 jobs fetched, unchanged).
- Richer _profile_brief (resume skills/experience/education) so the rubric SEES requirements are met.
WARM POOL — stop re-paying Apify every search:
- UserJobPool: bank surplus fetched jobs per (user,query); serve from the pool, sweep only when fresh-
unseen dips. Gate reorders at 80 / hard-floors at 70; background refill. (Saves Apify, not Opus.)
ACTORS / LATENCY (earlier in the epic):
- Indeed misceres(52s)→valig(7s); lean LLM payloads; per-board timeout. 48 tests pass.
The measurement showed the USER-FACING Opus score is already well-calibrated — the real bug was curate
FAILING and dropping to the bad white-box fallback:
- curate.py: _salvage_kept() brace-scans + parses each kept object independently, so one bad char or a
max-tokens truncation no longer drops the whole shortlist. max_tokens 4000 → 8000 (28 sifted jobs ×
full report cards overflowed). This fixed the sales case (was failing → fallback).
- curate.py: calibration nudge in the prompt — a genuinely strong current-state match is a real low-80s,
partial/stretch 60s-70s; don't under-sell strong matches into the 60s (without inflating weak ones).
- normalize.py + search.py: _seniority_from_title() infers seniority from the title (conservative) when a
board omits it, applied centrally in the sweep → the experience factor + Opus brief aren't blind.
Verified (live Opus on the regression cases): tech MAE 9.8→4.5, sales FAILED→MAE 5.3 (kept 13),
ops 4.2→3.3 — user-facing scores now MAE 3-5 with zero curate failures. 31 tests pass.
- rank.py: get_weights(profile) — when the profile reads non-tech (sales/finance/HR/ops/support/…),
tilt the weights to role(0.16)/industry(0.12)/semantic(0.12), drop skill(0.18). Tech weights
unchanged. (The competency-skill-backfill was tried + REVERTED — uniform per-role competencies
added no discrimination and Phase 2's real-cosine f_semantic already supplies the non-tech signal.)
- curate.py: non-tech instruction — role/responsibility fit is the spine; a missing skill list is NOT
a negative; a strong role+industry+responsibility+seniority match is a genuine ~80, not ~65.
- test_contracts.py: allowlist the new keyword-handled non-tech vocab (active boards keyword-fold;
numeric codes deferred since the active feed/recall paths don't use them).
Regression (120 pairs): MAE 18.4 → 16.1 (all from non-tech: sales 17.1→13.0, ops 16.3→11.7), recall
holds 0.92, tech unchanged. 31 tests pass.
The sift was feeding Opus a magnitude-blind, under-trusted, truncated shortlist — only 68% of the
genuinely-best jobs reached the curator. Fixes:
- embed.py: embed the FULL description (was desc[:600]) + role_category + industry; richer profile
text (current_role + experience summary + seniority). The best semantic signal, fed real content.
- sift.py: magnitude-preserving min-max fusion (was rank-position, which flattened cosine 0.95 vs
0.72) and embedding-LED weights (W_VIBE 0.6 / W_WHITEBOX 0.4). Embed runs FIRST so the white-box
f_semantic reuses the real cosine (threaded as job["_vibe_cosine"]) instead of token overlap.
- config.py: SIFT_TOP_K 18 → 28 (the gate was cutting good jobs before Opus).
- curate.py: richer Opus briefs — full responsibilities (was desc[:500]) + role_category/industry/
seniority/pay, so even the shortlist is fully described.
Regression (120 pairs): sift membership recall 0.68 → 0.92, score MAE 24.7 → 18.4. 31 tests pass.
Phase 0 — measurement gate:
- tests/test_contracts.py: BOARDS↔NORMALIZERS alignment, funnel-shape guards, overlap() None
contract, and the frontend↔backend EXACT-STRING vocab check (scout.ts options must resolve in
coerce.py, else silent keyword-fold).
- tests/run_regression.py + fixtures/regression_set.json (+ _seed_regression.py): a frozen 120-pair
set (tech + non-tech) Opus-judged for recruiter-fit targets. Two metrics: score MAE and SIFT TOP-K
MEMBERSHIP RECALL (do the best jobs reach Opus?). Baseline: MAE 24.7, recall 0.68 — i.e. ~32% of
the genuinely-best jobs are cut before the curator ever sees them (worse for non-tech).
Phase 1 — coverage (India non-tech):
- Enable Indeed (misceres) + enrich indeed_to_scoutjob with details.description + location_mode
(jobType is employment type, not skills → required_skills now []).
- Add WorkIndia (shahidirfan) — India blue/grey-collar non-tech: build_workindia_input +
workindia_to_scoutjob (real skills + description; per-job apply URL from job_id since source_url
is generic and would dedup-collapse the deck).
- Drop Wellfound from the stack (US-startup-heavy + 400s); kept registered-but-off.
- BOARDS_ENABLED = naukri,foundit,linkedin,indeed,workindia. Pool ~90 → ~140 jobs/search.
Regression scores unchanged (engine scoring untouched). 31 tests pass.
Emit a rotating progress message every ~5s while the Apify scrape + Opus curation run,
so the long (30-40s) silent gap can't idle the response relay and improves the wait UX.
- OpportunityState model + repo: persist applied / dismissed / restored status and the
generated resume-builder doc ids (never re-pay to re-tailor). annotate_and_filter drops
dismissed + annotates applied/docs on read; handlers for dismiss/restore/mark_applied/
save_apply_docs.
- Unique decks per run (spend only for NEW jobs, no cache, no re-fetch+dedup band-aid):
per-(user,query) search cursor → LinkedIn page++ and Naukri incremental+stateKey (async
run path in apify_client, since the actor's crawl exceeds the run-sync window); seen-net
excludes already-shown ids (covers Foundit, which can't paginate).
- normalize: every board offsite-first (apply_url prefers the employer/ATS redirect over the
board listing) + offsite_apply flag; real company logos (_logo_url across Naukri logoPath /
LinkedIn company.logo / etc.).
- suggest.py: stage-aware engine (broad -> narrow -> role) — LLM (claude-haiku-4-5)
generates bubbles per stage from the picks so far. Profile-grounded, generalizes
to any field (English professor -> Senior Instructional Designer, etc.); None on
no input -> frontend keeps seeded bubbles
- handle_suggest_bubbles action (run off the event loop) + SUGGEST_MODEL config
- bundled: LLM work (sift/curate/suggest) now via asyncio.to_thread so the Redis
response can publish (fixes the loader hang); get_feed handler -> get_scout_feed
(avoids the course-service action collision)
- A1: fire the hard filter (Apify is precise); embedding vibe-ranker (embed.py)
blends with the floor-free white-box (rank.py) to sift ~90 -> top 18
- A2: Opus curator (curate.py) reads the 18 -> honest <=count + Gemini-voiced
report cards; graceful fallback to the white-box + templated cards
- LLM via opencode.ai/zen gateway (llm.py): Opus chat + direct-OpenAI embeddings
- Run LLM work off the event loop (asyncio.to_thread) so the Redis response publishes
- C1: dedicated Postgres (app/db/) persists the per-user feed; get_scout_feed
replays it so matches survive navigation/refresh
- match contract + report-card fields (schema.py); skills.py; tests/
