commit 89ca9ad647d30c1e0bb8921738c84d6b9994fee2 Author: raulgupta Date: Thu Jun 18 19:22:03 2026 +0530 Initial commit: matchmaking-v2 on-demand multi-board job-search agent On-demand Scout service (replaces the nightly aggregator): - FastAPI agent-mesh service; card name "matchmaking-service" (HTTP /a2a/tasks + Redis-stream worker matching the sibling-service pattern) - run_search: parallel multi-board sweep (Naukri/blackfalcondata + Foundit + LinkedIn), city-filtered at the board, cheapest-per-result first - per-board adapters + normalizers -> ScoutJob with rich read-only details and offsite apply links; recall mode + MVQ guard - result cache for dev replay ($0); per-board cost knobs; retry-on-5xx - research/: engine design, board cost economics, India-actor shortlist, POC diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..5fd0966 --- /dev/null +++ b/.env.example @@ -0,0 +1,23 @@ +# matchmaking-v2 — copy to .env (git-ignored). On-demand: no corpus DB required. +ENV=development +LOG_LEVEL=INFO +API_PORT=8000 +AGENT_URL=http://localhost:8006 + +# A2A bearer the orchestrator presents (match the mesh's shared key in prod) +A2A_ALLOWED_KEYS=dev-a2a-key + +# Orchestrator comms (optional in dev — worker no-ops if unset/unreachable) +ORCHESTRATOR_REDIS_URL= +REDIS_URL=redis://localhost:6379/0 + +# Inputs / engine (wired in later slices) +APIFY_TOKEN= +OPENAI_API_KEY= + +# QScore (consumed) +QSCORE_BASE_URL=http://localhost:8004 +QSCORE_AUTH_TOKEN= + +# Feedback/label store (NOT the old corpus) — optional until the learning loop +DATABASE_URL= diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..03bf39b --- /dev/null +++ b/.gitignore @@ -0,0 +1,31 @@ +# ── Secrets / environment (NEVER commit) ── +.env +.env.* +*.env +!.env.example + +# ── Python ── +.venv/ +venv/ +env/ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.mypy_cache/ +.ruff_cache/ + +# ── Local dev result cache (saved Apify job sets) ── +.apify_cache/ + +# ── Tooling / local state ── +.claude/ +research/poc/**/.env +research/poc/**/node_modules/ +research/poc/**/__pycache__/ + +# ── OS / editor ── +.DS_Store +*.swp +.idea/ +.vscode/ diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..3ab2cbb --- /dev/null +++ b/Dockerfile @@ -0,0 +1,13 @@ +FROM python:3.12-slim + +WORKDIR /app + +ENV PYTHONUNBUFFERED=1 PYTHONDONTWRITEBYTECODE=1 + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY app ./app + +EXPOSE 8000 +CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"] diff --git a/README.md b/README.md new file mode 100644 index 0000000..d03426b --- /dev/null +++ b/README.md @@ -0,0 +1,46 @@ +# matchmaking-v2 (Scout) + +Fresh, **on-demand** matchmaking service replacing the dead nightly aggregator. Same agent-mesh +mold as the other GrowQR services (FastAPI · a2a card · /a2a/tasks · orchestrator-routed). Not a +drop-in — it keeps the existing 4 skills working and **adds new actions from scratch**. + +## Layout +``` +app/ + main.py FastAPI app (card + /a2a/tasks + /api/v1/health), lifespan worker + config.py lean settings (no corpus DB, no scrape schedule) + a2a/ card.py (discovery), auth.py (bearer), tasks.py (orchestrator entry) + agent/session.py Session: on_session_start / on_user_action dispatch ← the brain + adk/worker.py Redis-Streams worker (graceful no-op without Redis) + api/v1/health.py /api/v1/health + engine/ + board_adapters/ ScoutPrefs → Apify actor inputs (Naukri-first, verified maps) + ... the §3 cascade (normalize→filter→utility→fusion→rerank) lands here + contracts/ Pydantic contracts (user_context, transport, …) — added per slice +research/ docs/ (ENGINE_DESIGN, SIGNAL_AUDIT_V2, ENGINE_INPUTS, …) + poc/ (auto-apply) +``` + +## Contract (how it connects) +Frontend `useAgentSession` (page `"job-matching"`) → orchestrator (routes by card `name` += `matchmaking-service`) → `POST /a2a/tasks {action, params, user_context}` → `Session` pushes +`agent_data{action,data}` → orchestrator → frontend `latestData[action]`. + +**Skills:** existing `get_feed · sync_preferences · record_feedback · get_opportunity_detail`; +new `run_search · tailor_resume · submit_application · get_apply_proof` (stubbed). Each new action +needs: card skill (here) + orchestrator action-map entry + frontend `sendAction` wiring. + +## Run (local) +```bash +pip install -r requirements.txt +uvicorn app.main:app --reload --port 8006 +# card: GET http://localhost:8006/.well-known/agent-card.json +# health: GET http://localhost:8006/api/v1/health +# action: POST http://localhost:8006/a2a/tasks (Bearer dev-a2a-key) +``` + +## Build order (full-stack slices) +0. ✅ scaffold (this) — bootable skeleton, contract wired, handlers stubbed. +1. on-demand `run_search` — board_adapters → Apify → engine cascade → ranked feed (+ frontend wire). +2. feedback labels + `record_feedback`. 3. `tailor_resume`. 4. `submit_application` + `get_apply_proof`. +5. cut over from old `:8006`, decommission corpus. +``` diff --git a/app/__init__.py b/app/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/a2a/__init__.py b/app/a2a/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/a2a/auth.py b/app/a2a/auth.py new file mode 100644 index 0000000..9ea47fb --- /dev/null +++ b/app/a2a/auth.py @@ -0,0 +1,15 @@ +"""A2A (agent-to-agent) bearer auth — same scheme as the rest of the mesh.""" +from fastapi import Request, HTTPException + +from app.config import get_settings + + +async def verify_a2a_key(request: Request) -> str: + auth = request.headers.get("authorization", "") + if not auth.startswith("Bearer "): + raise HTTPException(status_code=401, detail="Missing A2A auth token") + token = auth[7:].strip() + allowed = [k.strip() for k in get_settings().A2A_ALLOWED_KEYS.split(",")] + if token not in allowed: + raise HTTPException(status_code=403, detail="Invalid A2A auth token") + return token diff --git a/app/a2a/card.py b/app/a2a/card.py new file mode 100644 index 0000000..7505a5e --- /dev/null +++ b/app/a2a/card.py @@ -0,0 +1,35 @@ +"""Agent card — how the orchestrator discovers this service (it routes by `name`).""" +from fastapi import APIRouter + +from app.config import get_settings + +router = APIRouter() + + +def agent_card() -> dict: + return { + "name": "matchmaking-service", # keep — orchestrator page/action maps target this name + "description": "On-demand opportunity matching: live board search → white-box ranking → auto-apply with proof.", + "url": get_settings().AGENT_URL, + "version": "2.0.0", + "capabilities": {"streaming": False, "pushNotifications": False}, + "skills": [ + # ── existing contract (keeps the current orchestrator routing working) ── + {"id": "get_feed", "name": "Personalized Feed", "description": "Return the ranked opportunity feed for a seeker."}, + {"id": "sync_preferences", "name": "Sync Preferences", "description": "Persist the seeker's search preferences."}, + {"id": "record_feedback", "name": "Record Feedback", "description": "Record SAVE / DISMISS / APPLY feedback (training labels)."}, + {"id": "get_opportunity_detail", "name": "Opportunity Detail", "description": "Opportunity detail with the match-score breakdown."}, + # ── NEW in v2 (stubbed; each needs an orchestrator action-map entry + frontend wiring) ── + {"id": "run_search", "name": "On-demand Search", "description": "Fetch live listings (Apify) + rank against the seeker now."}, + {"id": "tailor_resume", "name": "Tailor Resume", "description": "Tailor the resume to a chosen role (resume-builder)."}, + {"id": "submit_application", "name": "Submit Application", "description": "Auto-apply via the master-key engine (tier-aware)."}, + {"id": "get_apply_proof", "name": "Apply Proof", "description": "Return captured confirmation screenshots for submitted applications."}, + ], + "authentication": {"schemes": ["bearer"]}, + } + + +@router.get("/.well-known/agent.json") +@router.get("/.well-known/agent-card.json") +async def get_agent_card(): + return agent_card() diff --git a/app/a2a/tasks.py b/app/a2a/tasks.py new file mode 100644 index 0000000..d6532bc --- /dev/null +++ b/app/a2a/tasks.py @@ -0,0 +1,60 @@ +"""A2A task endpoint — the single HTTP entry the orchestrator calls per action. + +Collects the messages a Session pushes and returns them; the orchestrator forwards +each to the frontend WebSocket. (Same proven adapter as the rest of the mesh.) +""" +import logging +from uuid import uuid4 + +from fastapi import APIRouter, Depends +from pydantic import BaseModel, Field + +from app.a2a.auth import verify_a2a_key +from app.agent.session import MatchmakingAgentSession + +logger = logging.getLogger(__name__) +router = APIRouter() + + +class TaskRequest(BaseModel): + task_id: str = Field(default_factory=lambda: str(uuid4())) + action: str = "" + params: dict = Field(default_factory=dict) + user_id: str = "" + user_context: dict | None = None + session_start: bool = False + + +class TaskCollector: + """WebSocket-shaped sink that captures pushed messages into a list.""" + + def __init__(self): + self.messages: list[dict] = [] + + async def accept(self): + pass + + async def send_json(self, data: dict): + self.messages.append(data) + + +@router.post("/a2a/tasks") +async def handle_a2a_task(req: TaskRequest, _token: str = Depends(verify_a2a_key)): + collector = TaskCollector() + session = MatchmakingAgentSession(websocket=collector, user_id=req.user_id) + + params = {**req.params} + if req.user_context: + params["user_context"] = req.user_context + + try: + if req.session_start: + await session.on_session_start(params) + elif req.action: + await session.on_user_action(req.action, params) + else: + return {"task_id": req.task_id, "status": "error", "messages": [], "error": "No action or session_start"} + return {"task_id": req.task_id, "status": "completed", "messages": collector.messages} + except Exception as e: # noqa: BLE001 + logger.error("A2A task %s failed: %s", req.task_id, e, exc_info=True) + return {"task_id": req.task_id, "status": "error", "messages": collector.messages, "error": str(e)} diff --git a/app/adk/__init__.py b/app/adk/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/adk/emitter.py b/app/adk/emitter.py new file mode 100644 index 0000000..18e5736 --- /dev/null +++ b/app/adk/emitter.py @@ -0,0 +1,34 @@ +"""EventEmitter — async queue adapter (mirrors the established agent-mesh pattern). + +Agent session handlers call ``self.ws.send_json(data)`` expecting a WebSocket-like +interface. This adapter puts each message into an ``asyncio.Queue`` so the Redis Stream +worker can consume them one by one and publish to the per-task response channel. +""" + +import asyncio + + +class EventEmitter: + """WebSocket-compatible adapter backed by an asyncio.Queue.""" + + def __init__(self): + self.queue: asyncio.Queue = asyncio.Queue() + self._done = False + + async def accept(self): + pass + + async def send_json(self, data: dict): + """Called by session handlers via ``self.push()``.""" + await self.queue.put(data) + + async def receive_json(self): + raise StopIteration("EventEmitter does not support receive") + + def query_params(self): + return {} + + def mark_done(self): + """Signal that the session handler has finished.""" + self._done = True + self.queue.put_nowait(None) diff --git a/app/adk/worker.py b/app/adk/worker.py new file mode 100644 index 0000000..4ea43a4 --- /dev/null +++ b/app/adk/worker.py @@ -0,0 +1,133 @@ +"""Redis Stream worker for matchmaking-service — the established agent-mesh pattern. + +Consumes tasks from the ``tasks:matchmaking-service`` Redis Stream (consumer group), +executes each via MatchmakingAgentSession, and streams response messages to the +per-task pub/sub channel ``responses:{task_id}``, ending with a ``__task_complete__`` +sentinel. Mirrors the sibling services (../matchmaking, interview-service, pathways …). +""" + +from __future__ import annotations + +import asyncio +import json +import logging + +from app.adk.emitter import EventEmitter +from app.agent.session import MatchmakingAgentSession + +logger = logging.getLogger(__name__) + +AGENT_NAME = "matchmaking-service" + + +class RedisStreamWorker: + """Consumes tasks from a Redis Stream, executes via the session handler.""" + + def __init__(self, redis_url: str | None): + self.redis_url = redis_url + self.stream = f"tasks:{AGENT_NAME}" + self.group = f"{AGENT_NAME}_workers" + self.consumer = f"{AGENT_NAME}-1" + self._redis = None + self._task: asyncio.Task | None = None + + async def start(self): + if not self.redis_url: + logger.info("RedisStreamWorker: no REDIS_URL — idle (HTTP /a2a/tasks still serves).") + return + try: + import redis.asyncio as aioredis # lazy import + self._redis = aioredis.from_url(self.redis_url, decode_responses=True) + await self._redis.ping() + except Exception as e: # noqa: BLE001 — boot standalone if Redis is down + logger.warning("RedisStreamWorker: Redis unavailable (%s) — idle, HTTP /a2a/tasks still serves.", e) + self._redis = None + return + try: + await self._redis.xgroup_create(self.stream, self.group, id="$", mkstream=True) + logger.info("Created consumer group %s on %s", self.group, self.stream) + except Exception: # noqa: BLE001 — group already exists + pass + self._task = asyncio.create_task(self._consume_loop()) + logger.info("RedisStreamWorker started: stream=%s", self.stream) + + async def _consume_loop(self): + while True: + try: + messages = await self._redis.xreadgroup( + self.group, self.consumer, {self.stream: ">"}, count=10, block=2000, + ) + if not messages: + continue + for _stream_name, entries in messages: + for entry_id, data in entries: + asyncio.create_task(self._process_task(entry_id, data)) + except asyncio.CancelledError: + logger.info("%s worker shutting down", AGENT_NAME) + break + except Exception as e: # noqa: BLE001 + if "NOGROUP" in str(e): + logger.warning("Consumer group lost, recreating: %s", self.group) + try: + await self._redis.xgroup_create(self.stream, self.group, id="$", mkstream=True) + except Exception: # noqa: BLE001 + pass + else: + logger.error("Worker consume error: %s", e, exc_info=True) + await asyncio.sleep(1) + + async def _process_task(self, entry_id: str, data: dict): + task_id = data.get("task_id", "") + response_channel = f"responses:{task_id}" + try: + task_params = json.loads(data.get("payload", "{}")) + action = task_params.get("action", "") + params = {**task_params.get("params", {})} + user_id = task_params.get("user_id", "") + user_context = task_params.get("user_context") + session_start = task_params.get("session_start", False) + if user_context: + params["user_context"] = user_context + + emitter = EventEmitter() + session = MatchmakingAgentSession(websocket=emitter, user_id=user_id) + + async def run_session(): + try: + if session_start: + await session.on_session_start(params) + elif action: + await session.on_user_action(action, params) + finally: + emitter.mark_done() + + session_task = asyncio.create_task(run_session()) + + while True: + msg = await emitter.queue.get() + if msg is None: + break + await self._redis.publish(response_channel, json.dumps(msg)) + + await session_task + + except Exception as e: # noqa: BLE001 + logger.error("Task %s failed: %s", task_id, e, exc_info=True) + await self._redis.publish( + response_channel, json.dumps({"type": "agent_error", "message": str(e)})) + finally: + await self._redis.publish( + response_channel, json.dumps({"type": "__task_complete__", "task_id": task_id})) + await self._redis.xack(self.stream, self.group, entry_id) + + async def stop(self): + if self._task: + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + if self._redis: + await self._redis.aclose() + self._redis = None + logger.info("%s worker stopped", AGENT_NAME) diff --git a/app/agent/__init__.py b/app/agent/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/agent/session.py b/app/agent/session.py new file mode 100644 index 0000000..20fe232 --- /dev/null +++ b/app/agent/session.py @@ -0,0 +1,91 @@ +"""Matchmaking-v2 agent session. + +Mirrors the mesh contract: the orchestrator calls /a2a/tasks, which drives +on_session_start / on_user_action; handlers `push` messages back (agent_thinking, +agent_data{action,data}, agent_error) that the orchestrator forwards to the frontend. + +Step-0 scaffold: the dispatch + contract are wired; handlers are honest stubs that +say "not implemented yet" so we can fill them in one full-stack slice at a time. +""" +from __future__ import annotations + +import logging + +logger = logging.getLogger(__name__) + + +class MatchmakingAgentSession: + def __init__(self, websocket, user_id: str): + self.ws = websocket + self.user_id = user_id + self.context: dict = {} + + async def push(self, msg_type: str, **kwargs): + await self.ws.send_json({"type": msg_type, **kwargs}) + + async def _stub(self, action: str, note: str): + await self.push("agent_data", action=action, data={"status": "not_implemented", "note": note}) + + async def on_session_start(self, params: dict): + self.context["user_context"] = params.get("user_context") + await self.push("agent_thinking", message="Scout is warming up…") + # Real flow lands in a later slice: resolve prefs → run_search → emit feed. + await self.push("agent_data", action="session_ready", data={"service": "matchmaking-v2", "version": "2.0.0"}) + + async def on_user_action(self, action: str, params: dict): + handlers = { + # existing contract + "sync_preferences": self.handle_sync_preferences, + "get_feed": self.handle_get_feed, + "record_feedback": self.handle_record_feedback, + "get_opportunity_detail": self.handle_get_opportunity_detail, + # new in v2 + "run_search": self.handle_run_search, + "tailor_resume": self.handle_tailor_resume, + "submit_application": self.handle_submit_application, + "get_apply_proof": self.handle_get_apply_proof, + } + handler = handlers.get(action) + if not handler: + await self.push("agent_error", message=f"Unknown action: {action}") + return + try: + await handler(params) + except Exception as e: # noqa: BLE001 — surface to the client, log the trace + logger.error("action '%s' failed: %s", action, e, exc_info=True) + await self.push("agent_error", message=f"Failed to {action}: {e}") + + # ── handlers (stubs — replaced one full-stack slice at a time) ── + async def handle_sync_preferences(self, params: dict): + await self._stub("preferences_synced", "persist ScoutPrefs → label store") + + async def handle_get_feed(self, params: dict): + await self._stub("feed_loaded", "return cached ranked feed") + + async def handle_record_feedback(self, params: dict): + await self._stub("feedback_recorded", "store SAVE/DISMISS/APPLY label") + + async def handle_get_opportunity_detail(self, params: dict): + await self._stub("opportunity_detail", "return match-score breakdown") + + async def handle_run_search(self, params: dict): + """Slice-1a: ScoutPrefs → live multi-board fetch → cards. No engine, no user-arm.""" + from app.engine import search + + prefs = params # the frontend's ScoutPrefs arrive as params (user_context, if present, is ignored) + if not search.has_mvq(prefs): + await self.push("agent_data", action="search_complete", + data={"opportunities": [], "needs": ["title", "location"]}) + return + await self.push("agent_thinking", message="Scanning live roles across job boards…") + result = await search.run_sweep(prefs) + await self.push("agent_data", action="search_complete", data=result) + + async def handle_tailor_resume(self, params: dict): + await self._stub("resume_tailored", "resume-builder: tailor resume to the chosen role") + + async def handle_submit_application(self, params: dict): + await self._stub("application_submitted", "master-key auto-apply (tier-aware)") + + async def handle_get_apply_proof(self, params: dict): + await self._stub("apply_proof", "return captured confirmation screenshots") diff --git a/app/api/__init__.py b/app/api/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/api/v1/__init__.py b/app/api/v1/__init__.py new file mode 100644 index 0000000..3b69342 --- /dev/null +++ b/app/api/v1/__init__.py @@ -0,0 +1,7 @@ +"""v1 REST router aggregator.""" +from fastapi import APIRouter + +from app.api.v1.health import router as health_router + +router = APIRouter() +router.include_router(health_router) diff --git a/app/api/v1/health.py b/app/api/v1/health.py new file mode 100644 index 0000000..6d57952 --- /dev/null +++ b/app/api/v1/health.py @@ -0,0 +1,12 @@ +"""Health endpoint — used by deploy.sh's health checks.""" +from fastapi import APIRouter + +from app.config import get_settings + +router = APIRouter() + + +@router.get("/health") +async def health(): + s = get_settings() + return {"status": "ok", "service": s.SERVICE_NAME, "version": "2.0.0", "env": s.ENV} diff --git a/app/config.py b/app/config.py new file mode 100644 index 0000000..be38095 --- /dev/null +++ b/app/config.py @@ -0,0 +1,74 @@ +"""matchmaking-v2 settings — lean, on-demand. No corpus DB, no nightly scrape.""" +from __future__ import annotations + +from functools import lru_cache +import os + +from pydantic_settings import BaseSettings + + +class Settings(BaseSettings): + model_config = {"env_file": ".env", "env_file_encoding": "utf-8", "extra": "ignore"} + + ENV: str = "development" + LOG_LEVEL: str = "INFO" + SERVICE_NAME: str = "matchmaking-service" # MUST stay "matchmaking-service" — orchestrator routes by card name + API_PREFIX: str = "/api/v1" + API_HOST: str = "0.0.0.0" + API_PORT: int = 8000 + CORS_ORIGINS: str = "*" + + # Public URL the orchestrator discovers this agent at (one of AGENT_URLS). + AGENT_URL: str = "http://localhost:8006" + + # A2A (agent-to-agent) bearer auth — the orchestrator presents one of these. + A2A_ALLOWED_KEYS: str = "dev-a2a-key" + + # Orchestrator comms (Redis Streams). Optional in dev — the worker no-ops if absent. + ORCHESTRATOR_REDIS_URL: str | None = None + REDIS_URL: str = "redis://localhost:6379/0" + + # Inputs / engine + APIFY_TOKEN: str | None = None + OPENAI_API_KEY: str | None = None + + # Result cache (dev cost-saver). Keyed by (actor, exact input) → saved job set on disk. + # off = always live, never save (PRODUCTION default) + # readwrite = replay if saved else live + save (DEV: one live sweep, then $0 replays) + # read = replay only, error if missing (offline / deterministic) + # refresh = always live + overwrite the saved set + APIFY_CACHE_MODE: str = "off" + APIFY_CACHE_DIR: str = ".apify_cache" + # Active boards — "Balanced" India stack: all three filter to the exact city at the board + # (hyper-relevant) and surface offsite apply links. Naukri=blackfalcondata feed, + # Foundit=Monster India, LinkedIn=geoId/city. Others (ats/indeed/naukri_v1) stay registered, off. + BOARDS_ENABLED: str = "naukri,foundit,linkedin" + + # Per-board fetch budget (the #1 Apify cost lever). Demo-sized to conserve credits. + NAUKRI_MAX_JOBS: int = 15 # blackfalcondata maxResults (no floor); fetchDetails=True for offsite apply links + FOUNDIT_MAX_JOBS: int = 50 + LINKEDIN_MAX_JOBS: int = 25 + # ATS lane (off by default) — company-seeded global/remote "dream companies". + ATS_MAX_PER_COMPANY: int = 4 + ATS_COMPANIES: str = "stripe,databricks,gitlab,figma,ramp,notion,razorpay,zerodha" + INDEED_MAX_JOBS: int = 30 + + # QScore (competence proxy) — consumed, not computed + QSCORE_BASE_URL: str = "http://localhost:8004" + QSCORE_AUTH_TOKEN: str | None = None + + # Storage for feedback/labels (NOT the old corpus). Optional until the learning loop lands. + DATABASE_URL: str | None = None + + @property + def cors_origins_list(self) -> list[str]: + if self.CORS_ORIGINS.strip() == "*": + return ["*"] + return [o.strip() for o in self.CORS_ORIGINS.split(",") if o.strip()] + + +@lru_cache +def get_settings() -> Settings: + if os.environ.get("PYTEST_CURRENT_TEST"): + return Settings(_env_file=None) + return Settings() diff --git a/app/contracts/__init__.py b/app/contracts/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/engine/__init__.py b/app/engine/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/app/engine/apify_client.py b/app/engine/apify_client.py new file mode 100644 index 0000000..24a1014 --- /dev/null +++ b/app/engine/apify_client.py @@ -0,0 +1,93 @@ +"""Thin Apify client — run an actor synchronously and get the dataset items. + +One call reused by every board adapter. On-demand: we fetch live and score only what +we fetched (no corpus). `run-sync-get-dataset-items` blocks up to ~300s. +""" +from __future__ import annotations + +import asyncio +import hashlib +import json +import logging +from pathlib import Path + +import httpx + +from app.config import get_settings + +logger = logging.getLogger(__name__) + +_BASE = "https://api.apify.com/v2/acts" + + +def _cache_path(actor_id: str, run_input: dict) -> Path: + """Stable path per (actor, exact input) — same query replays the same saved set.""" + canon = json.dumps(run_input, sort_keys=True, ensure_ascii=False) + digest = hashlib.sha1(f"{actor_id}|{canon}".encode()).hexdigest()[:16] + safe = actor_id.replace("/", "~").replace("~", "_") + return Path(get_settings().APIFY_CACHE_DIR) / f"{safe}__{digest}.json" + + +def _cache_read(path: Path) -> list[dict] | None: + try: + return json.loads(path.read_text()) + except (OSError, ValueError): + return None + + +def _cache_write(path: Path, items: list[dict]) -> None: + try: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(items, ensure_ascii=False)) + except OSError as e: # noqa: BLE001 — caching is best-effort + logger.warning("apify cache write failed: %s", e) + +# Apify's run-sync endpoint occasionally throws a transient 5xx (esp. 502). Retry ONCE +# on that. We do NOT retry on empty results: a completed run is billed, so re-running it +# would double the cost for no gain (an empty result is usually a real "0 matches"). +_MAX_ATTEMPTS = 2 +_BACKOFF = 2.0 + + +async def run_actor(actor_id: str, run_input: dict, *, timeout: float = 240.0) -> list[dict]: + """POST input → return the dataset items (list). One retry on transient 5xx; raises otherwise. + + Result cache (APIFY_CACHE_MODE): dev replays a saved job set for $0; production stays live. + """ + mode = get_settings().APIFY_CACHE_MODE + path = _cache_path(actor_id, run_input) if mode != "off" else None + + if path and mode in ("readwrite", "read"): + cached = _cache_read(path) + if cached is not None: + logger.info("apify cache HIT %s (%d items) — $0", actor_id, len(cached)) + return cached + if mode == "read": + raise RuntimeError(f"APIFY_CACHE_MODE=read but no cached set for {actor_id} ({path.name})") + + token = get_settings().APIFY_TOKEN + if not token: + raise RuntimeError("APIFY_TOKEN not configured") + url = f"{_BASE}/{actor_id}/run-sync-get-dataset-items" + + async with httpx.AsyncClient(timeout=timeout) as client: + for attempt in range(1, _MAX_ATTEMPTS + 1): + try: + resp = await client.post(url, params={"token": token}, json=run_input) + if resp.status_code >= 500: + raise httpx.HTTPStatusError( + f"{resp.status_code} from Apify", request=resp.request, response=resp) + resp.raise_for_status() + items = resp.json() + items = items if isinstance(items, list) else [] + if path and mode in ("readwrite", "refresh"): + _cache_write(path, items) + logger.info("apify cache SAVE %s (%d items)", actor_id, len(items)) + return items + except (httpx.HTTPStatusError, httpx.TransportError) as e: + if attempt < _MAX_ATTEMPTS: + logger.warning("apify %s: %s — retrying once", actor_id, e) + await asyncio.sleep(_BACKOFF) + continue + raise + return [] # pragma: no cover diff --git a/app/engine/board_adapters/__init__.py b/app/engine/board_adapters/__init__.py new file mode 100644 index 0000000..161c1e3 --- /dev/null +++ b/app/engine/board_adapters/__init__.py @@ -0,0 +1,15 @@ +"""Board adapters — ScoutPrefs → Apify actor inputs (see docs/ENGINE_DESIGN.md §1.2).""" +from .coerce import has_mvq, title_of, parse_location, parsed_locations +from .adapters import ( + build_naukri_input, + build_linkedin_input, + build_indeed_input, + build_indeed_searches, + build_ats_input, +) + +__all__ = [ + "has_mvq", "title_of", "parse_location", "parsed_locations", + "build_naukri_input", "build_linkedin_input", "build_indeed_input", + "build_indeed_searches", "build_ats_input", +] diff --git a/app/engine/board_adapters/adapters.py b/app/engine/board_adapters/adapters.py new file mode 100644 index 0000000..09052a0 --- /dev/null +++ b/app/engine/board_adapters/adapters.py @@ -0,0 +1,223 @@ +""" +Per-board query builders — `ScoutPrefs` (dict) → each Apify actor's input. +See `docs/ENGINE_DESIGN.md` §1.2. Returns the `input` dict you POST to the actor's +`run-sync-get-dataset-items` endpoint. + +System knobs (budget / freshness) are arguments with sane defaults, not user inputs. +""" +from __future__ import annotations +from typing import Any + +from . import coerce as C + +Prefs = dict[str, Any] + + +# ── Naukri (India workhorse — richest structured filters) ──────────────────── +def build_naukri_input(prefs: Prefs, *, max_jobs: int = 80, freshness: str = "7", recall: bool = False) -> dict: + # recall=True (retrieve stage): keep keyword + city + work-mode only; drop the + # narrowing filters (experience/department/industry/salary/stage) — those become + # ENGINE ranking later, not hard board cuts. Wider freshness for recall. + if recall: + freshness = "30" + locs = C.parsed_locations(prefs) + cities: list[int] = [] + for lp in locs: + if lp["city"]: + cities += C.CITY_NAUKRI.get(lp["city"], []) # unmapped city → dropped (kept in keyword) + + out: dict[str, Any] = { + "keyword": C.title_of(prefs) or C.build_keyword(prefs), + "maxJobs": max_jobs, # REQUIRED — the budget + "freshness": freshness, # liveness gate + "sortBy": "date", + "fetchDetails": True, # → apply redirect / full details + } + if cities: + out["cities"] = [str(c) for c in sorted(set(cities))] # Naukri select codes are STRINGS + + wm = [C.WORKMODE_NAUKRI[w] for w in (prefs.get("workMode") or []) if w in C.WORKMODE_NAUKRI] + if C.is_remote(prefs) and "remote" not in wm: + wm.append("remote") + if wm: + out["workMode"] = wm + + if recall: + return out # retrieve stage stops here — keyword + cities + work-mode only + + yrs = C.years_of(prefs) + if yrs is not None: + out["experience"] = str(yrs) # numeric-string enum "0".."30" + + depts = [C.ROLE_NAUKRI_DEPT[r] for r in (prefs.get("role") or []) if r in C.ROLE_NAUKRI_DEPT] + if depts: + out["department"] = [str(d) for d in sorted(set(depts))] # codes → strings + + inds = [C.INDUSTRY_NAUKRI[i] for i in (prefs.get("industry") or []) if i in C.INDUSTRY_NAUKRI] + if inds: + out["industry"] = [str(i) for i in sorted(set(inds))] # codes → strings + + if prefs.get("targetComp") in C.COMP_NAUKRI_SALARY: + out["salaryRange"] = [C.COMP_NAUKRI_SALARY[prefs["targetComp"]]] + + if prefs.get("companyStage") in C.COMPANY_TYPE_NAUKRI: + out["companyType"] = [C.COMPANY_TYPE_NAUKRI[prefs["companyStage"]]] + + return out + + +# ── Naukri (blackfalcondata feed — city-precise, real offsite applyRedirectUrl) ── +def build_naukri_feed_input(prefs: Prefs, *, max_jobs: int = 50, fetch_details: bool = True) -> dict: + # City-filtered at the board (hyper-relevant + cheap). fetch_details=True surfaces the + # offsite `applyRedirectUrl`; skipReposts dedups for cost. Narrowing (dept/industry/exp) + # stays off here — that's engine ranking later. + city = next((lp["city"] for lp in C.parsed_locations(prefs) if lp["city"]), "") + return { + "keyword": C.title_of(prefs) or C.build_keyword(prefs), + "location": city, # precise city, e.g. "New Delhi" + "maxResults": max_jobs, + "fetchDetails": fetch_details, + "skipReposts": True, + "sortBy": "date", + } + + +# ── Foundit / Monster India (city-precise; apply_url is the offsite redirect) ── +# Foundit uses legacy city names — it returns 0 for "New Delhi"/"Bengaluru". +_FOUNDIT_CITY = { + "new delhi": "Delhi", "delhi ncr": "Delhi", "bengaluru": "Bangalore", + "gurugram": "Gurgaon", "mumbai": "Mumbai", "pune": "Pune", "hyderabad": "Hyderabad", + "chennai": "Chennai", "kolkata": "Kolkata", "noida": "Noida", +} + + +def build_foundit_input(prefs: Prefs, *, results_wanted: int = 50) -> dict: + city = next((lp["city"] for lp in C.parsed_locations(prefs) if lp["city"]), "") + city = _FOUNDIT_CITY.get(city.lower(), city) # alias to Foundit's canonical name + return { + "keyword": C.title_of(prefs) or C.build_keyword(prefs), + "location": city, + "results_wanted": results_wanted, + } + + +# ── LinkedIn (takes the band, not years; ₹ comp incompatible) ──────────────── +def build_linkedin_input(prefs: Prefs, *, max_items: int = 80, posted: str = "week", + recall: bool = False) -> dict: + # recall=True (retrieve stage): title + COUNTRY-level location only — drop the narrowing + # filters (easyApply / workplaceType / experienceLevel / industryIds / freshness), which + # become ENGINE ranking later. City-level location + a wrong industryId zero the actor out. + out: dict[str, Any] = { + "jobTitles": [C.title_of(prefs)], + "maxItems": max_items, + "sortBy": "date", + } + + cities: list[str] = [] # bare city names ("New Delhi") — harvestapi's location format + cities_q: list[str] = [] # city, country ("New Delhi, India") — used outside recall + countries: list[str] = [] + for lp in C.parsed_locations(prefs): + if lp["country"] and lp["country"] not in countries: + countries.append(lp["country"]) + if lp["city"]: + cities.append(lp["city"]) + cities_q.append(f"{lp['city']}, {lp['country']}" if lp["country"] else lp["city"]) + + if recall: + # City-level for precision (engine ranks finer later). Fall back to country if no city. + out["locations"] = cities or countries + return out + + out["postedLimit"] = posted # freshness + out["easyApply"] = False # favour offsite (auto-applyable) over login-gated easy-apply + if cities_q or countries: + out["locations"] = cities_q or countries + # TODO: geoIds = [lookup(loc)] is more reliable than free-text locations. + + wm = [C.WORKMODE_LINKEDIN[w] for w in (prefs.get("workMode") or []) if w in C.WORKMODE_LINKEDIN] + if C.is_remote(prefs) and "remote" not in wm: + wm.append("remote") + if wm: + out["workplaceType"] = wm + + band = C.seniority_band(prefs) + if band in C.SENIORITY_LINKEDIN: + out["experienceLevel"] = [C.SENIORITY_LINKEDIN[band]] + + ind_ids = [C.INDUSTRY_LINKEDIN_ID[i] for i in (prefs.get("industry") or []) if i in C.INDUSTRY_LINKEDIN_ID] + if ind_ids: + out["industryIds"] = ind_ids # TODO verify IDs; unmapped industries already fall to keyword + + return out + + +# ── Indeed (blunt: what + where + country only; everything else folds in) ──── +def build_indeed_input(prefs: Prefs, *, max_items: int = 80) -> dict: + loc = next((l for l in C.parsed_locations(prefs) if l["city"] or l["country"]), {}) + country_name = loc.get("country") or "India" + return { + # Indeed has no work-mode/seniority/industry params → fold them into the query. + "position": C.build_keyword(prefs, seniority=True, industry=True), + "location": loc.get("city") or "", + "country": C.COUNTRY_ISO2.get(country_name, "IN"), + "maxItemsPerSearch": max_items, + "followApplyRedirects": True, # → real offsite apply URL + "saveOnlyUniqueItems": True, + } + + +def build_indeed_searches(prefs: Prefs, *, max_items: int = 80) -> list[dict]: + """Fan-out: Indeed `position`/`location` are single strings, so multi title×city + becomes N runs to merge+dedup downstream.""" + titles = (prefs.get("targetTitles") or [])[:3] or [C.title_of(prefs)] + cities = [l["city"] for l in C.parsed_locations(prefs) if l["city"]] or [None] + base = build_indeed_input(prefs, max_items=max_items) + runs = [] + for t in titles: + for city in cities: + r = dict(base) + r["position"] = " ".join(x for x in [t, *(["remote"] if C.is_remote(prefs) else [])] if x) + if city: + r["location"] = city + runs.append(r) + return runs + + +# ── Direct-ATS (curated-company lane — `companies` is passed in, NOT from prefs) ── +def build_ats_input(prefs: Prefs, *, companies: list[dict], recent_days: int = 30, + max_per_company: int = 50, output_profile: str = "compact", + recall: bool = False) -> dict: + loc = next((l for l in C.parsed_locations(prefs) if l["city"]), {}) + inds = [i for i in (prefs.get("industry") or []) if i and i != "Any"] + out: dict[str, Any] = { + "companies": companies, # curated [{company}] — the lane's identity + "titleKeyword": C.title_of(prefs), # filters within those companies (role relevance) + "remoteOnly": C.is_remote(prefs), + "keywords": inds, # loose industry fold + "recentWindowDays": recent_days, + "maxJobsPerCompany": max_per_company, + "outputProfile": output_profile, # "compact" = cheaper (enough for display); "full" adds descriptions + } + # recall=True: skip the city filter — these curated companies are global/remote, so a + # city lock ("Bengaluru") empties the deck. Location becomes engine ranking later. + if not recall: + out["locationKeyword"] = loc.get("city", "") + return out + + +if __name__ == "__main__": + import json + demo: Prefs = { + "targetTitles": ["Payments PM"], "title": "Associate Product Manager", + "role": ["Product"], "industry": ["Fintech"], + "location": ["Bengaluru · India", "Delhi NCR · India"], + "workMode": ["Hybrid"], "experience": ["Junior"], "years": 2, + "targetComp": "₹16–22L", "companyStage": "Growth-stage", + # engine-side only (ignored by adapters): priorities/stretch/intent/availability/dealBreakers/sort + } + print("MVQ ok:", C.has_mvq(demo), "\n") + for name, fn in [("NAUKRI", build_naukri_input), ("LINKEDIN", build_linkedin_input), + ("INDEED", build_indeed_input)]: + print(f"── {name} ──"); print(json.dumps(fn(demo), indent=2, ensure_ascii=False), "\n") + print("── DIRECT-ATS ──") + print(json.dumps(build_ats_input(demo, companies=[{"ats": "ashby", "company": "razorpay"}]), indent=2)) diff --git a/app/engine/board_adapters/coerce.py b/app/engine/board_adapters/coerce.py new file mode 100644 index 0000000..4d8a34d --- /dev/null +++ b/app/engine/board_adapters/coerce.py @@ -0,0 +1,121 @@ +""" +Input coercion layer — turns the frontend `ScoutPrefs` (JSON dict) into each board +actor's exact input schema. See `docs/ENGINE_DESIGN.md` §1.2. + +All maps verified against the live Apify input schemas (2026-06-17). +GOLDEN RULE: a value that doesn't resolve to a board enum/code is NEVER sent as an +invalid enum — it degrades to a keyword fold (or post-filter) and is flagged. +Coverage gaps lose precision, never the call. +""" +from __future__ import annotations +import re +from typing import Any, Optional + +Prefs = dict[str, Any] + +# ── lookup maps (verified codes/enums) ─────────────────────────────────────── +COUNTRY_ISO2 = { # Indeed `country` + "India": "IN", "United States": "US", "United Kingdom": "GB", "Canada": "CA", + "Australia": "AU", "Germany": "DE", "Singapore": "SG", "UAE": "AE", +} +WORKMODE_NAUKRI = {"Remote": "remote", "Hybrid": "hybrid", "On-site": "office"} +WORKMODE_LINKEDIN = {"Remote": "remote", "Hybrid": "hybrid", "On-site": "office"} + +SENIORITY_YEARS = {"Fresher": 0, "Junior": 2, "Mid-level": 5, "Senior": 9, "Lead": 13} +SENIORITY_LINKEDIN = { # LinkedIn `experienceLevel` + "Fresher": "entry", "Junior": "associate", "Mid-level": "mid-senior", + "Senior": "director", "Lead": "executive", +} +COMP_NAUKRI_SALARY = { # ₹ band → Naukri `salaryRange` (lakhs bucket) + "₹10–14L": "10to15", "₹14–18L": "15to25", "₹16–22L": "15to25", + "₹22–30L": "25to50", "₹30L+": "50to75", +} +COMPANY_TYPE_NAUKRI = { # loose — funding stage ≠ company type + "Seed / early": "Startup", "Growth-stage": "Startup", + "Late / pre-IPO": "Corporate", "Enterprise": "MNC", +} + +# Naukri numeric codes (labels are display-only — the actor wants the code) +ROLE_NAUKRI_DEPT = { # role area → `department` + "Product": 10, "Engineering": 5, "Design": 15, "Data": 3, + "Marketing": 9, "Sales": 14, "Ops": 2, # "Open to any" → omit +} +INDUSTRY_NAUKRI = { # → `industry` + "Fintech": 114, "SaaS": 110, "E-commerce": 108, "EdTech": 133, + "HealthTech": 131, "Dev tools": 110, "Consulting": 126, +} +CITY_NAUKRI = { # → `cities` (1→N for metro regions) + "Bengaluru": [97], "Bangalore": [97], "Mumbai": [134], "New Delhi": [6], "Delhi": [6], + "Delhi NCR": [6, 73, 220, 350], "Gurgaon": [73], "Gurugram": [73], "Noida": [220], + "Hyderabad": [17], "Chennai": [183], "Pune": [139], "Kolkata": [232], +} +# LinkedIn industry IDs — best-effort, TODO verify vs LinkedIn taxonomy; unmapped → keyword fold +INDUSTRY_LINKEDIN_ID = { + "Fintech": "1742", "SaaS": "4", "E-commerce": "6", "EdTech": "68", + "HealthTech": "14", "Dev tools": "4", "Consulting": "11", +} + +# ── parsing + accessors ────────────────────────────────────────────────────── +def parse_location(loc: str) -> dict: + """'Bengaluru · India' → {city, country, remote, anywhere}.""" + s = (loc or "").strip() + low = s.lower() + if low == "remote": + return {"city": None, "country": None, "remote": True, "anywhere": False} + if low in ("anywhere", ""): + return {"city": None, "country": None, "remote": False, "anywhere": True} + parts = [p.strip() for p in re.split(r"[·,]", s) if p.strip()] + return { + "city": parts[0] if parts else None, + "country": parts[1] if len(parts) > 1 else None, + "remote": False, "anywhere": False, + } + +def parsed_locations(prefs: Prefs) -> list[dict]: + locs = [parse_location(l) for l in (prefs.get("location") or [])] + return locs or [{"city": None, "country": None, "remote": False, "anywhere": True}] + +def is_remote(prefs: Prefs) -> bool: + return any(l["remote"] for l in parsed_locations(prefs)) or "Remote" in (prefs.get("workMode") or []) + +def title_of(prefs: Prefs) -> str: + """Keyword precedence: specific fine-tune pick → typed title → first concrete role.""" + tt = prefs.get("targetTitles") or [] + if tt: + return tt[0] + if (prefs.get("title") or "").strip(): + return prefs["title"].strip() + for r in (prefs.get("role") or []): + if r and r != "Open to any": + return r + return "" + +def seniority_band(prefs: Prefs) -> Optional[str]: + exp = prefs.get("experience") or [] + return exp[0] if exp else None + +def years_of(prefs: Prefs) -> Optional[int]: + """Numeric years: explicit `years` → else derived from the seniority band.""" + y = prefs.get("years") + if isinstance(y, (int, float)): + return max(0, min(30, int(y))) + b = seniority_band(prefs) + return SENIORITY_YEARS.get(b) if b else None + +def build_keyword(prefs: Prefs, *, seniority: bool = False, industry: bool = False) -> str: + """Composite keyword for boards lacking department/industry filters.""" + parts: list[str] = [] + if seniority and seniority_band(prefs) in ("Senior", "Lead"): + parts.append(seniority_band(prefs)) # prefix only the high bands + parts.append(title_of(prefs)) + if industry: + inds = prefs.get("industry") or [] + if inds and inds[0] not in ("Any", None): + parts.append(inds[0]) + return " ".join(p for p in parts if p).strip() + +def has_mvq(prefs: Prefs) -> bool: + """Minimum Viable Query = a title + a resolvable location (city/country/remote).""" + if not title_of(prefs): + return False + return any(l["city"] or l["country"] or l["remote"] for l in parsed_locations(prefs)) diff --git a/app/engine/normalize.py b/app/engine/normalize.py new file mode 100644 index 0000000..ee8398f --- /dev/null +++ b/app/engine/normalize.py @@ -0,0 +1,364 @@ +"""Per-board normalizers: raw Apify item → the frontend `ScoutJob` shape. + +Slice-1a: no engine, so matchScore/note/qx are placeholders (matchScore is assigned +in search.py after merge). Each board's output differs — one mapper per board. +""" +from __future__ import annotations + +import html as _html +import json +import re + +_PALETTE = ["#4F46E5", "#EA580C", "#0F6E56", "#DB2777", "#0EA5E9", "#7C3AED", "#D97706", "#059669"] + + +def _plain(s, limit: int = 1600) -> str: + """HTML/Markdown → readable plain text for the detail modal (keeps paragraph breaks). + + We surface ONLY what the board/employer provided — and strip the scraper's own + aggregator disclaimer boilerplate (we add nothing of our own). + """ + if not s: + return "" + # HTML → text + s = re.sub(r"<\s*(br|/p|/li|/div|/h\d|/tr)\s*[^>]*>", "\n", str(s), flags=re.I) # breaks → newlines + s = re.sub(r"]*>", "• ", s, flags=re.I) + s = re.sub(r"<[^>]+>", "", s) # drop remaining tags + s = _html.unescape(s) + # Markdown leftovers (when the source is markdown, not HTML) + s = re.sub(r"(?m)^\s{0,3}#{1,6}\s*", "", s) # ATX headers → plain line + s = re.sub(r"(?m)^[ \t]*[-*+][ \t]+", "• ", s) # list bullets → • + s = re.sub(r"\*\*([^*]+)\*\*", r"\1", s) # **bold** → text + s = re.sub(r"`([^`]+)`", r"\1", s) # `code` → text + # Broken-tag remnants (e.g. a leaked "> from malformed HTML) + s = re.sub(r'"\s*>', " ", s) + s = s.lstrip(' "\'>') + # Drop the scraper's appended disclaimer boilerplate (NOT employer content) + s = re.split(r"\n?\s*Disclaimer\s*[:\-]", s, maxsplit=1, flags=re.I)[0] + # whitespace + s = re.sub(r"[ \t]+", " ", s) + s = re.sub(r"\n{3,}", "\n\n", s).strip() + return (s[:limit].rstrip() + "…") if len(s) > limit else s + + +def _compact(d: dict) -> dict: + """Drop empty values so the details payload stays lean.""" + return {k: v for k, v in d.items() if v not in (None, "", [], {})} + + +def _logo(org: str) -> dict: + org = (org or "").strip() + label = org[0].upper() if org else "?" + bg = _PALETTE[sum(map(ord, org)) % len(_PALETTE)] if org else "#9CA3AF" + return {"label": label, "bg": bg} + + +def _digits(s: str) -> int | None: + nums = re.sub(r"[^\d]", "", s or "") + return int(nums) if nums else None + + +def _salary_from_amount(amount: int | None, per_year: bool) -> tuple[float | None, str]: + """amount in ₹ → (lpa, payLabel).""" + if not amount: + return None, "Not disclosed" + annual = amount if per_year else amount * 12 + lpa = round(annual / 100_000, 1) + return lpa, f"₹{lpa:g}L/yr" + + +def _seniority_from_years(min_yrs: int | None): + if min_yrs is None: + return None + return ("junior" if min_yrs <= 1 else "junior" if min_yrs <= 4 else + "mid" if min_yrs <= 8 else "senior" if min_yrs <= 12 else "lead") + + +# free-text seniority (LinkedIn experienceLevel / ATS seniority) → our Seniority enum +_SENIORITY_TEXT = { + "internship": "junior", "entry level": "junior", "entry": "junior", "junior": "junior", + "associate": "mid", "mid": "mid", "mid-senior level": "senior", "senior": "senior", + "director": "lead", "executive": "lead", "lead": "lead", +} + + +def _seniority_from_text(s: str | None): + if not s: + return None + return _SENIORITY_TEXT.get(s.strip().lower()) + + +_WORKMODE = {"remote": "remote", "hybrid": "hybrid", "on_site": "onsite", "onsite": "onsite"} + + +def naukri_to_scoutjob(it: dict, board: str = "Naukri") -> dict | None: + # The naukri-job-scraper (fetchDetails=True) nests the real fields under `jobDetails`. + jd = it.get("jobDetails") or {} + title = (jd.get("title") or "").strip() + org = ((jd.get("companyDetail") or {}).get("name") or "").strip() + if not title or not org: + return None + locs = jd.get("locations") or [] + city = (locs[0].get("label") if locs and isinstance(locs[0], dict) else None) or None + # keySkills is {other:[{label}], preferred:[{label}], ...} — flatten the labels. + skills: list[str] = [] + ks = jd.get("keySkills") or {} + if isinstance(ks, dict): + for bucket in ks.values(): + for s in bucket or []: + lbl = (s.get("label") if isinstance(s, dict) else str(s)).strip() + if lbl and lbl not in skills: + skills.append(lbl) + # salaryDetail is already a dict here (not a JSON string). + lpa, pay = None, "Not disclosed" + sd = jd.get("salaryDetail") or {} + if isinstance(sd, dict) and not sd.get("hideSalary") and sd.get("maximumSalary"): + lpa, pay = _salary_from_amount(int(sd["maximumSalary"]), per_year=True) + return { + "id": f"naukri-{jd.get('jobId')}", + "title": title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": "India", + "matchScore": 0, # placeholder — set after merge + "salary_lpa": lpa, + "payLabel": pay, + "seniority_level": _seniority_from_years(_digits(str(jd.get("minimumExperience", "")))), + "required_skills": skills[:6], + "tags": skills[:3], + "applicants": jd.get("applyCount"), + "note": f"Live from {board}", + "apply_url": jd.get("staticUrl"), + "qx": 10, + } + + +def indeed_to_scoutjob(it: dict, board: str = "Indeed") -> dict | None: + if str(it.get("isExpired", "False")).lower() == "true": + return None + title = (it.get("positionName") or "").strip() + org = (it.get("company") or "").strip() + if not title or not org: + return None + # Indeed location is "Area, City, State" (or "City, State") — the city is the 2nd-to-last part. + parts = [p.strip() for p in (it.get("location") or "").split(",") if p.strip()] + city = (parts[-2] if len(parts) >= 2 else (parts[0] if parts else None)) + types = [] + try: + types = [t.strip() for t in eval(it.get("jobType") or "[]")] if it.get("jobType") else [] + except Exception: # noqa: BLE001 + pass + sal = it.get("salary") or "" + lpa, pay = _salary_from_amount(_digits(sal), per_year="year" in sal.lower()) if sal else (None, "Not disclosed") + apply_url = it.get("externalApplyLink") + if not apply_url or apply_url == "None": + apply_url = it.get("url") + return { + "id": f"indeed-{it.get('id')}", + "title": title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": "India", + "matchScore": 0, + "salary_lpa": lpa, + "payLabel": pay, + "required_skills": types, + "tags": types[:3], + "applicants": None, + "note": f"Live from {board}", + "apply_url": apply_url, + "qx": 10, + } + + +def linkedin_to_scoutjob(it: dict, board: str = "LinkedIn") -> dict | None: + title = (it.get("title") or "").strip() + org = ((it.get("company") or {}).get("name") or "").strip() + if not title or not org: + return None + loc = it.get("location") or {} + parsed = loc.get("parsed") or {} if isinstance(loc, dict) else {} + city = parsed.get("city") or None + country = parsed.get("country") or parsed.get("countryFull") or None + mode = _WORKMODE.get((it.get("workplaceType") or "").lower()) + industries = [s for s in (it.get("industries") or []) if isinstance(s, str)] + sal = it.get("salary") or {} + pay = sal.get("text") if isinstance(sal, dict) and sal.get("text") else "Not disclosed" + co = it.get("company") or {} + details = _compact({ + "description": _plain(it.get("descriptionText") or it.get("descriptionHtml")), + "experience": it.get("experienceLevel"), + "employment_type": (it.get("employmentType") or "").replace("_", " ").title() or None, + "industry": ", ".join(industries[:3]) or None, + "skills": industries, + "company_about": _plain(co.get("description"), 700) if isinstance(co, dict) else "", + }) + return { + "id": f"linkedin-{it.get('id')}", + "title": title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": country, + "location_mode": mode, + "matchScore": 0, + "salary_lpa": None, + "payLabel": pay, + "seniority_level": _seniority_from_text(it.get("experienceLevel")), + "required_skills": industries[:6], + "tags": industries[:3], + "applicants": it.get("applicants"), + "note": f"Live from {board}", + "apply_url": it.get("linkedinUrl"), + "details": details, + "qx": 10, + } + + +def ats_to_scoutjob(it: dict, board: str = "Direct") -> dict | None: + title = (it.get("title") or "").strip() + org = (it.get("company") or "").strip() + if not title or not org: + return None + org = org.replace("-", " ").replace("_", " ").title() # ats slugs ("stripe") → "Stripe" + parts = [p.strip() for p in (it.get("location") or "").split(",") if p.strip()] + city = parts[0] if parts else None + mode = _WORKMODE.get((it.get("remote_type") or "").lower()) or ("remote" if it.get("remote") else None) + dept = [d for d in [it.get("department"), it.get("team")] if d] + return { + "id": f"ats-{it.get('global_id') or it.get('job_id')}", + "title": title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": None, + "location_mode": mode, + "matchScore": 0, + "salary_lpa": None, + "payLabel": "Not disclosed", + "seniority_level": _seniority_from_text(it.get("seniority")), + "required_skills": dept[:6], + "tags": dept[:3], + "applicants": None, + "note": f"Live from {board}", # direct employer posting → real offsite apply link + "apply_url": it.get("apply_url") or it.get("url"), + "qx": 10, + } + + +def naukri_feed_to_scoutjob(it: dict, board: str = "Naukri") -> dict | None: + """blackfalcondata/naukri-jobs-feed (fetchDetails=True) — flat shape with a real + offsite `applyRedirectUrl` when the employer uses an external ATS.""" + title = (it.get("title") or "").strip() + org = (it.get("companyName") or it.get("staticCompanyName") or "").strip() + if not title or not org: + return None + loc = it.get("location") or "" + city = (loc.split("/")[0].split(",")[0].strip() or None) if isinstance(loc, str) else None + # keySkills = {preferred:[{label}], other:[{label}]} — flatten labels. + skills: list[str] = [] + ks = it.get("keySkills") or {} + if isinstance(ks, dict): + for bucket in ks.values(): + for s in bucket or []: + lbl = (s.get("label") if isinstance(s, dict) else str(s)).strip() + if lbl and lbl not in skills: + skills.append(lbl) + elif isinstance(ks, list): + skills = [str(s) for s in ks if s] + lpa, pay = None, "Not disclosed" + if not it.get("salaryHidden") and it.get("salaryMax"): + lpa, pay = _salary_from_amount(int(it["salaryMax"]), per_year=True) + redirect = it.get("applyRedirectUrl") + education = "; ".join(it.get("educationUG") or []) or it.get("degreeCombination") or None + details = _compact({ + "description": _plain(it.get("descriptionMarkdown") or it.get("description") or it.get("shortDescription")), + "experience": it.get("experienceText"), + "employment_type": it.get("employmentType"), + "education": education, + "industry": it.get("industry"), + "role_category": it.get("roleCategory"), + "skills": skills, + "company_about": _plain(it.get("companyDescription"), 700), + "vacancies": it.get("vacancy"), + }) + return { + "id": f"naukri-{it.get('jobId')}", + "title": title.title() if title.isupper() else title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": "India", + "location_mode": _WORKMODE.get((it.get("wfhType") or "").lower()), + "matchScore": 0, + "salary_lpa": lpa, + "payLabel": pay, + "seniority_level": _seniority_from_years(it.get("minimumExperience")), + "required_skills": skills[:6], + "tags": skills[:3], + "applicants": it.get("applyCount"), + "note": f"Live from {board}", + "apply_url": redirect or it.get("staticUrl") or it.get("portalUrl"), + "offsite_apply": bool(redirect), # True = real 3rd-party employer/ATS link (auto-applyable) + "details": details, + "qx": 10, + } + + +def foundit_to_scoutjob(it: dict, board: str = "Foundit") -> dict | None: + title = (it.get("title") or "").strip() + org = (it.get("company") or "").strip() + if not title or not org: + return None + loc = it.get("location") or "" + city = (loc.split("/")[0].split(",")[0].strip() or None) if isinstance(loc, str) else None + skills = it.get("skills") or [] + if isinstance(skills, str): + skills = [s.strip() for s in skills.split(",") if s.strip()] + skills = [str(s) for s in skills if s] + seniority = None + m = re.search(r"\d+", str(it.get("experience") or "")) + if m: + seniority = _seniority_from_years(int(m.group())) + redirect = it.get("apply_url") + details = _compact({ + "description": _plain(it.get("description_text") or it.get("description_html")), + "experience": it.get("experience"), + "employment_type": it.get("employment_type"), + "industry": it.get("industry"), + "role_category": it.get("function"), + "skills": skills, + }) + return { + "id": f"foundit-{it.get('job_id')}", + "title": title, + "organization": org, + "logo": _logo(org), + "location_city": city, + "location_country": "India", + "matchScore": 0, + "salary_lpa": None, + "payLabel": "Not disclosed", + "seniority_level": seniority, + "required_skills": skills[:6], + "tags": [it.get("function")][:1] if it.get("function") else skills[:3], + "applicants": None, + "note": f"Live from {board}", + "apply_url": redirect or it.get("url"), + "offsite_apply": bool(redirect), + "details": details, + "qx": 10, + } + + +# board key → normalizer +NORMALIZERS = { + "naukri": naukri_feed_to_scoutjob, # blackfalcondata (primary) + "foundit": foundit_to_scoutjob, + "linkedin": linkedin_to_scoutjob, + "naukri_v1": naukri_to_scoutjob, # muhammetakkurtt (fallback) + "indeed": indeed_to_scoutjob, + "ats": ats_to_scoutjob, +} diff --git a/app/engine/search.py b/app/engine/search.py new file mode 100644 index 0000000..2f40526 --- /dev/null +++ b/app/engine/search.py @@ -0,0 +1,85 @@ +"""On-demand multi-board sweep: ScoutPrefs → all boards (parallel) → merge + dedup → ScoutJob[]. + +This is the retrieve layer (final shape). Slice-1a has NO engine — ranking is a placeholder +spread; the §3 cascade slots in here later without changing the output contract. +Degrade-don't-break: a board that errors/needs setup is skipped; the others still return. +""" +from __future__ import annotations + +import asyncio +import logging + +from app.config import get_settings +from app.engine.apify_client import run_actor +from app.engine.board_adapters import adapters as A +from app.engine.board_adapters import coerce as C +from app.engine import normalize as N + +logger = logging.getLogger(__name__) + +def _ats_companies() -> list[dict]: + return [{"company": c.strip()} for c in get_settings().ATS_COMPANIES.split(",") if c.strip()] + + +# board key → (actor_id, build_input, normalizer). "Balanced" India stack = naukri+foundit+linkedin, +# all city-filtered at the board. The rest stay registered (code ready) but off unless enabled. +BOARDS = { + "naukri": ("blackfalcondata~naukri-jobs-feed", + lambda p: A.build_naukri_feed_input(p, max_jobs=get_settings().NAUKRI_MAX_JOBS), + N.naukri_feed_to_scoutjob), + "foundit": ("shahidirfan~Foundit-Jobs-Scraper", + lambda p: A.build_foundit_input(p, results_wanted=get_settings().FOUNDIT_MAX_JOBS), + N.foundit_to_scoutjob), + "linkedin": ("harvestapi~linkedin-job-search", + lambda p: A.build_linkedin_input(p, recall=True, max_items=get_settings().LINKEDIN_MAX_JOBS), + N.linkedin_to_scoutjob), + # ── registered fallbacks / other lanes (off unless added to BOARDS_ENABLED) ── + "ats": ("bovi~greenhouse-lever-ashby-job-scraper", + lambda p: A.build_ats_input(p, recall=True, companies=_ats_companies(), + max_per_company=get_settings().ATS_MAX_PER_COMPANY), + N.ats_to_scoutjob), + "naukri_v1": ("muhammetakkurtt~naukri-job-scraper", + lambda p: A.build_naukri_input(p, recall=True, max_jobs=get_settings().NAUKRI_MAX_JOBS), + N.naukri_to_scoutjob), + "indeed": ("misceres~indeed-scraper", + lambda p: A.build_indeed_input(p, max_items=get_settings().INDEED_MAX_JOBS), + N.indeed_to_scoutjob), +} + + +async def _fetch_board(key: str, prefs: dict): + actor, build, norm = BOARDS[key] + items = await run_actor(actor, build(prefs)) + jobs = [sj for it in items if (sj := norm(it))] + return key, jobs + + +async def run_sweep(prefs: dict) -> dict: + enabled = [b.strip() for b in get_settings().BOARDS_ENABLED.split(",") if b.strip() in BOARDS] + results = await asyncio.gather(*(_fetch_board(k, prefs) for k in enabled), return_exceptions=True) + + merged: list[dict] = [] + seen: set[str] = set() + sources: dict[str, int] = {} + for r in results: + if isinstance(r, Exception): + logger.warning("board fetch failed: %s", r) + continue + key, jobs = r + sources[key] = len(jobs) + for j in jobs: + dk = (j.get("apply_url") or "").strip() or f"{j['organization']}|{j['title']}|{j.get('location_city')}".lower() + if dk in seen: + continue + seen.add(dk) + merged.append(j) + + # placeholder ranking (no engine yet): a descending spread so cards aren't all the same % + for i, j in enumerate(merged): + j["matchScore"] = max(55, 95 - i) + + return {"opportunities": merged, "sources": sources} + + +def has_mvq(prefs: dict) -> bool: + return C.has_mvq(prefs) diff --git a/app/main.py b/app/main.py new file mode 100644 index 0000000..67f9d44 --- /dev/null +++ b/app/main.py @@ -0,0 +1,51 @@ +"""matchmaking-v2 — FastAPI app (course-service mold). + +Serves: the agent card (discovery), /a2a/tasks (orchestrator entry), /api/v1/health. +On-demand by design — no corpus DB, no nightly ingestion. +""" +from __future__ import annotations + +import logging +from contextlib import asynccontextmanager + +from fastapi import FastAPI +from fastapi.middleware.cors import CORSMiddleware + +from app.adk.worker import RedisStreamWorker +from app.api.v1 import router as api_router +from app.a2a.card import router as a2a_card_router +from app.a2a.tasks import router as a2a_tasks_router +from app.config import get_settings + +logging.basicConfig(level=get_settings().LOG_LEVEL) + +_settings = get_settings() +_worker = RedisStreamWorker(redis_url=_settings.ORCHESTRATOR_REDIS_URL or _settings.REDIS_URL) + + +@asynccontextmanager +async def lifespan(_: FastAPI): + await _worker.start() + try: + yield + finally: + await _worker.stop() + + +def create_app() -> FastAPI: + s = get_settings() + app = FastAPI(title="Matchmaking v2", version="2.0.0", lifespan=lifespan) + app.add_middleware( + CORSMiddleware, + allow_origins=s.cors_origins_list, + allow_credentials=s.cors_origins_list != ["*"], + allow_methods=["*"], + allow_headers=["*"], + ) + app.include_router(api_router, prefix=s.API_PREFIX) + app.include_router(a2a_card_router) + app.include_router(a2a_tasks_router) + return app + + +app = create_app() diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..21230dc --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,10 @@ +# Local dev for matchmaking-v2 in isolation (prod wiring lives in the growqr monorepo compose). +services: + matchmaking-v2: + build: . + container_name: matchmaking-v2 + ports: + - "8006:8000" + env_file: + - .env + restart: unless-stopped diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..c23f565 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,9 @@ +fastapi>=0.115 +uvicorn[standard]>=0.30 +pydantic>=2.7 +pydantic-settings>=2.3 +httpx>=0.27 +redis>=5.0 +# engine/auto-apply deps land in later slices: +# openai>=1.40 # GPT-5.4 rerank/explain + tailoring +# apify-client>=1.7 # on-demand board fetch (or call run-sync via httpx) diff --git a/research/docs/AUTOAPPLY_STATE.md b/research/docs/AUTOAPPLY_STATE.md new file mode 100644 index 0000000..356fd06 --- /dev/null +++ b/research/docs/AUTOAPPLY_STATE.md @@ -0,0 +1,89 @@ +# Auto-Apply Engine — State of the POC & Segment Thesis + +> Companion to `ENGINE_DESIGN.md`. Captures where the browser-use auto-apply POC stands as of +> 2026-06-15: what's proven, the honest boundaries, and the segment thesis. Code: `poc/browser-apply/`. +> Numbers are from small, directional samples — real direction, not statistical precision. + +--- + +## 0. What this is +The "heart" of Scout (`ENGINE_DESIGN.md` §6), built for real. We replaced the dead heuristic mock +(which proved nothing — fake ATS, fake liveness gate) with an **AI browser agent**: open-source +**browser-use** + **gpt-4.1 vision** driving real Chrome to fill and submit real job applications. +The design is a **board-agnostic "master key" engine** + a thin **adapter registry**, fed by Apify +apply-URLs. It applies on the *company's own ATS*, sidestepping the LinkedIn/Naukri login + ban zone. + +## 1. Architecture (what's built) +- `candidate.py` — board-agnostic candidate model: vision-parsed résumé + confirmed facts + a + per-job tailored-motivation rule. (Résumé parsing reuses the resume-builder vision pattern.) +- `apply_engine.py` — the **master key**: navigate → autofill → fill-every-field → **accuracy gate** + → verify → screenshot. Statuses incl. `no_form` / `login_required` so the long tail is legible. +- `adapters/` — registry. `GenericAdapter` (catch-all, pure vision) handles the long tail; + `AshbyAdapter` carries earned Ashby quirks. Route by URL domain. New boards = a few-line adapter. +- `apply_ashby.py` — the proven standalone (kept untouched). +- `apply_harness.py` — accuracy/cost/timing measurement rig. `apify_probe.py` — inbound yield rig. + +**Design principle:** the engine is generic; adapters are thin hints; **verification is first-class** +— screenshot-as-ground-truth + a hard gate, because the agent's self-report is optimistic. + +## 2. What's PROVEN (the receipts) +- **Real end-to-end submission.** Submitted a real application to OpenAI on AshbyHQ — verified by + the on-screen green success banner, not the agent's word. +- **The master key generalizes.** The *generic* adapter filled Greenhouse forms it had never seen + (GitLab, Figma, Anthropic's long adversarial form) + Ashby via its adapter — **4/5**, the 5th + honestly bounced on a login wall. **0 fake successes.** +- **India works, untuned.** Cracked **Zoho Recruit + Keka** (the big India-native ATS) with no India + tuning — even read their simple text-CAPTCHAs via vision — and honestly reported `no_form` on an + email-only careers page. +- **The loop closes, economics known.** Apify direct-ATS actor = 100% offsite Tier-1 links; + LinkedIn (harvestapi) = 60% offsite; combined ~43% offsite, ~14% immediately autonomous. Cost + **~$0.13/app clean ATS, ~$0.21–0.27 India ATS**; latency ~3–6 min/app; **vision-off ~43% cheaper** + with no accuracy loss on clean forms. Cheap models (gpt-4o-mini) are a **false economy** (0/2, + step-blowup). + +## 3. Honest boundaries (what's NOT yet solved) +- **Login-gated demand stays Tier-3.** LinkedIn Easy Apply, Naukri's one-click dashboard, + Workday/Oracle account-gates — need persisted session + human/Cloud, not the master key. +- **Submit-time anti-bot is the persistent threat.** Got spam-flagged once on Ashby; reCAPTCHA/ + hCaptcha would wall us; even the simple-CAPTCHA "solve" is **unverified** (we never submitted to + confirm acceptance). This is what breaks at scale. +- **Latency needs parallelization** (architecture, not a faster model). **ToS/account-safety** is a + real exposure on the login boards. +- **The matching engine (`ENGINE_DESIGN.md` core) is not built.** This arc was all *apply*. + +## 4. The segment (defined precisely) +Not "auto-apply to anything." The defensible segment is: + +> **Autonomous, honestly-verified applications to the public-ATS + India-native-ATS tier — +> applying on the company's own ATS, sidestepping the LinkedIn/Naukri login & account-ban problem.** + +**Differentiation:** existing tools (LazyApply, Simplify) are brittle per-board scripts that operate +*inside* LinkedIn (the ban zone). A **board-agnostic vision engine that goes around to the ATS**, +with **no-fake-success verification**, is a different, more durable product. **India is the sharpest +edge** — a huge, Naukri-dominated market where the direct-ATS + Zoho/Keka/Darwinbox path is +genuinely underserved. + +## 5. POC → segment (the path) +1. **Submit-time anti-bot lane** (make-or-break): Cloud stealth for spam/CAPTCHA; human-in-the-loop + final tap otherwise. Decides the true autonomous conversion rate. +2. **Parallelization** — concurrent sessions for acceptable latency at volume. +3. **Always-JSON output + answer-bank flywheel** (small): legible triage; engine learns answers over + time so human interventions shrink. +4. **Wrapper decision:** the *apply layer of Scout*, or a standalone *autonomous-applicant / apply-API* + product. The tech supports either. + +## 6. Honesty ledger +| Proven (de-risked) | Open / must-prove | +|---|---| +| Generic vision engine fills arbitrary forms across boards & countries | Submit-time anti-bot (spam flags, reCAPTCHA) at scale | +| Real verified submission; no fake successes | Login-gated tier (LinkedIn/Naukri dashboard/Workday) | +| India-native ATS (Zoho/Keka) with no tuning | CAPTCHA acceptance (unverified — needs a sandbox submit) | +| Known unit cost (~$0.13–0.27/app) + the vision-off lever | Latency at volume (needs parallelization) | +| Apify inbound returns usable apply-URLs (loop closes) | The matching engine (still on paper) | + +## Bottom line +A **proven, board-agnostic, honestly-verified apply engine** — real submission, generalizes across +US + India ATS, known economics. A strong POC and a legitimate **segment for the public-ATS / +India-ATS autonomous slice**. The realism check: "fully autonomous for *every* board" is not the +product — the login tier needs an assisted lane, and submit-time anti-bot is the risk that decides +scale. **Define the segment around the proven slice, lead with India.** diff --git a/research/docs/DECOMMISSION_PLAN.md b/research/docs/DECOMMISSION_PLAN.md new file mode 100644 index 0000000..4122c84 --- /dev/null +++ b/research/docs/DECOMMISSION_PLAN.md @@ -0,0 +1,46 @@ +# Old Matchmaking — Decommission & Slot-In Plan + +> How `matchmaking-v2` (Scout) replaces the dead `matchmaking` service. Companion to +> `ENGINE_DESIGN.md`. Status (2026-06-16): **planned, local teardown not yet executed.** + +## Current state (verified live 2026-06-16) +The old `matchmaking` runs identically in **both** local (`growqr-demo`) and prod (EC2): + +| Component | Local | Prod (EC2 `3.109.159.139`) | +|---|---|---| +| `matchmaking-api` | `:8006` → 8000, up | `:8006` → 8000, up 4d | +| `matchmaking-worker` | up | up 4d | +| `matchmaking-ingestion-worker` | up | up 4d | +| `matchmaking-postgres` (own DB, NOT RDS) | `:5439`, vol `matchmaking-pgdata` | `:5439`, vol `matchmaking-pgdata` | + +Containers are *running* but the **service is functionally dead** (ingestion chain broke 2026-05-27; +986,770 stale score rows for 21 users). `:8011` is free in both. AWS reachable: CLI as +`iam::189924572890:user/rahul`; SSH `ec2-user@3.109.159.139` (key `~/.ssh/growqr-key.pem`). + +## Locked decisions +1. **Reuse port `:8006`** (the old matchmaking's slot) — NOT a new `:8011`. This overrides + `ENGINE_DESIGN.md` §1's "12th agent on :8011". Rationale: in-place swap — v2 inherits the + existing `api.growqr.ai/matchmaking/*` nginx route + the frontend's existing matchmaking wiring, + so the Scout UI works with minimal rewiring. +2. **Drop the old corpus DB.** v2 is **on-demand, no corpus** (`ENGINE_DESIGN.md` §1) — so + `matchmaking-postgres` + its `matchmaking-pgdata` volume are discarded. The data is stale/worthless + (design doc verdict). Irreversible — confirm before deleting the volume. +3. **WS-agent via orchestrator.** v2 reuses `:8006`, registers in orchestrator `AGENT_URLS`, and the + frontend reaches it through `useAgentSession` (WS) — matching the mesh pattern (like + course-service `:8010`). The old `/matchmaking/*` nginx HTTP route stays vestigial or is removed + later. + +## Sequence +**A. Local — immediate (on go-ahead):** +- `docker compose stop` + `rm` the 4 matchmaking containers (api, worker, ingestion-worker, postgres). +- Drop the `matchmaking-pgdata` volume (decision #2). Frees `:8006` + `:5439` for v2 dev. +- Comment/remove the matchmaking services from the local compose; v2 takes `:8006`. + +**B. Prod — later (after v2 validated):** +- Stop the 4 prod matchmaking containers; remove from `docker-compose.prod.yml`. +- Deploy v2 on `:8006`; add to orchestrator `AGENT_URLS`; keep/retire the nginx `/matchmaking/*` route. +- Periodic `docker builder/image prune` (per aws-deployment.md) after the swap. + +## Rollback +Local teardown is reversible except the dropped volume. Keep the old `matchmaking` repo + compose +definition until v2 is validated in prod, so prod can be restored by re-adding the service block. diff --git a/research/docs/ENGINE_DESIGN.md b/research/docs/ENGINE_DESIGN.md new file mode 100644 index 0000000..ad033c1 --- /dev/null +++ b/research/docs/ENGINE_DESIGN.md @@ -0,0 +1,377 @@ +# Matchmaking v2 (Scout) — Engine Design & Locked Plan + +> Canonical design doc. Captures the engine research, the locked decisions, and the +> open items. Written 2026-06-12. **Revised 2026-06-16** with live-evidence findings. +> Supersedes the dead `matchmaking` service. +> +> **Companion docs (gathered evidence — read alongside):** +> - `SIGNAL_AUDIT_V2.md` — evidence-based signal inventory; **supersedes §2/§5 below**. +> - `AUTOAPPLY_STATE.md` — validated browser-use auto-apply engine; **supersedes/extends §6**. +> - `DECOMMISSION_PLAN.md` — how v2 slots in over the old matchmaking (port `:8006`). +> +> **2026-06-16 deltas in one line:** the engine is richer mostly by *wiring* (interview/roleplay/ +> assessment are LIVE data, not "gaps"); a new high-trust signal (**course watch-behavior**) was +> found; auto-apply is now a **validated board-agnostic vision agent** (not just hosted-submit/ +> handoff); v2 reuses port **`:8006`** (not `:8011`); **Pathways is cut**. + +--- + +## 0. Why v2 exists + +The old `matchmaking` service is being decommissioned. Diagnosis (verified June 11–12): + +- **Wrong shape:** it was a nightly *job aggregator* — scraped ~13.8k jobs into its own + Postgres and scored 21 users against the stale pile. 986,770 match-score rows for 21 users. +- **Dead scheduling:** not cron — a self-rescheduling RQ chain in Redis that went empty + (likely an ElastiCache eviction) and silently stopped. Ingestion dead since May 27. +- **Weak engine:** ranked on the *lowest-trust* signals (self-declared prefs + claimed skills + + an often-absent QScore that, at 30% weight, collapsed everyone to a neutral 60). + +v2 is a **fresh, purely on-demand** service. No aggregation. Fetch live listings when the +user searches, score only those. + +--- + +## 1. Locked architecture (the shape) + +- **On-demand only.** Per user search, fetch ~25–200 live listings from **LinkedIn + Naukri + + Indeed** via Apify, score only those. No corpus, no nightly batch. + - Sync fetch is viable: Apify `run-sync-get-dataset-items` (300s cap), cold start ~1.2s. + - 3-board search ≈ **$0.15–0.25 uncached**; shared cache keyed `role+location`, TTL 1–6h. + - **Vendor picks — VALIDATED 2026-06-16 (live probe, `poc/browser-apply/apify_probe.py`):** + - **Direct-ATS `bovi/greenhouse-lever-ashby-job-scraper` = the autonomous backbone** — public + JSON endpoints (no auth), returns clean ATS `apply_url`, **100% Tier-1 offsite** in-sample, + ~$1.50/1k. Company/preset-driven (not free-text). This is the new, highest-yield inbound. + - LinkedIn `harvestapi/linkedin-job-search` — **60% offsite** (`applyMethod.companyApplyUrl`), + real ATS diversity. Needs a one-time `approvePermissions` (full-account-access actor). + - Indeed `misceres/indeed-scraper` — ~35% external, but ~0% clean-ATS for India/ML (breadth, low + autonomous yield). + - Naukri → use **`muhammetakkurtt/naukri-job-scraper`** (11k users, no permission gate), NOT + `memo23` or the fishy `infinity_and_beyond`. Its `applyRedirectUrl` is the true external link, + but ~93% point to bespoke small-company career pages, not clean ATS. + - **Measured combined yield:** ~43% have an offsite link, **~14% immediately Tier-1 autonomous** + on a free-text search (more if you lead with direct-ATS). Full data → `apify-inbound` memory. +- **Surface:** ONE responsive web app, **no extension, no native app download.** + - The cross-origin limitation is now **solved a third way (validated):** a server-side/cloud + **AI browser agent (browser-use) fills any board's form by vision** — no extension, no ATS + creds needed. See §6 + `AUTOAPPLY_STATE.md`. (Hosted-submit/prepared-handoff remain fallbacks.) +- **Slots in over the OLD matchmaking — port `:8006`, not a new `:8011`** (revised; see + `DECOMMISSION_PLAN.md`). v2 inherits the existing `api.growqr.ai/matchmaking/*` route + the + frontend's matchmaking wiring. Still FastAPI in the `course-service` mold: Redis-Streams worker, + registered in orchestrator `AGENT_URLS`, reached via the orchestrator WS, frontend talks to it + through the existing `useAgentSession(...)` hook. The old matchmaking's own Postgres (corpus) is + dropped — v2 is on-demand, no corpus. + +--- + +## 1.1 Apify call contracts (verified live 2026-06-17) +Probed each actor's real input schema (`poc/browser-apply/apify_probe.py`, key loaded). Field-level +contract + the ScoutPrefs→params coverage matrix live in **`ENGINE_INPUTS.md`**. Headlines: +- **MVQ (Minimum Viable Query) = title + a resolvable location.** No MVQ → **do not call Apify** + (engine returns `{needs:[...]}`; frontend also gates "Find my matches"). Cache key = `title+geo`, TTL 1–6h. +- **Free-text lane:** `harvestapi/linkedin-job-search` (`jobTitles[]`,`locations[]`), `misceres/indeed-scraper` + (`position`,`location`,`country`), `muhammetakkurtt/naukri-job-scraper` (`keyword`,`cities[]`,`maxJobs*`). +- **Curated lane (separate):** `bovi/greenhouse-lever-ashby` is **company-list-driven**; user inputs only + *filter within* it (`titleKeyword`,`locationKeyword`,`remoteOnly`,`keywords[]`). +- **Naukri is the richest target for India** — uniquely exposes `department`, `industry`, ₹ `salaryRange`, + numeric `experience`. Lead with it. **Gotcha:** Naukri `industry`/`department`/`cities` take **numeric + codes**, not labels (labels are display-only). + +## 1.2 Input coercion layer (the board adapters) +Nothing from the UI goes straight to an actor — a per-board **adapter** parses/maps/converts each +`ScoutPrefs` field to that actor's exact schema. **Code: `engine/board_adapters/`** (`coerce.py` + +`build_{naukri,linkedin,indeed,ats}_input(prefs)`). + +**6 coercion dimensions:** ① parse/split (`"City · Country"`→`{city,country,remote}`) · ② enum-map +(label→board enum) · ③ unit-convert (₹→lakhs bucket; seniority⇄years) · ④ id-lookup (name→code: +LinkedIn `geoIds`/`industryIds`, **Naukri numeric codes**) · ⑤ cardinality (multi-select→array where +supported, else **fan-out N searches + dedup** — Indeed/bovi take single strings) · ⑥ presence +(`Any`→omit, `Remote`→flag, `Anywhere`→geo-only). + +**Verified lookup maps (built into `coerce.py`):** +| Map | Examples | +|---|---| +| `COUNTRY_ISO2` (Indeed) | India→`IN`, US→`US`, UK→`GB` | +| `WORKMODE_*` | Remote→`remote`, On-site→`office`, Any→omit | +| `SENIORITY_YEARS` (Naukri `experience`) | Fresher 0 · Junior 2 · Mid 5 · Senior 9 · Lead 13 | +| `SENIORITY_LINKEDIN` (`experienceLevel`) | Junior→`associate`, Mid→`mid-senior`, Senior→`director` | +| `COMP_NAUKRI_SALARY` (`salaryRange`) | ₹16–22L→`15to25`, ₹22–30L→`25to50` | +| `ROLE_NAUKRI_DEPT` (codes) | Product→`10` · Engineering→`5` · Design→`15` · Data→`3` · Sales→`14` · Ops→`2` | +| `INDUSTRY_NAUKRI` (codes) | Fintech→`114` · SaaS→`110` · E-commerce→`108` · EdTech→`133` · HealthTech→`131` · Consulting→`126` | +| `CITY_NAUKRI` (codes, 1→N) | Bengaluru→`[97]` · Mumbai→`[134]` · **Delhi NCR→`[6,73,220,350]`** · Hyderabad→`[17]` · Pune→`[139]` | +| `INDUSTRY_LINKEDIN_ID` | best-effort — **TODO verify** vs LinkedIn taxonomy; falls back to keyword fold | + +**System-set (not user):** `maxJobs`/`maxItems` (25–200 budget) · freshness (`freshness=7`/`postedLimit=week`/ +`recentWindowDays=30` — the liveness gate) · `sortBy=date` · `fetchDetails`/`followApplyRedirects=true` +(real offsite apply URL) · Naukri country implicit `IN`. + +**Composite-keyword fallback** for boards lacking `department`/`industry` (LinkedIn/Indeed/bovi): +`keyword = [seniorityPrefix] + title + [industryTerm]` (e.g. Indeed `position="Senior Payments PM fintech"`), +while Naukri stays clean (`keyword="Payments PM"` + structured `department/industry/salaryRange`). + +**Hard rule — degrade, don't break:** any value that doesn't resolve to a board enum/code is **never +sent as an invalid enum**. It falls back to the **keyword fold** or **post-fetch filter**, and is logged. +A missing lookup loses precision, never the call. + +**Engine-side-only inputs** (no board param on any actor — used for ranking/floors/exclusions/ordering): +`priorities`, `stretch`, `intent`, `availability`, `dealBreakers`, `sort`. See `ENGINE_INPUTS.md` §6. + +--- + +## 2. Signals — keep / discard (the input audit) + +> **➜ `SIGNAL_AUDIT_V2.md` supersedes this section** (evidence-based, from live-DB sampling +> 2026-06-16). Deltas: (a) **course watch-behavior** is a NEW high-trust booster — revealed +> preference ("completes React content → step toward frontend") beats stated; (b) interview/ +> roleplay/assessment are **LIVE data, not gaps** (§5) but are **already digested into QScore**, so +> v1 uses the QScore quotient as the competence proxy to avoid double-counting; (c) **Pathways is +> CUT** (feature *and* its questionnaire-as-signal — weak link: broken skill-gap, impractical roles). +> The trust⟂coverage discipline below is unchanged and is what keeps "more signals" from re-causing +> the v1 collapse. + +**Governing tension: trust ⟂ coverage.** High-trust signals have low coverage; high-coverage +signals have low trust. **Design rule: a signal earns SCORE WEIGHT only if its coverage is high +enough that absence isn't the norm; otherwise it is an ADDITIVE BOOSTER (bonus when present, +never a penalty when absent).** The old engine broke this rule — that's the v1 collapse bug. + +### KEEP + +**Tier A — backbone (high coverage; filter + light score)** +- `years_experience` (date-derived, **not** the self-declared band) +- `current_role` / title (prefer social-branding LinkedIn-verified) +- `preferred_locations` + work mode (hard filter) +- `target_roles` / `target_industries` (filter, not score — it's *want* not *is*) +- skills (normalized — see §3.2) + +**Tier B — verified enrichers (medium coverage; real weight when present)** +- verified title / company / **industry** (social-branding; industry only via BrightData path) +- degree / field_of_study / institution +- certifications +- assessment scores (`passed` / `percentage` — user-service, shipped Jun 11; earned, grows) +- QScore **quotient vector** (10-D; enricher/tiebreaker — not additive-only, coverage-penalized) + +**Tier C — premium boosters (low coverage; ADDITIVE ONLY, never penalize absence)** +- interview 8-D rubric (~5–10% coverage) +- roleplay 6-D rubric (<3%) +- improvement velocity / coachability +- video on-camera score (<1%) +- (Doubles as an adoption flywheel: "do a mock interview → sharper matches.") + +### DISCARD (noise for matching) +- **ATS `overall_score`** ⚠️ measures resume *quality/formatting*, **not job fit** — the trap. +- parser confidence, `resume_check` signals (QA, not the person) +- follower/connection counts, post engagement (vanity) +- `brand_score`, `pathway_archetype` (rare/external/unpopulated) +- self-declared `salary_range` (zeros/guesses — use the listing's market salary) +- engagement streak/trends (engagement ≠ fit) +- names, avatar, email; cover-letter analysis & coaching narratives (for humans, not the scorer) +- **institution_tier** — discarded deliberately; sidesteps the worst India pedigree-bias vector. + +--- + +## 3. THE ENGINE (the v2 redesign) + +### 3.0 Core verdict (research-backed) +- **Embedding/cosine similarity must NOT be the ranker** [proven, 3-0]. It rewards topical + word-overlap over qualification fit, can't enforce hard constraints (seniority, certs), and + *collapses on sparse profiles* — exactly the v1 failure mode. Embedding = **one input signal**. +- **Winner: a deterministic white-box utility score, fused with other rankers by + WEIGHTED-AVERAGE RANK** [proven: WAvg rank +31.9% nDCG@10; beats RRF +5.8% and every single + ranker]. "Retrieve cheap, rerank expensive" still applies — the boards did retrieval; we + spend compute reranking the 25–200 fetched listings. + +### 3.1 The cascade (per search) +``` +① NORMALIZE skills → ESCO canonical; titles → canonical role +② HARD FILTER location/work-mode, must-have certs, seniority floor (deterministic gate — + embeddings CAN'T do this) +③ WHITE-BOX U = Σ_{f∈present} c_f·w_f·φ_f / Σ_{f∈present} c_f·w_f + UTILITY factors: skill_match, experience, location, salary, semantic_sim (ONE input), + industry/company, + QScore-quotient / assessment boosters +④ FUSION weighted-average RANK over: utility ‖ embedding-sim ‖ LLM soft-score +⑤ LLM RERANK GPT-5.4 on top ~20: calibrated 0–1 fit vs a 6-level rubric; + + EXPLAIN ALSO logged as cold-start supervision; emits "why" + growth nudge +⑥ OUTPUT ranked list + breakdown + explanation + ↓ + LOG soft-scores + user save/apply → accrue labels → swap ③④ priors for a learned + pairwise/listwise ranker (margin loss / CMMD — NOT pointwise MSE) +``` + +### 3.2 Skill matching (the core accuracy lever) [proven method] +Exact taxonomy lookup is a dead end (only 41/202 soft skills exact-match ESCO; real-world +span-match F1 caps ~27%). The stack: +1. **Synonym canonicalization** to ESCO (`k8s→Kubernetes`, `ML→Machine Learning`) — seed from + ESCO + v1 role synonyms + India tech terms. +2. **Depth-2 knowledge-graph traversal** of `RELATED_TO` edges for implied/transferable skills + (Kubernetes → "container orchestration"). Powers STRETCH. +3. **Contrastive embedding alignment** to the taxonomy to beat the ~27% ceiling. + *Validate locally — the F1@5 0.72 result is on Chinese ads.* v1 ships synonym+KG+cosine-to-ESCO; + upgrade to a trained contrastive aligner later. +4. **Required vs preferred** weighting: required near-gating, preferred additive. +- **Taxonomy = ESCO.** EU-origin — sanity-check India role/skill coverage early; extend, don't swap. + +### 3.3 Coverage-aware combination [our design — no validated formula exists] +Renormalize over **present** signals only, each weighted by confidence `c_f` (coverage × trust +from the audit): `U = Σ_{present} c_f·w_f·φ_f / Σ_{present} c_f·w_f`. +A missing signal redistributes its weight to what we know — never penalizes, never flattens. +This makes the v1 "QScore absent → everyone 60" bug impossible by construction. **Validate `c_f` +offline** (confirm it doesn't over-reward sparse profiles). + +### 3.4 Cold-start weighting (no labels at launch) [proven] +- **Day 1:** expert-prior weights — starting point `skill .35 / experience .25 / location .15 / + salary .10 / semantic .10 / company .05`. **Priors to calibrate, not optima.** +- **From day 1, in parallel:** GPT-5.4 as **teacher** — calibrated 0–1 soft scores on a 6-level + rubric, logged on every match. Synthesizes supervision (study: discriminative specificity + 3.5× — 0.058 → 0.208) instead of waiting for outcomes (<0.05% dense). +- **As save/apply accrues:** train a **pairwise/listwise** ranker + contrastive/paraphrase + augmentation; shift fusion weight off the priors. + +### 3.5 FIT vs STRETCH [our design — research left open] +- **FIT:** skill-gap ≈ 0; score current-state overlap; mild over-qualification penalty. +- **STRETCH:** target = one level up / adjacent; score by **reachability** = fraction of the + job's required skills the user *has OR is within depth-2 KG hops of*. "Reach but reachable" = + small, bridgeable gap → this is the prototype's growth nudge. +- Maps onto the Scout intent picker: current-state → "more like this"; aspiration → "step up / new". + +--- + +## 4. Evaluation & model roles — **no human in the loop** + +- **Runtime is fully autonomous.** No human ever reviews, approves, or scores a user's matches. +- **Eval grading is automated and cross-model** to avoid self-grading inflation: + | Role | Model | + |---|---| + | Match generation, cold-start soft-score teacher, explanations/growth nudges | **GPT-5.4** | + | Independent accuracy grading (LLM-as-judge) | **Opus 4.8** | + - Cross-family grading kills self-preference bias (no model marks its own homework). + - Residual generic LLM-judge bias (position/verbosity) → prompt hygiene: randomize order, + pairwise, strip length cues. + - **Un-gameable ground truth = real save/apply rate once live.** Opus is the pre-launch proxy. +- **Metrics:** nDCG@10, MRR, P@5, calibration MAE on a synthetic + dogfood gold set + (~50–100 India-representative pairs; no hired annotators). Online: interleaving + A/B once live. +- Note: Opus-as-judge is a deliberate, fine exception to "always GPT-5.4" (it's Anthropic, not an + OpenAI call). Eval pipeline needs an Anthropic API key alongside OpenAI. + +--- + +## 5. Transport readiness (do the pipes exist?) +Backbone exists: orchestrator `context_resolver` builds `user_context` from resume-builder + +social-branding `/api/state/{clerk_id}`; matchmaking calls qscore-service directly. +**REVISED 2026-06-16 — the "gaps" were assumptions; the data is LIVE.** Verified by sampling the +prod DBs over SSH (see `SIGNAL_AUDIT_V2.md` §3): +- **✅ READY:** all Tier A + QScore quotient vector. +- **✅ LIVE (was "partial"):** verified LinkedIn title/company/industry — social-branding does a + **live LinkedIn fetch**; LinkedIn education; certifications. +- **✅ LIVE (was "❌ GAP"):** assessment scores (user-service), interview rubrics (`growqr_interview`, + 210 reviews — 8-D rubric + video presence + confidence-ratio), roleplay rubrics (`growqr_roleplay`, + Gemini video analysis), improvement velocity (`trend_data`/`historical_comparison`). **None are a + build — they're a query.** *Caveat: filter degenerate sessions (no-show/low-talk poison scores).* +- **Transport already exists:** interview & roleplay services **already receive résumé/LinkedIn + context** via the orchestrator `context_resolver`. **matchmaking-v2 reuses that exact pattern** — + no new transport to build. v1 wiring = consume the assembled `user_context` + one extra read of + course watch-behavior (`growqr_course`). +- **Still: QScore is the competence proxy for v1** (it already consumes interview/roleplay); read + raw rubrics later only if they add lift beyond the aggregate. + +--- + +## 6. Auto-apply (the heart) — "one tap at the end" + +> **➜ `AUTOAPPLY_STATE.md` supersedes/extends this section** (validated POC, 2026-06-16). Headline +> update: the original "full auto-submit = NO-GO" is **REVISED** — a server-side **AI browser agent +> (browser-use + gpt-4.1 vision)** is now a *validated, board-agnostic "master key"*: +> - **Proven:** a real application was submitted to OpenAI on AshbyHQ (on-screen success confirmed); +> the *generic* adapter then filled Greenhouse forms it had never seen (4/5) and India-native ATS +> (Zoho Recruit, Keka) with no tuning, honestly bouncing login-walled / no-form pages. +> - **Architecture:** one generic engine + a thin **adapter registry** (Ashby adapter carries earned +> quirks; generic adapter handles the long tail by vision). Verification-first: screenshot-as- +> ground-truth + a hard accuracy gate (never trust the agent's self-report). +> - **Tiered reality:** autonomous for **public ATS (Ashby/Greenhouse/Lever) + India-native ATS**; +> login-gated boards (LinkedIn Easy Apply, Naukri dashboard, Workday/Oracle) stay human/Cloud-assisted. +> - **Economics (measured):** gpt-4.1 **vision-off** ~$0.13/app (clean ATS), ~$0.21–0.27 (India ATS), +> ~3–6 min/app; **gpt-4o-mini is a false economy** (failed + not cheaper). Latency needs parallelism. +> - **Anti-bot is the make-or-break risk:** Ashby spam-flagged the *fast* LinkedIn-routed submit; +> the fix was direct-to-ATS + human-like pacing + verified upload. reCAPTCHA/hCaptcha remain a wall +> (simple text CAPTCHAs are vision-solvable). New POC code: `poc/browser-apply/` (the old +> `poc/auto-apply/` mock is retired). + +By the time the user taps, all the work is done (found, tailored, filled). The tap ends an +application, it doesn't start one. In the no-install web app, "one tap" resolves two ways: +- **🛰️ Hosted submit** — backend POSTs the application to an **ATS API** (Greenhouse / Lever / + Ashby). True one-tap-done, any device, zero install. **Gated by employer creds** → coverage = + the integrations we provision; grows over time. +- **🔗 Prepared handoff** — one tap opens the board's own page (LinkedIn/Naukri) with résumé + + cover letter + answers staged; user taps submit there. Zero install, any device, zero ban risk. +- **Phase 4 full auto-submit:** NO-GO for now (unattended submit ~46% reliable + ToS-exposed; + the survivor pattern is user-device extension autofill + human submit). Opt-in only when revisited. + +POC validates the hosted-submit path end-to-end → see `poc/auto-apply/`. +**Hard rule: never fire test applications at real employers' live postings.** POC-0 = mock ATS; +POC-1 = our own ATS sandbox/test job. + +### 6.1 LISTING FRESHNESS / LIVENESS — first-class gate (learned from the POC, June 12) +The personalized demo tapped a real LinkedIn posting that was already "no longer accepting +applications." Then we empirically checked ~7 recent AI-Architect postings: **essentially all +were dead** (closed / HTTP 404 / redirect-to-search = LinkedIn's expired-job behavior). Findings: +- **A perfect match for a dead job is worse than no match** — it's actively undelightful. +- **Permalinks rot in days.** Never hand a user a *cached* posting URL — it's probably dead. +- **Tap-time server-side liveness re-check is unreliable:** LinkedIn serves bot-walled / guest + content (our httpx check false-returned "open" for a job we knew was closed), and expired URLs + 302→search or 404. You cannot trust a scrape to tell you a posting is open. +Requirements for v2: +1. **Freshness at the DATA layer:** the on-demand Apify fetch returns posting date / `isExpired` / + applicant count — filter & rank by recency, drop/flag stale **before presenting**. The whole + point of on-demand: every shown listing was fetched seconds ago, so it's fresh by construction. +2. **Honest age signals** in the UI ("posted 3d ago", "closes soon") — derive from posting age + (no scraper returns a true deadline field). +3. **Apply handoff targets a live view**, never a stored permalink: the freshly-fetched listing + or the board's **live search** for the role (always current). Recover gracefully, never dead-end. + +--- + +## 7. Proven vs. validate-locally (honesty ledger) +| Locked & proven (adopt method) | Our design — must validate locally | +|---|---| +| Hybrid fusion > embedding-only ranker | Exact fusion weights (60/25/15 was tuned on a tiny set) | +| Weighted-avg rank > RRF | The coverage-renormalization formula + `c_f` values | +| Synonym + KG-expand + contrastive skills > exact match | FIT/STRETCH reachability scoring | +| LLM-as-teacher for cold-start; pairwise > pointwise | India pedigree/fairness mitigations | +| Embedding-only collapses on sparse profiles | ESCO coverage for India roles | + +**Headline accuracy numbers do NOT transfer** (single-paper, small-benchmark, domain-mismatched — +gig matching, Chinese job ads, offline corpora). Adopt the **patterns**, re-tune on GrowQR data. + +## 8. Open items + +**Resolved 2026-06-16:** +- ✅ Apify inbound validated (Naukri → `muhammetakkurtt`; direct-ATS = autonomous backbone; offsite- + yield measured). ✅ Auto-apply POC validated end-to-end (master key, real submission, India ATS). +- ✅ Signal audit refreshed against live evidence (`SIGNAL_AUDIT_V2.md`). ✅ Agent capability pass + (interview/roleplay strong, Pathways cut). ✅ Slot-in/decommission plan (`DECOMMISSION_PLAN.md`). + +**Still open:** +- **Build the engine itself** — wire the v2 input set (Backbone + verified LinkedIn + QScore + + course watch-behavior) via the reused `context_resolver`; implement the §3 cascade. +- Latency benchmark of the actors from ap-south-1; **parallelize apply** (3–6 min/app is too slow serial). +- **Submit-time anti-bot lane** (the make-or-break): Cloud stealth / CAPTCHA / human-final-tap. +- Build the ~50–100 pair India gold set; validate Opus-judge agreement. ESCO India coverage check. +- Label volume for the learned ranker (old corpus was 21 users — accrue via the live save/apply loop). + +**UI — in progress (2026-06-17, branch `job-ui-final`):** +- ✅ Scout UI built **in-place in `dashboard-ui/scout`** (not a separate `index.html` POC) around the + locked feature set; screen-by-screen decisions + as-shipped divergences tracked in `SCOUT_UI_SPEC.md` + ("BUILD PROGRESS" section). Re-verified honest against this doc — all cuts/improvements, no + unbacked claims. Notable: cascading **Country→City** location (feeds Tier-A `preferred_locations` + + ported location-fit); a **no-resume "Any/open" state** that is the UI face of §3.3 (sparse profile → + coverage renormalizes, never fabricates); loader + scope made honest (≤120, on-demand framing). +- 🔧 UI still waiting on the engine for: apply-result statuses (proof screenshots + auto/failed→manual + split), per-job resume-tailoring spinner, and the real signal/match-breakdown wiring behind the cards. + +## Key sources (engine research, 2026) +- Synapse (arXiv 2604.02539) — hybrid fusion > embedding-only; WAvg rank fusion +31.9% nDCG@10. +- JobMatchAI (arXiv 2603.14558) — white-box utility weights; ESCO synonym + depth-2 KG expansion. +- ConFit / ConFit v2 (arXiv 2401.16349) — contrastive + paraphrase augmentation; label sparsity <0.05%. +- Person-Job Fit distillation (arXiv 2601.10321) — LLM-as-teacher soft scores; specificity 0.058→0.208; CMMD. +- JobSkape (arXiv 2402.03242) — ESCO span-match F1 ~26–27% ceiling. +- CareerBERT (arXiv 2503.02056) — embedding degradation on short/sparse profiles (coverage collapse). +- Springer 12651-025-00409-x — only 41/202 soft skills exact-match ESCO. diff --git a/research/docs/ENGINE_INPUTS.md b/research/docs/ENGINE_INPUTS.md new file mode 100644 index 0000000..af956e4 --- /dev/null +++ b/research/docs/ENGINE_INPUTS.md @@ -0,0 +1,154 @@ +# Scout → Engine — Exhaustive Input Inventory + +> Every field the **new Scout UI** (`dashboard-ui`, branch `job-ui-final`) produces that is (or should +> be) an input to matchmaking-v2. Derived field-by-field from the code, cross-checked against what the +> handlers actually pass (`ScoutConfirm.go()`, `ScoutBubbleJourney.toPrefs()`). Written 2026-06-17. +> Pair with `ENGINE_DESIGN.md` §2 (signal tiers) and `SIGNAL_AUDIT_V2.md`. + +There are **three kinds of input**: +1. **Search prefs** — the `ScoutPrefs` object the UI hands the engine per search (`onDone(prefs)`). +2. **Profile/context** — assembled by the orchestrator `context_resolver` (resume + LinkedIn + QScore + + course-behavior); NOT picked in Scout, but the Confirm card *displays/edits* some of it. +3. **Behavioral events** — view/save/pass emitted while swiping (learning loop + activity rank). + +--- + +## 1. SEARCH PREFS — the `ScoutPrefs` contract (what `onDone` sends) + +| # | Field | Type | UI surface(s) | Options / shape | Engine role (per ENGINE_DESIGN §2/§3) | +|---|---|---|---|---|---| +| 1 | `role` | `string[]` | Bubble journey role stage (like/love); Confirm via intent | Product/Design/Engineering/Data/Marketing/Sales/Ops/Open-to-any **+ add-your-own** | Tier-A `target_roles` — **filter, not score** (want, not is) | +| 2 | `industry` | `string[]` | Confirm "Industry" select; Bubble journey industry stage | Fintech/SaaS/E-commerce/EdTech/HealthTech/Dev tools/Consulting **+ Any + add-your-own** | Tier-A `target_industries` — filter | +| 3 | `location` | `string[]` | Confirm "Where" (LocationPicker); Bubble constraints (LocationPicker) | `"City · Country"` strings (+ Remote/Anywhere) — cascading Country→City **+ custom** | Tier-A `preferred_locations` — **hard filter + location-fit** (city AND country substring; Remote→100; Anywhere→open) | +| 4 | `workMode` | `string[]` | Confirm "Work mode"; Bubble constraints | Remote/Hybrid/On-site (Any = none) | Tier-A work mode — hard filter | +| 5 | `experience` | `string[]` | Confirm "Seniority" select; Bubble constraints | Fresher/Junior/Mid-level/Senior/Lead **+ Any** | Tier-A seniority floor — hard filter | +| 6 | `targetComp` | `string?` | Confirm "Target comp" select | ₹10–14L … ₹30L+ **+ Any** | preference signal (NOT self-declared salary, which §2 discards; this is target band) | +| 7 | `companyStage` | `string?` | Confirm "Company stage" select | Seed/Growth/Late-pre-IPO/Enterprise **+ Any** | company-fit enricher | +| 8 | `availability` | `string?` | Confirm "Availability" select | Open now / 30d / 60–90d / Just exploring **+ Any** | timing context | +| 9 | `dealBreakers` | `string[]?` | Confirm "Deal-breakers" chips | Remote-first/No on-call/Equity/Visa/4-day/No relocation | hard exclusions | +| 10 | `priorities` | `string[]?` | Confirm "What matters most? · Pick 2" | up to 2 of PRIORITY_OPTIONS | **ranking weights** (soft — orders results) | +| 11 | `stretch` | `Stretch?` | Confirm "How far should I reach?" | safe / balanced / reach | FIT↔STRETCH reach band (§3.5) → match-score floor | +| 12 | `sort` | `"match"\|"salary"\|"closing"` | default "match"; Shortlist has a local sort | — | output ordering (not a match signal) | +| 13 | **`intent`** *(not stored on prefs; shapes 1/2/5)* | `"same"\|"levelup"\|"new"` | Confirm "What are you after?" | More like this / A step up / Something new | **FIT vs STRETCH selector** (§3.5) — current-state vs aspiration | +| 14 | **`title`** ✅ wired 2026-06-17 | `string?` | Confirm card "Title" input | free text | Tier-A `current_role` / title (filter + weight). Sole title source when no resume | +| 15 | **`years`** ✅ wired 2026-06-17 | `number?` | Confirm card "Experience (yrs)" input | number | Tier-A `years_experience` (weight). Sole years source when no resume | +| 16 | **`targetTitles`** ✅ added 2026-06-17 | `string[]` (single-select → ≤1) | Bubble journey **stage 3 "Roles that fit you"** — AI/resume+LinkedIn-generated specific titles, **pick one** | e.g. "Payments PM", "Risk & Fraud Product" | **High-signal Tier-A target** — the precise role the engine should prefer to **seed the on-demand board query** (specific > broad `role`). The "true fine-tuning" pick. *Generation is a STUB for the GPT-5.4 /suggest call, mirroring the courses AI flow.* | + +> Both `title` and `years` flow through **both** paths — `ScoutConfirm.buildPrefs()` and (via the +> carry-forward `base`) `ScoutBubbleJourney.toPrefs()`, so Fine-tune no longer drops them. + +--- + +## 2. ⚠️ Remaining gap — captured but not fully used + +| Field | Where it's collected | Status | Why it matters | +|---|---|---|---| +| **bubble `like` vs `love`** | Bubble journey (3-state) | **Flattened** — `toPrefs()` maps both → "selected" | "love" = top priority — a free **weight** signal still collapsed to binary. Optional: feed a per-pick weight | + +**Resolved 2026-06-17:** the bubble-journey path (`toPrefs`) used to drop everything except +role/industry/workMode/experience/location. It now starts from the **carry-forward `base`** the Confirm +card passes in, so title/years/comp/stage/availability/deal-breakers/priorities/stretch survive Fine-tune. + +--- + +## 3. ✅ RESOLVED — orphaned prefs DROPPED from the contract (2026-06-17) + +These had no UI surface left, so they were removed from `ScoutPrefs` entirely (interface, `emptyPrefs`, +`resumePrefs`, `filterSortScoutJobs`, `go()`), and the orphaned `ScoutFilterSheet.tsx` + its option +constants (`JOB_TYPE_OPTIONS`, `SALARY_BANDS`, `JOB_TYPE_TO_VALUE`, `DREAM_COMPANY_OPTIONS`) were deleted. + +| Field | Was | Now | +|---|---|---| +| `jobType` | Action filter sheet (removed) | **dropped** | +| `salaryBand` | Action filter sheet (removed) | **dropped** (`targetComp` is the surviving comp input) | +| `dreamCompanies` | Confirm card edit (cut) | **dropped** | +| `hideEmployer` | Confirm card toggle (cut) | **dropped** | + +--- + +## 4. BEHAVIORAL EVENTS — emitted while swiping (feed the learning loop, not the per-search query) + +| Event | UI trigger | Engine role | +|---|---|---| +| **VIEW** | open a card / tap info (expand QX breakdown) | funnel "Viewed"; activity rank; weak relevance label | +| **SAVE** | swipe right / Save (heart) | funnel "Shortlisted"; **strong positive label** (§3.1 ⑥ accrue save/apply → learned ranker) | +| **PASS** | swipe left / X | negative label | +| **UNDO** | undo button | corrects the last save/pass | +| **APPLY (hand-off)** | DeckEnd "approve once" → auto/manual apply | **un-gameable ground truth** (§4) — the real save/apply rate | + +--- + +## 5. PROFILE / CONTEXT — assembled by `context_resolver`, surfaced (not picked) in Scout + +The Confirm card *displays and lets you edit* the resume-derived defaults; the rest is read server-side. +A **no-resume user** sends all of §1 as "Any"/open → engine §3.3 coverage-renormalizes (never fabricates). + +| Source | Fields | Tier | +|---|---|---| +| **Resume (resume-builder parse, "Mira")** | title, seniority, industry, city, years, + pre-filled guesses for comp/stage/availability/deal-breakers/priorities/stretch (editable on the card) | A / B | +| **LinkedIn (social-branding, live fetch)** | verified title / company / industry; education; certifications | B | +| **QScore** | 10-D quotient vector (already digests interview + roleplay) | B | +| **Assessment** | passed / percentage (user-service) | B | +| **Course watch-behavior** (`growqr_course`) | revealed-preference booster (e.g. completes React → frontend signal) | C (additive) | +| **Interview / roleplay rubrics** | 8-D / 6-D (low coverage) | C (additive) — read later only if lift beyond QScore | + +--- + +## 6. DEEP AUDIT — every engine signal × frontend coverage (2026-06-17) + +Walks the **entire** `SIGNAL_AUDIT_V2.md` inventory (Tier A/B/C + labels) and assigns each signal an +owner. **Rule:** the frontend owns only what the *user states* (request prefs + editable overrides + +feedback events). Everything resume/identity/competence-derived is **server-assembled by +`context_resolver` / qscore / Perplexity / `growqr_course`** — by design NOT a Scout UI field. + +| Engine signal (SIGNAL_AUDIT_V2) | Owner | Frontend field / event | Status | +|---|---|---|---| +| **Tier A** | | | | +| `years_experience` | Frontend override + resume | `years` (Confirm) | ✅ WIRED (was the gap) | +| `current_role` / title | Frontend override + resume/LinkedIn | `title` (Confirm) | ✅ WIRED (was the gap) | +| `skills` → ESCO | **Server** (resume parse) | — (not a Scout pref) | ✅ correctly server-side | +| `preferred_locations` + work mode | Frontend (request prefs) | `location` (City·Country), `workMode` | ✅ WIRED | +| `target_roles` / `target_industries` | Frontend (request prefs) | `role`, `industry` | ✅ WIRED | +| **Tier B** | | | | +| verified title / company / industry | **Server** (social-branding LinkedIn) | industry/title *overridable* via Confirm | ✅ server; overrides wired | +| degree / field / institution | **Server** (resume/LinkedIn) | — | ✅ server-side | +| certifications | **Server** (resume/LinkedIn) | — | ✅ server-side | +| assessment scores | **Server** (user-service) | — | ✅ server-side | +| QScore quotient vector (10-D) | **Server** (qscore-service) | — | ✅ server-side | +| market context (demand/freshness/comp) | **Server** (Perplexity, cached) | — | ✅ server-side | +| **Tier C (additive boosters)** | | | | +| course watch-behavior | **Server** (`growqr_course`) | — | ✅ server-side | +| improvement velocity / coachability | **Server** (interview/roleplay) | — | ✅ server-side | +| video on-camera presence | **Server** (interview/roleplay) | — | ✅ server-side | +| **Soft ranking / FIT-STRETCH (engine §3.4/§3.5)** | | | | +| ranking weights | Frontend | `priorities` (Pick 2) | ✅ WIRED | +| reach band | Frontend | `stretch` | ✅ WIRED | +| FIT vs STRETCH selector | Frontend | `intent` | ✅ WIRED | +| comp / stage / availability / deal-breakers | Frontend | `targetComp`/`companyStage`/`availability`/`dealBreakers` | ✅ WIRED | +| **Labels (training targets)** | | | | +| SAVE / DISMISS / APPLY feedback | Frontend events | swipe deck + apply hand-off | ✅ emitted (§4) | + +### Verdict +**Every engine input the frontend is responsible for is now wired.** After this pass there are **no +dropped frontend fields** — `title` + `years` were the last gaps and are now sent on both paths. +Everything still "unwired" in the UI is **server-assembled context** (skills, LinkedIn identity, +QScore, assessment, course-behavior, market context) and *must not* be a Scout form field — capturing +it client-side would duplicate or fake what `context_resolver` already assembles. + +**Only non-blocking residual:** the bubble `like` vs `love` 3-state is flattened to binary "selected" +(`toPrefs`); wiring it as a per-pick weight is an optional enhancement, not a missing engine input. + +--- + +## Summary — counts +- **15 search-pref fields** actively sent (incl. intent). The `ScoutPrefs` contract is now exactly: + `role · location · workMode · experience · industry · sort · title · years · targetComp · + companyStage · availability · dealBreakers · priorities · stretch` (+ `intent` shapes role/industry/experience). +- **title + years WIRED** (2026-06-17) on both the Confirm and Fine-tune paths — were the last gaps. +- **1 non-blocking residual** (bubble like/love → binary) — optional weight enhancement, not a missing input. +- **4 orphaned prefs DROPPED** (jobType, salaryBand, dreamCompanies, hideEmployer) — contract trimmed + to only what the UI renders. +- **5 behavioral events** for the learning loop. +- **6 server-assembled context sources** (skills/LinkedIn/QScore/assessment/course-behavior/market) — + correctly NOT Scout fields. +- **Deep-audit verdict (§6):** every frontend-owned engine input is wired; no dropped fields remain. diff --git a/research/docs/SCOUT_UI_SPEC.md b/research/docs/SCOUT_UI_SPEC.md new file mode 100644 index 0000000..a1bcc43 --- /dev/null +++ b/research/docs/SCOUT_UI_SPEC.md @@ -0,0 +1,134 @@ +# Scout UI — Lean Finalization Spec + +> Screen-by-screen "what we surface" decisions for the Scout UI, made against what the **engine + +> deployed agents can actually produce** (see `ENGINE_DESIGN.md`, `SIGNAL_AUDIT_V2.md`, +> `agent-capability` evidence). Work happens **in-place in `dashboard-ui/scout`** (no rebuilds). +> Written 2026-06-16. + +## Guiding principles +- **Don't fake it; lock it.** Features that are real *later* are shown as a tasteful **"coming soon" + locked tile**, never with fabricated data. Nothing here is a lie. +- **Personas stay only where backed.** **Mira** = real (resume-builder parsed/tailored an existing + résumé) → keep. The bubble-journey **"Nova"** is being replaced wholesale (see Configure) → gone. +- Cut the three never-backed claims everywhere: **"live status", "in-network", invented deadlines.** + +--- + +## Summary tab (two states) + +### First-time view → **UNCHANGED** +Onboarding doorway (hero + 4-step "how it works" + CTA). Locked as-is — "812 live roles" reads as +*market scale*, not a corpus claim; "Mira" is backed. No edits. + +### Returning view (metrics dashboard) → leaned +- **KEEP (producible today):** Streak · Monthly report · Readiness·QX (qscore + assessment) · + Matches found · Top match · Salary band · Competition (avg applicants from listings) · Match range/ + distribution · Momentum / Readiness trend (qscore over time) · What your agents did (logged) · + **Application funnel** = `Matches → Viewed → Shortlisted → Applied`, **all user-activity** (Viewed = + jobs the user opened, Shortlisted = saved) · **Your Apply window** (freshness-based urgency from + listing posting-age, replaces generic "Golden window") · **Your rank** = activity percentile from + **Scout activity only** (searches/views/saves/applies in the matchmaking DB; "top X% by how much + you're working your search" — honest engagement, not a quality claim) · Search stage. +- **🔒 LOCK (coming soon):** Expert / human review. +- **CUT:** Search plan ("as you sleep") · Proof-ready · Market value · Response·tracked (no recruiter + callback) · Warm intros (no referral agent). + +--- + +## Configure tab → ~90% intact (the strongest screen) + +| Section | Decision | +|---|---| +| Hero "Here's the search I'd run for you" | ✅ keep | +| Profile summary ("parsed by **Mira**", title/seniority/industry/city/years) | ✅ keep — real (resume-builder + LinkedIn) | +| **"Matching you on N signals"** transparency bar | ✅ keep, **but fix the list** to match `SIGNAL_AUDIT_V2`: keep Role&level + Skills + What-you-value; **Market context is now BACKED via Perplexity** (keep); add verified-LinkedIn / QScore / course-behavior; drop any item with no source | +| "Also matching on · derived" (comp/stage/availability/deal-breakers/dream cos/hide-employer) | ✅ keep — real preference/filter inputs | +| **Video intro** | ✅ **KEEP** — reuse the existing `VideoResumeBuilder.tsx`; persist via interview-service's MinIO blob storage (record → store). Drop the "rides with in-network" copy | +| "What are you after?" (More like this / step up / new) | ✅ keep — **FIT vs STRETCH** core | +| "What matters most?" (pick 2) | ✅ keep — ranking weights | +| "How far should I reach?" (Safe/Balanced/Reach) | ✅ keep — reach band | +| "Work mode" | ✅ keep — hard filter | +| **The must-haves bubble box** ("Fine-tune everything") | ✅ **KEEP the box, swap the engine** → reuse course **`BubblePicker` + `useBubbleField`** (physics: grow-on-like +6 / love +12, live repack). **"Add your own" on EVERY row (crucial)** — esp. **Where/location** (tier-2/3 cities typed → real preferred-location bubble). **AI suggestions via `onGenerateMore` but STRIP the "ai" badge** — untagged, indistinguishable from presets. Replaces `ScoutBubbleJourney`/Nova | +| **Location** (the "Where" row, lives in the bubble box) | ✅ feeds the **ported location-fit logic** (`SIGNAL_AUDIT_V2`): substring city match, Remote→100, Anywhere→open. Add-your-own + that scoring = location finally handled right | +| Footer "Scout never applies without you" | ✅ keep — honest consent | + +**Net Configure:** keep almost everything; the only real changes are (1) fix the signal-transparency +list to the true set, (2) reuse `VideoResumeBuilder` for video-intro, (3) replace the bubble journey +with the course bubbler (add-your-own + untagged AI), (4) wire the ported location logic. + +--- + +## Action tab → leaned, with an honest apply flow + +**A) Loader** (`JobMatchingLoader`) — **rework copy to the on-demand truth:** *"Looking in 3+ job +boards · Scanning 100+ roles"* → normalize skills → score against your signals → rank. Drop the fake +"812 corpus scan." ("fresh by construction — pulled seconds ago.") + +**B) Filter bar** — ✅ keep (the hard filters + active chips + Clear all). + +**C) Match card** (`MatchCard`): +| Element | Decision | +|---|---| +| match % + **QX breakdown** | ✅ keep — but it IS **the engine's match breakdown**: the white-box signal dimensions (skill match, experience, **location-fit**, course-behavior, QScore competence, market context). "Why we matched you," powered by the real engine, not gamified QX | +| title · ₹pay · applicants | ✅ keep (from listings) | +| **"Closes in N days"** | ❌ **removed** → "posted Nd ago" (no real deadline data) | +| "+N QX on apply" · "Proof-ready" · "In-network · live status" | ❌ cut | +| "Video intro ✓" | ✅ keep (reuse Video Resume; drop in-network gating) | +| "Warm intro · can refer you" | ❌ cut (no mentor/referral agent) | +| "light on {gap} → 15-min mock / Expert mock / {course}" | ✅ keep (interview/roleplay/course are real) | +| "Why Scout picked this" + tags | ✅ keep — wire to the TRUE signals | + +**D) Swipe controls** (Pass / undo / info / Save) — ✅ keep. These emit the **VIEW / SAVE / PASS +events** that feed the funnel + activity rank. + +**E) Apply flow (the hand-off) — make it honest and concrete:** +1. **Approve once** — the single consent tap (keep). +2. **Per-job résumé tailoring** — for each saved role, resume-builder tailors the résumé + cover + letter. **NEW UX: a per-job spinner** ("Tailoring your résumé for {company}…") — this step had no UI. +3. **Auto-apply** — the master-key engine submits (tier-aware: public-ATS/India-ATS = autonomous; + login-gated = staged handoff). +4. **Result — split, with proof:** + - ✅ **Applied (auto):** show the **captured confirmation screenshot** per role — *verified, not + assumed* (the engine already screenshots the success page). + - ⚠️ **Couldn't auto-apply** (login_required / no_form / failed / spam-flagged): list these with + **"Apply manually →"** + the live link, so the user finishes them by hand. + +**Net Action:** real bones kept (match + engine breakdown + why + mock/course boosts + apply), the +recurring cuts applied (in-network/live-status, +QX framing, warm-intro/referral, add-to-pathway, +deadlines), loader made honest, and the apply flow gains the **resume-tailoring spinner** + **proof +screenshots** + **manual-apply fallback** for failures. + +--- + +## BUILD PROGRESS — as shipped on branch `job-ui-final` (updated 2026-06-17) + +Built in-place in `dashboard-ui/scout` (+ `components/job-matching`). Committed/pushed: +`job-ui-final` (head `780e09e`+). All states compile; `/scout` serves clean. The build went +**further than this spec in places** — recorded here so the spec stays the source of truth. +Honesty check: **every divergence is a cut or an improvement; nothing surfaces a claim the engine +can't back** (re-verified against `ENGINE_DESIGN.md`). + +**Configure (most changed):** +- ✅ Bubble box swapped to the ported course physics — `components/scout/bubble/{useBubbleField,ScoutBubbleField}`; like/love grow+repack, **add-your-own + untagged "Suggest more"** on taste rows. (`ScoutBubbleJourney` rewired off Nova/`NovaBubbles`.) +- ✅ Location upgraded **beyond spec**: cascading **Country→City** picker (`LocationPicker.tsx`, ported from the old matchmaking `FilterBar` — flags, searchable city, "select country first"). Stored as `"City · Country"`; the deck matcher now scores **city AND country** (`filterSortScoutJobs`). Feeds Tier-A `preferred_locations`. +- ✅ Profile card is now **editable in place** ("Edit details" ↔ "Done") + a **"Fine-tune" orange pill** (header) and **"Fine-tune everything"** (bottom) → the bubble journey. All dropdowns are one shared **searchable `CustomSelect`** (type-your-own → "Use 'X'") with an **"Any"** option that maps to *no constraint* for the engine. +- ✅ **No-resume / nothing-selected state** (NEW, not in original spec): card opens in edit mode, every field defaults to **"Any"**, headline "What are you looking for?", an **"Add a resume to pre-fill"** action (simulated Mira parse). This is the UI face of engine §3.3 (sparse profile → coverage renormalizes, never fabricates). Preview toggle to view both states. +- ⚠️ **DIVERGENCE — "Matching you on N signals" bar: REMOVED** (text-light pass), not "kept + fixed" as spec said. Signals still feed the engine; just not surfaced. *Re-add a 1-line honest version if we want the credibility back.* +- ⚠️ **DIVERGENCE — Video intro: REMOVED** from card + chip (Tier-C, not wired). Spec wanted keep. +- ⚠️ **DIVERGENCE — dream-companies + hide-employer editing dropped** (kept comp/stage/availability/deal-breakers as read-only chips → editable inline). +- ✅ Removed the lame **"Use my résumé instead"** shortcut everywhere. + +**Summary:** +- ✅ 9 headline cards (activity-led) + 3 behind "More metrics"; Expert/human review **locked "coming soon"** (Mira/Vera hand-offs stay live). +- ✅ Market signals trimmed to producible set (dropped layoff/funnel/hiring-velocity). +- ⚠️ **DIVERGENCE — Readiness·QX assessment card REMOVED** (+8 QX reward theatre gone). QScore/assessment remain real Tier-B engine signals; "Readiness trend" still in More-metrics. Spec wanted keep. +- ⚠️ **PENDING — "Golden apply window" not yet renamed** to the freshness/posting-age "Apply window" (§6.1). + +**Action / match:** +- ✅ Loader honest: `Looking across 3+ job boards · Scanning 100+ live roles · Scoring · Ranking`; **MATCH_SCOPE 812→120**. +- ✅ **Video-intro chip removed** from `MatchCard`. +- ✅ Swipe card **action-bar overlap fixed** — card now renders in normal flow (`AnimatePresence popLayout`), so expanding the QX breakdown grows the page and Pass/Undo/Info/Save stay below it (removed the brittle absolute+ResizeObserver container). +- ⚠️ **DIVERGENCE — Action-tab filter bar + chips + `ScoutFilterSheet`: REMOVED.** Configure is the single source of search criteria; empty deck → "Adjust in Configure". Spec wanted keep. +- 🔧 **PENDING (needs engine):** apply-flow proof screenshots + auto/failed→manual split; per-job resume-tailoring spinner. These wait on engine apply-result statuses. + +**Text/polish:** all accented "résumé" → plain **"resume"**; "Pick 2" casing. diff --git a/research/docs/SIGNAL_AUDIT_V2.md b/research/docs/SIGNAL_AUDIT_V2.md new file mode 100644 index 0000000..0babe7d --- /dev/null +++ b/research/docs/SIGNAL_AUDIT_V2.md @@ -0,0 +1,107 @@ +# Signal Audit v2 — Evidence-Based Input Inventory + +> Companion to `ENGINE_DESIGN.md`. Supersedes its §2/§5 signal audit. Written 2026-06-16, grounded +> in **live prod-DB evidence** (sampled over SSH) rather than the original conservative assumptions. +> Key correction: the v1 doc flagged interview/roleplay/assessment as "❌ GAP — build transport." +> The data is **live and queryable today** — richer engine is mostly *wiring + discipline*, not new builds. + +--- + +## 0. Governing discipline (unchanged — the v1-collapse guardrails) +More signals make the engine **stronger only if** we keep these rules. Breaking them is how v1 died. +1. **Trust ⟂ coverage.** A signal earns SCORE WEIGHT only if coverage is high enough that absence + isn't the norm. Otherwise it is an **ADDITIVE BOOSTER** (bonus when present, never a penalty when + absent). Most new signals are low-coverage → boosters, not weights. +2. **Filter degenerate data.** Verified live: the sampled interview review was a *no-show* (score 30), + the roleplay an *8-word low-talk*. These would poison the profile. Drop `no_show` / `low_talk` + sessions (the data already flags `exclude_from_trend: true` / `scoring_kind: no_show`). +3. **Coverage-aware renormalization** (`ENGINE_DESIGN.md` §3.3): score over *present* signals only; + a missing signal redistributes its weight, never flattens. + +## 1. The signal inventory (by tier) + +### Tier A — Backbone (high coverage → filter + weight) +| Signal | Source (verified) | Role | Flows in via | +|---|---|---|---| +| years_experience (date-derived) | resume-builder parse | weight | context-resolver | +| current_role / title | resume + verified LinkedIn | filter + weight | context-resolver | +| skills (→ ESCO normalized) | resume-builder parse | **core weight** (skill match) | context-resolver | +| **location + work mode** | user prefs (Configure must-haves) + user-service | **hard filter + 30% weight** | request prefs | +| target_roles / industries | Configure (user-stated *want*) | filter, not score | request prefs | + +> **Location logic — PORT from the old engine** (`matchmaking/app/engine/core.py`, it did this better +> than v1's one-liner). Bring forward `_score_location_fit` + `_location_filter_blocks_match`: +> remote job → 100 (0 if seeker is onsite-only); else **city+country substring match** vs preferred +> locations (so an add-your-own typed city "just works"); travel-mode adds travel locations; +> unknown → neutral 60. As a **hard gate**: block remote for onsite-only seekers, block no-fit jobs +> (clamp below threshold) unless the seeker set no location prefs ("Anywhere" = open). Weighted 30%. + +### Tier B — Verified enrichers (medium coverage → real weight when present) +| Signal | Source (verified) | Role | Notes | +|---|---|---|---| +| verified title / company / **industry** | **social-branding live LinkedIn fetch** | weight (present) | doc said "partial" — it's *live* | +| degree / field / institution | resume parse / LinkedIn education | weight (present) | | +| certifications | resume / LinkedIn | booster | | +| assessment scores | user-service (assessment tables) | booster | doc called it a "gap"; it exists | +| **QScore quotient vector (10-D)** | qscore-service | **competence proxy (weight, coverage-penalized)** | see §2 — this is the key aggregate | +| **Market context** (demand, hiring velocity, posting freshness, comp band) | **Perplexity API (online)** — TESTED 2026-06-16, returns sourced numbers + honest "can't-verify" on layoffs | enricher / context for ranking + the UI "why" | **cache per role+location** (latency/cost). Honest by design — Perplexity flags what it can't confirm (e.g. layoff signals). Replaces the fabricated market-intel that was in the UI signal list. | + +### Tier C — Behavioral / competence boosters (LOW coverage → ADDITIVE ONLY) +| Signal | Source (verified) | Role | Notes | +|---|---|---|---| +| **Course watch behavior** (watch-time, completion, interest profile) | `growqr_course` | **booster — target-role inference + FIT/STRETCH direction** | **NEW class. Highest-trust: revealed preference > stated.** "Completing React content → step toward frontend." | +| improvement velocity / coachability | interview/roleplay `trend_data`, `historical_comparison` | booster | growth trajectory | +| video on-camera presence | interview/roleplay `video_analysis` (Gemini 2.5 Pro) | booster | <1% relevant; rarely surfaced | + +> Interview 8-D rubric + roleplay EQ are **not listed as independent inputs** — see §2 (already folded into QScore). + +### Labels (training targets, NOT scoring inputs) +| Signal | Source | Use | +|---|---|---| +| SAVE / DISMISS / APPLY / RSVP feedback | old matchmaking (`Record Feedback`) + **our apply engine outcomes** | accrue labels → train pairwise/listwise ranker (`ENGINE_DESIGN.md` §3.1) | + +## 2. Entanglement: QScore already consumes interview/roleplay +**Do not double-count.** QScore is *consumed by* interview & roleplay, and the interview/roleplay +competence is *digested into* the QScore quotient vector. So for the engine: +- **v1 (recommended):** use the **QScore quotient vector as the competence proxy** — it already + incorporates the interview/roleplay rubric signals. This matches `ENGINE_DESIGN.md` §5: "launch on + READY rails (Tier A + QScore quotients), add direct rubric access later as a precision upgrade." +- **Later (precision upgrade):** read the **raw** interview/roleplay rubrics directly as boosters — + only once we've confirmed they add lift *beyond* the QScore aggregate (else it's redundant weight). + +This means course watch-behavior is the **one genuinely new, non-redundant signal** to wire in v1. + +## 3. Transport: mostly already solved +**The résumé + LinkedIn → service transport already exists** — interview & roleplay services already +receive résumé/LinkedIn context (they're résumé/LinkedIn-aware), assembled by the orchestrator's +`context_resolver` into `user_context`. **matchmaking-v2 reuses that exact pattern**: consume the +assembled `user_context` (resume + verified LinkedIn + QScore + assessment) the same way interview/ +roleplay do. So the rich user-side profile is a *wiring* job against a **proven, existing transport**, +not a new build. Course watch-behavior is the one extra read (from `growqr_course`). + +The user-side profile is then scored against the **on-demand fetched listings** (LinkedIn/Naukri/ +Indeed via Apify) — the job side stays on-demand; the *user* side is what these signals enrich. + +## 4. Cut / discard +- **Pathways (entire) — CUT.** Feature dropped (weak link: buggy skill-gap, impractical roles) AND + its questionnaire is **not used as a signal**. The Configure screen already captures stated prefs. +- Carry-over discards (`ENGINE_DESIGN.md` §2): ATS overall_score (resume quality ≠ fit), parser + confidence, vanity metrics, self-declared salary, **institution_tier** (pedigree bias). +- UI overclaims with **no backing signal**: "live status" (tap-time liveness unreliable, §6.1), + "in-network" (no referral agent), "Closes in N days" (no deadline data). + +## 5. Net: stronger engine, same discipline +v1 input set = **Backbone (resume skills/experience + Configure prefs/filters) + verified LinkedIn + +QScore quotient vector (competence proxy) + course watch-behavior (new, high-trust booster)**, fused +coverage-aware, scored against on-demand listings, with save/apply feedback accruing as labels. That +is materially richer than the doc's "Tier-A + QScore proxy" — and it's reachable now because the +transport exists and the discipline (boosters for sparse, filter degenerate) keeps it from collapsing. + +## Honesty ledger +| Now proven available (was assumed a gap) | Still must validate | +|---|---| +| Interview/roleplay competence (live in DB; via QScore) | Does raw rubric add lift beyond QScore? | +| Course watch-behavior as a signal | Coverage of active learners; interest→role mapping quality | +| Verified LinkedIn identity (live fetch) | India LinkedIn fetch reliability | +| Feedback labels exist (save/apply) | Volume sufficient to train a ranker (was 21 users) | +| Transport reuse (context_resolver) | matchmaking-v2 wiring into the mesh | diff --git a/research/poc/auto-apply/README.md b/research/poc/auto-apply/README.md new file mode 100644 index 0000000..cc103aa --- /dev/null +++ b/research/poc/auto-apply/README.md @@ -0,0 +1,35 @@ +# Auto-apply POC — "one tap at the end" + +Proves the heart of Scout's apply: **one tap → backend submits → confirmation → tracked**, +with the apply-router branching (hosted ATS submit vs prepared board handoff). + +## What it proves / doesn't +- ✅ End-to-end one-tap flow + the router that picks the mechanism + latency. +- ✅ Hosted-submit path posts a complete application to an ATS endpoint and gets a confirmation. +- ❌ Does **not** submit to a real employer. The ATS is a **mock** (`/mock-ats/...`). + +> **Hard rule:** never fire test applications at real employers' live postings. It pollutes real +> hiring pipelines and is abuse. POC-0 = mock ATS. POC-1 = our own ATS *sandbox* job. + +## Run +```bash +python3 -m venv venv && source venv/bin/activate +pip install fastapi uvicorn httpx +uvicorn app:app --port 8765 +# open http://localhost:8765 +``` + +## Prove it from the CLI +```bash +curl -s localhost:8765/api/kit/job_ats # kit staged & ready +curl -s -X POST localhost:8765/api/apply/job_ats # the one tap -> submitted + confirmation_id +curl -s -X POST localhost:8765/api/apply/job_board # prepared handoff -> deeplink + staged +curl -s localhost:8765/api/tracker # pipeline view +``` + +## POC-1 (real ATS sandbox) — what's needed from Rahul +1. Create a **free/trial ATS account** (Ashby has a clean `applicationForm.submit`; or + Greenhouse/Lever trial) and post a **test job** in it. +2. Get the submission endpoint + token/board-token for *that test job*. +3. Set `ATS_SUBMIT_URL` (and add auth) — the mock route is then deleted. Same flow, real ATS. + - Confirms: real field requirements, résumé upload, and that programmatic submit is accepted. diff --git a/research/poc/auto-apply/app.py b/research/poc/auto-apply/app.py new file mode 100644 index 0000000..e1b5d21 --- /dev/null +++ b/research/poc/auto-apply/app.py @@ -0,0 +1,233 @@ +""" +Auto-apply POC — proves the "one tap at the end" works end-to-end. + +What this proves (and what it does NOT): + ✅ One tap -> backend assembles the staged application kit -> apply-router picks the + right mechanism (hosted ATS submit vs prepared board handoff) -> submits -> confirmation + -> tracked, with measured latency. + ✅ The apply-router BRANCHING that is the architectural heart of the no-install web app. + ❌ It does NOT submit to a real employer. The "ATS" here is a MOCK we control + (route: /mock-ats/...). POC-1 swaps ATS_SUBMIT_URL for a real ATS *sandbox* job. + +HARD RULE: never fire test applications at real employers' live postings. + +Run: + pip install fastapi uvicorn httpx + uvicorn app:app --port 8765 + open http://localhost:8765 +""" +from __future__ import annotations + +import os +import time +import uuid +from typing import Literal + +import httpx +from fastapi import FastAPI, HTTPException +from fastapi.responses import HTMLResponse +from pydantic import BaseModel + +app = FastAPI(title="Scout auto-apply POC") + +# Swap this for a real ATS sandbox endpoint in POC-1. Default = our local mock. +ATS_SUBMIT_URL = os.environ.get( + "ATS_SUBMIT_URL", "http://localhost:8765/mock-ats/greenhouse/acme-test/eng-42" +) + +# --------------------------------------------------------------------------- +# Demo data: two jobs, one routed to an ATS (hosted submit), one to a board +# (prepared handoff). In production these come from the on-demand listings fetch. +# --------------------------------------------------------------------------- +JOBS = { + "job_ats": { + "title": "Product Analyst", + "company": "Acme (Greenhouse)", + "source": "ats", # -> hosted submit + "ats_vendor": "greenhouse", + }, + "job_board": { + "title": "Associate PM", + "company": "Globex (Naukri)", + "source": "board", # -> prepared handoff + "board": "naukri", + "apply_url": "https://www.naukri.com/job-listings-associate-pm-globex", + }, +} + +# The "application kit" — staged BEFORE the tap. In production: tailored résumé from +# resume-builder + field answers from the user's profile. +KIT = { + "candidate": { + "first_name": "Asha", + "last_name": "Rao", + "email": "asha.rao@example.com", + "phone": "+91-90000-00000", + }, + "resume_url": "https://files.growqr.ai/resumes/asha-rao-tailored.pdf", + "answers": { + "years_experience": "3", + "work_authorization": "India - citizen", + "notice_period_days": "30", + }, +} + +# In-memory application tracker (the "pipeline" view). +TRACKER: list[dict] = [] + + +class ApplyResult(BaseModel): + status: Literal["submitted", "prepared"] + mechanism: Literal["hosted_ats", "board_handoff"] + job_id: str + confirmation_id: str | None = None + deeplink: str | None = None + staged: dict | None = None + latency_ms: int + + +@app.get("/api/kit/{job_id}") +def get_kit(job_id: str): + """The frontend shows 'kit ready' before the tap.""" + if job_id not in JOBS: + raise HTTPException(404, "unknown job") + job = JOBS[job_id] + return { + "job": {"id": job_id, "title": job["title"], "company": job["company"]}, + "ready": { + "resume": True, + "cover_letter": True, + "fields_filled": len(KIT["answers"]), + }, + "mechanism": "hosted_ats" if job["source"] == "ats" else "board_handoff", + } + + +@app.post("/api/apply/{job_id}", response_model=ApplyResult) +async def apply(job_id: str): + """THE ONE TAP. Routes to hosted submit or prepared handoff.""" + if job_id not in JOBS: + raise HTTPException(404, "unknown job") + job = JOBS[job_id] + t0 = time.perf_counter() + + if job["source"] == "ats": + # Hosted submit: backend POSTs the full application to the ATS API. + payload = { + "first_name": KIT["candidate"]["first_name"], + "last_name": KIT["candidate"]["last_name"], + "email": KIT["candidate"]["email"], + "phone": KIT["candidate"]["phone"], + "resume_url": KIT["resume_url"], + "answers": KIT["answers"], + } + async with httpx.AsyncClient(timeout=30) as client: + resp = await client.post(ATS_SUBMIT_URL, json=payload) + if resp.status_code >= 400: + raise HTTPException(502, f"ATS rejected: {resp.status_code} {resp.text}") + confirmation_id = resp.json().get("confirmation_id") + latency_ms = int((time.perf_counter() - t0) * 1000) + result = ApplyResult( + status="submitted", + mechanism="hosted_ats", + job_id=job_id, + confirmation_id=confirmation_id, + latency_ms=latency_ms, + ) + else: + # Prepared handoff: stage everything, hand the user a deep link to the board. + latency_ms = int((time.perf_counter() - t0) * 1000) + result = ApplyResult( + status="prepared", + mechanism="board_handoff", + job_id=job_id, + deeplink=job["apply_url"], + staged={"resume_url": KIT["resume_url"], "answers": KIT["answers"]}, + latency_ms=latency_ms, + ) + + TRACKER.append( + { + "job_id": job_id, + "company": job["company"], + "status": result.status, + "confirmation_id": result.confirmation_id, + } + ) + return result + + +@app.get("/api/tracker") +def tracker(): + return {"applications": TRACKER} + + +# --------------------------------------------------------------------------- +# MOCK ATS — stands in for Greenhouse/Lever/Ashby. POC-1 deletes this and points +# ATS_SUBMIT_URL at a real *sandbox* job. The contract mirrors Greenhouse's job-board +# application submission shape (first/last/email/phone/resume + question answers). +# --------------------------------------------------------------------------- +@app.post("/mock-ats/greenhouse/{board}/{job}") +def mock_ats_submit(board: str, job: str, payload: dict): + required = ["first_name", "last_name", "email", "resume_url"] + missing = [f for f in required if not payload.get(f)] + if missing: + raise HTTPException(422, f"missing required fields: {missing}") + return { + "accepted": True, + "board": board, + "job": job, + "confirmation_id": f"GH-{uuid.uuid4().hex[:10]}", + } + + +@app.get("/", response_class=HTMLResponse) +def index(): + return INDEX_HTML + + +INDEX_HTML = """ +Scout — one-tap apply POC + +

🕵️ Scout — one-tap apply

+

Everything is staged before the tap. The tap ends the application.

+
+ +""" diff --git a/research/poc/auto-apply/scout_demo.py b/research/poc/auto-apply/scout_demo.py new file mode 100644 index 0000000..e787da5 --- /dev/null +++ b/research/poc/auto-apply/scout_demo.py @@ -0,0 +1,192 @@ +""" +Scout — personalized one-tap apply DEMO, now with a LIVENESS GATE. + +The lesson from the first run: job postings expire fast (we checked 3 recent AI-Architect +postings live — 2 were already closed). A perfect match for a DEAD job is worse than no match. + +So this version re-checks the posting is still accepting applications AT TAP-TIME: + • still open -> prepared handoff: open the real posting with materials staged (you submit). + • just closed -> Scout catches it and recovers to FRESH, currently-open matches. + It never sends you to a closed door. + +LinkedIn = a board, so Scout never auto-submits — it preps everything and you give the final tap. + +Run: + source venv/bin/activate + uvicorn scout_demo:app --port 8765 + open http://localhost:8765 +""" +from __future__ import annotations +import re +import httpx +from fastapi import FastAPI +from fastapi.responses import HTMLResponse, JSONResponse + +app = FastAPI(title="Scout demo — Rahul") + +# The matched posting (may be expired — that's the point; we re-check live). +JOB_URL = "https://www.linkedin.com/jobs/view/ai-testing-architect-remote-at-teksystems-4418606196" +# Always-current fallback: live LinkedIn search for fresh AI Architect roles (remote). +LIVE_SEARCH = "https://www.linkedin.com/jobs/search/?keywords=AI%20Architect&f_WT=2&sortBy=DD" + +COVER_LETTER = """Dear Hiring Team, + +I build production AI systems and the verification that keeps them honest — exactly the +intersection an AI Architect role sits at. + +Most recently I architected a 12-agent, event-driven platform (FastAPI, Redis Streams, an +agent-to-agent protocol) with LLMs and a pgvector retrieval layer in production on AWS. To +trust it, I built an adversarial verification harness: multi-model panels that fan out, fetch +evidence, and vote to refute claims before they ship — AI-enabled QA in practice, with +explainability and responsible-use guardrails built in. + +My stack — Python, FastAPI, LLM/GenAI, vector databases, CI/CD, and AWS/Azure/GCP — lines up +directly with the requirements, and I've shipped and mentored on all of it. + +Warm regards, +Rahul Gupta +rahulgupta2608@icloud.com""" + + +@app.post("/api/apply") +async def apply(): + """ + Lesson learned the hard way: individual posting permalinks rot within days (we checked ~7 + real AI-Architect postings — closed, 404, or redirect-to-search), and LinkedIn bot-walls + make tap-time liveness unverifiable server-side. So the honest handoff targets the LIVE, + newest-first results for the role (always current) with materials tailored + staged. + """ + return JSONResponse({ + "mechanism": "board_handoff", + "fresh_search": LIVE_SEARCH, + "cover_letter": COVER_LETTER, + "qx": 8, + }) + + +@app.get("/", response_class=HTMLResponse) +def index(): + return HTML + + +HTML = r""" +Scout · one-tap apply +
+ +
🕵️
Scout
your opportunity agent
+

Evening, Rahul. I found a role built for you.

+
Scanned LinkedIn · 1 strong match · already tailored
+ +
+
+
TEKSYSTEMS
AI Testing Architect
+
🌐 RemoteMid–Senior$50–100/hr
+
+ + + 91%
+ +

Why Scout picked this for you

+
✓Your FastAPI + LLM + vector-DB stack matches their core requirements.
+
✓You've built adversarial AI verification — that is AI-enabled testing.
+
✓Governance & explainability you already practice map to their standards mandate.
+
+ +
+
✓ Résumé tailored to this role
+
✓ Cover letter written for the role
+
✓ Contact & screening answers filled
+ Preview your cover letter ↓ +
+
+ + +
+
Tailoring your final résumé…
+
📎 Staging résumé + cover letter…
+
🔄 Pulling today's open roles…
+
+ +
+
✅

Tailored & staged

+

Your résumé + cover letter are ready. Here are today's open AI Architect roles — + freshly pulled, newest first. Your tailored application drops straight in.

+ +8 QX earned + See today's open AI Architect roles → +
+
+ +
🔒 Scout never submits on a board for you — it preps everything and re-checks the +role is still open, then you give the final tap. Safe for your account, and no dead ends.
+ + +
""".replace("__COVER__", COVER_LETTER) diff --git a/research/poc/browser-apply/.gitignore b/research/poc/browser-apply/.gitignore new file mode 100644 index 0000000..ab54274 --- /dev/null +++ b/research/poc/browser-apply/.gitignore @@ -0,0 +1,4 @@ +.env +venv/ +__pycache__/ +*.pyc diff --git a/research/poc/browser-apply/README.md b/research/poc/browser-apply/README.md new file mode 100644 index 0000000..87a1277 --- /dev/null +++ b/research/poc/browser-apply/README.md @@ -0,0 +1,77 @@ +# browser-apply — browser-use (open source) POC + +The pivot away from heuristics. Instead of mocking an apply flow, an **AI agent drives a +real Chrome browser** — it looks at the page and clicks/types like a human. No cross-origin +wall, because it *is* the browser. + +## The moving parts (what's actually running) + +``` + your task (plain English) + │ + ▼ + ┌─────────┐ "what do I click next?" ┌──────────────┐ + │ Agent │ ───────────────────────────────► │ OpenAI LLM │ + │(browser-│ ◄─────────────────────────────── │ (the brain) │ + │ use) │ "click element #3, type X" └──────────────┘ + │ + │ Chrome DevTools Protocol (cdp-use) + ▼ + ┌──────────────┐ + │ real Chrome │ ← navigates, reads the page, clicks, types, uploads + └──────────────┘ +``` + +- **browser-use** (v0.13.1) — the library. Runs the loop: screenshot/read page → ask the + LLM what to do → do it → repeat until the task is done. +- **The LLM** (OpenAI here) — the decision-maker. Sees the page, decides the next action. + It's swappable: OpenAI, Anthropic, Google, or local (Ollama). +- **Chrome** — your installed Google Chrome, driven over the DevTools Protocol. No separate + browser download needed. browser-use auto-adds ad/cookie-blocker extensions. + +## Setup (already done once) + +```bash +python3 -m venv venv # Python >= 3.11 (you have 3.13.2) +source venv/bin/activate +pip install browser-use # pulls openai/anthropic/google SDKs + cdp-use +# Chrome is reused from /Applications — nothing else to install +``` + +Secrets live in `.env` (git-ignored): +``` +OPENAI_API_KEY=sk-... +``` + +## Run the smoke test + +```bash +source venv/bin/activate +python verify.py # headless (no window) +HEADLESS=0 python verify.py # SEE Chrome open and the agent drive it +``` +Expected: it visits example.com and prints `The main heading on the page is 'Example Domain'.` + +## What's proven so far +- ✅ The full stack works: browser launches, LLM reasons, agent reads a live page (`verify.py`). +- ✅ Vision résumé parsing works on the real `Resume.pdf` (`parse_resume.py`). + +## Real job-apply flow (built) +Applies to one real posting: LinkedIn job → hand-off to OpenAI's official job board → fill & submit. + +| File | Role | +|---|---| +| `parse_resume.py` | Vision model (gpt-4o-mini + PyMuPDF) → structured `profile` from `Resume.pdf`. Mirrors resume-builder's `parse_service.py`. | +| `browser_config.py` | Shared `BrowserProfile` (dedicated `chrome-profile/`, real Chrome, visible) so login persists into apply. | +| `login_once.py` | One-time: open LinkedIn, you log in yourself, session saved. | +| `apply.py` | Parses résumé → drives the LinkedIn→OpenAI-board apply → auto-submits → structured result. | + +```bash +source venv/bin/activate +python login_once.py # one time: log into LinkedIn in the window +python apply.py # parses résumé, applies, auto-submits +AUTO_SUBMIT=0 python apply.py # fill everything but STOP before submit (human review) +``` + +Honesty rules kept: a **liveness gate** (dead posting → report, never fake a submit), résumé is +the source of truth (no invented data), and auto-submit is one flag away from review-first. diff --git a/research/poc/browser-apply/Resume.pdf b/research/poc/browser-apply/Resume.pdf new file mode 100644 index 0000000..b29d088 Binary files /dev/null and b/research/poc/browser-apply/Resume.pdf differ diff --git a/research/poc/browser-apply/action_shot.py b/research/poc/browser-apply/action_shot.py new file mode 100644 index 0000000..78f3b6a --- /dev/null +++ b/research/poc/browser-apply/action_shot.py @@ -0,0 +1,20 @@ +import asyncio +from browser_use import BrowserSession +from browser_config import make_profile +async def ev(s,e): + c=await s.get_or_create_cdp_session() + r=await c.cdp_client.send.Runtime.evaluate(params={"expression":e,"returnByValue":True},session_id=c.session_id) + return r.get("result",{}).get("value") +async def click(s,t): + return await ev(s,f"""(()=>{{const el=[...document.querySelectorAll('button,[role=tab]')].find(e=>(e.textContent||'').trim().toLowerCase().includes('{t.lower()}'));if(el){{el.click();return true}}return false}})()""") +async def main(): + s=BrowserSession(browser_profile=make_profile(headless=True,persist=False)); await s.start() + try: await s._cdp_set_viewport(width=1340,height=1080) + except: pass + await ev(s,"try{localStorage.removeItem('scout-onboarded')}catch(e){}") + await s.navigate_to("http://localhost:3001/scout"); await asyncio.sleep(4) + await click(s,"Configure"); await asyncio.sleep(1) + await click(s,"Find my matches"); await asyncio.sleep(8) + await s.take_screenshot(path="/tmp/action_deck.png",full_page=True); print("deck") + await s.kill() +asyncio.run(main()) diff --git a/research/poc/browser-apply/adapters/__init__.py b/research/poc/browser-apply/adapters/__init__.py new file mode 100644 index 0000000..4bf7e0d --- /dev/null +++ b/research/poc/browser-apply/adapters/__init__.py @@ -0,0 +1,23 @@ +""" +Adapter registry. pick_adapter(url) returns the first specialized adapter that claims the URL, +else the GenericAdapter (the master key's catch-all). Add new specialized adapters to _SPECIALIZED. +""" +from __future__ import annotations + +from .base import Adapter +from .generic import GenericAdapter +from .ashby import AshbyAdapter + +# Specialized adapters, checked in order. Generic is the fallback. +_SPECIALIZED: list[Adapter] = [ + AshbyAdapter(), + # GreenhouseAdapter(), LeverAdapter(), WorkdayAdapter(), ... ← grow here +] +_GENERIC = GenericAdapter() + + +def pick_adapter(url: str) -> Adapter: + for a in _SPECIALIZED: + if a.matches(url): + return a + return _GENERIC diff --git a/research/poc/browser-apply/adapters/ashby.py b/research/poc/browser-apply/adapters/ashby.py new file mode 100644 index 0000000..47294f2 --- /dev/null +++ b/research/poc/browser-apply/adapters/ashby.py @@ -0,0 +1,28 @@ +""" +AshbyAdapter — the hard-won Ashby knowledge from apply_ashby.py, as a thin hint layer. +This is how the Ashby progress is preserved without duplicating the engine. +""" +from __future__ import annotations + +from .base import Adapter + + +class AshbyAdapter(Adapter): + name = "ashby" + tier = "tier1-autonomous" + + def matches(self, url: str) -> bool: + return "ashbyhq.com" in (url or "").lower() + + def hints(self) -> str: + return ( + "BOARD = AshbyHQ (public form, no login needed). Known quirks:\n" + "- There is an 'Autofill from resume' control near the top — upload the résumé THERE\n" + " first to auto-populate fields, then verify the required 'Resume' field shows the\n" + " file attached (filename visible, no spinner).\n" + "- The required consent checkbox reads 'I confirm I have read the above.' (no asterisk\n" + " but REQUIRED) — tick it and verify it is checked.\n" + "- 'Where are you currently located?' is a typeahead: type the city, wait, click the\n" + " matching suggestion (typing alone does not commit).\n" + "- The page is a JS app; it can render blank for a few seconds on load." + ) diff --git a/research/poc/browser-apply/adapters/base.py b/research/poc/browser-apply/adapters/base.py new file mode 100644 index 0000000..5195b36 --- /dev/null +++ b/research/poc/browser-apply/adapters/base.py @@ -0,0 +1,20 @@ +""" +Adapter contract. An adapter is a THIN layer of board-specific knowledge — it does NOT +re-implement the apply loop. The generic engine does all the work; an adapter only: + - matches(url): claims a URL by its domain/ATS, and + - hints(): returns extra prompt text about that board's known quirks (controls, field names). +The GenericAdapter returns no hints — pure vision. Specialized adapters (Ashby, later Greenhouse/ +Lever/Workday) add hints that raise accuracy on that board. +""" +from __future__ import annotations + + +class Adapter: + name = "base" + tier = "unknown" # tier1-autonomous | account-gated | generic + + def matches(self, url: str) -> bool: + return False + + def hints(self) -> str: + return "" diff --git a/research/poc/browser-apply/adapters/generic.py b/research/poc/browser-apply/adapters/generic.py new file mode 100644 index 0000000..1151ef7 --- /dev/null +++ b/research/poc/browser-apply/adapters/generic.py @@ -0,0 +1,23 @@ +""" +GenericAdapter — the master key's default. Matches ANY URL, adds no board-specific assumptions. +This is what handles the long tail (bespoke company career pages, unknown ATS) purely by vision. +""" +from __future__ import annotations + +from .base import Adapter + + +class GenericAdapter(Adapter): + name = "generic" + tier = "generic" + + def matches(self, url: str) -> bool: + return True # catch-all fallback + + def hints(self) -> str: + # No board assumptions. The engine's universal logic does everything. + return ("BOARD = unknown / bespoke. No known shortcuts — read the page carefully. Look for " + "any 'autofill from resume' control and use it if present; otherwise fill fields " + "manually. Watch for a login/account wall (report login_required) or a page with no " + "real application form, e.g. a generic /careers page or 'email your CV' (report " + "no_form).") diff --git a/research/poc/browser-apply/apify_probe.py b/research/poc/browser-apply/apify_probe.py new file mode 100644 index 0000000..19d0747 --- /dev/null +++ b/research/poc/browser-apply/apify_probe.py @@ -0,0 +1,149 @@ +""" +apify_probe.py — measure OFFSITE-APPLY-LINK YIELD across job-board actors. + +Question: for an on-demand user search, how many jobs come back with a usable OFFSITE apply URL +(the employer's own ATS — Ashby/Greenhouse/Lever/Workday/...), i.e. the Tier-1 links our +autonomous apply engine can actually finish? We probe each actor with a small sample and report: + total jobs · #with offsite link · #Tier-1 ATS · breakdown by ATS. + +Small samples on purpose (this costs Apify credits). Tune SAMPLE below. + +Run: python apify_probe.py +""" +from __future__ import annotations + +import os +import json +from collections import Counter +from urllib.parse import urlparse + +import httpx +from dotenv import load_dotenv + +load_dotenv() +TOKEN = os.environ["APIFY_TOKEN"] +SAMPLE = int(os.environ.get("SAMPLE", "25")) +ONLY = os.environ.get("ONLY", "").lower() # run only actors whose label matches +FOLLOW = os.environ.get("FOLLOW_REDIRECTS", "0") == "1" # Indeed: resolve externalApplyLink + +# role/geo to probe — tweak freely +TITLE = "Machine Learning Engineer" +COUNTRY = "IN" +LOCATION = "India" + +# ATS domain -> (label, is_tier1_autonomous) +ATS = { + "ashbyhq.com": ("Ashby", True), "jobs.ashbyhq.com": ("Ashby", True), + "greenhouse.io": ("Greenhouse", True), "boards.greenhouse.io": ("Greenhouse", True), + "lever.co": ("Lever", True), "jobs.lever.co": ("Lever", True), + "smartrecruiters.com": ("SmartRecruiters", True), "recruitee.com": ("Recruitee", True), + "personio.de": ("Personio", True), "personio.com": ("Personio", True), + "workable.com": ("Workable", True), "breezy.hr": ("Breezy", True), + "myworkdayjobs.com": ("Workday", False), "icims.com": ("iCIMS", False), + "taleo.net": ("Taleo", False), "successfactors.com": ("SuccessFactors", False), + "bamboohr.com": ("BambooHR", False), "jobvite.com": ("Jobvite", False), +} + + +def classify(url: str | None): + """Return (ats_label, is_tier1) for an offsite URL, or (None, False) if no/empty url.""" + if not url or not isinstance(url, str) or not url.startswith("http"): + return None, False + host = (urlparse(url).hostname or "").lower() + for dom, (label, t1) in ATS.items(): + if host == dom or host.endswith("." + dom) or dom in host: + return label, t1 + return "other-ATS/site", False # an offsite link, just not a known ATS + + +def offsite_url(kind: str, item: dict) -> str | None: + if kind == "linkedin": + am = item.get("applyMethod") or {} + return am.get("companyApplyUrl") or item.get("companyApplyUrl") + if kind == "indeed": + return item.get("externalApplyLink") or item.get("applyUrl") + if kind == "ats": + return item.get("apply_url") or item.get("applyUrl") or item.get("url") + if kind == "naukri": + # applyRedirectUrl = the true external handoff. companyApplyUrl is Naukri's OWN apply API + # (naukri.com/cloudgateway-apply/...), i.e. on-platform — not an external link. + url = item.get("applyRedirectUrl") or item.get("companyApplyUrl") + if url and "naukri.com" in url: + return None + return url + return None + + +ACTORS = [ + {"label": "LinkedIn (harvestapi)", "kind": "linkedin", + "id": "harvestapi~linkedin-job-search", + "input": {"jobTitles": [TITLE], "locations": [LOCATION], "maxItems": SAMPLE, "sortBy": "date"}}, + {"label": "Indeed (misceres)", "kind": "indeed", + "id": "misceres~indeed-scraper", + "input": {"position": TITLE, "location": LOCATION, "country": COUNTRY, + "maxItemsPerSearch": SAMPLE, "followApplyRedirects": FOLLOW, + "saveOnlyUniqueItems": True}}, + {"label": "ATS-direct (bovi G/L/Ashby)", "kind": "ats", + "id": "bovi~greenhouse-lever-ashby-job-scraper", + "input": {"presetLists": ["ai-ml"], "maxJobsPerCompany": 2, "outputProfile": "compact"}}, + # muhammetakkurtt: 11k users, no full-permission gate, returns applyRedirectUrl (external) + # + companyApplyUrl (Naukri-internal) + companyApplyJob flag. Uses maxJobs (NOT maxItems). + # maxJobs has a min of 50; cities wants numeric city IDs (omit -> nationwide). + {"label": "Naukri (muhammetakkurtt)", "kind": "naukri", + "id": "muhammetakkurtt~naukri-job-scraper", + "input": {"keyword": TITLE, "maxJobs": max(50, SAMPLE)}}, +] + + +def run_actor(actor: dict) -> list[dict]: + url = f"https://api.apify.com/v2/acts/{actor['id']}/run-sync-get-dataset-items?token={TOKEN}" + r = httpx.post(url, json=actor["input"], timeout=300) + r.raise_for_status() + data = r.json() + return data if isinstance(data, list) else data.get("items", []) + + +def main(): + print(f"Probe: '{TITLE}' / {LOCATION} · sample≈{SAMPLE} per actor\n" + "=" * 68) + grand = {"jobs": 0, "offsite": 0, "tier1": 0} + grand_ats = Counter() + + for a in ACTORS: + if ONLY and ONLY not in a["label"].lower(): + continue + print(f"\n▶ {a['label']} [{a['id']}]") + try: + items = run_actor(a) + except Exception as e: + print(f" ✗ failed: {type(e).__name__}: {str(e)[:160]}") + continue + n = len(items) + ats_counter = Counter() + offsite = tier1 = 0 + for it in items: + label, t1 = classify(offsite_url(a["kind"], it)) + if label: + offsite += 1 + ats_counter[label] += 1 + if t1: + tier1 += 1 + grand["jobs"] += n + grand["offsite"] += offsite + grand["tier1"] += tier1 + grand_ats.update(ats_counter) + pct = (100 * offsite / n) if n else 0 + t1pct = (100 * tier1 / n) if n else 0 + print(f" jobs={n} offsite-link={offsite} ({pct:.0f}%) Tier-1 autonomous={tier1} ({t1pct:.0f}%)") + if ats_counter: + print(" breakdown:", ", ".join(f"{k}={v}" for k, v in ats_counter.most_common())) + + print("\n" + "=" * 68) + j, o, t = grand["jobs"], grand["offsite"], grand["tier1"] + print(f"COMBINED: jobs={j} offsite={o} ({(100*o/j if j else 0):.0f}%) " + f"Tier-1 autonomous={t} ({(100*t/j if j else 0):.0f}%)") + if grand_ats: + print("ATS mix:", ", ".join(f"{k}={v}" for k, v in grand_ats.most_common())) + + +if __name__ == "__main__": + main() diff --git a/research/poc/browser-apply/apply.py b/research/poc/browser-apply/apply.py new file mode 100644 index 0000000..4fdd8d1 --- /dev/null +++ b/research/poc/browser-apply/apply.py @@ -0,0 +1,192 @@ +""" +apply.py — apply to one real job with a browser-use agent. + +Flow: + 1. Parse Resume.pdf with a vision model -> structured `profile` (source of truth). + 2. Open the LinkedIn job in the dedicated (logged-in) Chrome profile. + 3. Liveness gate: if the posting is closed/expired, STOP and report — never fake a submit. + 4. Click Apply -> follow the hand-off to OpenAI's official job board (new tab). + 5. Upload Resume.pdf, fill every field from `profile`, answer screening questions truthfully. + 6. Submit (auto, per Rahul's choice) and report a structured result. + +Run (after `python login_once.py`): + python apply.py + +Toggle AUTO_SUBMIT=0 to make the agent fill everything but STOP before the final submit. +""" +from __future__ import annotations + +import asyncio +import json +import os + +from dotenv import load_dotenv + +from browser_use import Agent, BrowserSession, ChatOpenAI +from browser_config import make_profile, RESUME_PATH +from parse_resume import parse_resume_to_profile + +load_dotenv() + +JOB_URL = "https://www.linkedin.com/jobs/view/4418852767/" +APPLY_MODEL = os.environ.get("APPLY_MODEL", "gpt-4.1") # strong model; a real form needs reasoning +AUTO_SUBMIT = os.environ.get("AUTO_SUBMIT", "1") != "0" +MAX_STEPS = int(os.environ.get("MAX_STEPS", "55")) + +# Credentials are referenced by PLACEHOLDER NAMES only. browser-use injects the real values +# from .env at the keystroke, masks them in logs/screenshots, and scopes them to linkedin.com — +# the LLM never sees the real password. +LINKEDIN_USER = os.environ.get("LINKEDIN_USER", "") +LINKEDIN_PASS = os.environ.get("LINKEDIN_PASS", "") +SENSITIVE_DATA = { + "https://*.linkedin.com": { + "linkedin_user": LINKEDIN_USER, + "linkedin_pass": LINKEDIN_PASS, + } +} if (LINKEDIN_USER and LINKEDIN_PASS) else None + + +def build_task(profile: dict) -> str: + submit_clause = ( + "Submit the application (click the final Submit / Apply button)." + if AUTO_SUBMIT else + "Fill everything but DO NOT click the final submit button — stop at the review step " + "and report that it is ready for human review." + ) + login_clause = ( + "0. LOGIN: Go to https://www.linkedin.com/feed/. If you are already logged in, continue. " + "If you see a login form, log in using the placeholders linkedin_user and linkedin_pass " + "(type them into the email and password fields and submit). If LinkedIn shows a security " + "checkpoint, CAPTCHA, or 'is this you' verification code, STOP and report status=\"blocked\" " + "with notes saying a human needs to clear the checkpoint — do NOT attempt to bypass it.\n" + if SENSITIVE_DATA else + "0. LOGIN: Assume you are already logged into LinkedIn (a session was set up beforehand).\n" + ) + return f"""You are applying to ONE job on behalf of the candidate below. Use ONLY the +candidate's real data — never invent qualifications, employers, dates, or work authorization. + +CANDIDATE PROFILE (the source of truth, parsed from their résumé): +{json.dumps(profile, indent=2, ensure_ascii=False)} + +Résumé file to upload when asked: {RESUME_PATH} + +CONFIRMED FACTS (use these EXACT answers — they OVERRIDE any guess from the résumé): +- Preferred contact email: rahul.gupta2608@yahoo.com (the résumé email). +- The candidate lives in India and is an Indian citizen — authorized to work in India. +- The candidate is NOT authorized to work in the United States, and WOULD require visa + sponsorship for any US-based role. Do not overstate authorization — honesty over optimism. +- So answer work-authorization / sponsorship questions truthfully based on the job's actual + location as shown on the form: + * "Authorized to work in the country where the job is located?" -> US role: NO. India role: YES. + * "Will you require visa sponsorship?" -> US role: YES. India role: NO. +- Availability / start date: can start as soon as possible. If a date is required, pick the + earliest selectable date; if it is a text field, enter "Immediately available". Never leave a + required start-date field blank. + +STEPS: +{login_clause}1. Go to {JOB_URL}. +2. LIVENESS CHECK FIRST. If the page shows "No longer accepting applications", is a 404, or + redirects to a generic job search, then the job is DEAD: STOP immediately, do not apply, and + in your final answer report status="closed" with what you saw. A real apply to a dead job is + worse than no apply. +3. If the job is open, click the Apply button. This job hands off to OPENAI'S OFFICIAL JOB BOARD + (an external AshbyHQ careers site), usually opening a NEW TAB. Switch to that new tab. + IMPORTANT: the Ashby page is a JavaScript app that takes several seconds to render — it may + look BLANK at first. Do NOT conclude it failed on the first look: `wait` ~6 seconds and re-read + the page; if still blank, `wait` again and refresh the tab ONCE. Only report a load failure if + it is still blank after two waits AND a refresh. (If instead it opens LinkedIn "Easy Apply" + inline, handle that form. If it dead-ends or demands an account you cannot create, STOP and + report.) +4. RÉSUMÉ UPLOAD — do this FIRST and treat it as the make-or-break step (a half-finished upload + is what got the submission flagged as spam before). Follow this exactly: + a. Find the résumé / CV upload control and upload the file at the path above. + b. IMMEDIATELY use the `wait` action for ~8 seconds. Ashby uploads the file in the BACKGROUND + after you select it; the form is not valid until that finishes. + c. VERIFY the upload actually completed using CONCRETE signals (not a guess): + - the résumé FILENAME is displayed in the upload area, AND + - NO spinner / progress bar / "uploading" / "we're updating your application" text remains, AND + - ideally, Ashby has auto-parsed the résumé and pre-filled some fields (name / email / + experience) — that is strong proof the file reached their server. + d. If those signals are NOT all present, the upload FAILED or is pending: remove the file if + possible, re-upload, `wait` ~8s again, and re-verify. Retry up to 2 more times. + e. Only once the upload is CONFIRMED, continue. Remember whether it was confirmed for the + final report (resume_upload_confirmed). +5. Now fill the rest of the form top to bottom, never skipping a REQUIRED field. Two fields are + tricky widgets — fill them so the value actually COMMITS, then visually confirm it stuck: + - "Where are you currently located?" is a TYPEAHEAD: type the city (e.g. "New Delhi"), WAIT for + the suggestion dropdown, and CLICK the matching option. Typing alone does NOT commit — verify + the field now shows the selected location, not the "Start typing…" placeholder. + - "When can you start a new role?" is a DATE PICKER: open it, click an actual date (earliest + selectable), and verify the field now shows a date, not "Pick date…". + - Also fill name, the yahoo email, phone, LinkedIn/portfolio links from the profile, and any + experience/skills questions. +6. Screening questions: answer truthfully from the profile and CONFIRMED FACTS. For anything not + covered, choose the most truthful/neutral valid option, leave optional ones blank, and note it. + NEVER fabricate. Tick any required certification/consent checkbox. +7. Before submitting, re-scan the whole form: fix any empty REQUIRED field or validation error, + and CONFIRM the résumé upload is still showing as complete (not reverted to "uploading"). +8. {submit_clause} +9. AFTER clicking submit, VERIFY — never assume it worked. IMPORTANT: click Submit only ONCE, then + `wait` ~5s and read the page. Do NOT rapid-click Submit repeatedly — repeated clicks themselves + look like spam. + - If it says files are still uploading / "try again when finished": the upload was not done. + Go back, re-verify the upload per step 4, then click Submit ONE more time. + - If validation errors appear, fix the flagged fields and submit once more. + - If it says the submission was "flagged as possible spam": STOP, do NOT resubmit, and report + status="failed" with that exact message (resubmitting hardens the flag). + - Look for an EXPLICIT confirmation: a success page or text such as "Application submitted", + "Thank you for applying", or a confirmation/reference ID. Read its exact wording. + - ONLY report status="submitted" if you actually SEE such an explicit confirmation, and quote + it verbatim in "confirmation". If you clicked submit but NO confirmation/success page + appears, report status="failed" with exactly what was on screen. NEVER claim success + without on-screen proof. +10. Final answer = a JSON object: {{"status": "submitted"|"ready_for_review"|"closed"|"failed"|"blocked", + "surface": "openai_board"|"linkedin_easy_apply"|"none", "resume_upload_confirmed": true|false, + "confirmation": , + "answered": [...], "uncertain": [], + "notes": }}. +""" + + +async def main(): + print(f"[1/3] Parsing résumé via vision model ({RESUME_PATH})...") + profile = await parse_resume_to_profile(RESUME_PATH) + print(f" -> {profile.get('full_name')} · {profile.get('years_experience')} yrs · " + f"{len(profile.get('skills', []))} skills") + + print(f"[2/3] Launching Chrome (dedicated profile) and the apply agent " + f"(model={APPLY_MODEL}, auto_submit={AUTO_SUBMIT})...") + session = BrowserSession(browser_profile=make_profile()) + agent = Agent( + task=build_task(profile), + llm=ChatOpenAI(model=APPLY_MODEL), + browser_session=session, + available_file_paths=[RESUME_PATH], + sensitive_data=SENSITIVE_DATA, + ) + print(f" login mode: {'credentials via sensitive_data' if SENSITIVE_DATA else 'persisted session'}") + + try: + history = await agent.run(max_steps=MAX_STEPS) + print("\n[3/3] ========== AGENT RESULT ==========") + print(history.final_result()) + print("=====================================") + # Capture the final page as proof of the confirmation (or of whatever is on screen). + try: + shot = os.path.join(os.path.dirname(os.path.abspath(__file__)), "confirmation.png") + await session.take_screenshot(path=shot, full_page=True) + print(f"📸 Saved final-page screenshot -> {shot}") + except Exception as e: + print(f"(could not save screenshot: {e})") + finally: + try: + await asyncio.get_event_loop().run_in_executor( + None, input, "\nPress Enter to close the browser... " + ) + except Exception: + pass + await session.kill() + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/research/poc/browser-apply/apply_ashby.py b/research/poc/browser-apply/apply_ashby.py new file mode 100644 index 0000000..8e56819 --- /dev/null +++ b/research/poc/browser-apply/apply_ashby.py @@ -0,0 +1,204 @@ +""" +apply_ashby.py — master the Ashby application in ISOLATION (no LinkedIn). + +Goes DIRECTLY to the OpenAI / AshbyHQ application URL. That page is public — no login, so no +LinkedIn new-device alerts and no ban vector. "5 applications ≠ 5 logins": here it's 0 logins. + +This is the focused "master Ashby" script. The LinkedIn discovery+login flow is preserved as a +COMMENTED block below (we'll re-enable it and merge the two into one unit once Ashby is solid). + +ACCURACY IS GO/NO-GO: the agent must satisfy a hard checklist — NO required field left empty — +before it will submit or hand off. It reports every required field and whether all were filled. + +Run: + ASHBY_URL="https://jobs.ashbyhq.com/openai//application" python apply_ashby.py + AUTO_SUBMIT=0 ASHBY_URL="..." python apply_ashby.py # fill + gate, STOP before submit (you submit) +""" +from __future__ import annotations + +import asyncio +import json +import os +from datetime import date, timedelta + +from dotenv import load_dotenv + +from browser_use import Agent, BrowserSession, ChatOpenAI +from browser_config import make_profile, RESUME_PATH +from parse_resume import parse_resume_to_profile + +load_dotenv() + +ASHBY_URL = os.environ.get("ASHBY_URL", "") # REQUIRED: the exact application page URL +APPLY_MODEL = os.environ.get("APPLY_MODEL", "gpt-4.1") +AUTO_SUBMIT = os.environ.get("AUTO_SUBMIT", "1") != "0" +MAX_STEPS = int(os.environ.get("MAX_STEPS", "55")) + +# --------------------------------------------------------------------------------------------- +# FUTURE — LinkedIn discovery + login (kept for when we merge into one unit; intentionally OFF). +# Do NOT delete. When combining: prepend a login step using these, then navigate from the +# LinkedIn job to Apply, then fall through to the Ashby steps below. +# +# LINKEDIN_USER = os.environ.get("LINKEDIN_USER", "") +# LINKEDIN_PASS = os.environ.get("LINKEDIN_PASS", "") +# SENSITIVE_DATA = { +# "https://*.linkedin.com": {"linkedin_user": LINKEDIN_USER, "linkedin_pass": LINKEDIN_PASS} +# } if (LINKEDIN_USER and LINKEDIN_PASS) else None +# LOGIN_STEP = ( +# "0. LOGIN: Go to https://www.linkedin.com/feed/. If a login form appears, log in with the " +# "placeholders linkedin_user / linkedin_pass. If a security checkpoint/CAPTCHA appears, STOP " +# "and report status='blocked' (a human must clear it). NOTE: prefer a pre-logged-in session " +# "to avoid repeated automated logins (each triggers a LinkedIn new-device alert)." +# ) +# --------------------------------------------------------------------------------------------- + + +def build_task(profile: dict) -> str: + # Compute dates at runtime so this never goes stale. Suggest a realistic ~2-week-out start. + today = date.today() + start_target = today + timedelta(days=14) + today_str = today.strftime("%Y-%m-%d") + start_str = start_target.strftime("%Y-%m-%d (%B %-d, %Y)") + submit_clause = ( + "Submit the application (click the final Submit / Apply button) — but only AFTER the " + "accuracy gate in step 6 passes." + if AUTO_SUBMIT else + "DO NOT click the final submit button. Stop at the review step and report it is ready for " + "a human to submit — but only AFTER the accuracy gate in step 6 passes." + ) + return f"""You are completing ONE job application on a public OpenAI / AshbyHQ careers page on +behalf of the candidate below. Use ONLY the candidate's real data — never invent qualifications, +employers, dates, or work authorization. + +CANDIDATE PROFILE (source of truth, parsed from their résumé): +{json.dumps(profile, indent=2, ensure_ascii=False)} + +Résumé file to upload: {RESUME_PATH} + +CONFIRMED FACTS (use these EXACT answers — they OVERRIDE any résumé guess): +- Preferred contact email: rahul.gupta2608@yahoo.com (the résumé email). +- The candidate lives in India, is an Indian citizen, and is authorized to work in India. +- This role is India-based, so: "authorized to work in the country where the job is located" = YES; + "require visa sponsorship" = NO. (If the page clearly shows a non-India location, answer + truthfully instead: not authorized / needs sponsorship for a US role. Honesty over optimism.) +- Availability: available to start soon. Today's date is {today_str}. For a "when can you start" + date field, pick a date ON OR AFTER today — NEVER a past date. A good choice is about two weeks + out: {start_str}. Use that unless the picker only allows a different valid future date. +- MOTIVATION (use this VERBATIM for any "what interests you about this job" / "why do you want to + work here" / "Additional Information" free-text field — a strong application never leaves this + blank): + "I want to help organizations actually put frontier AI to work, and the AI Deployment Engineer, + Codex role is the most direct place I've seen to do that. My background is hands-on ML and data + engineering — I shipped machine-learning systems at WalmartLabs and have built with LLM/GenAI + APIs, vector retrieval, and cloud infrastructure across AWS, Azure, and GCP. Codex is changing + how software gets written, and I'm excited to work directly with teams to turn that capability + into real, reliable deployments, then carry what I learn on the ground back into the product. + Being based in India, I'm especially motivated to help bring this technology to engineering + teams here. Deploying AI that people can trust and genuinely depend on is the work I most want + to be doing." + +STEPS: +1. Go DIRECTLY to the application page: {ASHBY_URL}. No login is needed. The Ashby page is a + JavaScript app and may look BLANK for a few seconds — `wait` ~6s and re-read; refresh ONCE if + still blank. Only report a load failure after two waits AND a refresh. +2. LIVENESS CHECK: if the page shows "no longer accepting applications", a 404, or the role is + closed, STOP and report status="closed" with what you saw. Never fake a submit on a dead job. +3. AUTOFILL FIRST (most reliable way to fill fields): find the "Autofill from resume" control near + the top of the form and upload the résumé file THERE. `wait` ~8s for Ashby to parse it and + pre-fill fields. This populates name/email/experience automatically. +4. RÉSUMÉ ATTACHMENT (make-or-break — a half-finished upload got us flagged before): + - Ensure the required "Resume" field shows the file ATTACHED: the filename is visible and NO + spinner / "uploading" / "we're updating your application" text remains. + - If autofill did not attach it to the Resume field, upload the file there, `wait` ~8s, verify. + Retry up to 2 times. Do not continue until the attachment is CONFIRMED. +5. FILL EVERY FIELD THAT STRENGTHENS THE APPLICATION — not only the "*" required ones. "Required" + and "needed for a good application" are different: a blank motivation box is technically allowed + but makes a WEAK application. So fill every field you reasonably can from the profile + CONFIRMED + FACTS, and leave a field blank ONLY if it is truly irrelevant or asks for something the candidate + genuinely does not have. Specifics: + - "Where are you currently located?" = TYPEAHEAD: type "New Delhi", WAIT for the dropdown, and + CLICK the matching suggestion. Verify it no longer shows "Start typing…". + - "When can you start a new role?" = DATE PICKER: open it, CLICK an actual date, verify it no + longer shows "Pick date…". + - Work-authorization = YES, sponsorship = NO (India). + - "What interests you about this job" / "Additional Information" free-text box: fill it with the + MOTIVATION text from CONFIRMED FACTS verbatim. NEVER leave this blank — it is what makes or + breaks the application even though it has no "*". + - REQUIRED CONSENT CHECKBOX: tick "I confirm I have read the above." (the certification near the + bottom). This is REQUIRED even though it has no "*". After clicking it, VERIFY the box is + actually checked (a check mark / filled box) — re-click if it did not register. +6. ACCURACY GATE — GO/NO-GO (the whole point). Before doing step 7, go through the form field by + field and build a checklist of every field that must be filled. This includes: every "*" field; + every required consent CHECKBOX (e.g. "I confirm I have read the above") even with no "*"; AND + the motivation / "what interests you" / "Additional Information" free-text box. For each, record + its current value/state. A field counts as FILLED only if: + - text/select fields: a real value, NOT a placeholder ("Start typing…", "Pick date…", + "Type here…", "hello@example.com…"); + - checkboxes/toggles: actually CHECKED/selected; + - the motivation box: contains the MOTIVATION text (not empty). + If ANY of these is empty, a placeholder, or an unchecked required box, it is NOT done: fill/tick + it now and re-check. You may NOT proceed to step 7 until EVERY one passes. Then CLOSE any open + date-picker/dropdown so the form is settled. Set all_required_filled=true only if every item + (including the consent checkbox and the motivation box) passes. +7. {submit_clause} +8. AFTER submit (if you submitted), VERIFY — never assume. Click Submit only ONCE, `wait` ~5s, read + the page. Do NOT rapid-click Submit (looks like spam). + - "still uploading / try again": upload wasn't done — re-verify step 4, then Submit once more. + - validation errors: fix the flagged fields, Submit once more. + - "flagged as possible spam": STOP, do NOT resubmit, report status="failed" with the exact text. + - Only report status="submitted" if you SEE an explicit confirmation ("Application submitted", + "Thank you for applying", or a reference id) — quote it verbatim. No proof => status="failed". +9. Final answer = JSON: {{"status": "submitted"|"ready_for_review"|"closed"|"failed"|"blocked", + "all_required_filled": true|false, "resume_upload_confirmed": true|false, + "required_fields": {{: , ...}}, + "confirmation": , + "uncertain": [], "notes": }}. +""" + + +async def main(): + if not ASHBY_URL or "ashby" not in ASHBY_URL.lower(): + print("✋ ASHBY_URL is not set (or doesn't look like an Ashby URL).") + print(" Open the OpenAI application tab and copy its address bar URL, then run:") + print(' ASHBY_URL="https://jobs.ashbyhq.com/openai//application" python apply_ashby.py') + return + + print(f"[1/3] Parsing résumé via vision model ({RESUME_PATH})...") + profile = await parse_resume_to_profile(RESUME_PATH) + print(f" -> {profile.get('full_name')} · {profile.get('years_experience')} yrs · " + f"{len(profile.get('skills', []))} skills") + + print(f"[2/3] Direct to Ashby (NO LinkedIn). model={APPLY_MODEL}, auto_submit={AUTO_SUBMIT}") + print(f" url: {ASHBY_URL}") + session = BrowserSession(browser_profile=make_profile()) + agent = Agent( + task=build_task(profile), + llm=ChatOpenAI(model=APPLY_MODEL), + browser_session=session, + available_file_paths=[RESUME_PATH], + # sensitive_data not needed — no login on the Ashby page. + ) + + try: + history = await agent.run(max_steps=MAX_STEPS) + print("\n[3/3] ========== AGENT RESULT ==========") + print(history.final_result()) + print("=====================================") + try: + shot = os.path.join(os.path.dirname(os.path.abspath(__file__)), "ashby_result.png") + await session.take_screenshot(path=shot, full_page=True) + print(f"📸 Saved final-page screenshot -> {shot}") + except Exception as e: + print(f"(could not save screenshot: {e})") + finally: + try: + await asyncio.get_event_loop().run_in_executor( + None, input, "\nPress Enter to close the browser... " + ) + except Exception: + pass + await session.kill() + + +if __name__ == "__main__": + asyncio.run(main()) diff --git a/research/poc/browser-apply/apply_engine.py b/research/poc/browser-apply/apply_engine.py new file mode 100644 index 0000000..fc3f8bf --- /dev/null +++ b/research/poc/browser-apply/apply_engine.py @@ -0,0 +1,121 @@ +""" +apply_engine.py — the MASTER KEY. One board-agnostic engine for any job board. + + any apply URL -> pick_adapter(url) -> generic fill loop + adapter hints -> accuracy gate + -> verify + screenshot. Ashby is just one adapter; the GenericAdapter handles the + long tail (bespoke career pages) by vision alone. + +Statuses it can report (the last three are what make the long-tail measurable): + submitted | ready_for_review | closed | login_required | no_form | failed | blocked + +Run: + APPLY_URL="https://...apply..." AUTO_SUBMIT=0 python apply_engine.py +""" +from __future__ import annotations + +import asyncio +import json +import os + +from dotenv import load_dotenv + +from browser_use import Agent, BrowserSession, ChatOpenAI +from browser_config import make_profile, RESUME_PATH +from candidate import build_candidate, candidate_prompt_block +from adapters import pick_adapter + +load_dotenv() + +APPLY_URL = os.environ.get("APPLY_URL", "") +APPLY_MODEL = os.environ.get("APPLY_MODEL", "gpt-4.1") +AUTO_SUBMIT = os.environ.get("AUTO_SUBMIT", "0") != "0" # default SAFE (fill, don't submit) +MAX_STEPS = int(os.environ.get("MAX_STEPS", "55")) + + +def build_task(cand: dict, url: str, hints: str) -> str: + submit_clause = ( + "Submit the application (click the final Submit/Apply button) — ONLY after the gate passes." + if AUTO_SUBMIT else + "DO NOT click final submit. Stop at review and report ready_for_review — ONLY after the gate passes." + ) + return f"""You are completing ONE job application on the careers page below, on behalf of the +candidate. Use ONLY the candidate's real data — never fabricate qualifications, employers, or dates. + +{candidate_prompt_block(cand)} + +{hints} + +STEPS: +1. Go to {url}. It may be a JS app that renders blank for a few seconds — wait ~6s and re-read; + refresh ONCE if still blank. +2. TRIAGE the page: + - If it shows "no longer accepting applications" / 404 / closed -> STOP, status="closed". + - If applying REQUIRES creating an account or logging in -> STOP, status="login_required" + (we do not create accounts; that is the human/Cloud lane). + - If there is NO real per-job application form (just a generic /careers page, or "email your + CV to...") -> STOP, status="no_form". + - Otherwise there is a fillable application form -> continue. +3. RÉSUMÉ: if an "autofill from resume" control exists, upload there first. Then ensure the résumé + file is ATTACHED to the resume field — filename shown, NO spinner/"uploading" text. Uploads are + async; wait ~8s and verify; retry up to 2x. A half-finished upload causes failures. +4. FILL EVERY FIELD THAT STRENGTHENS THE APPLICATION (not just "*" required ones): + - contact details, links, work-authorization/sponsorship per CONFIRMED FACTS; + - typeaheads: type then CLICK the suggestion (verify it committed, not a placeholder); + - date pickers: open and click a valid FUTURE date; + - motivation / "why" / "additional information": write the tailored answer (never blank). +5. Tick every required consent checkbox (verify it is actually checked). +6. ACCURACY GATE — GO/NO-GO. List EVERY required field: every "*" field, every required consent + checkbox, and the motivation box. A field passes only if it has a real value (not a placeholder + "Start typing…"/"Pick date…"/"Type here…") or, for checkboxes, is CHECKED. If any fails, fix it + and re-check. Do NOT proceed until all pass. Then close any open picker so the form is settled. +7. {submit_clause} +8. If you submitted: click Submit ONCE, wait ~5s, read the page (don't rapid-click — looks like + spam). Handle "still uploading"->re-verify upload & retry once; validation errors->fix & retry; + "flagged as spam"->STOP, status="failed". Report status="submitted" ONLY with an explicit on- + screen confirmation (quote it); otherwise status="failed". +9. Final answer = JSON: {{"status": "submitted"|"ready_for_review"|"closed"|"login_required"| + "no_form"|"failed"|"blocked", "all_required_filled": true|false, + "resume_upload_confirmed": true|false, "fields": {{