growqr-app/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
docs/upskilling/extras/meeting_notes.md
-Puter e6685203fe feat: initial docs repo with project inventory and all documentation 
- Added REPO_INVENTORY.md with all repos, branches, remotes, and staging info
- Added .gitignore
- Synced all existing docs from local workspace
- Centralized documentation hub for GrowQR team
2026-06-22 15:04:27 +05:30

11 KiB
Raw Permalink Blame History

GrowQR — Interview Service — Meeting Notes (Decision & Tradeoff Doc)

Primary purpose: help the team make high-quality engineering decisions for the Interview Service (and how it composes into the Upscaling module: Interview + Roleplay + Courses + Assessment).

Updated after meeting: 2026-02-27

Related cross-service doc: growqr/plans/upscaling_redundancy_breakdown.md

0) What we learned from the meeting (delta from earlier plan)

0.1 Compliance / security baseline is mandatory

You called this “DITA compliant” in the meeting. I’m interpreting that as: baseline privacy + security controls suitable for US/EU/India markets (not full SOC2-from-day-1, but not a hobby project).

0.2 Timeline changed

  • Previous assumption: ~6–7 working days
  • New agreed direction: ~3 weeks so we can optimize latency, consistency, and security properly.

0.3 Latency is now a core requirement

We need the experience to be:

  • fast to interact
  • seamless (no awkward waits)
  • less robotic (voice prosody + conversational pacing)

This affects where we put STT/TTS/avatar/lip-sync (client vs server) and whether we do streaming.

0.4 Client vs server split was debated

  • Team leaned server-side for “control”
  • You argued client-side for cost, scalability, compliance, and lower infra
  • Current direction: client-heavy media + near-real-time text to server (recommended compromise)

1) Baseline compliance & security checklist (US/EU/India)

1.1 What we should implement in v1 (minimum)

  • TLS everywhere
  • Encryption at rest (DB + object storage; KMS-managed keys where possible)
  • Least privilege IAM (per-service roles; scoped permissions)
  • Presigned URLs for uploads/downloads (short expiry; scoped paths)
  • PII minimization: explicitly classify data
    • Tier A: account identifiers
    • Tier B: transcripts (sensitive)
    • Tier C: raw recordings (highly sensitive)
    • Tier D: derived metrics (lower sensitivity)
  • Retention controls: configurable retention for recordings and transcripts
  • Deletion workflow: delete user data on request; delete expired artifacts automatically
  • Access audit logs: log access to raw recordings and reports
  • Vendor DPAs (if using paid AI providers): confirm acceptable processing + region

1.2 Security guardrails that improve decision-making

  • Prefer client-side processing for sensitive media where possible.
  • If we must use paid providers, prefer:
    • providers that support enterprise controls and DPAs
    • region pinning/data residency where required

2) What is a “real-time streaming pipeline” (and what we actually need)

A real-time streaming pipeline processes data continuously while it’s produced.

Level 0 — Batch (simplest)

  • Record → upload → analyze.
  • Low engineering risk, high latency.

Level 1 — Near-real-time (recommended)

  • Client produces transcript quickly and sends text turn immediately.
  • Server generates next question immediately (optionally with token streaming).
  • Media uploads continue in background and are used for post-session analysis.

Level 2 — True streaming (complex/expensive)

  • Stream audio/video to server (WebRTC/WebSocket)
  • Streaming STT → streaming LLM → streaming TTS
  • Maximum complexity, maximum compliance surface area.

Recommendation: Level 1. It achieves “fast and seamless” without heavy streaming of user audio/video.

3) Optimal client vs server split (for cost, latency, and compliance)

3.1 Client-side (recommended)

These reduce backend infra cost and reduce compliance risk.

  1. Media capture + encoding
  • camera/mic recording
  • screen share (specific window)
  • local compression
  1. STT (speech-to-text) where feasible
  • produce transcript locally and send text; enables fast turn-taking
  1. TTS (text-to-speech) for AI persona
  • fast, avoids voice vendor calls; BUT voice quality can vary across devices
  1. Avatar rendering + lip sync
  • best in client renderer (2D/3D) for low latency
  • accuracy improves if we can get viseme/phoneme timing from TTS
  1. Optional CV feature extraction
  • MediaPipe landmarks → send compact features/metrics rather than raw frames

3.2 Server-side (should remain centralized)

  1. Session orchestration + persistence
  • state machine, timing, idempotency
  1. Question selection/generation
  • interview type logic, persona rules
  1. Scoring + report generation
  • deterministic numeric scoring
  • LLM-generated narrative sections (structured)
  1. Artifact registry + storage broker
  • presigned URLs, metadata, retention/deletion

  1. History/trends

  2. Integrations

  • emit results to Dashboard/Courses/Assessment/Roleplay

4) On-device models (browser): realistic options

You mentioned bundling “a small whisper model for TTS” — Whisper is STT, not TTS.

4.1 STT client options

  • Fastest (low engineering): Web Speech Recognition
  • Self-contained: whisper.cpp / WASM Whisper (tiny/base)
    • Pros: privacy + no vendor
    • Cons: model download size + CPU cost
  • Hybrid fallback: try local STT; if unavailable/slow, user opts-in to server STT

4.2 TTS client options

  • Simple: Web Speech Synthesis / native OS voices
  • Higher quality + lip sync friendly (paid): Azure Speech TTS (supports visemes)
    • Note: only the assistant text is sent; user audio/video still stays client-side

4.3 Lip sync options (for “not robotic”)

  • Best for latency: client-rendered avatar + viseme timing (from TTS or heuristic)
  • Worst for latency: pre-rendered avatar videos per utterance (often seconds delay)

5) Modules required (composition modules)

  1. API layer (HTTP)
  2. AuthN/AuthZ verification
  3. Session State Machine & Orchestrator
  4. Question Bank + Rubrics
  5. Turn ingestion (Q/A turns)
  6. Media broker (presigned URLs, metadata)
  7. Background jobs (analysis worker)
  8. Signal processing (optional)
  9. Scoring engine (deterministic)
  10. Report generator (structured feedback)
  11. History/trends
  12. Outbound integrations (events/webhooks + outbox)
  13. Observability
  14. Compliance controls (retention/deletion/audit)

6) Implementation options per module (OSS vs Paid vs Paid-simple)

Legend

  • OSS = self-hostable
  • Paid = best quality / flexible
  • Paid-simple = easiest integration / fewest moving parts

6.1 API layer

  • OSS: FastAPI / NestJS
  • Paid: AWS API Gateway
  • Paid-simple: Cloudflare Workers

Decision (LOCKED): FastAPI

6.2 Database

  • OSS: Postgres self-host
  • Paid: RDS/Cloud SQL
  • Paid-simple: Supabase Postgres

Decision (LOCKED): Self-hosted Postgres (on GCP/compute)

6.3 Storage

  • OSS: MinIO
  • Paid: S3 / GCS
  • Paid-simple: Cloudflare R2

Decision (LOCKED): Google Cloud Storage (GCS) (S3-compatible bucket)

6.4 Jobs/queue

  • OSS: RQ/Celery + Redis
  • Paid: SQS + ECS/Lambda
  • Paid-simple: Step Functions

Decision (LOCKED): RQ + Redis (built into the service)

6.5 LLM (questions + report narrative)

  • OSS: Ollama / vLLM
  • Paid: OpenAI / Anthropic / Gemini
  • Paid-simple: single-provider adapter

Decision (LOCKED): OpenRouter / OpenAI key (via an adapter)

6.6 STT

  • OSS: faster-whisper (server) / whisper.cpp WASM (client)
  • Paid: Deepgram / AssemblyAI / Google STT
  • Paid-simple: AWS Transcribe

Decision (LOCKED): Whisper WASM on the client side

6.7 TTS

  • OSS: Piper / Coqui
  • Paid: ElevenLabs / Azure Speech
  • Paid-simple: native/browser TTS

Decision (LOCKED): Google Cloud TTS (we have an account, simple integration)

6.8 Facial/body/technical quality

  • OSS: MediaPipe client or server
  • Paid: Rekognition/Azure Face
  • Paid-simple: client-only metrics

Decision (LOCKED): MediaPipe on the client (no AWS Rekognition)

6.9 Screen-share / coding analysis

  • OSS: structured artifacts + tree-sitter
  • Paid: LLM code feedback; optional sandbox
  • Paid-simple: LLM-only feedback from code text

Decision (LOCKED): Simple feedback only for now (no heavy OCR/analysis)

7) Two best-path approaches

Option A — Client-heavy + Paid AI (recommended)

  • Client: capture + encode + (STT/TTS/avatar/lip-sync)
  • Server: orchestration + scoring + report + history + compliance + integrations
  • Paid LLM for narrative + follow-ups

Pros: best speed-to-quality tradeoff, minimal infra, easier compliance. Cons: variable LLM cost; requires careful client contract.

Option B — OSS/self-hosted ML

  • Self-host LLM + STT + TTS + CV

Pros: maximum control, lower marginal cost at scale. Cons: GPU ops + longer timeline.

8) Tradeoffs (Paid vs OSS)

Dimension Client-heavy + Paid AI OSS-first
Time-to-ship Best Slower
Latency UX Best (if we stream text + client TTS) Depends on infra
Compliance surface Smaller (less user media sent) Smallest if fully local
Ops burden Low High
Cost profile Usage-based Fixed infra + ops

9) Timeline estimate (updated)

9.1 Recommended plan — ~3 weeks

Week 1: foundations + compliance

  • finalize client/server contract
  • DB schema + migrations
  • state machine
  • artifact broker (presigned URLs)
  • retention + deletion hooks + audit logging

Week 2: latency + interaction path

  • “fast path” transcript turn-taking
  • SSE streaming responses (if using LLM)
  • deterministic scoring engine + report generator v1
  • background worker pipeline

Week 3: reliability + integrations + polish

  • idempotency, retries, outbox
  • perf budget enforcement (≤60s analysis)
  • events to Dashboard/Courses/Assessment/Roleplay
  • security review checklist + load test

9.2 If we insist on true server-side streaming (Level 2)

Add +3 to +6 weeks (complexity explodes: streaming STT/TTS, infra, compliance).

10) Cost model (decision-friendly, not exact pricing)

10.1 What costs money

  • LLM tokens (follow-ups + narrative report)
  • STT minutes (if server-side fallback)
  • TTS characters (if server-side paid voices)
  • storage (video/audio/screen)
  • compute for workers

10.2 Cheapest sustainable shape

  • Keep user audio/video processing client-side
  • Send only text turns to server for interaction
  • Upload media in background for post-analysis (or make recording storage opt-in)

10.3 “Buy vs build” money tradeoff (qualitative)

  • Buying LLM/STT/TTS reduces engineering and ops cost immediately, but increases variable cost.
  • OSS reduces variable cost at scale, but increases:
    • GPU infra costs
    • reliability/monitoring burden
    • engineering time

11) Cross-service redundancy note

Interview/Roleplay/Courses/Assessment share platform concerns (credits, artifacts, certificates, events, skill taxonomy, compliance).

See: growqr/plans/upscaling_redundancy_breakdown.md

12) Open questions (need answers to lock implementation)

  1. When we say “DITA compliant”, confirm the target:

    • baseline GDPR/DPDP/CCPA + security controls?
    • or a specific internal standard?

  2. Do we store raw recordings by default or make it opt-in with retention caps?

  3. For “less robotic voice”: do we accept platform TTS variance, or require consistent paid TTS (Azure Speech, etc.)?

  4. Client STT: commit to WASM Whisper now, or as a progressive enhancement/fallback?