docs/upskilling/extras/server_client_split.md

GrowQR Interview Service - Server/Client Responsibility Split

Version: 1.0
Date: March 2, 2026
Purpose: Define clear boundaries between server and client implementations

---

🎯 Quick Summary

Aspect	Server	Client
Primary Role	Orchestrator & Intelligence	Media Processor & User Interface
Heavy Lifting	AI logic, business rules, data persistence	Recording, encoding, speech processing
Data Flow	Text-based (transcripts, questions, metadata)	Media-based (video, audio, screen capture)
Technology	Python + FastAPI	JavaScript/TypeScript + Browser APIs
Scalability	Horizontal scaling via stateless design	Scales with user devices

---

📦 SERVER-SIDE: What Goes on Server

Core Principle

Server = Stateless orchestrator that handles AI, business logic, auth, and data persistence

Server Responsibilities Overview

┌─────────────────────────────────────────────────────────────┐
│                      SERVER SIDE                             │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  1. AUTHENTICATION & AUTHORIZATION                   │  │
│  │     • JWT token validation                           │  │
│  │     • User identity verification                     │  │
│  │     • Permission checks                              │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  2. BUSINESS LOGIC & ORCHESTRATION                   │  │
│  │     • Session state machine                          │  │
│  │     • Interview flow control                         │  │
│  │     • Credits/entitlements management                │  │
│  │     • Timer and progress tracking                    │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  3. AI LOGIC (LLM Integration)                       │  │
│  │     • Question generation                            │  │
│  │     • Follow-up question logic                       │  │
│  │     • Transcript analysis                            │  │
│  │     • Narrative feedback generation                  │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  4. STORAGE BROKER                                   │  │
│  │     • Pre-signed URL generation                      │  │
│  │     • Artifact metadata registry                     │  │
│  │     • Retention policy enforcement                   │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  5. POST-PROCESSING & ANALYSIS                       │  │
│  │     • Scoring algorithms (7 metrics)                 │  │
│  │     • Report generation                              │  │
│  │     • Historical trends calculation                  │  │
│  │     • Event emission (integrations)                  │  │
│  └──────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

---

🖥️ SERVER MODULES (Detailed Breakdown)

Module 1: Authentication & Authorization

Purpose: Verify user identity and permissions

Responsibilities:

• ✅ Validate JWT tokens (signature, expiry, claims)
• ✅ Extract user_id from token
• ✅ Check user permissions (org visibility, feature access)
• ✅ Rate limiting per user

Technology Stack:

• JWT library (PyJWT)
• Middleware pattern (FastAPI dependencies)
• Redis for rate limiting

Inputs:

• JWT token (from Authorization header)

Outputs:

• Validated User object
• HTTP 401 if invalid
• HTTP 429 if rate limited

TODO:

• Implement JWT validation middleware
• Define token claims structure
• Setup rate limiting rules

---

Module 2: Credits & Entitlements

Purpose: Manage user credits and interview quotas

Responsibilities:

• ✅ Check available credits before starting interview
• ✅ Deduct credits atomically (prevent double-spend)
• ✅ Handle different interview types (different credit costs)
• ✅ Support refunds (if interview fails)

Technology Stack:

• PostgreSQL transactions
• Database-level locks
• Optional: External credits service API

Database Schema:

user_credits:
  - user_id
  - credits_balance
  - last_updated

credit_transactions:
  - id
  - user_id
  - session_id
  - amount (positive or negative)
  - transaction_type (deduct, refund, purchase)
  - created_at

Inputs:

• User ID
• Interview type

Outputs:

• Success/failure
• Remaining credits

TODO:

• Implement atomic deduction logic
• Define credit costs per interview type
• Create refund workflow

---

Module 3: Session State Machine

Purpose: Manage interview session lifecycle

Responsibilities:

• ✅ Create new sessions
• ✅ Track session state transitions
• ✅ Enforce state rules (can't submit turn if not started)
• ✅ Handle timeouts and abandonment

States:

configured → in_progress → completed → processing → ready
                    ↓
                abandoned/timeout

Technology Stack:

• PostgreSQL for persistence
• Redis for active session cache
• FSM (Finite State Machine) pattern

Database Schema:

interview_sessions:
  - id (UUID)
  - user_id
  - interview_type
  - status (enum: configured, in_progress, completed, processing, ready, abandoned)
  - started_at
  - completed_at
  - created_at
  - metadata (JSONB)

Inputs:

• Session ID
• State transition request

Outputs:

• Updated session state
• Validation errors

TODO:

• Implement state machine logic
• Define transition rules
• Create timeout handling (background job)

---

Module 4: Question Generation (AI)

Purpose: Generate personalized interview questions using LLM

Responsibilities:

• ✅ Generate 8-10 questions based on interview type
• ✅ Personalize based on user profile (role, experience, skills)
• ✅ Create avatar scripts (conversational versions)
• ✅ Mix difficulty levels (easy/medium/hard)
• ✅ Store questions in database

Technology Stack:

• Anthropic Claude 3.5 Sonnet OR OpenAI GPT-4
• LLM client library
• Prompt templates

Prompt Strategy:

Input:
- Interview type (technical, behavioral, coding, warmup)
- User profile (role, experience, skills, previous interviews)
- Programming language (for coding interviews)

Output (JSON):
[
  {
    "text": "Question for interviewer",
    "avatar_script": "Conversational version",
    "category": "technical|behavioral|problem_solving",
    "difficulty": "easy|medium|hard",
    "expected_duration": 120
  }
]

Database Schema:

interview_questions:
  - id (UUID)
  - session_id
  - question_order (int)
  - question_text
  - avatar_script
  - category
  - difficulty
  - expected_duration_seconds
  - is_followup (boolean)
  - parent_question_id (UUID, nullable)

Inputs:

• Interview type
• User profile
• Programming language (optional)

Outputs:

• List of questions (JSON)
• Stored in database

TODO:

• Create prompt templates
• Implement LLM API integration
• Define question validation rules
• Handle LLM failures (retry logic)

---

Module 5: Follow-up Question Logic (AI)

Purpose: Decide whether to ask follow-up or proceed to next question

Responsibilities:

• ✅ Analyze user's answer transcript
• ✅ Determine if follow-up is needed (incomplete, interesting, or unclear answer)
• ✅ Generate contextual follow-up question
• ✅ Limit follow-ups (max 1-2 per main question)

Technology Stack:

• LLM (Claude/GPT-4)
• Context window management
• Prompt templates

Decision Logic:

Input:
- Previous question
- User's transcript
- Session context (time remaining, questions completed)

LLM Analysis:
1. Is the answer complete?
2. Did user mention something interesting worth exploring?
3. Is clarification needed?
4. Do we have time for follow-up?

Output:
- followup_needed: true/false
- followup_question: "..." (if needed)

Inputs:

• Session ID
• Previous question + answer transcript
• Time remaining

Outputs:

• Next question (follow-up or sequential)

TODO:

• Create follow-up analysis prompt
• Implement decision logic
• Set follow-up limits per session

---

Module 6: Pre-signed URL Generation

Purpose: Generate secure, time-limited upload URLs for client

Responsibilities:

• ✅ Generate S3/R2 pre-signed URLs
• ✅ Scope URLs to session + chunk number
• ✅ Set expiry (15 minutes)
• ✅ Generate URLs in batches (5-10 at a time)
• ✅ Support URL refresh requests

Technology Stack:

• Boto3 (S3 SDK) or Cloudflare R2 SDK
• S3-compatible API (signature v4)

URL Structure:

https://storage.growqr.com/sessions/{session_id}/{media_type}/chunk_{n}.webm?
  X-Amz-Algorithm=AWS4-HMAC-SHA256&
  X-Amz-Credential=...&
  X-Amz-Date=...&
  X-Amz-Expires=900&
  X-Amz-Signature=...

Security:

• PUT-only permission
• Content-Type restricted (video/webm)
• Path scoped to session
• Short expiry (15 min)

Inputs:

• Session ID
• Chunk numbers
• Media type (video, screen, audio)

Outputs:

• Array of pre-signed URLs with metadata

TODO:

• Implement URL generation function
• Configure S3/R2 bucket permissions
• Create URL refresh endpoint

---

Module 7: Turn Management

Purpose: Store and manage Q&A turns during interview

Responsibilities:

• ✅ Store transcript for each answer
• ✅ Record metadata (duration, chunks uploaded, confidence)
• ✅ Link turn to question
• ✅ Validate turn completeness

Technology Stack:

• PostgreSQL

Database Schema:

interview_turns:
  - id (UUID)
  - session_id
  - question_id
  - transcript (TEXT)
  - duration_seconds (INT)
  - chunks_uploaded (INT[])
  - metadata (JSONB) - STT confidence, language detected, etc.
  - created_at

Inputs:

• Session ID
• Question ID
• Transcript
• Duration
• Chunks uploaded

Outputs:

• Stored turn record
• Validation result

TODO:

• Implement turn storage endpoint
• Add validation rules
• Create turn query APIs

---

Module 8: Media Artifact Registry

Purpose: Track uploaded media chunks and metadata

Responsibilities:

• ✅ Register chunk uploads (after client uploads to S3)
• ✅ Track upload completeness
• ✅ Store metadata (size, duration, storage path)
• ✅ Support retention policies

Technology Stack:

• PostgreSQL

Database Schema:

media_chunks:
  - id (UUID)
  - session_id
  - chunk_number
  - media_type (video, screen, audio)
  - storage_key (S3 path)
  - file_size_bytes
  - duration_seconds
  - uploaded_at
  - expires_at (based on retention policy)

Inputs:

• Session ID
• Chunk metadata (from client callback)

Outputs:

• Registered chunk record

TODO:

• Implement chunk registration endpoint
• Create retention policy job
• Setup deletion workflow

---

Module 9: Scoring Engine

Purpose: Calculate 7 performance metrics from interview data

Responsibilities:

• ✅ Communication clarity (0-100)
• ✅ Technical accuracy (0-100)
• ✅ Problem-solving approach (0-100)
• ✅ Confidence (0-100)
• ✅ Professionalism (0-100)
• ✅ Engagement (0-100)
• ✅ Overall score (weighted average)

Scoring Algorithms:

1. Communication Clarity:

• Word count / duration = words per minute
• Filler word detection (um, uh, like, you know)
• Penalty for excessive fillers
• Score: 100 - (filler_count × 2)

2. Technical Accuracy:

• LLM analysis of answers vs expected concepts
• Keyword/concept matching
• Code correctness (if applicable)

3. Problem-Solving:

• Answer structure analysis
• Breaking down complex problems
• Logical flow

4. Confidence:

• Speech rate consistency
• Pause analysis
• Filler word ratio

5. Professionalism:

• Language formality
• No inappropriate content
• Proper terminology

6. Engagement:

• Answer completeness
• Enthusiasm markers
• Eye contact (if using video analysis)

Technology Stack:

• Python (NumPy for calculations)
• LLM for qualitative analysis
• Optional: Speech analysis libraries

Inputs:

• All turns (transcripts)
• Session metadata
• Optional: Media files for advanced analysis

Outputs:

• Scores object (JSON)
• Metric breakdowns

TODO:

• Implement deterministic scoring logic
• Integrate LLM for qualitative metrics
• Define scoring weights
• Create testing suite for score consistency

---

Module 10: Report Generation (AI)

Purpose: Generate narrative feedback using LLM

Responsibilities:

• ✅ Analyze complete interview transcript
• ✅ Generate strengths (top 3 with examples)
• ✅ Generate improvements (top 3 with examples)
• ✅ Overall assessment (2-3 sentences)
• ✅ Actionable recommendations

Technology Stack:

• LLM (Claude/GPT-4)
• Structured output (JSON)
• Report templates

Prompt Strategy:

Input:
- Full interview transcript (all Q&A pairs)
- Quantitative scores
- User profile

Prompt:
"Analyze this interview performance. Provide:
1. Top 3 strengths (with specific examples from transcript)
2. Top 3 areas for improvement (with specific examples)
3. Overall assessment (2-3 sentences)
4. Actionable next steps

Be specific, constructive, and reference actual quotes."

Output (JSON):
{
  "strengths": ["...", "...", "..."],
  "improvements": ["...", "...", "..."],
  "overall_assessment": "...",
  "recommendations": ["...", "..."]
}

Database Schema:

interview_reports:
  - id (UUID)
  - session_id (unique)
  - scores (JSONB)
  - feedback (JSONB)
  - metrics (JSONB)
  - processing_duration_seconds
  - created_at

Inputs:

• Session ID
• All turns
• Scores

Outputs:

• Complete report (JSON)
• Stored in database

TODO:

• Create feedback prompt template
• Implement LLM integration
• Define report structure
• Add caching for identical transcripts

---

Module 11: Historical Trends & Comparison

Purpose: Track user progress over time

Responsibilities:

• ✅ Retrieve previous interview scores
• ✅ Calculate improvement/decline trends
• ✅ Identify consistent strengths/weaknesses
• ✅ Generate progress charts data

Technology Stack:

• PostgreSQL aggregations
• Window functions (SQL)

Calculations:

- Average score change (last 5 interviews)
- Metric-specific trends (e.g., communication improving)
- Consistency analysis (score variance)
- Time between interviews (engagement)

Inputs:

• User ID
• Current session scores

Outputs:

• Trend data (JSON)
• Comparison with previous average

TODO:

• Implement trend calculation queries
• Define minimum interviews for trends (need 2+)
• Create visualization data format

---

Module 12: Background Jobs (Celery Workers)

Purpose: Async processing for heavy tasks

Responsibilities:

• ✅ Post-interview analysis (fetch media, score, report)
• ✅ Retention policy enforcement (delete old media)
• ✅ Session timeout handling
• ✅ Abandoned session cleanup

Technology Stack:

• Celery (Python task queue)
• Redis (broker + result backend)
• Worker processes

Jobs:

1. analyze_interview(session_id)

• Fetch all turns
• Run scoring engine
• Generate LLM report
• Calculate trends
• Update session status → "ready"
• Emit events

2. cleanup_expired_media()

• Query expired chunks
• Delete from S3
• Delete from database

3. handle_session_timeout(session_id)

• Mark session as "abandoned"
• Partial refund (if applicable)
• Cleanup resources

Inputs:

• Job-specific (session_id, etc.)

Outputs:

• Task results
• Updated database records

TODO:

• Setup Celery configuration
• Implement worker tasks
• Define retry policies
• Setup monitoring (Flower)

---

Module 13: Event Publishing & Integrations

Purpose: Share interview results with other services

Responsibilities:

• ✅ Emit events after interview completion
• ✅ Outbox pattern for reliability
• ✅ Integration with Dashboard, Courses, Assessment, Roleplay

Technology Stack:

• Outbox table pattern
• Message queue (RabbitMQ, Kafka, or SQS)
• Event schemas

Event Schema:

{
  "event_type": "interview.completed",
  "event_version": "1.0",
  "timestamp": "2026-03-02T15:30:00Z",
  "user_id": "uuid",
  "session_id": "uuid",
  "payload": {
    "interview_type": "technical",
    "overall_score": 81,
    "scores": {...},
    "skill_gaps": [
      {
        "skill_id": "python-async",
        "proficiency": 65,
        "target": 80
      }
    ],
    "recommended_courses": ["course-id-1", "course-id-2"]
  }
}

Database Schema:

outbox_events:
  - id (UUID)
  - event_type
  - payload (JSONB)
  - published (boolean)
  - published_at
  - created_at

Inputs:

• Session results

Outputs:

• Published events

TODO:

• Define event schemas
• Implement outbox pattern
• Setup message queue
• Create consumer contracts

---

Module 14: Compliance & Audit

Purpose: GDPR/CCPA compliance and security auditing

Responsibilities:

• ✅ Audit log for sensitive operations
• ✅ Data deletion workflow
• ✅ Retention policy enforcement
• ✅ Access logging

Technology Stack:

• PostgreSQL (audit log)
• S3 lifecycle policies
• Scheduled jobs

Database Schema:

audit_logs:
  - id (UUID)
  - user_id
  - action (viewed_report, deleted_session, etc.)
  - resource_id (session_id, etc.)
  - ip_address
  - user_agent
  - created_at

Operations to Audit:

• Session creation
• Report viewing
• Media access
• Session deletion
• Admin actions

TODO:

• Implement audit logging middleware
• Create deletion API
• Setup retention jobs
• Define retention periods per market

---

🌐 CLIENT-SIDE: What Goes on Client

Core Principle

Client = Smart media processor that handles all recording, encoding, and speech processing

Client Responsibilities Overview

┌─────────────────────────────────────────────────────────────┐
│                      CLIENT SIDE                             │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  1. MEDIA CAPTURE                                     │  │
│  │     • Camera/Microphone recording                    │  │
│  │     • Screen capture (for coding)                    │  │
│  │     • Device permission management                   │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  2. MEDIA PROCESSING                                 │  │
│  │     • Video/audio compression                        │  │
│  │     • Chunking (30-second segments)                  │  │
│  │     • Format encoding (WebM/VP9/Opus)                │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  3. SPEECH PROCESSING                                │  │
│  │     • Speech-to-Text (Whisper WASM)                  │  │
│  │     • Text-to-Speech (Web Speech API)                │  │
│  │     • Real-time transcription                        │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  4. AVATAR RENDERING                                 │  │
│  │     • 2D/3D avatar display                           │  │
│  │     • Lip sync (viseme mapping)                      │  │
│  │     • Facial expressions                             │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  5. UPLOAD MANAGEMENT                                │  │
│  │     • Direct S3/R2 uploads                           │  │
│  │     • Background upload (non-blocking)               │  │
│  │     • Retry logic                                    │  │
│  │     • Progress tracking                              │  │
│  └──────────────────────────────────────────────────────┘  │
│                          ↓                                   │
│  ┌──────────────────────────────────────────────────────┐  │
│  │  6. USER INTERFACE                                   │  │
│  │     • Interview flow UI                              │  │
│  │     • Real-time feedback                             │  │
│  │     • Progress indicators                            │  │
│  │     • Error handling                                 │  │
│  └──────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

---

💻 CLIENT MODULES & TECHNOLOGIES

Module 1: Media Recorder

Purpose: Capture video, audio, and screen

Technology:

• MediaRecorder API (native browser)
• getUserMedia API (camera/mic access)
• getDisplayMedia API (screen capture)

Features:

• Configure video resolution (1280x720)
• Audio quality settings (48kHz, echo cancellation)
• Chunked recording (30-second timeslices)
• Format: WebM with VP9 video codec and Opus audio codec

Implementation:

navigator.mediaDevices.getUserMedia({
  video: { width: 1280, height: 720 },
  audio: { echoCancellation: true, noiseSuppression: true }
})

TODO:

• Handle device permissions
• Implement fallback for unsupported browsers
• Add device selection UI

---

Module 2: Speech-to-Text (STT)

Purpose: Convert speech to text in real-time

Technology Options:

Primary: Whisper.cpp WASM

• Local processing (privacy)
• Model sizes: tiny (40MB), base (140MB)
• Offline capable
• High accuracy

Fallback: Web Speech Recognition API

• Native browser API
• Online only
• Variable accuracy across browsers

Decision: Start with Web Speech API, add Whisper WASM as enhancement

Features:

• Real-time or near-real-time transcription
• Confidence scores
• Language detection
• Punctuation

TODO:

• Integrate Whisper.cpp WASM bindings
• Implement Web Speech API fallback
• Add language selection
• Handle low-confidence results

---

Module 3: Text-to-Speech (TTS)

Purpose: Synthesize avatar voice from text

Technology Options:

Primary: Web Speech Synthesis API

• Native browser voices
• Zero cost
• Immediate playback
• Variable quality across platforms

Premium: Azure Speech SDK (optional)

• Consistent quality
• Neural voices
• Viseme data (for lip sync)
• Cost: ~$0.016 per 1K characters

Features:

• Voice selection
• Rate/pitch/volume control
• Playback callbacks
• SSML support (optional)

TODO:

• Implement Web Speech Synthesis
• Add voice selection UI
• Optionally integrate Azure SDK for premium users
• Handle speech interruption

---

Module 4: Avatar Renderer

Purpose: Render AI interviewer with lip sync

Technology Options:

Option 1: Canvas 2D

• Simple 2D animated avatar
• Lightweight
• Good browser support

Option 2: WebGL/Three.js

• 3D realistic avatar
• More engaging
• Higher performance requirements

Lip Sync Technology:

• Viseme mapping (phoneme → mouth shape)
• Sync with TTS timing
• Facial expressions (optional)

Features:

• Smooth animations
• Lip sync accuracy
• Idle animations
• Expression changes (friendly, thinking, etc.)

TODO:

• Choose 2D vs 3D approach
• Design/source avatar assets
• Implement viseme mapping
• Add facial expressions

---

Module 5: Upload Manager

Purpose: Upload media chunks to S3/R2 via pre-signed URLs

Technology:

• Fetch API (native)
• Chunked uploads
• Background processing

Features:

• Direct PUT to pre-signed URLs (no server proxy)
• Progress tracking per chunk
• Retry logic (exponential backoff)
• Queue management (upload in order)
• Handle URL expiry (refresh from server)

Flow:

1. Receive pre-signed URL from server
2. Chunk ready → PUT request to URL
3. Track upload progress
4. On success: notify server, mark chunk complete
5. On failure: retry 3 times, then report error
6. If URL expired: request new URL, retry

TODO:

• Implement upload queue
• Add retry logic with exponential backoff
• Create progress indicator UI
• Handle network failures gracefully

---

Module 6: State Management

Purpose: Manage client-side interview state

Technology:

• React Context API (if React)
• Zustand / Redux (for complex state)
• LocalStorage (persistence)

State to Manage:

• Current session config
• Current question
• Recording status
• Upload progress
• User responses (transcripts)
• Timer/progress
• Connection status

TODO:

• Define state structure
• Implement state management
• Add state persistence (survive page refresh)
• Handle state sync with server

---

Module 7: WebSocket Handler

Purpose: Real-time state sync with server

Technology:

• Native WebSocket API
• Reconnection logic

Events from Server:

• Next question
• Time warnings
• Session end
• Error messages

Events to Server:

• Ready for next question
• Recording status updates
• Pause requests

TODO:

• Implement WebSocket connection
• Add reconnection logic
• Handle connection failures
• Define message protocols

---

Module 8: User Interface Components

Purpose: Interview experience UI

Technology:

• React / Vue / Svelte (framework choice)
• Tailwind CSS / Material-UI (styling)
• Responsive design

Components:

• Interview setup screen
• Question display
• Avatar display
• Recording indicator
• Timer/progress bar
• Answer input (transcript display)
• Navigation controls
• Results display

TODO:

• Design UI mockups
• Implement component library
• Add responsive layouts
• Create loading states
• Add error states

---

Module 9: Optional - MediaPipe Integration

Purpose: Extract facial and body landmarks (client-side CV)

Technology:

• MediaPipe Face Mesh (468 landmarks)
• MediaPipe Pose (33 landmarks)
• WASM-based (runs in browser)

Use Case:

• Extract features locally
• Send compact feature data (not raw frames)
• Privacy-friendly
• Reduces upload bandwidth

Features:

• Real-time landmark detection
• Feature extraction (eye contact, posture)
• Export compact JSON (not video frames)

TODO:

• Evaluate if needed for v1
• Integrate MediaPipe libraries
• Define features to extract
• Create feature export format

---

📊 Data Flow Summary

During Interview

CLIENT                           SERVER
  │                                │
  │  1. POST /configure           │
  │ ───────────────────────────> │ Generate questions
  │ <─────────────────────────── │ Return config + URLs
  │                                │
  │  2. Record user response      │
  │     (local processing)        │
  │                                │
  │  3. PUT chunk to S3          │
  │ ──────────────────────> [S3] │ (direct, no server)
  │                                │
  │  4. POST /turn (transcript)   │
  │ ───────────────────────────> │ Store transcript
  │ <─────────────────────────── │ Return next question
  │                                │
  │  5. Repeat steps 2-4          │
  │                                │
  │  6. POST /complete            │
  │ ───────────────────────────> │ Queue analysis job
  │ <─────────────────────────── │ Return job ID
  │                                │
  │  7. Poll GET /report          │
  │ ───────────────────────────> │
  │ <─────────────────────────── │ Return report (when ready)

Media Never Touches Server

CLIENT                    S3/R2                SERVER
  │                         │                     │
  │  Record & Encode        │                     │
  │  ─────────────>         │                     │
  │                         │                     │
  │  Upload (PUT)           │                     │
  │  ──────────────────────>│                     │
  │                         │                     │
  │  (text only) POST /turn │                     │
  │  ─────────────────────────────────────────> │
  │                         │                     │
  │                         │  Fetch for analysis │
  │                         │ <─────────────────  │

---

🔧 Technology Stack Summary

Server Stack

Layer	Technology	Purpose
Language	Python 3.11+	Backend logic
Framework	FastAPI	REST API
Package Manager	UV	Fast dependency management
Database	PostgreSQL 15+	Data persistence
Cache	Redis	Session cache, queue broker
Storage	S3 / Cloudflare R2	Media storage
Queue	Celery + Redis	Background jobs
LLM	Claude 3.5 / GPT-4	AI logic
Auth	JWT	Stateless authentication
Deployment	Docker + ECS/EC2	Containerized deployment

Client Stack

Layer	Technology	Purpose
Language	JavaScript/TypeScript	Frontend logic
Framework	React / Vue / Svelte	UI framework
Recording	MediaRecorder API	Native recording
STT	Whisper.cpp WASM / Web Speech	Speech-to-text
TTS	Web Speech Synthesis / Azure	Text-to-speech
Avatar	Canvas 2D / Three.js	Avatar rendering
Upload	Fetch API	Direct S3 uploads
State	Context / Zustand	State management
Styling	Tailwind CSS	UI styling
Build	Vite / Webpack	Bundling

---

✅ Decision Checklist

Server Decisions Needed:

• LLM provider: Claude vs GPT-4?
• Storage: S3 vs Cloudflare R2?
• Deployment: AWS vs GCP vs Azure?
• Auth: Internal JWT or external service?
• Database: Self-hosted PostgreSQL or managed (RDS)?

Client Decisions Needed:

• Framework: React vs Vue vs Svelte?
• STT: Start with Web Speech or go straight to Whisper WASM?
• TTS: Web Speech only or add Azure premium option?
• Avatar: 2D simple or 3D realistic?
• MediaPipe: Include in v1 or later?

---

📋 Implementation Priority

Phase 1 (Week 1): Foundation

Server:

• Basic API structure
• JWT validation (TODO: design)
• Database setup
• Session management
• Pre-signed URL generation

Client:

• Media recording
• Basic UI
• Upload manager
• Web Speech API integration

Phase 2 (Week 2): Intelligence

Server:

• LLM question generation
• Follow-up logic
• Celery worker setup
• Scoring algorithms (basic)

Client:

• Avatar rendering
• TTS integration
• WebSocket connection
• State management

Phase 3 (Week 3): Polish

Server:

• Complete scoring (7 metrics)
• Report generation
• Historical trends
• Event publishing
• Compliance features

Client:

• UI polish
• Error handling
• Progress indicators
• Report display
• Offline handling

---

🎯 Success Criteria

Server:

• ✅ API latency p95 < 200ms
• ✅ Can handle 100 concurrent sessions
• ✅ Analysis completes in < 5 minutes
• ✅ Zero data loss
• ✅ GDPR/CCPA compliant

Client:

• ✅ Recording works on Chrome, Firefox, Safari, Edge
• ✅ STT accuracy > 90%
• ✅ Upload success rate > 99%
• ✅ Avatar lip sync smooth (no lag)
• ✅ Works on 5 Mbps connection

---

📝 Notes

1. Authentication is placeholder: JWT validation needs to be designed based on your auth system
2. Cost optimization: Client-side processing saves ~70% on server costs
3. Privacy: Media stays client-side or in storage, never processed on server in real-time
4. Scalability: Stateless server design enables horizontal scaling
5. Offline capability: Client can record offline, upload later (future enhancement)