growqr-app/docs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
docs/staging-vps-service-deployment-runbook.md
-Puter e6685203fe feat: initial docs repo with project inventory and all documentation 
- Added REPO_INVENTORY.md with all repos, branches, remotes, and staging info
- Added .gitignore
- Synced all existing docs from local workspace
- Centralized documentation hub for GrowQR team
2026-06-22 15:04:27 +05:30

67 KiB
Raw Permalink Blame History

GrowQR Staging VPS Service Deployment Runbook

Date: 2026-06-05 VPS SSH alias: gqr-temp VPS IP: 168.144.123.127 Staging wildcard DNS: *.gqr.puter.wtf Deployment root on VPS: /opt/growqr

User instructions followed

  • Deploy services one by one to the staging VPS.
  • For each service repo:
    • Check Git status, current branch, remotes, and Gitea remote.
    • If deployable work exists on a non-main branch, create a staging branch from the current branch.
    • Push staging to Gitea.
    • Deploy the staging branch/content to the VPS.
  • Most repos are expected to have a Gitea remote named gitea; if not, handle case by case.
  • Copy the local .env file to the VPS deployment.
  • Use Caddy as reverse proxy for APIs.
  • Keep room for many services/ports:
    • bind service/internal ports to 127.0.0.1 only
    • expose publicly only through Caddy subdomains
  • Use the wildcard DNS under *.gqr.puter.wtf for service URLs.

VPS base setup performed

Installed required packages on gqr-temp:

apt-get update
apt-get install -y git docker.io docker-compose caddy ca-certificates curl
systemctl enable --now docker caddy

Notes:

  • Debian 13 package name was docker-compose, not docker-compose-plugin.
  • Caddy is managed by systemd.
  • Docker Compose command available as docker-compose.

Git deployment branch procedure

For each service repo locally:

cd <service-repo>
git status --short --branch
git remote -v
git fetch --all --prune
git remote show gitea
git rev-list --left-right --count HEAD...gitea/main
git ls-remote --heads gitea staging origin staging

If there is no existing staging branch and the current branch contains the intended deployable changes:

git switch -c staging
git push -u gitea staging

Source transfer procedure used

Initial direct HTTPS clone from Gitea on the VPS failed because the remote requires credentials:

fatal: could not read Username for 'https://git.openputer.com': No such device or address

So we deployed by archiving the local staging worktree and copying it to the VPS:

cd <service-repo>
COPYFILE_DISABLE=1 tar \
  --exclude='./.venv' \
  --exclude='./__pycache__' \
  --exclude='*/__pycache__' \
  --exclude='./.pytest_cache' \
  -czf /tmp/<service>-staging.tgz .

scp /tmp/<service>-staging.tgz gqr-temp:/tmp/

ssh gqr-temp '
  rm -rf /opt/growqr/<service>
  mkdir -p /opt/growqr/<service>
  tar --no-same-owner -xzf /tmp/<service>-staging.tgz -C /opt/growqr/<service>
  cd /opt/growqr/<service>
  find . -name "._*" -type f -delete
  chown -R root:root .
  chmod 600 .env
  git status --short --branch
'

Important cleanup:

  • macOS archive metadata files like ._* were deleted on the VPS.
  • .env permissions were set to 600.
  • Ownership was normalized to root:root.

Port allocation strategy

To leave public port space clean, service ports were patched on the VPS so Docker only binds to loopback.

Pattern:

Purpose Port range style
API public reverse-proxy backend 127.0.0.1:180xx
Postgres internal/debug only 127.0.0.1:154xx
MinIO API internal/debug only 127.0.0.1:190xx
MinIO console internal/debug only 127.0.0.1:190xx+1

Only Caddy listens publicly on 80/443.

Caddy procedure

For each service, append a site block to /etc/caddy/Caddyfile:

<service-subdomain>.gqr.puter.wtf {
    encode gzip zstd
    reverse_proxy 127.0.0.1:<local-api-port>
}

Then validate, format, and reload:

cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak.$(date +%Y%m%d%H%M%S)
caddy fmt --overwrite /etc/caddy/Caddyfile
caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

Caddy automatically obtained Let's Encrypt certificates via the wildcard DNS-resolved subdomains.

Service 1: interview-service

Local repo status

Path:

/Users/puter/Workspace/growqr/interview-service

Before staging:

  • Branch: dashboard-service-rest-integration
  • Worktree: clean
  • Gitea remote: https://git.openputer.com/growqr-app/interview-service.git
  • GitHub origin also exists: https://github.com/GrowQR-Code/interview-service.git
  • Gitea default branch: main
  • Current branch was 1 commit ahead of gitea/main
  • No existing staging branch found

Created and pushed:

cd interview-service
git switch -c staging
git push -u gitea staging

VPS deployment

Remote path:

/opt/growqr/interview-service

Patched Docker Compose bindings on VPS:

api:      127.0.0.1:18007 -> 8000
postgres: 127.0.0.1:15440 -> 5432
minio:    127.0.0.1:19000 -> 9000
minio:    127.0.0.1:19001 -> 9001

Also changed Redis URL for this staging compose from host Redis to internal compose Redis:

REDIS_URL: redis://redis:6379/0

Started with:

cd /opt/growqr/interview-service
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d --build

Caddy URL:

https://interview-staging.gqr.puter.wtf

Health check:

curl -fsS https://interview-staging.gqr.puter.wtf/health

Expected response:

{"status":"ok","service":"interview-service","a2a":true,"agent":true}

Final container state:

interview-service-api-1        healthy   127.0.0.1:18007->8000
interview-service-postgres-1   healthy   127.0.0.1:15440->5432
interview-service-minio-1      healthy   127.0.0.1:19000->9000, 127.0.0.1:19001->9001
interview-service-redis-1      healthy   internal only

Service 2: roleplay-service

Local repo status

Path:

/Users/puter/Workspace/growqr/roleplay-service

Before staging:

  • Branch: dashboard-service-rest-integration
  • Worktree: clean
  • Gitea remote: https://git.openputer.com/growqr-app/roleplay-service.git
  • GitHub origin also exists: https://github.com/GrowQR-Code/roleplay-service.git
  • Gitea default branch: main
  • Current branch was 1 commit ahead of gitea/main
  • No existing staging branch found

Created and pushed:

cd roleplay-service
git switch -c staging
git push -u gitea staging

VPS deployment

Remote path:

/opt/growqr/roleplay-service

Patched Docker Compose bindings on VPS:

api:      127.0.0.1:18008 -> 8000
postgres: 127.0.0.1:15441 -> 5432
minio:    127.0.0.1:19010 -> 9000
minio:    127.0.0.1:19011 -> 9001

Started with:

cd /opt/growqr/roleplay-service
docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d --build

Caddy URL:

https://roleplay-staging.gqr.puter.wtf

Health check:

curl -fsS https://roleplay-staging.gqr.puter.wtf/health

Expected response:

{"status":"ok","service":"roleplay-service","a2a":true,"agent":true}

Final container state:

roleplay-service-api-1        healthy   127.0.0.1:18008->8000
roleplay-service-postgres-1   healthy   127.0.0.1:15441->5432
roleplay-service-minio-1      healthy   127.0.0.1:19010->9000, 127.0.0.1:19011->9001

qscore_service deployment (2026-06-05)

Local Git state

Repository:

/Users/puter/Workspace/growqr/qscore_service

Observed before deployment:

current branch: feature/quotients-from-pillars
modified: docker-compose.yml
untracked: .DS_Store, uv.lock
local .env: not present

Gitea remote was present as SSH fetch URL but SSH push failed with public-key auth. The push URL was changed to HTTPS and staging was pushed successfully:

cd qscore_service
git switch -C staging
git remote set-url --push gitea https://git.openputer.com/puter/qscore-service.git
git push -u gitea staging

VPS deployment

Remote path:

/opt/growqr/qscore_service

No local .env existed, so .env.example was copied to .env on the VPS and locked down with chmod 600.

Patched Docker Compose bindings on VPS:

api:      127.0.0.1:18009 -> 8000
postgres: 127.0.0.1:15442 -> 5432
redis:    127.0.0.1:16380 -> 6379

Started with:

cd /opt/growqr/qscore_service
docker-compose -p qscore-service-staging up -d --build

Caddy URL:

https://qscore-staging.gqr.puter.wtf

Health check:

curl -fsS https://qscore-staging.gqr.puter.wtf/health

Expected response:

{"status":"ok"}

Final container state:

qscore-service-staging-api-1        up       127.0.0.1:18009->8000
qscore-service-staging-postgres-1   healthy  127.0.0.1:15442->5432
qscore-service-staging-redis-1      healthy  127.0.0.1:16380->6379
qscore-service-staging-worker-1     up

growqr-app resume-builder and user-service deployment (2026-06-05)

These services live inside the monorepo:

/Users/puter/Workspace/growqr/growqr-app

Local Git state

Current branch before staging was dashboard-service-rest-integration, with modified service files under resume-builder/ and user-service/.

Created/pushed monorepo staging branch:

cd growqr-app
git switch -C staging
git push -u gitea staging

user-service VPS deployment

Remote path:

/opt/growqr/user-service

Copied local .env to the VPS and locked it down with chmod 600.

Patched Docker Compose bindings on VPS:

api:      127.0.0.1:18011 -> 8003
postgres: 127.0.0.1:15444 -> 5432
redis:    127.0.0.1:16382 -> 6379

Staging service URLs patched in compose:

PUBLIC_BASE_URL=https://user-staging.gqr.puter.wtf
RESUME_SERVICE_URL=https://resume-staging.gqr.puter.wtf
QSCORE_SERVICE_URL=https://qscore-staging.gqr.puter.wtf

Started with:

cd /opt/growqr/user-service
docker-compose -p user-service-staging up -d --build

Caddy URL:

https://user-staging.gqr.puter.wtf

Health check:

curl -fsS https://user-staging.gqr.puter.wtf/health

Expected response:

{"status":"healthy","service":"user-service","mcp":true,"agent":true,"a2a":true}

resume-builder VPS deployment

Remote path:

/opt/growqr/resume-builder

Copied local .env to the VPS and locked it down with chmod 600.

Patched Docker Compose bindings on VPS:

api:      127.0.0.1:18010 -> 8000
postgres: 127.0.0.1:15443 -> 5432
redis:    internal compose network only, exposed as 6379/tcp inside Docker

Patched USER_SERVICE_URL in compose:

USER_SERVICE_URL=https://user-staging.gqr.puter.wtf

Started with:

cd /opt/growqr/resume-builder
docker-compose -p resume-builder-staging up -d --build

Caddy URL:

https://resume-staging.gqr.puter.wtf

Health check:

curl -fsS https://resume-staging.gqr.puter.wtf/health

Expected response:

{"status":"healthy","service":"resume-builder","mcp":true,"agent":true,"a2a":true}

Final container state:

growqr_resume_api       healthy  127.0.0.1:18010->8000
growqr_resume_db        healthy  127.0.0.1:15443->5432
growqr_resume_redis     up       6379/tcp
growqr_user_service     up       127.0.0.1:18011->8003
growqr_users_db         healthy  127.0.0.1:15444->5432
growqr_users_redis      up       127.0.0.1:16382->6379

growqr-backend deployment

Local repo:

/Users/puter/Workspace/growqr/growqr-backend

Branch and commit deployed:

staging @ d10ef2a feat: personalize home feed suggestions

Started from chore/release, committed the local home-feed changes, created staging, and pushed to Gitea:

cd /Users/puter/Workspace/growqr/growqr-backend
npm run typecheck
git switch -c staging
git add src/home/home-feed-agent.ts src/home/home-feed.ts src/home/types.ts src/routes/home.ts
git commit -m "feat: personalize home feed suggestions"
git push -u origin staging

Remote path:

/opt/growqr/growqr-backend

Copied local .env to the VPS via the tar deployment and locked it down with chmod 600.

Patched Docker Compose bindings on VPS:

backend api: 127.0.0.1:18012 -> 4000
postgres:    127.0.0.1:15445 -> 5432
gitea http:  127.0.0.1:13001 -> 3000
gitea ssh:   127.0.0.1:12222 -> 2222
rivet api:   127.0.0.1:16420 -> 6420
rivet guard: 127.0.0.1:16421 -> 6421

Patched staging URLs in .env/compose:

GITEA_PUBLIC_URL=https://backend-gitea-staging.gqr.puter.wtf
GITEA_ROOT_URL=https://backend-gitea-staging.gqr.puter.wtf
RIVET_CLIENT_ENDPOINT=https://backend-staging.gqr.puter.wtf/api/rivet
INTERVIEW_SERVICE_URL=https://interview-staging.gqr.puter.wtf
ROLEPLAY_SERVICE_URL=https://roleplay-staging.gqr.puter.wtf
QSCORE_SERVICE_URL=https://qscore-staging.gqr.puter.wtf
RESUME_SERVICE_URL=https://resume-staging.gqr.puter.wtf
FRONTEND_ORIGIN=https://dashboard-staging.gqr.puter.wtf

Started with:

cd /opt/growqr/growqr-backend
docker-compose -p growqr-backend-staging up -d --build

The first backend boot failed because the fresh Postgres volume had no tables. Applied migrations from the compiled runtime image:

docker-compose -p growqr-backend-staging run --rm backend node dist/db/migrate.js
docker-compose -p growqr-backend-staging up -d backend

Caddy URLs:

https://backend-staging.gqr.puter.wtf
https://backend-gitea-staging.gqr.puter.wtf

Health checks:

curl -fsS https://backend-staging.gqr.puter.wtf/healthz
curl -fsS https://backend-gitea-staging.gqr.puter.wtf/api/v1/version

Expected responses:

{"ok":true}
{"version":"1.22.6"}

growqr-dashboard frontend deployment

Local repo:

/Users/puter/Workspace/growqr/growqr-dashboard

Branch deployed:

staging @ 45ce954 updates

growqr-dashboard was on main ahead of origin/main by one commit. Created and pushed staging to Gitea:

cd /Users/puter/Workspace/growqr/growqr-dashboard
npm run build
git switch -c staging
git push -u origin staging

Remote path:

/opt/growqr/growqr-dashboard

Copied local .env to the VPS and locked it down with chmod 600.

Patched staging env values on VPS:

NEXT_PUBLIC_RIVET_ENDPOINT=https://backend-staging.gqr.puter.wtf/api/rivet
NEXT_PUBLIC_GROWQR_BACKEND_URL=https://backend-staging.gqr.puter.wtf
GROWQR_BACKEND_URL=https://backend-staging.gqr.puter.wtf
NEXT_PUBLIC_INTERVIEW_WS_URL=wss://interview-staging.gqr.puter.wtf/api/v1
NEXT_PUBLIC_ROLEPLAY_WS_URL=wss://roleplay-staging.gqr.puter.wtf/api/v1

Created Dockerfile.staging and docker-compose.staging.yml on the VPS. Dashboard binds only to localhost:

dashboard: 127.0.0.1:13000 -> 3000

Build note: npm ci failed because @clerk/nextjs@7.4.2 peer requirements conflict with next@14.2.18, while local build already uses the existing dependency resolution. The staging Dockerfile uses:

npm install --legacy-peer-deps

Started with:

cd /opt/growqr/growqr-dashboard
docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build

Caddy URL:

https://dashboard-staging.gqr.puter.wtf

Verification:

curl -fsSI https://dashboard-staging.gqr.puter.wtf

Expected status:

HTTP/2 200

Verification commands

Check all running services:

ssh gqr-temp 'docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"'

Check local listening ports:

ssh gqr-temp 'ss -ltnp | grep -E ":(80|443|13000|18007|18008|18009|18010|18011|18012|15440|15441|15442|15443|15444|15445|16380|16382|19000|19001|19010|19011|12222|13001|16420|16421)" || true'

Check Caddy logs:

ssh gqr-temp 'journalctl -u caddy --no-pager -n 80'

Check service logs:

ssh gqr-temp 'cd /opt/growqr/interview-service && docker-compose -f docker-compose.yml -f docker-compose.staging.yml logs --tail=100 api'
ssh gqr-temp 'cd /opt/growqr/roleplay-service && docker-compose -f docker-compose.yml -f docker-compose.staging.yml logs --tail=100 api'
ssh gqr-temp 'cd /opt/growqr/qscore_service && docker-compose -p qscore-service-staging logs --tail=100 api'
ssh gqr-temp 'cd /opt/growqr/user-service && docker-compose -p user-service-staging logs --tail=100 user-service'
ssh gqr-temp 'cd /opt/growqr/resume-builder && docker-compose -p resume-builder-staging logs --tail=100 api'
ssh gqr-temp 'cd /opt/growqr/growqr-backend && docker-compose -p growqr-backend-staging logs --tail=100 backend'

Service dependency map

Staging services point at each other via public HTTPS URLs (not host.docker.internal):

dashboard-staging.gqr.puter.wtf
  ├─→ backend-staging.gqr.puter.wtf (API + Gitea + Rivet)
  ├─→ interview-staging.gqr.puter.wtf
  ├─→ roleplay-staging.gqr.puter.wtf
  ├─→ resume-staging.gqr.puter.wtf
  ├─→ user-staging.gqr.puter.wtf
  └─→ qscore-staging.gqr.puter.wtf

backend-staging.gqr.puter.wtf
  ├─→ interview-staging.gqr.puter.wtf
  ├─→ roleplay-staging.gqr.puter.wtf
  ├─→ qscore-staging.gqr.puter.wtf
  └─→ resume-staging.gqr.puter.wtf

user-service-staging
  ├─→ resume-staging.gqr.puter.wtf
  └─→ qscore-staging.gqr.puter.wtf

resume-builder-staging
  └─→ user-staging.gqr.puter.wtf

Port allocation reference

Service API Postgres Redis MinIO API MinIO Console Other
interview-service 18007 15440 internal 19000 19001 —
roleplay-service 18008 15441 internal 19010 19011 —
qscore_service 18009 15442 16380 — — —
resume-builder 18010 15443 internal — — —
user-service 18011 15444 16382 — — —
growqr-backend 18012 15445 — — — Gitea 13001/12222, Rivet 16420/16421
growqr-dashboard 13000 — — — — —

Public ports: only Caddy on 80/443.

Health check endpoints quick reference

Service Endpoint Expected response
interview-service GET /health {"status":"ok","service":"interview-service","a2a":true,"agent":true}
roleplay-service GET /health {"status":"ok","service":"roleplay-service","a2a":true,"agent":true}
qscore_service GET /health {"status":"ok"}
resume-builder GET /health {"status":"healthy","service":"resume-builder","mcp":true,"agent":true,"a2a":true}
user-service GET /health {"status":"healthy","service":"user-service","mcp":true,"agent":true,"a2a":true}
growqr-backend GET /healthz {"ok":true}
growqr-gitea GET /api/v1/version {"version":"1.22.6"}
growqr-dashboard GET / (root page) HTTP/2 200

Post-reboot / resize recovery

After a VPS reboot or resize, Docker containers with restart: unless-stopped or compose restart policies may not auto-start if they were not running when systemd shut down. To recover all services:

# Backend services
ssh gqr-temp '
  cd /opt/growqr/interview-service && docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
  cd /opt/growqr/roleplay-service && docker-compose -f docker-compose.yml -f docker-compose.staging.yml up -d
  cd /opt/growqr/qscore_service && docker-compose -p qscore-service-staging up -d
  cd /opt/growqr/user-service && docker-compose -p user-service-staging up -d
  cd /opt/growqr/resume-builder && docker-compose -p resume-builder-staging up -d
  cd /opt/growqr/growqr-backend && docker-compose -p growqr-backend-staging up -d
  cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d
'

Also ensure Caddy is running:

ssh gqr-temp 'systemctl is-active caddy || systemctl start caddy'

Common issues & fixes encountered

1. Gitea clone requires credentials on VPS

Direct HTTPS clone from git.openputer.com fails because the VPS has no stored credentials. Fix: archive the local worktree, scp it, and extract on the VPS.

2. Gitea SSH push fails with public-key auth

qscore_service had a Gitea SSH remote but push failed. Fix: switch the push URL to HTTPS:

git remote set-url --push gitea https://git.openputer.com/puter/qscore-service.git

3. Fresh backend Postgres has no tables

On first growqr-backend deploy, the backend container crashed with:

PostgresError: relation "user_stacks" does not exist

Fix: run the compiled migration script inside the built image before starting the backend:

cd /opt/growqr/growqr-backend
docker-compose -p growqr-backend-staging run --rm backend node dist/db/migrate.js
docker-compose -p growqr-backend-staging up -d backend

4. Next.js peer dependency conflict during Docker build

npm ci failed because @clerk/nextjs@7.4.2 peer requirements conflict with next@14.2.18. Fix in the Dockerfile:

RUN npm install --legacy-peer-deps

5. VPS overloaded during Next.js build

The initial VPS size (1 vCPU / 2 GB RAM) caused the next build to hang and the VPS became unreachable. Fix: resize the VPS to at least 4 vCPU / 8 GB RAM for frontend builds, or build locally and copy only the .next output.

6. Caddy certificate issuance delay

New subdomains may take 10–30 seconds for Let's Encrypt TLS-ALPN-01 validation. During that window, curl may return SSL: TLS alert internal error. Fix: wait for Caddy logs to show certificate obtained successfully before verifying HTTPS.

Environment variable staging checklist

When deploying a new service, the following env vars usually need changing from localhost/host.docker.internal to staging URLs:

  • GITEA_PUBLIC_URL / GITEA_ROOT_URL → https://backend-gitea-staging.gqr.puter.wtf
  • RIVET_CLIENT_ENDPOINT → https://backend-staging.gqr.puter.wtf/api/rivet
  • GROWQR_BACKEND_URL / NEXT_PUBLIC_GROWQR_BACKEND_URL → https://backend-staging.gqr.puter.wtf
  • FRONTEND_ORIGIN → https://dashboard-staging.gqr.puter.wtf
  • INTERVIEW_SERVICE_URL / NEXT_PUBLIC_INTERVIEW_WS_URL → https://interview-staging.gqr.puter.wtf / wss://interview-staging.gqr.puter.wtf
  • ROLEPLAY_SERVICE_URL / NEXT_PUBLIC_ROLEPLAY_WS_URL → https://roleplay-staging.gqr.puter.wtf / wss://roleplay-staging.gqr.puter.wtf
  • QSCORE_SERVICE_URL → https://qscore-staging.gqr.puter.wtf
  • RESUME_SERVICE_URL → https://resume-staging.gqr.puter.wtf
  • USER_SERVICE_URL / PUBLIC_BASE_URL → https://user-staging.gqr.puter.wtf
  • DATABASE_URL / Redis URLs → use internal compose service names (postgres, redis) unless binding to host for debugging

Docker build tips

  • Always add a .dockerignore on the VPS to reduce build context:
node_modules
.next
.git
screenshots
tsconfig.tsbuildinfo
.DS_Store
._*
*.tgz
  • For Node.js projects, if package-lock.json is stale, npm ci may fail. Use npm install --legacy-peer-deps as a fallback.
  • For Next.js frontends, the build requires significant RAM. If the VPS is small, consider building locally with npm run build and copying only the .next/ directory and public/ to the VPS.

Things to preserve for future services

  • Always check Git state before deployment.
  • Prefer Gitea remote gitea for staging branches.
  • Create staging branch from the currently validated deploy branch.
  • Push staging to Gitea before deploying.
  • Copy .env to the VPS with restrictive permissions.
  • Bind Docker service ports to 127.0.0.1, not 0.0.0.0.
  • Use unique local port ranges for each service.
  • Expose APIs only through Caddy subdomains.
  • Validate Caddy before reload.
  • Confirm both localhost health and public HTTPS health.
  • Keep deployment directories under /opt/growqr/<service>.
  • After a VPS resize/reboot, remember to restart all stopped Docker Compose stacks.
  • Run database migrations for new backends before starting the API container.

Final staging polish: auth, CORS, service URLs, and restart policy

Performed on 2026-06-05 after the dashboard exposed a stale localhost:8007 browser request and backend /home/feed returned 500s from miswired user-service calls.

Canonical staging URLs now used consistently:

Surface URL
Dashboard https://dashboard-staging.gqr.puter.wtf
Backend API https://backend-staging.gqr.puter.wtf
Backend Gitea https://backend-gitea-staging.gqr.puter.wtf
Interview https://interview-staging.gqr.puter.wtf
Roleplay https://roleplay-staging.gqr.puter.wtf
QScore https://qscore-staging.gqr.puter.wtf
Resume Builder https://resume-staging.gqr.puter.wtf
User Service https://user-staging.gqr.puter.wtf

Applied consistency fixes:

  • Copied the same Clerk publishable/secret/JWKS/issuer values from local envs into all deployed service envs that use or may validate Clerk tokens.
  • Set NODE_ENV=production for backend/dashboard and ENV=production for FastAPI-style staging services.
  • Replaced stale downstream URLs (localhost, host.docker.internal) with staging HTTPS URLs where cross-service calls happen through Caddy.
  • Set CORS origins to include at least https://dashboard-staging.gqr.puter.wtf and https://backend-staging.gqr.puter.wtf across services.
  • Patched dashboard leaderboard/artifact defaults so browser code falls back to /api/growqr/services/*, not http://localhost:8007 / :8008.
  • Rebuilt growqr-dashboard so NEXT_PUBLIC_* values were baked into the production bundle.
  • Recreated service containers with updated env and set Docker restart policy to unless-stopped for deployed service containers.

Verification commands:

# No baked frontend localhost service URLs
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "grep -R \"localhost:800[2378]\|127.0.0.1:800[2378]\" -n .next/static .next/server 2>/dev/null | head || true"'

# Public health checks
for url in \
  https://dashboard-staging.gqr.puter.wtf/ \
  https://backend-staging.gqr.puter.wtf/healthz \
  https://backend-gitea-staging.gqr.puter.wtf/api/v1/version \
  https://interview-staging.gqr.puter.wtf/health \
  https://roleplay-staging.gqr.puter.wtf/health \
  https://qscore-staging.gqr.puter.wtf/health \
  https://resume-staging.gqr.puter.wtf/health \
  https://user-staging.gqr.puter.wtf/health; do
  curl -ksS -o /tmp/out -w "%{http_code} %{url_effective}\n" "$url"
  head -c 160 /tmp/out; echo
done

# CORS preflight example
curl -ksSI -X OPTIONS https://backend-staging.gqr.puter.wtf/healthz \
  -H 'Origin: https://dashboard-staging.gqr.puter.wtf' \
  -H 'Access-Control-Request-Method: GET' \
  -H 'Access-Control-Request-Headers: authorization,content-type'

Expected state after polish:

  • All public health checks return 200.
  • CORS preflights include access-control-allow-origin: https://dashboard-staging.gqr.puter.wtf.
  • growqr-backend can fetch health from user, resume, interview, roleplay, and qscore via their staging HTTPS URLs.
  • Dashboard production bundle contains no localhost:8007, localhost:8008, 127.0.0.1:8007, or 127.0.0.1:8008 service URLs.

Additional actor/runtime polish:

  • Added/passed RIVET_RUNNER_VERSION=staging-20260605-polish for the backend image/build so RivetKit self-hosted runner stops warning about unversioned actors.
  • Verified growqr-rivet is running and backend starts without the RIVET_RUNNER_VERSION is not set warning.
  • Verified backend /home/feed returns 200 with a trusted service token smoke test after the user-service URL fix.

Staging fix: onboarding QScore baseline and Clerk avatar image loading

Performed on 2026-06-05 after a fresh-account onboarding showed three different Q Score states:

  • onboarding completion screen: static QX Score · 35
  • dashboard header: home-feed fallback score 47
  • /agents/qscore: Not ready / Not computed yet

Code changes deployed:

Repo Staging commit Purpose
growqr-backend 213987a fix: persist onboarding qscore baseline Persists the onboarding baseline as a real Q Score projection when onboarding is completed and the account has no existing Q Score signals.
growqr-dashboard 08f0250 fix: align qscore header and avatar images Removes the fake header default score, shows an awaiting state until the backend returns a score, and bypasses Next image optimization for Clerk avatars.

Backend behavior after this fix:

  • On PATCH /users/me, if preferences.onboarding.completed_at is present and the user has no existing Q Score signals/projection, backend seeds:
    • grow_qscore_latest.signal_id = onboarding.completed_baseline
    • score = 35
    • source = onboarding
    • grow_qscore_projection_state.score = 35
  • /home/feed also lazily seeds the same baseline from user-service preferences before building the identity rail.
  • /services/qscore/current also lazily seeds before reading the Q Score page data, so direct navigation to /agents/qscore becomes consistent after a fresh onboarding.
  • Existing users with any Q Score signal or non-zero projection are not overwritten.

Dashboard behavior after this fix:

  • The top bar no longer initializes QX Score to the fake 76 or keeps stale fallback values after failed fetches.
  • Header score now renders — / Awaiting baseline until the backend returns a real score.
  • Clerk avatars use next/image with unoptimized so remote Clerk/Google avatar URLs do not fail through Next's image optimizer.
  • next.config.mjs also allows common Clerk/Google/Gravatar remote hosts for other optimized image usage.

Deployment commands used:

# Local verification before deploy
cd /Users/puter/Workspace/growqr/growqr-backend
npm run typecheck
npm run build

cd /Users/puter/Workspace/growqr/growqr-dashboard
npm run build

# Push staging branches
cd /Users/puter/Workspace/growqr/growqr-backend
git push origin staging
cd /Users/puter/Workspace/growqr/growqr-dashboard
git push origin staging

# Copy changed files without overwriting remote .env / compose staging patches
scp growqr-backend/src/events/onboarding-qscore.ts gqr-temp:/opt/growqr/growqr-backend/src/events/onboarding-qscore.ts
scp growqr-backend/src/routes/users.ts growqr-backend/src/routes/services.ts gqr-temp:/opt/growqr/growqr-backend/src/routes/
scp growqr-backend/src/home/home-feed.ts gqr-temp:/opt/growqr/growqr-backend/src/home/
scp growqr-dashboard/components/layout/TopBar.tsx gqr-temp:/opt/growqr/growqr-dashboard/components/layout/TopBar.tsx
scp growqr-dashboard/next.config.mjs gqr-temp:/opt/growqr/growqr-dashboard/next.config.mjs

# Rebuild/recreate only affected services
ssh gqr-temp 'cd /opt/growqr/growqr-backend && docker-compose -p growqr-backend-staging build backend && docker-compose -p growqr-backend-staging up -d backend'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml build dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d dashboard'

Verification commands:

ssh gqr-temp 'docker ps --filter name=growqr-backend --filter name=growqr-dashboard --format "{{.Names}} {{.Status}}"'
ssh gqr-temp 'curl -fsS http://127.0.0.1:18012/healthz && echo'
curl -fsS https://backend-staging.gqr.puter.wtf/healthz
curl -fsSI https://dashboard-staging.gqr.puter.wtf | head -n 1

# Ensure new backend helper is in the running image
ssh gqr-temp 'docker exec growqr-backend sh -lc "grep -R \"completed onboarding baseline\" -n /app/dist | head -5"'

# Ensure old fake dashboard score initializer is gone from built assets
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "grep -R \"useState(76)\" -n /app/.next/server /app/.next/static 2>/dev/null || true"'

Expected user-facing result:

  • A fresh onboarded account that has no service activity should consistently show Q Score 35 in the onboarding baseline, dashboard header, and Q Score page after refresh/navigation.
  • Once real service signals arrive, the backend projection replaces the empty state with the computed readiness score and signal ledger.
  • The top-right account image should render instead of showing a broken optimized image response.

Staging fix: hide mission mocks in production and fix AppShell sidebar avatar

Performed on 2026-06-05 after the staging Missions page showed mock Interview-to-Offer Accelerator data to a fresh production-like account.

Code changes deployed:

Repo Staging commit Purpose
growqr-dashboard c7bbfb8 fix: hide demo missions in production Gates mission mock fallbacks behind NEXT_PUBLIC_SHOW_DEMO_MISSIONS=true; production/staging now render real backend data or empty states.
growqr-dashboard 95f0e75 fix: render Clerk avatars without optimizer Applies the same Clerk avatar image fix to the AppShell sidebar and the top-bar dropdown avatar.

Behavior after this fix:

  • /missions/active no longer falls back to MOCK_ACTIVE_SNAPSHOTS unless NEXT_PUBLIC_SHOW_DEMO_MISSIONS=true.
  • /missions/available no longer falls back to MOCK_AVAILABLE_MISSIONS unless NEXT_PUBLIC_SHOW_DEMO_MISSIONS=true.
  • /missions/[id] no longer resolves mock mission details unless the demo flag is enabled.
  • Production/staging empty state for no active missions: Nothing active yet with a CTA to browse available missions.
  • Production/staging empty state for no available missions: No missions available yet.
  • Sidebar and account dropdown Clerk avatars use next/image with unoptimized to avoid broken Next image optimizer responses for Clerk/remote account-image URLs.

Demo mode:

# Only set this in local/dev/demo deployments, not staging production-like deployments.
NEXT_PUBLIC_SHOW_DEMO_MISSIONS=true

Deployment commands used:

cd /Users/puter/Workspace/growqr/growqr-dashboard
npm run build
git push origin staging

scp 'growqr-dashboard/app/missions/(list)/active/page.tsx' 'gqr-temp:/opt/growqr/growqr-dashboard/app/missions/(list)/active/page.tsx'
scp 'growqr-dashboard/app/missions/(list)/available/page.tsx' 'gqr-temp:/opt/growqr/growqr-dashboard/app/missions/(list)/available/page.tsx'
scp 'growqr-dashboard/app/missions/[id]/page.tsx' 'gqr-temp:/opt/growqr/growqr-dashboard/app/missions/[id]/page.tsx'
scp growqr-dashboard/lib/demo-flags.ts gqr-temp:/opt/growqr/growqr-dashboard/lib/demo-flags.ts
scp growqr-dashboard/components/layout/Sidebar.tsx growqr-dashboard/components/layout/TopBar.tsx gqr-temp:/opt/growqr/growqr-dashboard/components/layout/

ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml build dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d dashboard'

Verification commands:

ssh gqr-temp 'docker ps --filter name=growqr-dashboard --format "{{.Names}} {{.Status}}"'
curl -ksS -o /dev/null -w "%{http_code}\n" https://dashboard-staging.gqr.puter.wtf/

# Staging should not set this flag.
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "printenv NEXT_PUBLIC_SHOW_DEMO_MISSIONS || true; printenv NODE_ENV"'

# The active missions build should include the flag-gated path, not unconditional mock fallback.
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "grep -R \"NEXT_PUBLIC_SHOW_DEMO_MISSIONS\|showDemoMissions\" -n /app/.next/server/app/missions /app/.next/static/chunks/app/missions 2>/dev/null | head"'

Staging fix: resume upload parsing state

Performed on 2026-06-05 after uploaded resumes showed only Uploaded, while the background parser was still working, and clicking Edit could surface a blocking Failed to parse resume browser alert.

Code change deployed:

Repo Staging commit Purpose
growqr-dashboard be9671e fix: show resume parsing state Treats uploaded resumes as parsing/processing, polls the resume API until parsed, and replaces blocking alerts with an in-UI parsing status modal.

Behavior after this fix:

  • A newly uploaded resume with status=uploaded now displays as Parsing with a spinner.
  • The resume card shows a processing overlay instead of implying it is ready.
  • The resume list polls /resumes every 3s while any resume remains uploaded or an edit-triggered parse is open.
  • Clicking Edit on an uploaded resume opens a non-blocking Preparing resume modal.
  • If parse-on-demand fails while background parsing is still underway, the UI keeps the card visible, shows the message inline, and continues polling instead of using alert().
  • When the API reports the resume has moved out of uploaded, the modal clears and the card becomes editable.

Deployment commands used:

cd /Users/puter/Workspace/growqr/growqr-dashboard
npm run build
git push origin staging

scp growqr-dashboard/components/resume/ResumeCard.tsx growqr-dashboard/components/resume/ResumeList.tsx gqr-temp:/opt/growqr/growqr-dashboard/components/resume/
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml build dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d dashboard'

Verification commands:

ssh gqr-temp 'docker ps --filter name=growqr-dashboard --format "{{.Names}} {{.Status}}"'
curl -ksS -o /dev/null -w "%{http_code}\n" https://dashboard-staging.gqr.puter.wtf/
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "grep -R \"We.ll unlock editing\|Large PDFs can take\|Parsing resume\" -n /app/.next/static/chunks /app/.next/server 2>/dev/null | head"'

Dashboard staging UI review deploy — PRM-32/33/34/37/44

Date: 2026-06-05 Repo: /Users/puter/Workspace/growqr/growqr-dashboard Branch: staging Commit: 61d0bea fix: polish dashboard review items Remote: origin → https://git.openputer.com/puter/growqr-dashboard.git VPS path: /opt/growqr/growqr-dashboard Compose project: growqr-dashboard-staging Container: growqr-dashboard Public URL: https://dashboard-staging.gqr.puter.wtf

Changes deployed:

  • PRM-33: Added GettingStartedHero above AgentSquad; uses Clerk user data, useDay(), and getRewards(day) for streak; camera button opens Clerk UserProfile via openUserProfile().
  • PRM-44: Added global growqr:toggle-chat event wiring and mounted HelperChatToggle from AppShell; TopBar chat button dispatches the event and collapses label on mobile.
  • PRM-37: Added QScoreWheel to IdentityRail; growth drivers display weight %, score %, chevron affordance, and clickable rows.
  • PRM-34: Standardized resume and cover-letter template thumbnail previews to aspect-[8.5/11].
  • PRM-32: Added docs/user-qr-code-plan.md design plan.

Local verification:

cd /Users/puter/Workspace/growqr/growqr-dashboard
npm run build
# Result: compiled successfully; zero build errors.
git diff --check
# Result: clean.

Deployment commands used:

cd /Users/puter/Workspace/growqr/growqr-dashboard
git add app/page.tsx components/AppShell.tsx \
  components/coverLetter/CoverLetterEditor/tabs/CoverLetterDesignTab.tsx \
  components/home/IdentityRail.tsx components/home/GettingStartedHero.tsx \
  components/layout/HelperChatToggle.tsx components/layout/TopBar.tsx \
  components/qscore/QScoreWheel.tsx \
  components/resume/ResumeEditor/tabs/DesignTab.tsx \
  components/resume/TemplateCard.tsx docs/user-qr-code-plan.md
git commit -m "fix: polish dashboard review items"
git push origin staging

TMP=/tmp/growqr-dashboard-staging-$(date +%s).tar.gz
tar --exclude='.git' --exclude='.next' --exclude='node_modules' --exclude='.env' --exclude='._*' -czf "$TMP" .
scp "$TMP" gqr-temp:/tmp/growqr-dashboard-staging.tar.gz
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && tar -xzf /tmp/growqr-dashboard-staging.tar.gz && find . -name "._*" -type f -delete && chown -R root:root . && chmod 600 .env && echo 61d0bea > .deployed-dashboard-git-sha'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build'

Post-deploy verification:

curl -I http://127.0.0.1:13000/
curl -I https://dashboard-staging.gqr.puter.wtf/
ssh gqr-temp 'docker inspect -f "{{.Name}} {{.State.Status}} {{.State.Running}} restart={{.HostConfig.RestartPolicy.Name}}" growqr-dashboard'
ssh gqr-temp 'docker exec growqr-dashboard sh -lc "find /app/.next -path /app/.next/cache -prune -o -type f -print | xargs grep -n \"localhost:8007\\|localhost:8008\\|127.0.0.1:8007\\|127.0.0.1:8008\" 2>/dev/null || true"'

Results:

  • Local and remote Docker builds succeeded.
  • Public dashboard returned HTTP/2 200.
  • Local dashboard returned HTTP/1.1 200 OK on 127.0.0.1:13000.
  • Container running with restart=unless-stopped.
  • Runtime bundle audit excluding .next/cache found no old localhost:8007/8008 service references.

Note: Docker build emitted transient Google Fonts fetch retry messages, then completed successfully.

Dashboard idle/data-fetching stability fix (2026-06-05)

Root cause found in staging: dashboard server-side proxy requests were going from the dashboard container to https://backend-staging.gqr.puter.wtf, causing public HTTPS/Caddy hairpin timeouts from inside Docker after idle/reconnect periods. This made /api/growqr/* intermittently return backend_timeout, which caused the home/resume UI to replace real state with empty fallback state.

Fix applied:

  • Dashboard runtime env now keeps public client URLs public, but uses internal Docker DNS for server-side proxying:
    • NEXT_PUBLIC_GROWQR_BACKEND_URL=https://backend-staging.gqr.puter.wtf
    • GROWQR_BACKEND_URL=http://growqr-backend:4000
  • /opt/growqr/growqr-dashboard/docker-compose.staging.yml attaches the dashboard container to the backend compose network:
services:
  dashboard:
    networks:
      - default
      - growqr-backend

networks:
  growqr-backend:
    external: true
    name: growqr-backend-staging_default

Verification commands:

ssh gqr-temp 'docker exec growqr-dashboard sh -lc "wget -qO- --timeout=5 http://growqr-backend:4000/healthz"'
ssh gqr-temp 'docker logs --since=5m growqr-dashboard 2>&1 | grep -i backend_timeout || true'

Frontend robustness shipped in growqr-dashboard commit 0be0b65:

  • Client API retries once with a fresh Clerk token (getToken({ skipCache: true })) on 401/403.
  • Home feed and TopBar send Clerk bearer tokens to /api/growqr/*, preserve last good data on transient failures, and refresh on focus/online/visibility restore.
  • Resume list preserves last good data instead of blanking after idle, refreshes in the background, and reconciles uploaded resumes before showing parse errors.
  • Resume edit flow no longer treats an on-demand parse race/timeout as terminal; it keeps polling and opens the editor automatically once the stored resume reaches draft/complete.

Resume upload/edit parsing correction (2026-06-05)

Problem: staging had drifted away from the original growqr-app/frontend resume flow. Uploads could trigger/background-track parsing and the dashboard UI waited/polled as if parsing had already begun. The intended flow is: upload stores the PDF as uploaded; parsing starts only when the user clicks Edit Resume; successful parse transitions the resume to draft and opens the editor.

Source commits pushed:

  • growqr-dashboard staging: 42ba346 fix: parse uploaded resumes on edit
  • growqr-app staging: e1b38d2 fix: defer resume parsing until edit

Files deployed to gqr-temp:

  • /opt/growqr/resume-builder/app/api/v1/resumes.py
  • /opt/growqr/growqr-dashboard/app/api/growqr/[...path]/route.ts
  • /opt/growqr/growqr-dashboard/components/resume/ResumeList.tsx
  • /opt/growqr/growqr-dashboard/components/resume/ResumeUpload.tsx

Backups were written with .pre-edit-parse-flow-bak before replacing files.

Docker Compose deployment commands used:

ssh gqr-temp 'cd /opt/growqr/resume-builder && docker-compose -p resume-builder-staging up -d --build api'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard'

Verification:

ssh gqr-temp 'cd /opt/growqr/resume-builder && docker-compose -p resume-builder-staging ps'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml ps'
curl -fsS https://resume-staging.gqr.puter.wtf/health
curl -fsS -I https://dashboard-staging.gqr.puter.wtf/features/resume

Results:

  • growqr_resume_api rebuilt via Docker Compose and is healthy.
  • growqr-dashboard rebuilt via Docker Compose and returns HTTP/2 200 on the public resume route.
  • Resume upload endpoint no longer dispatches the orchestrator background parse task.
  • Dashboard proxy timeout default increased to 180 seconds so on-edit parsing can complete through /api/growqr/services/resume without a premature proxy backend_timeout.

Resume stored-PDF parse proxy fix (2026-06-05)

Follow-up issue from browser testing: clicking an uploaded resume showed Failed to parse resume; DevTools showed a FastAPI validation error:

{"detail":[{"type":"missing","loc":["body","file"],"msg":"Field required"}]}

Root cause: the dashboard called the correct client path:

/api/growqr/services/resume/parse/resume/<resume_id>/parse

but growqr-backend extracted the resume-service subpath with:

c.req.path.split("/resume/")[1]

Because the target path itself contains /resume/, this truncated the forwarded route to parse, so the resume service received:

POST /api/v1/parse

instead of:

POST /api/v1/parse/resume/<resume_id>/parse

Fixes shipped:

  • growqr-backend c47e6de fix: preserve nested resume service proxy paths
    • Uses the first /resume/ marker index and slices the remainder, preserving nested /resume/ path segments.
  • growqr-dashboard 624c5af fix: show uploaded resumes as ready to parse
    • Uploaded resumes now display as Uploaded — click Edit to parse.
    • The spinner overlay only appears during an actual Edit-triggered parse request.

Docker Compose deployment commands used:

ssh gqr-temp 'cd /opt/growqr/growqr-backend && docker-compose -p growqr-backend-staging up -d --build backend'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard'

Safe route verification used a non-existent UUID, so no resume was parsed or mutated:

curl -X POST \
  -H "Authorization: Bearer $SERVICE_TOKEN" \
  -H "x-growqr-user: user_3EX1LG3gBk3KY6kfD9PiTkWubs4" \
  http://127.0.0.1:18012/services/resume/parse/resume/00000000-0000-0000-0000-000000000000/parse

Expected/observed result:

404 {"detail":"Resume not found"}

Resume-service log confirmed the corrected forwarded path:

POST /api/v1/parse/resume/00000000-0000-0000-0000-000000000000/parse HTTP/1.1

Public verification:

curl -I https://backend-staging.gqr.puter.wtf/healthz
curl -I https://dashboard-staging.gqr.puter.wtf/features/resume

Both returned HTTP/2 200.

Interview live-avatar, personalization, MediaPipe, and video-analysis fix (2026-06-05)

Problem: after moving the interview UI from growqr-app/frontend/orchestrator to growqr-dashboard/growqr-backend, several orchestration behaviors were missing or incomplete:

  • The dashboard build did not ship the MediaPipe runtime files used by useFaceFeedback, so live camera coaching/presence analysis could not reliably start in production.
  • growqr-backend called interview-service directly but did not recreate the orchestrator's enriched user_context flow (candidate_name, resume skills/experience/education, LinkedIn headline/summary/experience, and opt-in candidate_profile).
  • interview-service page-state always returned resume_available=false and linkedin_available=false, hiding the personalization checkbox.
  • The browser video-analysis upload flow posted with an empty body while growqr-backend tried to parse JSON before proxying the request, so recorded session video registration could fail before the presigned upload flow began.
  • LiveAvatar envs were not enabled on the staging interview/roleplay containers.
  • Interview-service supported only Docker-internal MinIO presign URLs; browsers need public HTTPS presign endpoints.

Source commits pushed:

  • growqr-dashboard staging: 4c95dcb fix: restore interview video analysis assets
  • growqr-backend staging: aa8f285 fix: enrich interview service context
  • interview-service staging: 78c7070 fix: support public interview artifact presigning
  • roleplay-service staging: 6e64519 fix: support liveavatar api alias

Key source changes:

  • growqr-dashboard/package.json
    • Added @mediapipe/tasks-vision.
    • Added copy-mediapipe, predev, and prebuild scripts.
    • copy-mediapipe copies vision_bundle.mjs and the MediaPipe WASM files from node_modules into public/mediapipe at build time.
  • growqr-dashboard/public/mediapipe/face_landmarker.task
    • Checked in the face landmark model required by useFaceFeedback.
  • growqr-dashboard/lib/api/interview.ts and lib/api/roleplay.ts
    • Video upload-url/uploaded calls now send {} JSON bodies for compatibility with older backend deployments.
  • growqr-dashboard/app/features/interview/preview/page.tsx and app/agents/interview/preview/page.tsx
    • Regenerate preserves personalize=1 so regenerated plans keep the grounded candidate profile.
  • growqr-backend/src/routes/services.ts
    • Recreates the orchestrator context enrichment flow by reading user-service profile plus resume/social state.
    • Enriches interview configure payloads with candidate_name and, when requested, a compact candidate_profile built from resume + LinkedIn data.
    • Enriches /services/interview/page-state with real resume_available and linkedin_available flags.
    • Video upload proxy routes no longer require JSON parsing before proxying.
  • growqr-backend/src/services/product-service-clients.ts
    • Video upload proxy calls are explicit POSTs even when no body is required.
  • interview-service/app/core/config.py, app/services/storage.py, and app/api/artifacts.py
    • Added S3_PUBLIC_ENDPOINT_URL support for browser-reachable presigned URLs.
    • Added compatibility with the existing LIVEAVATAR_API env alias.
  • roleplay-service/app/core/config.py
    • Added compatibility with the existing LIVEAVATAR_API env alias.

Staging env changes applied on gqr-temp (values redacted in this runbook):

/opt/growqr/interview-service/.env
  LIVEAVATAR_ENABLED=true
  LIVEAVATAR_API_KEY=<redacted>
  LIVEAVATAR_API=<redacted>
  S3_PUBLIC_ENDPOINT_URL=https://interview-minio-staging.gqr.puter.wtf

/opt/growqr/roleplay-service/.env
  LIVEAVATAR_ENABLED=true
  LIVEAVATAR_API_KEY=<redacted>
  LIVEAVATAR_API=<redacted>
  S3_PUBLIC_ENDPOINT_URL=https://roleplay-minio-staging.gqr.puter.wtf

Backups were created with:

.env.pre-interview-fix-bak-<timestamp>

Caddy blocks added for browser-reachable, presigned MinIO upload/download URLs:

interview-minio-staging.gqr.puter.wtf {
    reverse_proxy 127.0.0.1:19000
}

roleplay-minio-staging.gqr.puter.wtf {
    reverse_proxy 127.0.0.1:19010
}

Validated and reloaded Caddy:

caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

Docker Compose deployment commands used:

ssh gqr-temp 'cd /opt/growqr/growqr-backend && docker-compose -p growqr-backend-staging up -d --build backend'
ssh gqr-temp 'cd /opt/growqr/interview-service && docker-compose -p interview-service up -d --build api'
ssh gqr-temp 'cd /opt/growqr/roleplay-service && docker-compose -p roleplay-service up -d --build api'
ssh gqr-temp 'cd /opt/growqr/growqr-dashboard && docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard'

Verification commands:

curl -fsS http://127.0.0.1:18012/healthz
curl -fsS http://127.0.0.1:18007/health
curl -fsS http://127.0.0.1:18008/health
curl -I https://dashboard-staging.gqr.puter.wtf/mediapipe/vision_bundle.mjs
curl -I https://dashboard-staging.gqr.puter.wtf/mediapipe/wasm/vision_wasm_internal.wasm
curl -I https://dashboard-staging.gqr.puter.wtf/mediapipe/face_landmarker.task
curl -I https://interview-minio-staging.gqr.puter.wtf/minio/health/live
curl -I https://roleplay-minio-staging.gqr.puter.wtf/minio/health/live

Observed:

  • Backend, interview-service, roleplay-service, and dashboard all healthy.
  • Dashboard route returns HTTP/2 200.
  • MediaPipe bundle, WASM, and face_landmarker.task all return HTTP/2 200.
  • Both MinIO public health endpoints return HTTP/2 200.
  • /services/interview/page-state now returns real personalization availability (verified with service-token probe: resume_available=true for the probed staging user).
  • A presign probe against an existing interview session returned a browser-reachable URL beginning with https://interview-minio-staging.gqr.puter.wtf/....

LiveAvatar sandbox-mode correction and VIP codes (2026-06-05)

Follow-up from browser comparison with the production interview page: staging was still showing the static interviewer headshot even when avatar_mode=video was stored on the interview session.

Root cause:

  • LIVEAVATAR_ENABLED=true and the API key were present, but LIVEAVATAR_SANDBOX was still at the service default of true.
  • The selected Payal/Emma/John persona avatar IDs are not supported in LiveAvatar sandbox mode. LiveAvatar returned 400 Bad Request with This avatar is not supported in sandbox mode, so the service correctly degraded to the static-photo path.

Fix:

/opt/growqr/interview-service/.env
  LIVEAVATAR_SANDBOX=false
  INTERVIEW_UNLIMITED_PHRASE=GQR-VIDEO-VIP-2026,GQR-LIVEAVATAR-TEST,PUTER-VIDEO-TEST

/opt/growqr/roleplay-service/.env
  LIVEAVATAR_SANDBOX=false
  ROLEPLAY_UNLIMITED_PHRASE=GQR-VIDEO-VIP-2026,GQR-LIVEAVATAR-TEST,PUTER-VIDEO-TEST

Source commits:

  • interview-service staging: 7c67ceb fix: support multiple interview vip codes
  • roleplay-service staging: 90392e0 fix: support multiple roleplay vip codes

Deployment:

ssh gqr-temp 'cd /opt/growqr/interview-service && docker-compose -p interview-service up -d --build api'
ssh gqr-temp 'cd /opt/growqr/roleplay-service && docker-compose -p roleplay-service up -d --build api'

Verification:

  • Container settings now show LIVEAVATAR_ENABLED=True, LIVEAVATAR_SANDBOX=False, API key present, and all three VIP codes loaded.
  • A staging backend configure call using unlimited_phrase=GQR-LIVEAVATAR-TEST produced config.avatar_mode=video for a user already at the video cap.
  • A direct websocket smoke test for that configured session returned:
session.ready has_avatar=True provider=liveavatar livekit_url=wss://heygen-...

User-facing test instruction: choose Video mode on Interview Setup, expand/enter VIP code, and use any of:

GQR-VIDEO-VIP-2026
GQR-LIVEAVATAR-TEST
PUTER-VIDEO-TEST

2026-06-05 — Roleplay LiveAvatar parity polish

Goal: make staging Roleplay behave like the production GrowQR app/orchestrator path, same as the Interview LiveAvatar polish.

Source changes

  • growqr-backend/src/routes/services.ts

    • Added roleplay configure enrichment equivalent to the production orchestrator/A2A flow.
    • POST /services/roleplay/configure now resolves Grow user context from user-service, resume-builder, and social-branding, then forwards:
      • metadata.candidate_name
      • metadata.target_role
      • metadata.candidate_role
      • metadata.difficulty
      • metadata.candidate_profile
      • metadata.context_notes
      • user_context
      • qscore fallback/defaults
    • GET /services/roleplay/page-state now augments roleplay page-state with resume_available and linkedin_available just like interview page-state.

  • roleplay-service/app/services/storage.py

    • Fixed public MinIO presigning so explicit staging domains like https://roleplay-minio-staging.gqr.puter.wtf are preserved.
    • The old fallback rewrite now only changes the exact Docker-internal hostname minio to localhost; it no longer rewrites public domains containing the word minio.

Commits deployed

growqr-backend staging: 170d358 fix: enrich roleplay service context
roleplay-service staging: e3d50e5 fix: preserve public roleplay minio presign host

Staging deployment notes

During tar-based redeploy, the staging-only compose/env patches must be preserved or re-applied:

  • Backend API port: 127.0.0.1:18012:4000
  • Roleplay API port: 127.0.0.1:18008:8000
  • Roleplay Postgres: 127.0.0.1:15441:5432
  • Roleplay MinIO: 127.0.0.1:19010:9000, 127.0.0.1:19011:9001
  • Backend env URLs must point at staging HTTPS service domains, not local worktree defaults.
  • Backend RIVET_RUNNER_VERSION must be set to a non-dev staging value to avoid Rivet runner version collisions.
  • Roleplay env must include:
BASE_URL=https://roleplay-staging.gqr.puter.wtf
S3_ENDPOINT_URL=http://minio:9000
S3_PUBLIC_ENDPOINT_URL=https://roleplay-minio-staging.gqr.puter.wtf
LIVEAVATAR_ENABLED=true
LIVEAVATAR_SANDBOX=false
ROLEPLAY_UNLIMITED_PHRASE=GQR-VIDEO-VIP-2026,GQR-LIVEAVATAR-TEST,PUTER-VIDEO-TEST
CORS_ORIGINS=https://dashboard-staging.gqr.puter.wtf,https://backend-staging.gqr.puter.wtf

Verification performed

Health:

curl -fsS http://127.0.0.1:18012/healthz
curl -fsS http://127.0.0.1:18008/health

Roleplay configure smoke through backend with VIP video mode returned:

session_id=<uuid>
avatar_mode=video
metadata includes candidate_profile/context_notes

Roleplay presigned upload URL now starts with the browser-reachable HTTPS MinIO domain:

https://roleplay-minio-staging.gqr.puter.wtf/roleplay-artifacts/...

End-to-end WebSocket smoke:

wss://roleplay-staging.gqr.puter.wtf/api/v1/roleplays/session/<session_id>
→ send {"type":"session.start"}
→ received session.ready with avatar.provider=liveavatar and livekit_url present

Tested VIP code:

GQR-LIVEAVATAR-TEST

2026-06-06 — Dashboard mission runtime + home feed wiring polish

Goal: keep the existing dashboard UI, but wire it to the mission-first backend runtime and make Home/Missions production-ready on staging.

Changes deployed:

  • growqr-dashboard staging commit d52c160 (fix: wire mission runtime to dashboard)
    • /api/growqr/* now guards against stale loopback backend envs in production Docker. If GROWQR_BACKEND_URL is accidentally localhost:4000 / 127.0.0.1:4000, the proxy uses http://growqr-backend:4000 on the shared compose network.
    • lib/grow-api.ts now understands actionsByMission and mission action endpoints:
      • approve/reject/run/answer/snooze
      • manual daily scrum: POST /missions/active/:instanceId/scrum/run
    • Active mission cards now render backend mission_actions inside the existing card-deck UI and can execute HITL actions without adding new navigation/sidebar surfaces.
  • growqr-backend staging commit dd48321 (fix: keep home feed responsive)
    • Product service page-state calls used by Home feed now have a short timeout (PRODUCT_SERVICE_TIMEOUT_MS, default 3500ms) so one slow service cannot block the dashboard.
    • Home Feed Agent LLM refinement has a bounded timeout (HOME_FEED_AGENT_TIMEOUT_MS, default 8000ms) and falls back to deterministic notifications.
  • Staging dashboard env fixed in /opt/growqr/growqr-dashboard/.env:
    • GROWQR_BACKEND_URL=http://growqr-backend:4000
    • NEXT_PUBLIC_GROWQR_BACKEND_URL=https://backend-staging.gqr.puter.wtf
    • GROWQR_BACKEND_TIMEOUT_MS=180000

Deployment commands used:

# Dashboard: copy only committed mission/proxy files, preserving unrelated working-tree UI edits
cd growqr-dashboard
git archive --format=tar HEAD \
  'app/api/growqr/[...path]/route.ts' \
  'app/missions/(list)/active/page.tsx' \
  components/mission/ActiveMissionView.tsx \
  components/mission/AgentCard.tsx \
  components/mission/adapter.ts \
  lib/grow-api.ts \
  > /tmp/growqr-dashboard-mission-wire.tar
scp /tmp/growqr-dashboard-mission-wire.tar gqr-temp:/tmp/

# Backend: copy only responsive home-feed files
cd growqr-backend
git archive --format=tar HEAD \
  src/home/home-feed-agent.ts \
  src/services/product-service-clients.ts \
  > /tmp/growqr-backend-home-responsive.tar
scp /tmp/growqr-backend-home-responsive.tar gqr-temp:/tmp/

ssh gqr-temp
cd /opt/growqr/growqr-dashboard
cp .env .env.pre-mission-wire-bak.$(date +%Y%m%d%H%M%S)
tar -xf /tmp/growqr-dashboard-mission-wire.tar
chmod 600 .env

cd /opt/growqr/growqr-backend
tar -xf /tmp/growqr-backend-home-responsive.tar

docker-compose -p growqr-backend-staging up -d --build backend
cd /opt/growqr/growqr-dashboard
docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard

Verification performed:

# Dashboard container can reach backend over Docker DNS
docker exec growqr-dashboard wget -qO- http://growqr-backend:4000/healthz
# => {"ok":true}

# Public app and backend are healthy
curl -I https://dashboard-staging.gqr.puter.wtf/
curl https://backend-staging.gqr.puter.wtf/healthz

# Dashboard proxy works for mission definitions, active mission runtime, and home feed
curl -H "Authorization: Bearer $SERVICE_TOKEN" -H "x-growqr-user: $USER_ID" \
  http://127.0.0.1:13000/api/growqr/missions/available
curl -H "Authorization: Bearer $SERVICE_TOKEN" -H "x-growqr-user: $USER_ID" \
  http://127.0.0.1:13000/api/growqr/missions/active
curl -H "Authorization: Bearer $SERVICE_TOKEN" -H "x-growqr-user: $USER_ID" \
  http://127.0.0.1:13000/api/growqr/home/feed

# Manual mission scrum endpoint works through dashboard proxy
curl -X POST -H "Authorization: Bearer $SERVICE_TOKEN" -H "x-growqr-user: $USER_ID" \
  http://127.0.0.1:13000/api/growqr/missions/active/<instanceId>/scrum/run

Expected browser result after hard refresh:

  • Home tiles should show real backend notification counts/content instead of all-zero fallback tiles.
  • /missions/available should show the five backend mission definitions.
  • /missions/active should show active missions and backend action/HITL cards in the existing active mission card UI.

2026-06-06 — Interview/Roleplay staging WebSocket endpoint fix

Symptom: the dashboard preview/session UI showed WebSocket connection error after successful Interview configure calls.

Root cause: growqr-dashboard staging .env still baked loopback WebSocket URLs into the Next.js browser bundle:

NEXT_PUBLIC_INTERVIEW_WS_URL=ws://127.0.0.1:8007/api/v1
NEXT_PUBLIC_ROLEPLAY_WS_URL=ws://127.0.0.1:8040/api/v1

Those URLs work only from the VPS, not from a user's browser. The public browser endpoints must use the Caddy HTTPS domains with wss://.

Fix deployed:

  • growqr-dashboard commit 6604ff8 (fix: use staging websocket endpoints)
  • Runtime/source fallback now maps *.gqr.puter.wtf browser sessions to:
    • wss://interview-staging.gqr.puter.wtf/api/v1
    • wss://roleplay-staging.gqr.puter.wtf/api/v1
  • Staging .env patched to explicitly bake:
NEXT_PUBLIC_INTERVIEW_WS_URL=wss://interview-staging.gqr.puter.wtf/api/v1
NEXT_PUBLIC_ROLEPLAY_WS_URL=wss://roleplay-staging.gqr.puter.wtf/api/v1
NEXT_PUBLIC_INTERVIEW_API_URL=/api/growqr/services/interview
NEXT_PUBLIC_ROLEPLAY_API_URL=/api/growqr/services/roleplay
NEXT_PUBLIC_INTERVIEW_ARTIFACTS_URL=/api/growqr/services/interview
NEXT_PUBLIC_ROLEPLAY_ARTIFACTS_URL=/api/growqr/services/roleplay

Deployment command:

cd /opt/growqr/growqr-dashboard
docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard

Verification:

# Dashboard bundle must not contain browser-invalid loopback WS URLs
docker exec growqr-dashboard sh -lc '
  for p in "localhost:8007" "127.0.0.1:8007" "localhost:8008" "127.0.0.1:8040"; do
    echo "$p:$(grep -R -l "$p" /app/.next/static /app/.next/server 2>/dev/null | wc -l)"
  done
'
# all counts should be 0

# Public websocket handshake should succeed; fake session should connect then return Session not found
# Interview: wss://interview-staging.gqr.puter.wtf/api/v1/session/<session_id>
# Roleplay:  wss://roleplay-staging.gqr.puter.wtf/api/v1/roleplays/session/<session_id>

Browser action required: hard-refresh https://dashboard-staging.gqr.puter.wtf so the new Next.js chunks are loaded.

2026-06-06 — Dashboard onboarding first-login trigger fix

Symptom: after signing in through the dashboard modal, a not-yet-onboarded user landed on the dashboard and the onboarding modal only appeared after a hard refresh.

Root cause: immediately after modal sign-in, Clerk client state can become signed-in before a usable session token/cookie is available to the Next.js /api/growqr/users/bootstrap proxy. The first user bootstrap could fail once, leaving UserProvider with user=null; OnboardingGate therefore had no user profile to inspect until a page refresh retried bootstrap with a settled session.

Fix deployed:

  • growqr-dashboard commit 1140d43 (fix: trigger onboarding immediately after sign-in)
  • UserProvider now retries user bootstrap with getToken({ skipCache: true }) before giving up.
  • GrowBootstrap sends an explicit Clerk bearer token instead of relying only on server-side cookies.
  • useOnboarding now merges saves with DEFAULT_ONBOARDING_DATA and the freshest preferences.onboarding, so every onboarding step persists a complete onboarding object to user-service.

Deployment command:

cd /opt/growqr/growqr-dashboard
docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard

Verification performed:

cd growqr-dashboard
npm run build

curl -I https://dashboard-staging.gqr.puter.wtf/
# => HTTP/2 200

Browser verification:

  1. Hard-refresh once to load commit 1140d43.
  2. Sign out / sign in with a user whose preferences.onboarding.completed_at is missing.
  3. Confirm the onboarding modal opens automatically without manually refreshing.
  4. Complete onboarding and confirm preferences.onboarding.completed_at plus the collected fields are saved in user-service.

2026-06-06 — Onboarding privacy-screen loop fix

Symptom: the first onboarding step opened, but clicking Continue after accepting privacy terms sent the user back to the same privacy screen.

Root cause: each onboarding step save calls refreshUser() after persisting to user-service. The previous first-login fix made UserProvider.fetchUser() set global isLoading=true for every refresh. OnboardingGate returned null while isLoading was true, which unmounted AdaptiveOnboarding; when the profile refresh completed, the modal remounted from local step=0, creating a loop back to the privacy screen.

Fix deployed:

  • growqr-dashboard commit dbdd681 (fix: keep onboarding mounted during saves)
  • UserProvider now shows global loading only before the first user profile arrives, not during refreshes.
  • OnboardingGate keeps the modal mounted while an existing user profile is refreshing.

Deployment command:

cd /opt/growqr/growqr-dashboard
docker-compose -p growqr-dashboard-staging -f docker-compose.staging.yml up -d --build dashboard

Verification performed:

cd growqr-dashboard
npm run build
curl -I https://dashboard-staging.gqr.puter.wtf/
# => HTTP/2 200

Browser action required: hard-refresh the dashboard once so the new bundle is used, then continue onboarding. Step saves should no longer reset the modal back to the privacy screen.

2026-06-06 — Onboarding Q Score baseline correction

Symptom: immediately after onboarding, the dashboard showed QX Score 100 even though a new user should start around the baseline (35).

Root cause: growqr-backend projected a plain resume.uploaded event as a perfect 100 signal. Users who uploaded a resume during onboarding therefore got a projection of 100 before any parsed resume analysis, interview review, or roleplay review existed.

Fix deployed:

  • growqr-backend commit 9fd478c (fix: keep onboarding qscore baseline at 35)
  • resume.uploaded now projects as baseline 35, not 100.
  • The onboarding baseline seeder also repairs users affected by the old upload-only 100 projection.

Deployment note:

  • Docker Hub returned 429 Too Many Requests while resolving node:22-alpine.
  • Workaround used on staging: tag the existing backend image as the local node:22-alpine base, then rebuild without pulling.
docker tag growqr-backend-staging-backend:latest node:22-alpine
cd /opt/growqr/growqr-backend
DOCKER_BUILDKIT=0 docker-compose -p growqr-backend-staging up -d --build backend

Verification performed for sumit26696@gmail.com:

resume.uploaded: 100 -> 35
home/feed identity.qx: { from: 35, to: 42, baseline: 35 }

The user's current score was 42 because later roleplay review signals existed; a fresh onboarding-only user with only the resume-upload baseline should now show 35, not 100.