163 lines
16 KiB
Markdown
163 lines
16 KiB
Markdown
# Deployment Architecture Recommendation
|
||
|
||
> **Status:** Partially implemented — staging environment is live as of 2026-08-03. The public Vercel/Convex endpoints and VDS services are operational; the declarative Pulumi/Ansible/CI backlog remains.
|
||
>
|
||
> **Goal:** retain an instant local iteration loop while operating one stable public staging environment with reproducible, declarative infrastructure.
|
||
|
||
## Decision
|
||
|
||
Use a deliberately split stack:
|
||
|
||
```text
|
||
Fast iteration (private development) Stable staging (public)
|
||
──────────────────────────────────── ────────────────────────
|
||
Mac + Tailscale Vercel + Contabo VDS
|
||
Web / Flue + AgentOS registry (one process) Web SSR Agents VDS
|
||
│ │ │
|
||
personal Convex dev deployment staging Convex deployment
|
||
```
|
||
|
||
| Concern | Iteration | Staging |
|
||
| --- | --- | --- |
|
||
| User-visible URL | Existing Tailscale URL on the Mac | `https://zopu-staging.vercel.app` on Vercel |
|
||
| Product backend | Personal Convex **dev** deployment | Dedicated long-lived Convex deployment |
|
||
| Web | Vite HMR on the Mac | Vercel React Router SSR |
|
||
| Agent worker | Local Flue + AgentOS registry (one Node process) + Engine | Contabo VDS: one Flue + AgentOS registry Node process + Rivet Engine |
|
||
| State/credentials | Development-only | Independent staging secrets, OAuth app, and volumes |
|
||
|
||
**Do not create a second remote “fast iteration” stack.** It duplicates the slowest part of the loop—building and replacing container images—while the existing Mac/Tailscale stack already exercises the complete topology from a phone. Staging is the public link and integration gate; local development is the fast environment.
|
||
|
||
## Why this split
|
||
|
||
The repository has three materially different runtime needs:
|
||
|
||
1. **Web** is a React Router SSR Node application. Its current production artifact is `react-router build` plus `react-router-serve`, and `apps/web/Dockerfile` already serves it on Node. Vercel supports React Router SSR and streaming, and is the appropriate managed host for this stateless application. [Vercel React Router guide](https://vercel.com/docs/frameworks/frontend/react-router)
|
||
2. **Convex** owns authentication, durable product data, workflows, and reactive client projections. It is not an application container to run on the VDS. The architecture explicitly requires clients to communicate only with Convex.
|
||
3. **Flue + in-process AgentOS registry + Rivet Engine** are the worker topology. Flue owns the registry request handler in the same Node process; the Engine remains a separate private service with durable state. Their filesystem-backed workspaces require a long-lived volume, so they are not appropriate for Vercel or Cloudflare Workers. [Flue Node deployment](https://flueframework.com/docs/ecosystem/deploy/node) · [Rivet runtime modes](https://rivet.dev/docs/general/runtime-modes)
|
||
|
||
Cloudflare remains valuable for authoritative DNS, TLS/DDoS controls, and optionally a later public-edge layer. It is **not** the first frontend runtime choice: moving the current Node SSR artifact to Workers requires a Cloudflare-specific React Router/workerd build and `nodejs_compat`; it does not host the private worker stack. [Cloudflare React Router guide](https://developers.cloudflare.com/workers/framework-guides/web-apps/react-router/)
|
||
|
||
## Environment isolation — required, not optional
|
||
|
||
The present shared Convex deployment is incompatible with two simultaneously operating environments. `SITE_URL` is a single deployment-scoped Better Auth base/trusted origin and GitHub OAuth callback origin; switching it between hosts breaks the other host.
|
||
|
||
Create these independent Convex deployments:
|
||
|
||
| Deployment | Type | Purpose |
|
||
| --- | --- | --- |
|
||
| `dev:<developer>` | Convex dev | Local loop only; each developer owns one |
|
||
| `staging` (`joyous-cat-297`) | Long-lived production-type deployment | Public integration/staging; never reused for local testing |
|
||
| Branch previews | Ephemeral preview deployment | Optional later, only for frontend/backend changes that do not need OAuth |
|
||
|
||
Convex supports a named long-lived production-type deployment (`convex deployment create staging --type prod`), per-deployment environment variables, and branch preview deployments. [Convex multiple deployments](https://docs.convex.dev/production/multiple-deployments) · [Convex environment variables](https://docs.convex.dev/production/environment-variables)
|
||
|
||
Every environment gets its own:
|
||
|
||
- `SITE_URL`, exact browser origin;
|
||
- `FLUE_URL` / `AGENT_BACKEND_URL`, pointing at only that environment’s Flue worker (e.g. `https://agents.zopu.puter.wtf`);
|
||
- `FLUE_DB_TOKEN` shared only with that environment’s agent service;
|
||
- model and Git provider credentials;
|
||
- GitHub OAuth application/client credentials where GitHub login is enabled.
|
||
|
||
A GitHub OAuth App permits one callback URL. Use a staging OAuth App with `https://zopu-staging.vercel.app/api/auth/callback/github`; keep local development on its own OAuth app or login mechanism. Do not promise OAuth on throwaway Vercel preview domains. [GitHub OAuth Apps](https://docs.github.com/en/apps/oauth-apps/building-oauth-apps/creating-an-oauth-app)
|
||
|
||
## Staging topology
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
Browser[Browser] -->|HTTPS| Vercel[Vercel: zopu-staging.vercel.app]
|
||
Browser -->|queries & mutations| Convex[Convex: joyous-cat-297]
|
||
Vercel -->|/api/auth rewrite| ConvexSite[Convex Site: joyous-cat-297.convex.site]
|
||
Convex -->|FLUE_DB_TOKEN + org header| Flue[Contabo: Flue + AgentOS registry one process]
|
||
Flue -->|control plane + outbound envoy| Engine[Rivet Engine: 127.0.0.1:6420]
|
||
Engine --> Flue
|
||
Engine --> Volumes[/srv/zopu/data/rivet]
|
||
Flue --> Workspaces[/srv/zopu/workspaces + /srv/zopu/data/flue]
|
||
CF[Cloudflare DNS] --> Vercel
|
||
CF --> Flue
|
||
```
|
||
|
||
### Public surface
|
||
|
||
- `zopu-staging.vercel.app` → Vercel.
|
||
- `agents.zopu.puter.wtf` → Contabo VDS (`zopu-staging-1`, `158.220.110.74`) Caddy → Flue only.
|
||
- Only ports 80/443 (and Tailscale/managed SSH) enter the VDS. Do **not** publish Rivet ports `6420` or `6421`, registry internals, workspace directories, or any engine dashboard.
|
||
- The Flue route is reachable from Convex, but its middleware must continue to require the per-environment bearer `FLUE_DB_TOKEN` and organization/turn correlation headers. This is a private worker protocol exposed narrowly for Convex callbacks—not a browser API.
|
||
- Browser clients continue to talk only to Convex. Do not restore browser-to-Flue traffic or a generic `/api/*` agent proxy.
|
||
|
||
### Private VDS services
|
||
|
||
The current VDS (`zopu-staging-1`, `158.220.110.74`) runs a deliberately small private topology:
|
||
|
||
| Service | Runtime | Network exposure | Persistent data | Notes |
|
||
| --- | --- | --- | --- | --- |
|
||
| `engine` | Existing Docker Compose (`/srv/zopu/compose/docker-compose.yml`) | private loopback `127.0.0.1:6420` | `/srv/zopu/data/rivet` bind mount | Single-node RocksDB backend. Health: `http://127.0.0.1:6420/health`. |
|
||
| `agents` | systemd (`/etc/systemd/system/zopu-agents.service`) | Caddy upstream only (`127.0.0.1:3000`) | `/srv/zopu/workspaces` + `/srv/zopu/data/flue` | One Node process runs Flue **and** hosts the AgentOS registry in native envoy Mode A (`registry.startAndWait()`). `RIVET_ENDPOINT` is the Engine control plane; the registry opens an outbound connection to the Engine, which routes actor calls back over it. No second runner process and no inbound serverless callback route. |
|
||
| `caddy` | systemd (`/etc/systemd/system/caddy.service`) | 80/443 only | Caddy certificate/config volumes | `/etc/caddy/Caddyfile` terminates TLS for `agents.zopu.puter.wtf` and forwards only the documented Flue paths to `127.0.0.1:3000`. |
|
||
|
||
Release artifacts are source controlled: `scripts/release-agents.sh` builds a curated bundle locally and atomically promotes `/srv/zopu/current`; `deploy/vds/` contains the systemd unit and secret-free environment template. The VDS materializes Linux production dependencies from the frozen lockfile, so macOS-native binaries are never uploaded.
|
||
|
||
Rivet’s filesystem backend is explicitly appropriate for single-node deployments; multi-node/HA later requires PostgreSQL and NATS. Configure the engine with a persistent `/data` bind mount, admin token, resource limits, and a `:6420/health` probe. [Rivet Docker Compose](https://rivet.dev/docs/self-hosting/docker-compose) · [Rivet production checklist](https://rivet.dev/docs/self-hosting/production-checklist)
|
||
|
||
## IaC model: Pulumi + Ansible + systemd/Compose, each at the right seam
|
||
|
||
Dokploy is intentionally excluded: it has already proved too slow and imperative for this stack. One tool should not be forced to manage three different concerns poorly.
|
||
|
||
| Layer | Source of truth | Tool | Reason |
|
||
| --- | --- | --- | --- |
|
||
| VDS baseline | `infra/ansible` | **Ansible** | Idempotent OS convergence: service user, Docker, firewall, Tailscale, directories, Caddy prerequisites, and backup timer. No Pulumi SSH-command pseudo-provider. |
|
||
| | VDS application topology | `deploy/vds` + `/srv/zopu/compose` | **systemd + Docker Compose** | Engine stays in its existing private Compose service; one checked-in systemd unit runs the single Flue + AgentOS registry Node process (native envoy Mode A). |
|
||
| Release execution | `scripts/release-agents.sh` now; CI later | **Local release bundle promotion** | Builds a curated cross-platform bundle, installs Linux dependencies on the VDS, atomically swaps `current`, then restarts `zopu-agents`. |
|
||
|
||
> **Current state (2026-08-03):** the Engine and Caddy retain their manually bootstrapped VDS services. The Node agents process is now represented by source-controlled `deploy/vds` artifacts and `scripts/release-agents.sh`; Pulumi/Ansible and CI are still future work.
|
||
|
||
Pulumi state is meaningful only with a shared backend—use Pulumi Cloud or a managed/self-hosted state backend, **not** a developer-local `file://` state file. Pulumi’s state is what enables previews, refreshes, encrypted secret tracking, and drift detection. [Pulumi state and backends](https://www.pulumi.com/docs/iac/concepts/state-and-backends/) · [Pulumi secrets](https://www.pulumi.com/docs/iac/concepts/secrets/)
|
||
|
||
Ansible is not redundant: it makes the Contabo host reproducible without pretending that SSH command resources are declarative infrastructure. [Ansible basic concepts](https://docs.ansible.com/projects/ansible/latest/getting_started/)
|
||
|
||
## Delivery workflow
|
||
|
||
### Local iteration
|
||
|
||
1. Keep using the current root `pnpm dev:tailscale` stack and personal Convex dev deployment.
|
||
2. Use the Mac’s Tailscale URL for phone testing.
|
||
3. Never edit staging Convex variables, staging OAuth callback settings, or staging VDS volumes from the local loop.
|
||
4. Merge only when targeted runtime smoke testing and repository checks pass.
|
||
|
||
### Staging release
|
||
|
||
1. Run repository validation and build the staging web artifact for the staging Convex deployment (`joyous-cat-297`, `https://joyous-cat-297.convex.cloud`), then deploy it to `zopu-staging` (`https://zopu-staging.vercel.app`).
|
||
2. From the repository root, create and promote the agents release with `scripts/release-agents.sh <vds-host> <release-id>`; it installs Linux production dependencies on the VDS, atomically swaps `/srv/zopu/current`, and leaves the running process untouched until restart.
|
||
3. Restart `zopu-agents.service` on the VDS. The unit waits for Engine health, then starts the single Node process that serves Flue and starts the AgentOS registry in native envoy Mode A.
|
||
4. Verify: `https://zopu-staging.vercel.app` returns 200, Convex auth origin `https://zopu-staging.vercel.app` is accepted, `https://agents.zopu.puter.wtf/health` returns 200, Engine health returns 200 at `http://127.0.0.1:6420/health` inside the private network, `zopu-agents.service` is active, the registry envoy connected to the engine (check the process log for the startup line), and a real signed-in staging conversation receives an agent response.
|
||
5. Roll back by atomically repointing `/srv/zopu/current` at the prior release then restarting `zopu-agents.service`.
|
||
|
||
## Required implementation backlog
|
||
|
||
The staging environment is live. Remaining work is to close the gap between the manual, on-host configuration and the declarative, CI-driven target described above:
|
||
|
||
1. **Convex staging deployment** — ✅ Done as `joyous-cat-297`. `SITE_URL` is set to `https://zopu-staging.vercel.app`, `FLUE_URL`/`AGENT_BACKEND_URL` to `https://agents.zopu.puter.wtf`, and `FLUE_DB_TOKEN` to the VDS agents token.
|
||
2. **Auth/webhook origin seams** — ✅ Vercel rewrites `/api/auth/*` to `https://joyous-cat-297.convex.site/api/auth/*` in `vercel.json`. Verify that Puter/Gitea webhooks are configured to post directly to the Convex Site HTTP action URL (`https://joyous-cat-297.convex.site/api/git/webhooks/...`) rather than the web origin.
|
||
3. **Vercel React Router staging project** — ✅ Done as `zopu-staging` (`https://zopu-staging.vercel.app`).
|
||
4. **VDS agent topology** — ✅ Source-controlled VDS unit, env template, and atomic release bundle now model the single Flue + AgentOS registry Node process running in native envoy Mode A. Apply them to the VDS and retire the former separate runner service.
|
||
5. **Engine deployment, backups, and CI** — 🔄 Engine health and VDS backups exist. Move the existing Engine Compose service and Caddy configuration into source-controlled provisioning; add CI to run validation, promote release bundles, and deploy the web.
|
||
6. **Declarative IaC** — ⏳ Create `infra/pulumi` (Cloudflare DNS + Vercel project) and `infra/ansible` (host convergence).
|
||
7. **End-to-end staging smoke** — ⏳ Execute: sign in via GitHub OAuth, create/connect a project, send a conversation, and complete a disposable issue-to-PR job against the live `zopu-staging` + `agents.zopu.puter.wtf` stack.
|
||
|
||
## Security and operations guardrails
|
||
|
||
- Keep environment secrets in CI environment secret stores, Pulumi encrypted configuration/ESC where appropriate, Convex deployment variables, and Vercel environment variables. Never commit `.env` files or place secrets in image layers.
|
||
- Use unique `FLUE_DB_TOKEN`, Rivet admin token, workspace token, model credential, and Git token per environment.
|
||
- The VDS runs code-writing agents. Use a dedicated service user; do not mount the host home directory; expose only scoped repository credentials to individual attempts; and keep staging separate from any production host.
|
||
- Back up Rivet engine state and Caddy configuration. Treat workspaces as reproducible/ephemeral unless an active job requires retention; prune completed worktrees deliberately.
|
||
- A single Flue + AgentOS registry Node instance (native envoy Mode A) is the correct initial staging shape. Its durable Convex adapter survives restarts, but each conversation still requires one live owner—do not add replicas until ownership routing is designed. [Flue database guide](https://flueframework.com/docs/guide/database)
|
||
|
||
## Sources
|
||
|
||
- Repository architecture: `docs/LOCAL_SETUP.md`; `docs/auth-proxy.md`; `vercel.json`; `apps/web/Dockerfile`; `deploy/vds/`; `scripts/release-agents.sh`.
|
||
- [Convex environments and deployments](https://docs.convex.dev/production/multiple-deployments), [hosting on Vercel](https://docs.convex.dev/production/hosting/vercel), and [HTTP actions](https://docs.convex.dev/functions/http-actions).
|
||
- [Vercel React Router](https://vercel.com/docs/frameworks/frontend/react-router) and [Vercel environments](https://vercel.com/docs/deployments/environments).
|
||
- [Rivet self-hosted Docker Compose](https://rivet.dev/docs/self-hosting/docker-compose), [production checklist](https://rivet.dev/docs/self-hosting/production-checklist), and [runtime modes](https://rivet.dev/docs/general/runtime-modes).
|
||
- [Flue Node deployment](https://flueframework.com/docs/ecosystem/deploy/node) and [durable database ownership](https://flueframework.com/docs/guide/database).
|
||
- [Pulumi state](https://www.pulumi.com/docs/iac/concepts/state-and-backends/), [Pulumi secrets](https://www.pulumi.com/docs/iac/concepts/secrets/), and [Ansible](https://docs.ansible.com/projects/ansible/latest/getting_started/).
|