feat: single-node Zopu runtime deployment for Debian

Lane D deliverable: reproducible execution-plane deployment on a fresh Debian dedicated server (~12 cores, 40 GB RAM, single-node). Artifacts: - bootstrap.sh: Docker Engine, Bun, repo clone, build, systemd install, firewall (deny-by-default), optional Tailscale, disk monitor cron - .env.template: all 8 environment groups documented (Convex, Gitea, model gateway, AgentOS/RivetKit, Zopu agent, daemon, Docker sandbox, service auth) - systemd units: zopu-daemon, zopu-agent (both Restart=always), zopu-health timer (60s), zopu-docker-cleanup timer (daily) - scripts: health-check, update, rollback, docker-cleanup, disk-monitor - Caddyfile: optional reverse proxy (documentation-only by default) - README.md: full runbook with start/stop/log/update/rollback/cleanup procedures, topology documentation, and boundary notes Verified topology (from installed RivetKit source): - registry.start() boots an in-process RivetKit engine (envoy mode) with a native Rust sidecar at http://localhost:6420 (DEFAULT_ENDPOINT in chunk-YDUQHING.js line 4751). createClient() connects back to it. - No separate Rivet Engine process required for single-node operation. - Linux x64 sidecar packages (@rivet-dev/agentos-sidecar-linux-x64-gnu) are available as optional dependencies. Docker / Orb boundary: - Docker Engine installed and zopu user in docker group. - No Docker-backed sandbox code wired; Orb lane contract not yet landed. - Volume pruning omitted from docker-cleanup.sh to protect durable data. Validated: bash -n on all 6 shell scripts; mocked health-check smoke (nc-based TCP probes, mocked systemctl/docker); systemd unit structural checks. No JS/TS/agent files changed.
2026-07-24 20:44:47 +05:30
parent b5c40755cf
commit 8f0b915f4b
15 changed files with 1193 additions and 0 deletions
--- a/deploy/zopu-runtime/README.md
+++ b/deploy/zopu-runtime/README.md
@@ -0,0 +1,322 @@
+# Zopu Single-Node Runtime Deployment
+
+Deployment artifacts for the Zopu execution plane on a single Debian dedicated
+server. Convex is already deployed as the control plane; this lane deploys only
+the execution plane: the daemon (Bun/Effect + AgentOS/RivetKit), the Flue agent
+service, Docker Engine for future Orb sandboxes, and supporting infrastructure.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    Debian Dedicated Server                   │
+│  ~12 CPU cores · ~40 GB RAM · single-node                    │
+│                                                               │
+│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐ │
+│  │ zopu-daemon  │     │ zopu-agent   │     │  Docker      │ │
+│  │ (systemd)    │     │ (systemd)    │     │  Engine      │ │
+│  │              │     │              │     │              │ │
+│  │ Effect daemon│     │ Flue Node    │     │ Orb sandboxes│ │
+│  │ + RivetKit   │     │ server.mjs   │     │ (future)     │ │
+│  │   in-process │     │ :3583        │     │              │ │
+│  │   engine     │     │              │     │              │ │
+│  │ + native     │     │              │     │              │ │
+│  │   sidecar    │     │              │     │              │ │
+│  │   :6420      │     │              │     │              │ │
+│  └──────┬───────┘     └──────┬───────┘     └──────────────┘ │
+│         │                    │                               │
+│         └────────┬───────────┘                               │
+│                  │                                            │
+│         ┌────────▼─────────┐                                 │
+│         │   Convex (cloud)  │  ← control plane (pre-deployed)│
+│         └──────────────────┘                                 │
+│                                                               │
+│  systemd timers: health-check (60s), docker-cleanup (daily)  │
+│  cron: disk-monitor (daily 06:00)                             │
+│  ufw: deny-by-default, SSH only                              │
+│  Tailscale: optional private overlay                          │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Single-node RivetKit topology
+
+The daemon calls `registry.start()` from `rivetkit`, which boots an
+**in-process RivetKit engine** (envoy mode) backed by a **native Rust
+sidecar** binary (`@rivet-dev/agentos-sidecar`, platform-resolved). The engine
+listens on `RIVET_ENDPOINT` (default `http://localhost:6420`). The daemon then
+calls `createClient(RIVET_ENDPOINT)` to connect back to its own in-process
+engine for actor dispatch.
+
+Evidence: RivetKit source `chunk-YDUQHING.js` line 4751 —
+`DEFAULT_ENDPOINT = "http://localhost:6420"`. The `Registry.start()` method
+calls `#startEnvoy()` → `runtime.serveRegistry()` for serverful mode (Mode A).
+The `createClient()` function reads `RIVET_ENDPOINT` env or defaults to the
+same `http://localhost:6420`.
+
+**No separate Rivet Engine process is required.** The engine, actor envoy, and
+sidecar all run inside the daemon process. A future multi-node deployment
+would externalize the engine, but that is out of scope.
+
+### Docker / Orb boundary
+
+Docker Engine is installed and the `zopu` service user is in the `docker`
+group. The daemon's systemd unit includes `SupplementaryGroups=docker`.
+However, **no Docker-backed sandbox code is currently wired**. The Orb
+sandbox lane contract has not landed; Docker access is provisioned now so the
+boundary is ready. The current agent uses the in-process AgentOS VM (Wasm/V8)
+sandbox, not Docker.
+
+## Files
+
+```
+deploy/zopu-runtime/
+├── bootstrap.sh              # One-shot Debian installer
+├── .env.template             # Environment template (all groups documented)
+├── README.md                 # This file (runbook)
+├── systemd/
+│   ├── zopu-daemon.service   # Daemon systemd unit
+│   ├── zopu-agent.service    # Agent systemd unit
+│   ├── zopu-health.service   # Health check oneshot
+│   ├── zopu-health.timer     # Health check every 60s
+│   ├── zopu-docker-cleanup.service
+│   └── zopu-docker-cleanup.timer
+├── scripts/
+│   ├── health-check.sh       # TCP/process health probes
+│   ├── update.sh             # Update to branch or commit
+│   ├── rollback.sh           # Roll back to previous commit
+│   ├── docker-cleanup.sh     # Prune stopped containers/images/networks
+│   └── disk-monitor.sh       # Disk usage alerting
+└── caddy/
+    └── Caddyfile             # Optional reverse proxy config (documentation)
+```
+
+## Fresh install
+
+```bash
+# 1. SSH into the fresh Debian 12 server as root.
+
+# 2. Set environment overrides (optional):
+export ZOPU_REPO_URL="ssh://git@git.openputer.com:2222/puter/zopu-code.git"
+export ZOPU_REPO_BRANCH="dogfood/v0"
+# export TAILSCALE_AUTHKEY="tskey-..."
+# export TAILSCALE_HOSTNAME="zopu-runtime"
+
+# 3. Run the bootstrap script:
+bash bootstrap.sh
+
+# 4. Edit .env with real values:
+nano /opt/zopu/.env
+
+# 5. Start services:
+systemctl start zopu-daemon
+sleep 3
+systemctl start zopu-agent
+
+# 6. Enable timers:
+systemctl enable --now zopu-health.timer zopu-docker-cleanup.timer
+
+# 7. Verify:
+/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh
+```
+
+## Start / stop / restart
+
+```bash
+# Start both services
+systemctl start zopu-daemon zopu-agent
+
+# Stop both services
+systemctl stop zopu-agent zopu-daemon
+
+# Restart (daemon first — it owns the RivetKit engine)
+systemctl restart zopu-daemon && sleep 3 && systemctl restart zopu-agent
+
+# Enable on boot
+systemctl enable zopu-daemon zopu-agent
+
+# Disable on boot
+systemctl disable zopu-daemon zopu-agent
+```
+
+## Log inspection
+
+All service logs go to journald with `SyslogIdentifier` tags.
+
+```bash
+# Daemon logs (live follow)
+journalctl -u zopu-daemon -f
+
+# Agent logs (live follow)
+journalctl -u zopu-agent -f
+
+# Last 100 lines of daemon
+journalctl -u zopu-daemon -n 100
+
+# Logs since boot
+journalctl -u zopu-daemon -b
+
+# Health check timer logs
+journalctl -u zopu-health.service -n 50
+
+# Docker cleanup logs
+journalctl -u zopu-docker-cleanup.service -n 50
+
+# Disk monitor logs (cron → file)
+tail -100 /var/log/zopu/disk-monitor.log
+
+# All Zopu syslog identifiers
+journalctl -t zopu-daemon -t zopu-agent --since "1 hour ago"
+```
+
+## Health checks
+
+```bash
+# Manual health check (prints all probes)
+/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh
+
+# Quiet mode (exit code only)
+/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh --quiet
+
+# Check systemd timer is running
+systemctl status zopu-health.timer
+systemctl list-timers zopu-health.timer
+```
+
+The health check probes:
+1. `zopu-daemon` systemd unit is active
+2. `zopu-agent` systemd unit is active
+3. RivetKit engine port (default 6420) accepts TCP connections
+4. Flue agent port (default 3583) accepts TCP connections
+5. Docker daemon responds to `docker info`
+
+No HTTP health endpoints are assumed. Flue does not expose one by design
+(per Flue docs: "Flue does not add a health endpoint"). RivetKit's health
+route is internal to the registry runtime and not documented as
+publicly addressable on the engine endpoint.
+
+## Update to commit
+
+```bash
+# Update to latest of dogfood/v0 (default)
+/opt/zopu/deploy/zopu-runtime/scripts/update.sh
+
+# Update to a specific branch
+/opt/zopu/deploy/zopu-runtime/scripts/update.sh dogfood/runtime-deploy
+
+# Update to a specific commit
+/opt/zopu/deploy/zopu-runtime/scripts/update.sh abc123def456
+```
+
+The update script:
+1. Records current HEAD to `.last-deployed-sha`
+2. Fetches, resolves branch-or-commit, checks out
+3. `bun install`, builds daemon and agent
+4. Restarts daemon, waits, restarts agent
+5. Runs health check; reports failure and rollback instructions
+
+## Rollback
+
+```bash
+# Roll back to the previously deployed commit
+/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh
+
+# Roll back to a specific commit
+/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh abc123def456
+```
+
+Rollback reads `.last-deployed-sha` (written by `update.sh`), checks out that
+commit, rebuilds, and restarts services. The pre-rollback SHA is saved to
+`.pre-rollback-sha` for re-rollback if needed.
+
+## Docker cleanup
+
+```bash
+# Manual cleanup
+/opt/zopu/deploy/zopu-runtime/scripts/docker-cleanup.sh
+
+# Check Docker disk usage
+docker system df
+
+# Timer runs daily; check its schedule
+systemctl list-timers zopu-docker-cleanup.timer
+```
+
+Cleanup prunes:
+- Stopped containers older than 24 hours
+- Dangling (untagged) images
+- Unused networks
+
+
+Named volumes and running containers are never removed.
+
+## Disk-space monitoring
+
+```bash
+# Manual check
+/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh
+
+# Custom threshold (90%)
+/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh --warn-percent 90
+```
+
+A cron job runs at 06:00 daily and writes to `/var/log/zopu/disk-monitor.log`.
+Default alert threshold is 80%.
+
+## Firewall and private networking
+
+The firewall (`ufw`) is deny-by-default:
+- SSH (port 22) is allowed
+- All other incoming traffic is denied
+
+The RivetKit engine (`:6420`) and Flue agent (`:3583`) ports are **not**
+exposed publicly. Access is via:
+
+1. **Tailscale** (recommended): private overlay network. Set
+   `TAILSCALE_AUTHKEY` before running bootstrap to configure automatically.
+2. **Direct private IP**: if the server is on a private network, the ports
+   are reachable from other machines on the same subnet.
+3. **Caddy** (optional): install Caddy and use the annotated Caddyfile in
+   `caddy/` if you need TLS termination or a public ingress point.
+
+## Environment groups
+
+See [`.env.template`](./.env.template) for the full annotated template. The
+eight required groups:
+
+| Group | Variables |
+|-------|-----------|
+| Convex | `CONVEX_URL`, `CONVEX_SITE_URL`, `SITE_URL` |
+| Gitea | `GITEA_URL`, `GITEA_TOKEN` |
+| Model gateway | `AGENT_MODEL_*` |
+| AgentOS/RivetKit | `RIVET_ENDPOINT` (optional) |
+| Zopu agent | `FLUE_DB_TOKEN`, `PORT` |
+| Daemon | `DAEMON_ID`, `DAEMON_NAME`, `DAEMON_VERSION`, `DAEMON_HEARTBEAT_MS`, `DAEMON_COMMAND_LEASE_MS` |
+| Docker sandbox | group membership (no env vars) |
+| Service auth | `AUTH_SECRET` (if needed) |
+
+## What is NOT deployed
+
+- **Web app**: the web frontend is not deployed in this lane.
+- **Kubernetes**: no container orchestration.
+- **PostgreSQL for Rivet**: the in-process RivetKit engine uses its own
+  storage; no external PostgreSQL is required.
+- **Multi-node coordination**: single-node only.
+- **Public administration endpoints**: no admin HTTP surface.
+- **Secrets in source**: `.env` is never committed; `.env.template` contains
+  only placeholder values.
+- **Production generated-app hosting**: only the execution plane runs here.
+- **Docker-backed Orb sandboxes**: Docker is installed and access is
+  provisioned, but no Orb sandbox code is wired. This is a boundary
+  prepared for the Orb lane, not a working feature.
+
+## Reproducibility
+
+The deployment does not require the developer's MacBook to remain online.
+Once bootstrap completes and `.env` is filled in:
+
+1. Services run under systemd with `Restart=always`.
+2. Logs persist in journald.
+3. Health checks run every 60 seconds via systemd timer.
+4. Docker cleanup runs daily.
+5. Disk usage is monitored daily.
+6. Unattended-upgrades handles Debian security patches.