zopu-code/deploy/zopu-runtime/README.md

# Zopu Single-Node Runtime Deployment

Deployment artifacts for the complete Zopu stack on a single Debian dedicated server: the React Router web app, a persistent self-hosted Convex control plane, the daemon (Bun/Effect + AgentOS/RivetKit), the Flue agent service, Docker Engine, and supporting infrastructure.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Debian Dedicated Server                   │
│  ~12 CPU cores · ~40 GB RAM · single-node                    │
│                                                               │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐ │
│  │ zopu-web     │     │ zopu-agent   │     │  Docker      │ │
│  │ (systemd)    │     │ (systemd)    │     │  Engine      │ │
│  │              │     │              │     │              │ │
│  │ React Router │     │ Flue Node 22 │     │ Convex       │ │
│  │ :13100       │     │ server.mjs   │     │ backend      │ │
│  │              │     │ :3583        │     │ :3210/:3211 │ │
│  └──────┬───────┘     └──────┬───────┘     └──────────────┘ │
│         │                    │                               │
│  ┌──────▼───────┐                                            │
│  │ zopu-daemon  │  Bun + Effect + RivetKit :6420             │
│  └──────────────┘                                            │
│                                                               │
│  systemd timers: health-check (60s), docker-cleanup (daily)  │
│  cron: disk-monitor (daily 06:00)                             │
│  ufw: deny-incoming, SSH + tailscale0                              │
│  Tailscale: optional private overlay                          │
└─────────────────────────────────────────────────────────────┘
```

### Single-node RivetKit topology

The daemon calls `registry.start()` from `rivetkit`, which boots an **in-process RivetKit engine** (envoy mode) backed by a **native Rust sidecar** binary (`@rivet-dev/agentos-sidecar`, platform-resolved). The engine listens on `RIVET_ENDPOINT` (default `http://localhost:6420`). The daemon then calls `createClient(RIVET_ENDPOINT)` to connect back to its own in-process engine for actor dispatch.

Evidence: RivetKit source `chunk-YDUQHING.js` line 4751 — `DEFAULT_ENDPOINT = "http://localhost:6420"`. The `Registry.start()` method calls `#startEnvoy()` → `runtime.serveRegistry()` for serverful mode (Mode A). The `createClient()` function reads `RIVET_ENDPOINT` env or defaults to the same `http://localhost:6420`.

**No separate Rivet Engine process is required.** The engine, actor envoy, and sidecar all run inside the daemon process. A future multi-node deployment would externalize the engine, but that is out of scope.

### Docker / Orb boundary

Docker Engine is installed and the `zopu` service user is in the `docker` group. The daemon's systemd unit includes `SupplementaryGroups=docker`. However, **no Docker-backed sandbox code is currently wired**. The Orb sandbox lane contract has not landed; Docker access is provisioned now so the boundary is ready. The current agent uses the in-process AgentOS VM (Wasm/V8) sandbox, not Docker.

## Files

```
deploy/zopu-runtime/
├── bootstrap.sh              # One-shot Debian installer
├── .env.template             # Environment template (all groups documented)
├── README.md                 # This file (runbook)
├── systemd/
│   ├── zopu-daemon.service   # Daemon systemd unit
│   ├── zopu-agent.service    # Agent systemd unit
│   ├── zopu-web.service      # React Router web systemd unit
│   ├── zopu-health.service   # Health check oneshot
│   ├── zopu-health.timer     # Health check every 60s
│   ├── zopu-docker-cleanup.service
│   └── zopu-docker-cleanup.timer
├── scripts/
│   ├── health-check.sh       # TCP/process health probes
│   ├── update.sh             # Update to branch or commit
│   ├── rollback.sh           # Roll back to previous commit
│   ├── docker-cleanup.sh     # Prune stopped containers/images/networks
│   └── disk-monitor.sh       # Disk usage alerting
└── caddy/
    └── Caddyfile             # Optional reverse proxy config (documentation)
```

## Fresh install

```bash
# 1. SSH into the fresh Debian 12 server as root.

# 2. Set environment overrides (optional):
export ZOPU_REPO_URL="ssh://git@git.openputer.com:2222/puter/zopu-code.git"
export ZOPU_REPO_BRANCH="dogfood/v0"
# export TAILSCALE_AUTHKEY="tskey-..."
# export TAILSCALE_HOSTNAME="zopu-runtime"

# 3. Run the bootstrap script:
bash bootstrap.sh

# 4. Edit .env with real values:
nano /opt/zopu/.env

# 5. Start services:
systemctl start zopu-web zopu-daemon
sleep 3
systemctl start zopu-agent

# 6. Enable timers:
systemctl enable --now zopu-health.timer zopu-docker-cleanup.timer

# 7. Verify:
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh
```

## Start / stop / restart

```bash
# Start all services
systemctl start zopu-web zopu-daemon zopu-agent

# Stop all services
systemctl stop zopu-agent zopu-daemon zopu-web

# Restart (daemon first — it owns the RivetKit engine)
systemctl restart zopu-web zopu-daemon && sleep 3 && systemctl restart zopu-agent

# Enable on boot
systemctl enable zopu-web zopu-daemon zopu-agent

# Disable on boot
systemctl disable zopu-web zopu-daemon zopu-agent
```

## Log inspection

All service logs go to journald with `SyslogIdentifier` tags.

```bash
# Daemon logs (live follow)
journalctl -u zopu-daemon -f

# Agent logs (live follow)
journalctl -u zopu-agent -f

# Last 100 lines of daemon
journalctl -u zopu-daemon -n 100

# Logs since boot
journalctl -u zopu-daemon -b

# Health check timer logs
journalctl -u zopu-health.service -n 50

# Docker cleanup logs
journalctl -u zopu-docker-cleanup.service -n 50

# Disk monitor logs (cron → file)
tail -100 /var/log/zopu/disk-monitor.log

# All Zopu syslog identifiers
journalctl -t zopu-daemon -t zopu-agent --since "1 hour ago"
```

## Health checks

```bash
# Manual health check (prints all probes)
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh

# Quiet mode (exit code only)
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh --quiet

# Check systemd timer is running
systemctl status zopu-health.timer
systemctl list-timers zopu-health.timer
```

The health check probes:

1. `zopu-daemon` systemd unit is active
2. `zopu-agent` systemd unit is active
3. RivetKit engine port (default 6420) accepts TCP connections
4. Flue agent port (default 3583) accepts TCP connections
5. Docker daemon responds to `docker info`

No HTTP health endpoints are assumed. Flue does not expose one by design (per Flue docs: "Flue does not add a health endpoint"). RivetKit's health route is internal to the registry runtime and not documented as publicly addressable on the engine endpoint.

## Update to commit

```bash
# Update to latest of dogfood/v0 (default)
/opt/zopu/deploy/zopu-runtime/scripts/update.sh

# Update to a specific branch
/opt/zopu/deploy/zopu-runtime/scripts/update.sh dogfood/runtime-deploy

# Update to a specific commit
/opt/zopu/deploy/zopu-runtime/scripts/update.sh abc123def456
```

The update script:

1. Records current HEAD to `.last-deployed-sha`
2. Fetches, resolves branch-or-commit, checks out
3. `bun install`, builds daemon and agent
4. Restarts daemon, waits, restarts agent
5. Runs health check; reports failure and rollback instructions

## Rollback

```bash
# Roll back to the previously deployed commit
/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh

# Roll back to a specific commit
/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh abc123def456
```

Rollback reads `.last-deployed-sha` (written by `update.sh`), checks out that commit, rebuilds, and restarts services. The pre-rollback SHA is saved to `.pre-rollback-sha` for re-rollback if needed.

## Docker cleanup

```bash
# Manual cleanup
/opt/zopu/deploy/zopu-runtime/scripts/docker-cleanup.sh

# Check Docker disk usage
docker system df

# Timer runs daily; check its schedule
systemctl list-timers zopu-docker-cleanup.timer
```

Cleanup prunes:

- Stopped containers older than 24 hours
- Dangling (untagged) images
- Unused networks

Named volumes and running containers are never removed.

## Disk-space monitoring

```bash
# Manual check
/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh

# Custom threshold (90%)
/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh --warn-percent 90
```

A cron job runs at 06:00 daily and writes to `/var/log/zopu/disk-monitor.log`. Default alert threshold is 80%.

## Firewall and private networking

The firewall (`ufw`) is deny-by-default:

- SSH (port 22) is allowed on all interfaces
- All traffic on `tailscale0` is allowed (Tailscale private overlay)
- All other incoming traffic is denied

The RivetKit engine (`:6420`) and Flue agent (`:3583`) ports are **not** exposed on public interfaces. Reachability options:

1. **Tailscale** (recommended): bootstrap runs `ufw allow in on tailscale0` so all ports are reachable over the private overlay. Set `TAILSCALE_AUTHKEY` before running bootstrap to configure automatically. Other Tailscale-connected machines can reach the agent at `http://zopu-runtime:3583` and the engine at `http://zopu-runtime:6420`.
2. **Custom private interface**: if you have a non-Tailscale private network (e.g. a VLAN or wireguard interface), add an explicit rule:
   ```bash
   ufw allow in on eth1  # or your private interface name
   ```
   Do NOT assume direct private IP access works by default — the deny-incoming policy blocks it until an interface-specific rule is added.
3. **Caddy** (optional): install Caddy and use the annotated Caddyfile in `caddy/` if you need TLS termination or a public ingress point.

## Environment groups

See [`.env.template`](./.env.template) for the full annotated template. The eight required groups:

| Group | Variables |
| --- | --- |
| Convex | `CONVEX_URL`, `CONVEX_SITE_URL`, `SITE_URL` |
| Gitea | `GITEA_URL`, `GITEA_TOKEN` |
| Model gateway | `AGENT_MODEL_*` |
| AgentOS/RivetKit | `RIVET_ENDPOINT` (optional) |
| Zopu agent | `FLUE_DB_TOKEN` (`zopu-agent.service` sets port 3583) |
| Daemon | `DAEMON_ID`, `DAEMON_NAME`, `DAEMON_VERSION`, `DAEMON_HEARTBEAT_MS`, `DAEMON_COMMAND_LEASE_MS` |
| Docker sandbox | group membership (no env vars) |
| Service auth | `AUTH_SECRET` (if needed) |

## Public single-node routes

The checked-in Caddy example assumes Cloudflare Tunnel terminates TLS and sends the four Zopu hosts to Caddy on loopback:

- `zopu.sai-onchain.me` → React Router web app on `127.0.0.1:13100`
- `zopu-api.sai-onchain.me` → Convex API on `127.0.0.1:3210`
- `zopu-site.sai-onchain.me` → Convex HTTP actions on `127.0.0.1:3211`
- `zopu-agent.sai-onchain.me` → Flue on `127.0.0.1:3583`

Self-hosted Convex state lives in the `zopu-convex-data` Docker volume. Generate the CLI admin key after the backend is healthy:

```bash
docker compose \
  --env-file /opt/zopu/.env \
  -f /opt/zopu/deploy/zopu-runtime/convex/docker-compose.yml \
  exec backend ./generate_admin_key.sh
```

## What is NOT deployed

- **Kubernetes**: no container orchestration.
- **PostgreSQL for Rivet**: the in-process RivetKit engine uses its own storage; no external PostgreSQL is required.
- **Multi-node coordination**: single-node only.
- **Public administration endpoints**: no admin HTTP surface.
- **Secrets in source**: `.env` is never committed; `.env.template` contains only placeholder values.
- **Docker-backed Orb sandboxes**: Docker is installed and access is provisioned, but no Orb sandbox code is wired. This is a boundary prepared for the Orb lane, not a working feature.

## Reproducibility

The deployment does not require the developer's MacBook to remain online. Once bootstrap completes and `.env` is filled in:

1. Services run under systemd with `Restart=always`.
2. Logs persist in journald.
3. Health checks run every 60 seconds via systemd timer.
4. Docker cleanup runs daily.
5. Disk usage is monitored daily.
6. Unattended-upgrades handles Debian security patches.