zopu-code/deploy/zopu-runtime/README.md

# Zopu Single-Node Runtime Deployment

Deployment artifacts for the Zopu execution plane on a single Debian dedicated
server. Convex is already deployed as the control plane; this lane deploys only
the execution plane: the daemon (Bun/Effect + AgentOS/RivetKit), the Flue agent
service, Docker Engine for future Orb sandboxes, and supporting infrastructure.

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                    Debian Dedicated Server                   │
│  ~12 CPU cores · ~40 GB RAM · single-node                    │
│                                                               │
│  ┌──────────────┐     ┌──────────────┐     ┌──────────────┐ │
│  │ zopu-daemon  │     │ zopu-agent   │     │  Docker      │ │
│  │ (systemd)    │     │ (systemd)    │     │  Engine      │ │
│  │              │     │              │     │              │ │
│  │ Effect daemon│     │ Flue Node    │     │ Orb sandboxes│ │
│  │ + RivetKit   │     │ server.mjs   │     │ (future)     │ │
│  │   in-process │     │ :3583        │     │              │ │
│  │   engine     │     │              │     │              │ │
│  │ + native     │     │              │     │              │ │
│  │   sidecar    │     │              │     │              │ │
│  │   :6420      │     │              │     │              │ │
│  └──────┬───────┘     └──────┬───────┘     └──────────────┘ │
│         │                    │                               │
│         └────────┬───────────┘                               │
│                  │                                            │
│         ┌────────▼─────────┐                                 │
│         │   Convex (cloud)  │  ← control plane (pre-deployed)│
│         └──────────────────┘                                 │
│                                                               │
│  systemd timers: health-check (60s), docker-cleanup (daily)  │
│  cron: disk-monitor (daily 06:00)                             │
│  ufw: deny-incoming, SSH + tailscale0                              │
│  Tailscale: optional private overlay                          │
└─────────────────────────────────────────────────────────────┘
```

### Single-node RivetKit topology

The daemon calls `registry.start()` from `rivetkit`, which boots an
**in-process RivetKit engine** (envoy mode) backed by a **native Rust
sidecar** binary (`@rivet-dev/agentos-sidecar`, platform-resolved). The engine
listens on `RIVET_ENDPOINT` (default `http://localhost:6420`). The daemon then
calls `createClient(RIVET_ENDPOINT)` to connect back to its own in-process
engine for actor dispatch.

Evidence: RivetKit source `chunk-YDUQHING.js` line 4751 —
`DEFAULT_ENDPOINT = "http://localhost:6420"`. The `Registry.start()` method
calls `#startEnvoy()` → `runtime.serveRegistry()` for serverful mode (Mode A).
The `createClient()` function reads `RIVET_ENDPOINT` env or defaults to the
same `http://localhost:6420`.

**No separate Rivet Engine process is required.** The engine, actor envoy, and
sidecar all run inside the daemon process. A future multi-node deployment
would externalize the engine, but that is out of scope.

### Docker / Orb boundary

Docker Engine is installed and the `zopu` service user is in the `docker`
group. The daemon's systemd unit includes `SupplementaryGroups=docker`.
However, **no Docker-backed sandbox code is currently wired**. The Orb
sandbox lane contract has not landed; Docker access is provisioned now so the
boundary is ready. The current agent uses the in-process AgentOS VM (Wasm/V8)
sandbox, not Docker.

## Files

```
deploy/zopu-runtime/
├── bootstrap.sh              # One-shot Debian installer
├── .env.template             # Environment template (all groups documented)
├── README.md                 # This file (runbook)
├── systemd/
│   ├── zopu-daemon.service   # Daemon systemd unit
│   ├── zopu-agent.service    # Agent systemd unit
│   ├── zopu-health.service   # Health check oneshot
│   ├── zopu-health.timer     # Health check every 60s
│   ├── zopu-docker-cleanup.service
│   └── zopu-docker-cleanup.timer
├── scripts/
│   ├── health-check.sh       # TCP/process health probes
│   ├── update.sh             # Update to branch or commit
│   ├── rollback.sh           # Roll back to previous commit
│   ├── docker-cleanup.sh     # Prune stopped containers/images/networks
│   └── disk-monitor.sh       # Disk usage alerting
└── caddy/
    └── Caddyfile             # Optional reverse proxy config (documentation)
```

## Fresh install

```bash
# 1. SSH into the fresh Debian 12 server as root.

# 2. Set environment overrides (optional):
export ZOPU_REPO_URL="ssh://git@git.openputer.com:2222/puter/zopu-code.git"
export ZOPU_REPO_BRANCH="dogfood/v0"
# export TAILSCALE_AUTHKEY="tskey-..."
# export TAILSCALE_HOSTNAME="zopu-runtime"

# 3. Run the bootstrap script:
bash bootstrap.sh

# 4. Edit .env with real values:
nano /opt/zopu/.env

# 5. Start services:
systemctl start zopu-daemon
sleep 3
systemctl start zopu-agent

# 6. Enable timers:
systemctl enable --now zopu-health.timer zopu-docker-cleanup.timer

# 7. Verify:
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh
```

## Start / stop / restart

```bash
# Start both services
systemctl start zopu-daemon zopu-agent

# Stop both services
systemctl stop zopu-agent zopu-daemon

# Restart (daemon first — it owns the RivetKit engine)
systemctl restart zopu-daemon && sleep 3 && systemctl restart zopu-agent

# Enable on boot
systemctl enable zopu-daemon zopu-agent

# Disable on boot
systemctl disable zopu-daemon zopu-agent
```

## Log inspection

All service logs go to journald with `SyslogIdentifier` tags.

```bash
# Daemon logs (live follow)
journalctl -u zopu-daemon -f

# Agent logs (live follow)
journalctl -u zopu-agent -f

# Last 100 lines of daemon
journalctl -u zopu-daemon -n 100

# Logs since boot
journalctl -u zopu-daemon -b

# Health check timer logs
journalctl -u zopu-health.service -n 50

# Docker cleanup logs
journalctl -u zopu-docker-cleanup.service -n 50

# Disk monitor logs (cron → file)
tail -100 /var/log/zopu/disk-monitor.log

# All Zopu syslog identifiers
journalctl -t zopu-daemon -t zopu-agent --since "1 hour ago"
```

## Health checks

```bash
# Manual health check (prints all probes)
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh

# Quiet mode (exit code only)
/opt/zopu/deploy/zopu-runtime/scripts/health-check.sh --quiet

# Check systemd timer is running
systemctl status zopu-health.timer
systemctl list-timers zopu-health.timer
```

The health check probes:
1. `zopu-daemon` systemd unit is active
2. `zopu-agent` systemd unit is active
3. RivetKit engine port (default 6420) accepts TCP connections
4. Flue agent port (default 3583) accepts TCP connections
5. Docker daemon responds to `docker info`

No HTTP health endpoints are assumed. Flue does not expose one by design
(per Flue docs: "Flue does not add a health endpoint"). RivetKit's health
route is internal to the registry runtime and not documented as
publicly addressable on the engine endpoint.

## Update to commit

```bash
# Update to latest of dogfood/v0 (default)
/opt/zopu/deploy/zopu-runtime/scripts/update.sh

# Update to a specific branch
/opt/zopu/deploy/zopu-runtime/scripts/update.sh dogfood/runtime-deploy

# Update to a specific commit
/opt/zopu/deploy/zopu-runtime/scripts/update.sh abc123def456
```

The update script:
1. Records current HEAD to `.last-deployed-sha`
2. Fetches, resolves branch-or-commit, checks out
3. `bun install`, builds daemon and agent
4. Restarts daemon, waits, restarts agent
5. Runs health check; reports failure and rollback instructions

## Rollback

```bash
# Roll back to the previously deployed commit
/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh

# Roll back to a specific commit
/opt/zopu/deploy/zopu-runtime/scripts/rollback.sh abc123def456
```

Rollback reads `.last-deployed-sha` (written by `update.sh`), checks out that
commit, rebuilds, and restarts services. The pre-rollback SHA is saved to
`.pre-rollback-sha` for re-rollback if needed.

## Docker cleanup

```bash
# Manual cleanup
/opt/zopu/deploy/zopu-runtime/scripts/docker-cleanup.sh

# Check Docker disk usage
docker system df

# Timer runs daily; check its schedule
systemctl list-timers zopu-docker-cleanup.timer
```

Cleanup prunes:
- Stopped containers older than 24 hours
- Dangling (untagged) images
- Unused networks


Named volumes and running containers are never removed.

## Disk-space monitoring

```bash
# Manual check
/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh

# Custom threshold (90%)
/opt/zopu/deploy/zopu-runtime/scripts/disk-monitor.sh --warn-percent 90
```

A cron job runs at 06:00 daily and writes to `/var/log/zopu/disk-monitor.log`.
Default alert threshold is 80%.

## Firewall and private networking

The firewall (`ufw`) is deny-by-default:
- SSH (port 22) is allowed on all interfaces
- All traffic on `tailscale0` is allowed (Tailscale private overlay)
- All other incoming traffic is denied

The RivetKit engine (`:6420`) and Flue agent (`:3583`) ports are **not**
exposed on public interfaces. Reachability options:

1. **Tailscale** (recommended): bootstrap runs `ufw allow in on tailscale0`
   so all ports are reachable over the private overlay. Set
   `TAILSCALE_AUTHKEY` before running bootstrap to configure automatically.
   Other Tailscale-connected machines can reach the agent at
   `http://zopu-runtime:3583` and the engine at `http://zopu-runtime:6420`.
2. **Custom private interface**: if you have a non-Tailscale private network
   (e.g. a VLAN or wireguard interface), add an explicit rule:
   ```bash
   ufw allow in on eth1  # or your private interface name
   ```
   Do NOT assume direct private IP access works by default — the deny-incoming
   policy blocks it until an interface-specific rule is added.
3. **Caddy** (optional): install Caddy and use the annotated Caddyfile in
   `caddy/` if you need TLS termination or a public ingress point.

## Environment groups

See [`.env.template`](./.env.template) for the full annotated template. The
eight required groups:

| Group | Variables |
|-------|-----------|
| Convex | `CONVEX_URL`, `CONVEX_SITE_URL`, `SITE_URL` |
| Gitea | `GITEA_URL`, `GITEA_TOKEN` |
| Model gateway | `AGENT_MODEL_*` |
| AgentOS/RivetKit | `RIVET_ENDPOINT` (optional) |
| Zopu agent | `FLUE_DB_TOKEN`, `PORT` |
| Daemon | `DAEMON_ID`, `DAEMON_NAME`, `DAEMON_VERSION`, `DAEMON_HEARTBEAT_MS`, `DAEMON_COMMAND_LEASE_MS` |
| Docker sandbox | group membership (no env vars) |
| Service auth | `AUTH_SECRET` (if needed) |

## What is NOT deployed

- **Web app**: the web frontend is not deployed in this lane.
- **Kubernetes**: no container orchestration.
- **PostgreSQL for Rivet**: the in-process RivetKit engine uses its own
  storage; no external PostgreSQL is required.
- **Multi-node coordination**: single-node only.
- **Public administration endpoints**: no admin HTTP surface.
- **Secrets in source**: `.env` is never committed; `.env.template` contains
  only placeholder values.
- **Production generated-app hosting**: only the execution plane runs here.
- **Docker-backed Orb sandboxes**: Docker is installed and access is
  provisioned, but no Orb sandbox code is wired. This is a boundary
  prepared for the Orb lane, not a working feature.

## Reproducibility

The deployment does not require the developer's MacBook to remain online.
Once bootstrap completes and `.env` is filled in:

1. Services run under systemd with `Restart=always`.
2. Logs persist in journald.
3. Health checks run every 60 seconds via systemd timer.
4. Docker cleanup runs daily.
5. Disk usage is monitored daily.
6. Unattended-upgrades handles Debian security patches.