237 lines
6.7 KiB
Markdown
237 lines
6.7 KiB
Markdown
# Kubernetes
|
|
|
|
> Source: `src/content/docs/self-hosting/kubernetes.mdx`
|
|
> Canonical URL: https://rivet.dev/docs/self-hosting/kubernetes
|
|
> Description: Deploy production-ready Rivet Engine to Kubernetes with PostgreSQL storage.
|
|
|
|
---
|
|
PostgreSQL is the recommended backend for multi-node self-hosted deployments today, but it remains experimental. For a production-ready single-node Rivet deployment, use the file system backend (RocksDB-based). Enterprise teams can contact [enterprise support](https://rivet.dev/sales) about FoundationDB for the most scalable production-ready deployment.
|
|
|
|
## Prerequisites
|
|
|
|
- Kubernetes cluster
|
|
- `kubectl` configured
|
|
- [Metrics server](https://github.com/kubernetes-sigs/metrics-server) (required for HPA) — included by default in most distributions (k3d, GKE, EKS, AKS)
|
|
|
|
## Deploy Rivet Engine
|
|
|
|
|
|
### Download Manifests
|
|
|
|
Download the `self-host/k8s/engine` directory from the Rivet repository:
|
|
|
|
```bash
|
|
npx giget@latest gh:rivet-dev/rivet/self-host/k8s/engine rivet-k8s
|
|
cd rivet-k8s
|
|
```
|
|
|
|
|
|
|
|
### Configure Engine
|
|
|
|
In `02-engine-configmap.yaml`, set `public_url` to your engine's external URL.
|
|
|
|
|
|
|
|
### Configure PostgreSQL
|
|
|
|
In `11-postgres-secret.yaml`, update the PostgreSQL password. See [Using a Managed PostgreSQL Service](#using-a-managed-postgresql-service) for external databases.
|
|
|
|
|
|
|
|
### Configure Admin Token
|
|
|
|
Generate a secure admin token and save it somewhere safe:
|
|
|
|
```bash
|
|
openssl rand -hex 32
|
|
```
|
|
|
|
Create the namespace and store the token as a Kubernetes secret:
|
|
|
|
```bash
|
|
kubectl create namespace rivet-engine
|
|
kubectl -n rivet-engine create secret generic rivet-secrets --from-literal=admin-token=YOUR_TOKEN_HERE
|
|
```
|
|
|
|
|
|
|
|
### Deploy
|
|
|
|
```bash
|
|
# Apply all manifests
|
|
kubectl apply -f .
|
|
|
|
# Wait for all pods to be ready
|
|
kubectl -n rivet-engine wait --for=condition=ready pod -l app=nats --timeout=300s
|
|
kubectl -n rivet-engine wait --for=condition=ready pod -l app=postgres --timeout=300s
|
|
kubectl -n rivet-engine wait --for=condition=ready pod -l app=rivet-engine --timeout=300s
|
|
|
|
# Verify all pods are running
|
|
kubectl -n rivet-engine get pods
|
|
```
|
|
|
|
|
|
|
|
### Access the Engine
|
|
|
|
Visit `/ui` on your `public_url` to access the dashboard.
|
|
|
|
|
|
## Connecting Your Project
|
|
|
|
### Create your server
|
|
|
|
Follow the [quickstart](/docs/actors/quickstart/backend) to create a working server with actors.
|
|
|
|
### Create Kubernetes manifests
|
|
|
|
Create these manifest files for your app:
|
|
|
|
```yaml deployment.yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: rivetkit-app
|
|
namespace: your-namespace
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: rivetkit-app
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: rivetkit-app
|
|
spec:
|
|
# Allow enough time for actors to gracefully stop on SIGTERM.
|
|
# The runner waits up to 30m for actors to finish.
|
|
# Add buffer for runner shutdown overhead after actors stop.
|
|
# See: /docs/actors/versions#graceful-shutdown-sigterm
|
|
terminationGracePeriodSeconds: 2100
|
|
containers:
|
|
- name: rivetkit-app
|
|
image: registry.example.com/your-team/rivetkit-app:latest
|
|
envFrom:
|
|
- secretRef:
|
|
name: rivetkit-secrets
|
|
```
|
|
|
|
```yaml service.yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: rivetkit-app
|
|
namespace: your-namespace
|
|
spec:
|
|
selector:
|
|
app: rivetkit-app
|
|
ports:
|
|
- name: http
|
|
port: 8080
|
|
targetPort: 8080
|
|
```
|
|
|
|
### Configure the endpoint
|
|
|
|
Create `rivetkit-secrets.yaml` with `RIVET_ENDPOINT` pointing to the engine. This tells your app to connect as a runner instead of running standalone. See [Endpoints](/docs/general/endpoints) for all options.
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Secret
|
|
metadata:
|
|
name: rivetkit-secrets
|
|
namespace: your-namespace
|
|
type: Opaque
|
|
stringData:
|
|
RIVET_ENDPOINT: http://my-app:your-admin-token@your-engine.example.com
|
|
```
|
|
|
|
### Deploy your app
|
|
|
|
```bash
|
|
kubectl apply -f rivetkit-secrets.yaml
|
|
kubectl apply -f deployment.yaml
|
|
kubectl apply -f service.yaml
|
|
```
|
|
|
|
### Register your runner with the engine
|
|
|
|
### Dashboard
|
|
|
|
1. Open the Rivet Engine dashboard in your browser.
|
|
2. Enter your admin token when prompted.
|
|
3. In the namespace sidebar, click **Settings**.
|
|
4. Click **Add Provider**, then choose **Custom**.
|
|
5. Click **Next**.
|
|
6. Go to **Confirm Connection**, enter your app endpoint (e.g. `http://rivetkit-app.your-namespace:8080/api/rivet`), then click **Add**.
|
|
|
|
### CLI (curl)
|
|
|
|
Register your runner programmatically via the engine API:
|
|
|
|
```bash @nocheck
|
|
curl -X PUT "https://your-engine.example.com/runner-configs/default?namespace=default" \
|
|
-H "Content-Type: application/json" \
|
|
-H "Authorization: Bearer your-admin-token" \
|
|
-d '{
|
|
"datacenters": {
|
|
"default": {
|
|
"normal": {}
|
|
}
|
|
}
|
|
}'
|
|
```
|
|
|
|
## Advanced
|
|
|
|
### Using a Managed PostgreSQL Service
|
|
|
|
If you prefer to use a managed PostgreSQL service (e.g. Amazon RDS, Cloud SQL, Azure Database) instead of the bundled Postgres deployment:
|
|
|
|
- Update the `postgres.url` connection string in `02-engine-configmap.yaml` to point to your managed instance
|
|
- Delete the bundled PostgreSQL manifests:
|
|
- `10-postgres-configmap.yaml`
|
|
- `11-postgres-secret.yaml`
|
|
- `12-postgres-statefulset.yaml`
|
|
- `13-postgres-service.yaml`
|
|
|
|
### Applying Configuration Updates
|
|
|
|
When making subsequent changes to `02-engine-configmap.yaml`, restart the engine pods to pick up the new configuration:
|
|
|
|
```bash
|
|
kubectl apply -f 02-engine-configmap.yaml
|
|
kubectl -n rivet-engine rollout restart deployment/rivet-engine
|
|
```
|
|
|
|
### Ingress and Load Balancer Timeouts
|
|
|
|
Rivet uses long-lived WebSocket connections for both client traffic (browsers, SDKs) and envoy traffic (actor hosts connecting back to the engine). Default Ingress and cloud load balancer idle timeouts (typically 30 to 60 seconds) drop these connections and cause reconnect storms.
|
|
|
|
Raise the idle / read / send timeout on every Ingress and load balancer in front of the engine to at least 1 hour (`3600` seconds). Examples:
|
|
|
|
- **NGINX Ingress** (annotations on the Ingress):
|
|
|
|
```yaml
|
|
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
|
|
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
|
|
```
|
|
|
|
- **AWS Load Balancer Controller (ALB)**:
|
|
|
|
```yaml
|
|
alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=3600
|
|
```
|
|
|
|
- **GCE Ingress (GKE)**: set `timeoutSec: 3600` on the `BackendConfig` referenced by the Service.
|
|
|
|
The same guidance applies to the load balancer fronting your RivetKit app, since envoys connect to it over WebSocket.
|
|
|
|
## Next Steps
|
|
|
|
- Review the [Production Checklist](/docs/self-hosting/production-checklist) before going live
|
|
- See [Configuration](/docs/self-hosting/configuration) for all engine config options
|
|
|
|
_Source doc path: /docs/self-hosting/kubernetes_
|