mirror of
https://github.com/getpaseo/paseo.git
synced 2026-07-29 12:01:31 +00:00
1d1132de9c
Preserve application frame identity through encryption so negotiated binary traffic avoids base64 expansion while mixed-version peers remain compatible.
46 lines
4.9 KiB
Markdown
46 lines
4.9 KiB
Markdown
# Terminal performance
|
|
|
|
How terminal output stays low-latency, what the invariants are, and how to measure before/after any change to the pipeline. Read this before touching anything under `packages/server/src/terminal/` or `packages/app/src/terminal/runtime/`.
|
|
|
|
## The pipeline
|
|
|
|
```
|
|
pty (node-pty, forked worker process)
|
|
→ headless xterm parse (worker, snapshot fidelity)
|
|
→ TerminalOutputCoalescer (worker, ≤1 IPC message per 5ms per terminal)
|
|
→ process.send IPC → daemon main process
|
|
→ TerminalOutputCoalescer (per client stream, terminal-session-controller.ts)
|
|
→ binary ws frame (2-byte header + raw bytes)
|
|
→ client decode (daemon-client.ts) → stream router → emulator runtime
|
|
→ xterm.write (back-to-back; xterm batches internally)
|
|
```
|
|
|
|
Terminal frames share the daemon main event loop with all agent traffic. The `eventLoopDelay` block in the `ws_runtime_metrics` log line (every 30s in `daemon.log`) is the ground truth for "the daemon is busy" — p99/max there directly bound worst-case terminal frame delay.
|
|
|
|
## Invariants (the easy-to-break ones)
|
|
|
|
- **Coalescers are leading+trailing throttles.** The first chunk after an idle window flushes immediately (synchronously); only sustained bursts wait for the trailing timer. Reverting to trailing-only adds a full window (~5ms) to every keystroke echo.
|
|
- **Output coalescing happens in the worker, before IPC.** One `process.send` per pty chunk was a main-loop flood under build output. Non-output messages (snapshot/snapshotReady/titleChange/exit) must flush the coalescer first so ordering is preserved.
|
|
- **Coalesced output carries the LAST chunk's revision.** Snapshot replay dedup (`replayTerminalOutputAfterSnapshot`) skips buffered output with `revision <= replayRevision`; a merged batch with a lower revision would be wrongly skipped (lost output).
|
|
- **The input-mode tracker runs once per process boundary, not per hop.** The worker owns the authoritative tracker; the daemon caches the replay preamble from `getTerminalState` responses and `snapshotReady` messages. Do not reintroduce a per-chunk `feed()` on the daemon main loop.
|
|
- **Snapshot catch-up is backpressure-gated.** A stream falls back to a full snapshot only when `outputBytesSinceSnapshot > MAX_TERMINAL_OUTPUT_FRAME_BYTES` (256KB) **and** the client transport reports `bufferedAmount > MAX_CLIENT_BUFFERED_BYTES` (4MB). A client that keeps draining streams continuously, no matter how much output is produced. Before this gate existed, every 256KB of build output dropped a frame and forced a full JSON cell-grid snapshot (~200k objects across IPC) — the historical source of spiky lag and GC hitches.
|
|
- **Client output writes are not serialized per frame.** The emulator runtime drains contiguous plain writes straight into xterm (which buffers internally). Only barrier ops (`clear`, `snapshot`, `suppressInput` writes) wait — behind a zero-length sentinel write — so resets can't interleave with in-flight output.
|
|
|
|
## Measuring
|
|
|
|
- **Node-only benchmark (fast iteration, server pipeline):** `npx tsx scripts/benchmark-terminal-latency.ts`. Boots an isolated daemon (fresh `PASEO_HOME`, random port — never 6767), measures echo latency percentiles, burst jitter, and snapshot counts under ramped mock-agent load. Writes JSON to `/tmp/paseo-terminal-bench/`. Healthy numbers (2026-06): echo p50 ~2.3ms, p95 ~3.3ms, a 2MB burst fully streamed with `snap=0`.
|
|
- **Browser perf specs (user-perceived path):** gated behind `PASEO_TERMINAL_PERF_E2E=1` —
|
|
`packages/app/e2e/terminal-performance.spec.ts` and `packages/app/e2e/terminal-keystroke-stress.spec.ts` (per-stage keydown→xterm-commit breakdown under mock-agent load). Healthy: keydown→commit p50 ~18ms under 600-key burst.
|
|
- **Production:** grep `daemon.log` for `ws_runtime_metrics` and read `eventLoopDelay` + `bufferedAmount`.
|
|
- **Git pressure:** the same log line includes `git.commands` (limiter occupancy, queue age,
|
|
queue wait, execution time, failures, timeouts, and top operations),
|
|
`git.workspaceService` (daemon-global Git observer ownership), and per-session workspace Git
|
|
subscription totals under `runtime`. Queue wait and execution time are separate because the Git
|
|
command timeout begins only after a command acquires a limiter slot.
|
|
|
|
## Known remaining contention (follow-up candidates)
|
|
|
|
- A single large `agent_stream` message (e.g. a 250KB diff payload) measurably delays terminal echo (~100ms-class dips) — cost is split between daemon serialization and app-side parse/render on the shared browser main thread.
|
|
- Relay-attached clients pay pure-JS tweetnacl encryption on the daemon main loop (`packages/relay/src/encrypted-channel.ts`). Negotiated binary application frames stay binary ciphertext and avoid base64 encode/decode; text and mixed-version traffic remain base64 WebSocket text frames.
|
|
- `sendToClient` re-stringifies session messages per socket; only matters for multi-socket connections.
|