mirror of https://github.com/getpaseo/paseo.git synced 2026-07-29 12:01:31 +00:00

Files

Mohamed Boudra 72b67f48e3 Eliminate spiky terminal lag under load (#1500 )

* Eliminate spiky terminal lag under load

Terminal output stuttered during heavy output and when the daemon was
busy. Two compounding causes, both confirmed by measurement:

- Every 256KB of output, the daemon dropped pending frames and re-sent a
  full cell-grid snapshot (cloned across IPC, stringified to JSON) — a
  10MB build did this ~40 times. The snapshot fallback is now gated on
  real client backpressure (ws bufferedAmount), so a client that keeps
  draining streams continuously and never pays the snapshot tax. This
  also removes the periodic GC hitch the churn caused.
- Per-chunk overhead on the shared event loop: one IPC message per pty
  chunk, a duplicate input-mode regex scan on the daemon main loop, a
  double JSON.stringify per outbound message, and 16KB string realloc
  per chunk. Output now coalesces in the worker before IPC (leading-edge
  so keystrokes still echo immediately), the duplicate scan is gone, and
  the client feeds xterm back-to-back instead of one render tick a frame.

Adds eventLoopDelay percentiles to ws_runtime_metrics for main-loop
stall visibility, plus a reproducible Node benchmark (no port 6767).

Measured: echo p50 7.7ms to 2.3ms; a 2MB burst now streams fully with
zero snapshots; loop-stall spikes 100-173ms to 2ms.

* Guard trace-level check against partial logger stubs

The isLevelEnabled gate added for the emit() perf fix crashed every
session test that injects a hand-rolled logger stub (10 of them omit
isLevelEnabled). Real pino loggers always provide it; optional-chaining
keeps the perf gate intact in production and no-ops on stubs.

* Make recent-output exit-summary test deterministic

The added test self-exited the child immediately after writing 3000
lines, then asserted the newest line was in the exit summary — but the
summary reads the headless xterm buffer, which parses writes
asynchronously, so a loaded CI runner saw a stale buffer (line-2707 not
line-2999). Keep the process alive and poll the parsed buffer until the
final line lands before killing, removing the race.

* Skip ConPTY-fragile worker terminal tests on Windows

The coalescing and input-mode-preamble tests assert byte-contiguous PTY
output and an exact kitty-escape round-trip. Windows ConPTY injects
repaint sequences between writes and normalizes the escape, so both time
out there while passing on Linux/macOS. Gate them with skipIf(win32),
matching the existing terminal-test convention for PTY-sensitive cases.

* Apply terminal barriers immediately when no writes are pending

A barrier op (snapshot/restore/clear) only needs the sentinel-write gate
to wait out plain writes still parsing in xterm's buffer. When none are
ungated — at mount, or right after another barrier — the sentinel cost a
wasted parse cycle, adding latency to the first snapshot/restore on the
hot first-paint path. Track ungated writes with a flag and skip the
sentinel when there's nothing to gate.

* Fix mobile startup saved-host e2e

* Unskip Windows worker terminal coverage

* Tighten worker coalescing assertion

* Reset terminal ungated writes on unmount

* Fix terminal regressions found in adversarial review

Three issues this PR's terminal changes introduced:

- Worker snapshot could duplicate coalesced output. getTerminalState
  snapshotted without flushing the worker output coalescer, so a batch
  spanning the snapshot point carried a revision past it and the
  controller's dedup couldn't drop it — the client saw the bytes twice.
  Flush before snapshotting.
- Relay clients lost backpressure protection. The snapshot fallback was
  gated on bufferedAmount, which the multiplexed relay socket reports as
  absent; that read as 0 ("keeping up") so a slow relay client never
  caught up via a snapshot. Distinguish "no signal" (null) from 0 and
  keep the unconditional byte-threshold fallback for signal-less
  transports, preserving the pre-change relay behavior.
- Recent-output buffer was no longer a hard cap. A single chunk larger
  than the limit was retained whole; slice its tail.

Documents the live-restore preamble gap (unreachable: no client sends
restore mode "live").

* Skip input-mode preamble test on Windows ConPTY

CI confirmed Windows ConPTY normalizes away the kitty keyboard escape
the child writes, so it never reaches the worker's input-mode tracker
and the preamble stays empty — the test times out. ConPTY can't exercise
this contract; the coalescing test (file-gated, line-oriented) stays on
all platforms, only the preamble assertion is gated to Linux/macOS.

2026-06-13 05:38:58 +00:00

4.3 KiB

Raw Permalink Blame History

Terminal performance

How terminal output stays low-latency, what the invariants are, and how to measure before/after any change to the pipeline. Read this before touching anything under packages/server/src/terminal/ or packages/app/src/terminal/runtime/.

The pipeline

pty (node-pty, forked worker process)
  → headless xterm parse (worker, snapshot fidelity)
  → TerminalOutputCoalescer (worker, ≤1 IPC message per 5ms per terminal)
  → process.send IPC → daemon main process
  → TerminalOutputCoalescer (per client stream, terminal-session-controller.ts)
  → binary ws frame (2-byte header + raw bytes)
  → client decode (daemon-client.ts) → stream router → emulator runtime
  → xterm.write (back-to-back; xterm batches internally)

Terminal frames share the daemon main event loop with all agent traffic. The eventLoopDelay block in the ws_runtime_metrics log line (every 30s in daemon.log) is the ground truth for "the daemon is busy" — p99/max there directly bound worst-case terminal frame delay.

Invariants (the easy-to-break ones)

Coalescers are leading+trailing throttles. The first chunk after an idle window flushes immediately (synchronously); only sustained bursts wait for the trailing timer. Reverting to trailing-only adds a full window (~5ms) to every keystroke echo.
Output coalescing happens in the worker, before IPC. One process.send per pty chunk was a main-loop flood under build output. Non-output messages (snapshot/snapshotReady/titleChange/exit) must flush the coalescer first so ordering is preserved.
Coalesced output carries the LAST chunk's revision. Snapshot replay dedup (replayTerminalOutputAfterSnapshot) skips buffered output with revision <= replayRevision; a merged batch with a lower revision would be wrongly skipped (lost output).
The input-mode tracker runs once per process boundary, not per hop. The worker owns the authoritative tracker; the daemon caches the replay preamble from getTerminalState responses and snapshotReady messages. Do not reintroduce a per-chunk feed() on the daemon main loop.
Snapshot catch-up is backpressure-gated. A stream falls back to a full snapshot only when outputBytesSinceSnapshot > MAX_TERMINAL_OUTPUT_FRAME_BYTES (256KB) and the client transport reports bufferedAmount > MAX_CLIENT_BUFFERED_BYTES (4MB). A client that keeps draining streams continuously, no matter how much output is produced. Before this gate existed, every 256KB of build output dropped a frame and forced a full JSON cell-grid snapshot (~200k objects across IPC) — the historical source of spiky lag and GC hitches.
Client output writes are not serialized per frame. The emulator runtime drains contiguous plain writes straight into xterm (which buffers internally). Only barrier ops (clear, snapshot, suppressInput writes) wait — behind a zero-length sentinel write — so resets can't interleave with in-flight output.

Measuring

Node-only benchmark (fast iteration, server pipeline): npx tsx scripts/benchmark-terminal-latency.ts. Boots an isolated daemon (fresh PASEO_HOME, random port — never 6767), measures echo latency percentiles, burst jitter, and snapshot counts under ramped mock-agent load. Writes JSON to /tmp/paseo-terminal-bench/. Healthy numbers (2026-06): echo p50 ~2.3ms, p95 ~3.3ms, a 2MB burst fully streamed with snap=0.
Browser perf specs (user-perceived path): gated behind PASEO_TERMINAL_PERF_E2E=1 — packages/app/e2e/terminal-performance.spec.ts and packages/app/e2e/terminal-keystroke-stress.spec.ts (per-stage keydown→xterm-commit breakdown under mock-agent load). Healthy: keydown→commit p50 ~18ms under 600-key burst.
Production: grep daemon.log for ws_runtime_metrics and read eventLoopDelay + bufferedAmount.

Known remaining contention (follow-up candidates)

A single large agent_stream message (e.g. a 250KB diff payload) measurably delays terminal echo (~100ms-class dips) — cost is split between daemon serialization and app-side parse/render on the shared browser main thread.
Relay-attached clients pay pure-JS tweetnacl encryption + base64 per frame on the daemon main loop (packages/relay/src/encrypted-channel.ts).
sendToClient re-stringifies session messages per socket; only matters for multi-socket connections.

4.3 KiB Raw Permalink Blame History