mirror of
https://github.com/getpaseo/paseo.git
synced 2026-11-11 02:46:29 +00:00
Terminals survive daemon restarts, crashes and updates (#7644)
* M1: terminals survive daemon restarts: stop reasons, detached terminal worker, kill cleanup
restart_server_request.killTerminals / shutdown_server_request.keepTerminals and server_info.features.terminalsSurviveRestart; CLI daemon restart --kill-terminals and daemon stop --keep-terminals (capability-gated). The terminal worker runs detached over an owner-only, secret-authenticated socket and is adopted on daemon start. One kill cleanup contract (the reaper) by birth time, used by the daemon, the supervisor and stopDaemonInstance. Settings offers Restart and close terminals.
* M1: fix review findings: reaper session/table retry, short-command creation order, remote kill-restart cleanup report, stop test port
* M1: fix review findings: reaper-owned cleanup record, session member identity by terminal id, unreadable cleanup records, Settings restart error rendered in the card
* M2: worker protocol versions and draining, worker runs from its own copy, desktop upgrades keep terminals, out-of-date daemon notice, docs/terminal-worker.md
- Daemon classifies live workers by protocol: adopts current, drains one version old, shuts
down older, leaves a newer one running and answers list/create with
terminal_worker_incompatible (daemon status prints it). Never more than two workers.
- esbuild worker bundle (build:lib); a built daemon copies bundle, shell integration and
node-pty into $PASEO_HOME/terminal-worker/<version>-<hash>/ and launches from it; copies no
record names are pruned.
- Desktop app_update/version_mismatch stop with keep terminals on macOS/Linux; Windows and a
daemon without the capability close them.
- Out-of-date daemon sidebar notice with Update button, worded by terminalsSurviveRestart.
- PASEO-13: unreadable worker records fail status and kill-restart retry.
- PASEO-14: a session member without a terminal id is unknown, not unrelated.
- Smoke test 46 with A/B installation fixtures and per-fixture CLI shims (PASEO-9 fold).
* M2: track in-flight worker requests without leaving an unhandled rejection
detach({ afterPendingRequests }) tracked each request with request.finally(), which rejects
with the request; a fire-and-forget send that timed out surfaced as an unhandled rejection
(worker-terminal-manager.test.ts). Track settlement only.
* M2: CI round 1 fixes: worker terminal manager in the MCP key test, status cleanup field, shared socket dir
- mcp-server.test.ts (new upstream test) built the worker manager with the old fork API;
it now starts a real detached worker in a throwaway home.
- daemon-session.test.ts expected status without terminalCleanupUnconfirmed (added in M1).
- paseo-daemon.test.ts: a home under the scratch temp dir is too long for a unix socket, so
the worker listens in the per-user paseo-<uid> directory, which outlives workers by design;
the test now requires it to be empty instead of absent.
* M2: run Windows PowerShell by its install path for process identity
server-tests (windows-latest): bootstrap-provider-availability.test.ts starts a daemon with a
PATH that holds only git, and the terminal worker failed to start: 'cannot read its own process
identity: spawn powershell.exe ENOENT'. Process inspection (managed-processes.ts inspect and the
terminal worker's process table) now runs %SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe.
* M2: lifecycle e2e plain stop uses the shutdown RPC on Windows
CI round 2, server-integration (windows-latest): 'stop and stop --force never signal processes
a stale or earlier-boot record names' called stopDaemonInstance without a shutdown channel,
which Windows refuses ('has no graceful shutdown channel'). Plain stops in the file now pass
one, as the CLI does.
* M2: native notice outlet, Windows desktop-update wording, worker cap counts unconfirmed cleanups, create recovers after a drain
* M2: phone sidebar mounts the callout slot (iOS notice verified), worker record reads retry transient errors
* M2: native notice preconditions use explicit working directories, one block per terminal
* M1: compact sidebar shows only native callouts on every platform; PTY round trips in the MCP key test get a 5 s window
* ci: pin Playwright browser jobs to ubuntu-24.04
The pinned Playwright (1.58) has no ubuntu-26.04 build, and forks whose
ubuntu-latest already resolves to 26.04 fail at browser install. Job names
are unchanged, so required-check contexts still match.
* ci: use ubuntu-latest for Playwright jobs, same as upstream
Reverts the ubuntu-24.04 pin; upstream main's Playwright shards pass on ubuntu-latest.
* fix(server): keep terminal worker record IO off the daemon event loop
* fix(server): restore terminal creation after worker failure
A disconnected terminal worker remained the current manager indefinitely, leaving dead sessions listed and every later create bound to a closed socket. Confirm process death before retiring the member and use the existing single launch path for replacement. Reconnect live workers and replay their screens while preserving terminal identity.
* test(server): verify orphan cleanup during worker recovery
Recovery reaps orphaned programs before a later daemon stop. The previous fixed-delay assertion raced that cleanup and failed when it completed promptly. Verify orphan removal, same-daemon replacement creation and cleanup on normal or forced stop instead.
* test(server): isolate Windows dependency fixture setup
Build the local package registry in setup so its npm packing processes do not consume the installation journey deadline. Install the platform package used by Windows runtime fixtures directly, avoiding an unused wrapper postinstall probe without a timeout.
* test(server): wait for pending watchers before disposal
Observation admission starts before asynchronous path setup reaches watcher subscription. Disposing at the admission counter races setup and never exercises late watcher cleanup. Wait for the subscriptions to be pending before disposing.
---------
Co-authored-by: Mohamed Boudra <boudra.moha@gmail.com>
This commit is contained in:
Sebastian Arena
co-authored by
Mohamed Boudra
Mohamed Boudra
parent
dee0d138e9
commit
41582eb5fd
@@ -3,8 +3,11 @@ import {
|
||||
readDaemonInstance,
|
||||
isSameDaemonInstance,
|
||||
DaemonInstanceError,
|
||||
retryUnconfirmedTerminalCleanup,
|
||||
TerminalCleanupNotConfirmedError,
|
||||
} from "@getpaseo/server/daemon-control";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import type { DaemonGetStatusResponse } from "@getpaseo/protocol/messages";
|
||||
import { connectToDaemon } from "../../utils/client.js";
|
||||
import { withOutput, type CommandOptions } from "../../output/index.js";
|
||||
import { addJsonAndDaemonHostOptions } from "../../utils/command-options.js";
|
||||
@@ -20,13 +23,30 @@ export function daemonRestartCommand(): Command {
|
||||
),
|
||||
)
|
||||
.option("--timeout <seconds>", "Replacement readiness deadline (default: 600)")
|
||||
.option("--kill-terminals", "Close every terminal instead of keeping them across the restart")
|
||||
.action(withOutput(runRestartCommand));
|
||||
}
|
||||
|
||||
type DaemonStatus = DaemonGetStatusResponse["payload"];
|
||||
|
||||
export async function runRestartCommand(options: CommandOptions, _command: Command) {
|
||||
const target = options.daemonTarget;
|
||||
const deadline = Date.now() + parseTimeoutMs(options.timeout);
|
||||
const remaining = () => Math.max(1, deadline - Date.now());
|
||||
const killTerminals = options.killTerminals === true;
|
||||
// The supervisor reaps before the respawn and the replacement retries what that left;
|
||||
// anything still named by a closing record survived both. A local caller retries once more
|
||||
// itself; a remote one has only what the replacement reports.
|
||||
const confirmTerminalsClosed = async (status: DaemonStatus) => {
|
||||
if (!killTerminals) return;
|
||||
if (target.kind === "instance") {
|
||||
await retryUnconfirmedTerminalCleanup(target.home);
|
||||
return;
|
||||
}
|
||||
// Absent on daemons without terminalsSurviveRestart, which have no terminal cleanup.
|
||||
const unconfirmed = status.terminalCleanupUnconfirmed ?? [];
|
||||
if (unconfirmed.length > 0) throw new TerminalCleanupNotConfirmedError(unconfirmed);
|
||||
};
|
||||
const instance = target.kind === "instance" ? await readDaemonInstance(target.home) : null;
|
||||
async function checkSupervisor() {
|
||||
if (target.kind !== "instance") return;
|
||||
@@ -54,7 +74,10 @@ export async function runRestartCommand(options: CommandOptions, _command: Comma
|
||||
workerPid = (await client.getDaemonStatus({ timeout: remaining() })).pid;
|
||||
await checkSupervisor();
|
||||
try {
|
||||
await client.restartServer("cli_restart", undefined, { timeout: remaining() });
|
||||
await client.restartServer("cli_restart", undefined, {
|
||||
timeout: remaining(),
|
||||
killTerminals,
|
||||
});
|
||||
acknowledged = true;
|
||||
} catch (error) {
|
||||
if (!isReconnectFailure(error)) throw error;
|
||||
@@ -77,6 +100,7 @@ export async function runRestartCommand(options: CommandOptions, _command: Comma
|
||||
const status = await replacement.getDaemonStatus({ timeout: remaining() });
|
||||
if (status.pid !== workerPid) {
|
||||
await checkSupervisor();
|
||||
await confirmTerminalsClosed(status);
|
||||
return {
|
||||
type: "single" as const,
|
||||
data: {
|
||||
|
||||
@@ -103,6 +103,7 @@ async function probeDaemonStatus(
|
||||
daemonNode: status.nodePath,
|
||||
providers: status.providers,
|
||||
relay: status.relay,
|
||||
...(status.terminalsUnavailable ? { terminals: status.terminalsUnavailable } : {}),
|
||||
};
|
||||
}
|
||||
} finally {
|
||||
|
||||
@@ -0,0 +1,110 @@
|
||||
import { spawn, type ChildProcess } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { hostname, tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { Command } from "commander";
|
||||
import { afterEach, describe, expect, test } from "vitest";
|
||||
import { createStopCommandRunner, type StopCommandDeps } from "./stop.js";
|
||||
|
||||
// The daemon the gate is checked against: a connection whose server_info may or may not
|
||||
// advertise terminalsSurviveRestart, and that records any shutdown it is asked for.
|
||||
function createFakeDaemon(features: { terminalsSurviveRestart?: boolean } = {}) {
|
||||
const shutdownRequests: Array<{ keepTerminals?: boolean }> = [];
|
||||
const deps: StopCommandDeps = {
|
||||
connectToDaemon: async () => ({
|
||||
getLastServerInfoMessage: () => ({
|
||||
status: "server_info",
|
||||
serverId: "fake-daemon",
|
||||
version: "0.11.0",
|
||||
features,
|
||||
}),
|
||||
shutdownServer: async (options) => {
|
||||
shutdownRequests.push({ keepTerminals: options?.keepTerminals });
|
||||
return { status: "shutdown_requested", clientId: "fake-client", requestId: "fake-request" };
|
||||
},
|
||||
close: async () => undefined,
|
||||
}),
|
||||
};
|
||||
return { deps, shutdownRequests };
|
||||
}
|
||||
|
||||
const cleanups: Array<() => void> = [];
|
||||
|
||||
afterEach(() => {
|
||||
for (const cleanup of cleanups.splice(0)) cleanup();
|
||||
});
|
||||
|
||||
function stop(options: Record<string, unknown>, deps: StopCommandDeps) {
|
||||
return createStopCommandRunner(deps)(options as never, new Command("stop"));
|
||||
}
|
||||
|
||||
// A home whose lock names a live stand-in supervisor, so the local stop path runs for real.
|
||||
function homeWithRunningSupervisor(): { home: string; supervisor: ChildProcess } {
|
||||
const home = mkdtempSync(path.join(tmpdir(), "paseo-stop-keep-"));
|
||||
const supervisor = spawn(process.execPath, ["-e", "setInterval(() => {}, 1e9)"], {
|
||||
stdio: "ignore",
|
||||
});
|
||||
writeFileSync(
|
||||
path.join(home, "paseo.pid"),
|
||||
JSON.stringify({
|
||||
pid: supervisor.pid,
|
||||
startedAt: new Date().toISOString(),
|
||||
hostname: hostname(),
|
||||
uid: process.getuid?.() ?? 0,
|
||||
listen: "127.0.0.1:1",
|
||||
}),
|
||||
);
|
||||
cleanups.push(() => {
|
||||
supervisor.kill("SIGKILL");
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
return { home, supervisor };
|
||||
}
|
||||
|
||||
describe("daemon stop --keep-terminals", () => {
|
||||
test("refuses an endpoint daemon that does not advertise terminalsSurviveRestart", async () => {
|
||||
const daemon = createFakeDaemon();
|
||||
await expect(
|
||||
stop(
|
||||
{ daemonTarget: { kind: "endpoint", host: "127.0.0.1:7777" }, keepTerminals: true },
|
||||
daemon.deps,
|
||||
),
|
||||
).rejects.toMatchObject({ code: "TERMINALS_KEEP_UNSUPPORTED" });
|
||||
expect(daemon.shutdownRequests).toEqual([]);
|
||||
});
|
||||
|
||||
test.skipIf(process.platform === "win32")(
|
||||
"refuses a local daemon without the capability and never signals its supervisor",
|
||||
async () => {
|
||||
const daemon = createFakeDaemon();
|
||||
const { home, supervisor } = homeWithRunningSupervisor();
|
||||
await expect(
|
||||
stop({ daemonTarget: { kind: "instance", home }, keepTerminals: true }, daemon.deps),
|
||||
).rejects.toMatchObject({ code: "TERMINALS_KEEP_UNSUPPORTED" });
|
||||
expect(daemon.shutdownRequests).toEqual([]);
|
||||
expect(supervisor.exitCode).toBeNull();
|
||||
expect(supervisor.signalCode).toBeNull();
|
||||
},
|
||||
);
|
||||
|
||||
test("asks a capable daemon to keep its terminals", async () => {
|
||||
const daemon = createFakeDaemon({ terminalsSurviveRestart: true });
|
||||
await stop(
|
||||
{ daemonTarget: { kind: "endpoint", host: "127.0.0.1:7777" }, keepTerminals: true },
|
||||
daemon.deps,
|
||||
);
|
||||
expect(daemon.shutdownRequests).toEqual([{ keepTerminals: true }]);
|
||||
});
|
||||
|
||||
test("rejects --force with --keep-terminals before contacting the daemon", async () => {
|
||||
const daemon = createFakeDaemon({ terminalsSurviveRestart: true });
|
||||
const { home } = homeWithRunningSupervisor();
|
||||
await expect(
|
||||
stop(
|
||||
{ daemonTarget: { kind: "instance", home }, keepTerminals: true, force: true },
|
||||
daemon.deps,
|
||||
),
|
||||
).rejects.toMatchObject({ code: "INVALID_OPTIONS" });
|
||||
expect(daemon.shutdownRequests).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -1,6 +1,10 @@
|
||||
import { Command } from "commander";
|
||||
import {
|
||||
requireTerminalKeepCapability,
|
||||
type DaemonClient,
|
||||
} from "@getpaseo/client/internal/daemon-client";
|
||||
import { stopDaemonInstance, type DaemonInstance } from "@getpaseo/server/daemon-control";
|
||||
import { connectToDaemon } from "../../utils/client.js";
|
||||
import { connectToDaemon, type ConnectOptions } from "../../utils/client.js";
|
||||
import { withOutput, type CommandOptions } from "../../output/index.js";
|
||||
import { addJsonAndDaemonHostOptions } from "../../utils/command-options.js";
|
||||
import { describeDaemonTarget } from "../../utils/daemon-target.js";
|
||||
@@ -10,19 +14,43 @@ export function daemonStopCommand(): Command {
|
||||
return addJsonAndDaemonHostOptions(new Command("stop").description("Stop the selected daemon"))
|
||||
.option("--timeout <seconds>", "Graceful exit deadline (default: 15)")
|
||||
.option("--force", "Permit forced local process cleanup")
|
||||
.option(
|
||||
"--keep-terminals",
|
||||
"Leave terminals running; the next daemon started for this home takes them back",
|
||||
)
|
||||
.option("--kill-timeout <seconds>", "Forced exit deadline (default: 3)")
|
||||
.action(withOutput(runStopCommand));
|
||||
}
|
||||
|
||||
export async function runStopCommand(options: CommandOptions, _command: Command) {
|
||||
// The daemon connection stop needs; stop.test.ts passes an in-memory adapter.
|
||||
export type StopDaemonConnection = Pick<
|
||||
DaemonClient,
|
||||
"getLastServerInfoMessage" | "shutdownServer" | "close"
|
||||
>;
|
||||
|
||||
export interface StopCommandDeps {
|
||||
connectToDaemon(options: ConnectOptions): Promise<StopDaemonConnection>;
|
||||
}
|
||||
|
||||
export const runStopCommand = createStopCommandRunner({ connectToDaemon });
|
||||
|
||||
export function createStopCommandRunner(deps: StopCommandDeps) {
|
||||
return (options: CommandOptions, _command: Command) => runStop(options, deps);
|
||||
}
|
||||
|
||||
async function runStop(options: CommandOptions, deps: StopCommandDeps) {
|
||||
const target = options.daemonTarget;
|
||||
const timeoutMs = parseTimeoutMs(options.timeout, 15_000);
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
const remaining = () => Math.max(1, deadline - Date.now());
|
||||
const keepTerminals = options.keepTerminals === true;
|
||||
const requestShutdown = async (instance?: DaemonInstance) => {
|
||||
const client = await connectToDaemon({ target, timeout: remaining(), instance });
|
||||
const client = await deps.connectToDaemon({ target, timeout: remaining(), instance });
|
||||
try {
|
||||
await client.shutdownServer({ timeout: remaining() });
|
||||
// Checked on the connection that carries the shutdown: a daemon without the capability
|
||||
// would close the terminals it was asked to keep.
|
||||
if (keepTerminals) requireTerminalKeepCapability(client, describeDaemonTarget(target));
|
||||
await client.shutdownServer({ timeout: remaining(), keepTerminals });
|
||||
} finally {
|
||||
await client.close();
|
||||
}
|
||||
@@ -32,6 +60,11 @@ export async function runStopCommand(options: CommandOptions, _command: Command)
|
||||
code: "INVALID_OPTIONS",
|
||||
message: "--force requires a local --home; an endpoint gives no remote process authority.",
|
||||
};
|
||||
if (options.force && keepTerminals)
|
||||
throw {
|
||||
code: "INVALID_OPTIONS",
|
||||
message: "--force kills terminals; it cannot be combined with --keep-terminals.",
|
||||
};
|
||||
const result: {
|
||||
action: string;
|
||||
home?: string;
|
||||
@@ -44,6 +77,7 @@ export async function runStopCommand(options: CommandOptions, _command: Command)
|
||||
? {
|
||||
...(await stopDaemonInstance(target.home, {
|
||||
force: options.force === true,
|
||||
keepTerminals,
|
||||
timeoutMs,
|
||||
killTimeoutMs: parseTimeoutMs(options.killTimeout, 3_000),
|
||||
requestShutdown,
|
||||
|
||||
@@ -0,0 +1,324 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
|
||||
/**
|
||||
* Terminals survive `paseo daemon restart` and `stop --keep-terminals`; `restart
|
||||
* --kill-terminals`, `stop` and `stop --force` close them. Runs the built CLI and daemon in a
|
||||
* throwaway home; every process it starts is stopped in `finally`.
|
||||
*/
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn } from "node:child_process";
|
||||
import { existsSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { mkdtemp, mkdir, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { readDaemonInstance } from "@getpaseo/server/daemon-control";
|
||||
import { runLocalPaseo } from "./helpers/local-cli.ts";
|
||||
import { getAvailablePort } from "./helpers/network.ts";
|
||||
|
||||
if (process.platform === "win32") {
|
||||
console.log("Skipping terminal survival CLI test on Windows (POSIX shell terminals).");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log("=== Daemon terminals survive restart ===\n");
|
||||
|
||||
const paseoHome = await mkdtemp(join(tmpdir(), "paseo-terminals-survive-"));
|
||||
const workDir = join(paseoHome, "work");
|
||||
await mkdir(workDir, { recursive: true });
|
||||
const port = await getAvailablePort();
|
||||
const env = {
|
||||
PASEO_HOME: paseoHome,
|
||||
PASEO_LOCAL_SPEECH_AUTO_DOWNLOAD: "0",
|
||||
PASEO_DICTATION_ENABLED: "0",
|
||||
PASEO_VOICE_MODE_ENABLED: "0",
|
||||
// The daemon inherits this; terminals then run a plain POSIX shell with no startup files.
|
||||
SHELL: "/bin/sh",
|
||||
};
|
||||
|
||||
function isAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function waitFor<T>(
|
||||
probe: () => Promise<T | null | false> | T | null | false,
|
||||
message: string,
|
||||
timeoutMs = 30_000,
|
||||
): Promise<T> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const value = await probe();
|
||||
if (value) return value;
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
}
|
||||
throw new Error(message);
|
||||
}
|
||||
|
||||
async function paseo(args: string[], expectSuccess = true) {
|
||||
const result = await runLocalPaseo(args, env);
|
||||
if (expectSuccess) {
|
||||
assert.equal(
|
||||
result.exitCode,
|
||||
0,
|
||||
`paseo ${args.join(" ")} failed:\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`,
|
||||
);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
async function json<T>(args: string[]): Promise<T> {
|
||||
return JSON.parse((await paseo([...args, "--json"])).stdout) as T;
|
||||
}
|
||||
|
||||
async function capture(terminalId: string): Promise<string> {
|
||||
const { lines } = await json<{ lines: string[] }>(["terminal", "capture", terminalId]);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function waitForScreen(terminalId: string, text: string): Promise<void> {
|
||||
await waitFor(
|
||||
async () => (await capture(terminalId)).includes(text),
|
||||
`terminal ${terminalId} never showed ${text}:\n${await capture(terminalId)}`,
|
||||
);
|
||||
}
|
||||
|
||||
async function terminalIds(): Promise<string[]> {
|
||||
return (await json<Array<{ id: string }>>(["terminal", "ls", "--all"])).map((t) => t.id);
|
||||
}
|
||||
|
||||
// Worker records and the cleanup records a kill that could not finish leaves beside them.
|
||||
function workerRecordCount(): number {
|
||||
const dir = join(paseoHome, "terminal-worker");
|
||||
return existsSync(dir)
|
||||
? readdirSync(dir).filter((name) => /^(\d+|closing-.+)\.json$/.test(name)).length
|
||||
: 0;
|
||||
}
|
||||
|
||||
// A shell terminal that reports its pid and prints a marker.
|
||||
async function openTerminal(name: string): Promise<{ id: string; pid: number }> {
|
||||
const terminal = await json<{ id: string }>([
|
||||
"terminal",
|
||||
"create",
|
||||
"--cwd",
|
||||
workDir,
|
||||
"--name",
|
||||
name,
|
||||
]);
|
||||
const pidFile = join(workDir, `${name}.pid`);
|
||||
await paseo([
|
||||
"terminal",
|
||||
"send-keys",
|
||||
terminal.id,
|
||||
"-l",
|
||||
`echo $$ > '${pidFile}'; echo MARKER-${name}`,
|
||||
]);
|
||||
await paseo(["terminal", "send-keys", terminal.id, "Enter"]);
|
||||
const pid = await waitFor(
|
||||
() =>
|
||||
existsSync(pidFile) && readFileSync(pidFile, "utf8").trim()
|
||||
? Number(readFileSync(pidFile, "utf8"))
|
||||
: null,
|
||||
`terminal ${name} never wrote its pid`,
|
||||
);
|
||||
await waitForScreen(terminal.id, `MARKER-${name}`);
|
||||
return { id: terminal.id, pid };
|
||||
}
|
||||
|
||||
async function expectKept(terminal: { id: string; pid: number }, name: string, trigger: string) {
|
||||
assert((await terminalIds()).includes(terminal.id), `${trigger}: terminal is still listed`);
|
||||
assert(isAlive(terminal.pid), `${trigger}: terminal program is still alive`);
|
||||
assert((await capture(terminal.id)).includes(`MARKER-${name}`), `${trigger}: marker is kept`);
|
||||
await paseo(["terminal", "send-keys", terminal.id, "-l", `echo after-${trigger}`]);
|
||||
await paseo(["terminal", "send-keys", terminal.id, "Enter"]);
|
||||
await waitForScreen(terminal.id, `\nafter-${trigger}`);
|
||||
}
|
||||
|
||||
async function expectClosed(pid: number, trigger: string) {
|
||||
await waitFor(() => !isAlive(pid), `${trigger}: terminal program ${pid} is still alive`, 15_000);
|
||||
}
|
||||
|
||||
// A zombie: a real process with a birth time that SIGKILL cannot remove while its parent (an
|
||||
// exec'd `sleep` that never waits) lives. Cleanup can only report it as still alive.
|
||||
async function startZombie(): Promise<{ zombiePid: number; parentPid: number }> {
|
||||
const parent = spawn("/bin/sh", ["-c", "sleep 0 & echo $!; exec sleep 600"], {
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
});
|
||||
const zombiePid = await new Promise<number>((resolve) => {
|
||||
parent.stdout!.once("data", (chunk: Buffer) => resolve(Number(chunk.toString().trim())));
|
||||
});
|
||||
parent.unref();
|
||||
await waitFor(
|
||||
async () => (await psField(zombiePid, "stat")).startsWith("Z"),
|
||||
`process ${zombiePid} never became a zombie`,
|
||||
);
|
||||
return { zombiePid, parentPid: parent.pid! };
|
||||
}
|
||||
|
||||
async function psField(pid: number, field: "stat" | "lstart"): Promise<string> {
|
||||
const child = spawn("ps", ["-o", `${field}=`, "-p", String(pid)], {
|
||||
env: { ...process.env, LC_ALL: "C" },
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
});
|
||||
let out = "";
|
||||
child.stdout.on("data", (chunk: Buffer) => (out += chunk.toString()));
|
||||
await new Promise((resolve) => child.once("close", resolve));
|
||||
return out.trim();
|
||||
}
|
||||
|
||||
async function goneProcessPid(): Promise<number> {
|
||||
const child = spawn(process.execPath, ["-e", ""], { stdio: "ignore" });
|
||||
await new Promise((resolve) => child.once("exit", resolve));
|
||||
return child.pid!;
|
||||
}
|
||||
|
||||
// A record as a dead terminal worker leaves it, naming one terminal program that is a zombie.
|
||||
async function plantRecordNaming(programPid: number): Promise<void> {
|
||||
const workerPid = await goneProcessPid();
|
||||
const record = {
|
||||
recordId: "planted-record",
|
||||
pid: workerPid,
|
||||
bootId: null,
|
||||
createdAt: new Date().toISOString(),
|
||||
identity: { startedAt: "Thu Jan 1 00:00:00 1970", commandLine: "planted-worker" },
|
||||
protocolVersion: 1,
|
||||
paseoVersion: "0.0.0-test",
|
||||
socketPath: join(paseoHome, "planted-worker.sock"),
|
||||
secret: "planted-secret",
|
||||
terminals: [
|
||||
{ id: "planted-terminal", pid: programPid, startedAt: await psField(programPid, "lstart") },
|
||||
],
|
||||
};
|
||||
writeFileSync(
|
||||
join(paseoHome, "terminal-worker", `${workerPid}.json`),
|
||||
`${JSON.stringify(record, null, 2)}\n`,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
}
|
||||
|
||||
const trackedPids: number[] = [];
|
||||
|
||||
try {
|
||||
for (const [field, value] of [
|
||||
["daemon.listen", `127.0.0.1:${port}`],
|
||||
["daemon.relay.enabled", "false"],
|
||||
["features.dictation.enabled", "false"],
|
||||
["features.voiceMode.enabled", "false"],
|
||||
] as const) {
|
||||
await paseo(["daemon", "config", "set", field, value]);
|
||||
}
|
||||
await paseo(["daemon", "start"]);
|
||||
|
||||
console.log("Test 1: daemon restart keeps the terminal");
|
||||
const kept = await openTerminal("kept");
|
||||
trackedPids.push(kept.pid);
|
||||
await paseo(["daemon", "restart", "--home", paseoHome]);
|
||||
await expectKept(kept, "kept", "restart");
|
||||
console.log("✓ same id, same program, marker and input after restart\n");
|
||||
|
||||
console.log("Test 2: stop --keep-terminals, then start, gives the terminal back");
|
||||
const forceAndKeep = await paseo(
|
||||
["daemon", "stop", "--home", paseoHome, "--force", "--keep-terminals"],
|
||||
false,
|
||||
);
|
||||
assert.notEqual(forceAndKeep.exitCode, 0, "--force with --keep-terminals is refused");
|
||||
await paseo(["daemon", "stop", "--home", paseoHome, "--keep-terminals"]);
|
||||
assert.equal(await readDaemonInstance(paseoHome), null, "the supervisor stopped");
|
||||
assert(isAlive(kept.pid), "the program outlives the stopped daemon");
|
||||
await paseo(["daemon", "start"]);
|
||||
await expectKept(kept, "kept", "keep-stop");
|
||||
console.log("✓ adopted with the same id and still usable\n");
|
||||
|
||||
console.log("Test 3: a plain stop with no daemon running closes kept terminals");
|
||||
await paseo(["daemon", "stop", "--home", paseoHome, "--keep-terminals"]);
|
||||
assert(isAlive(kept.pid), "kept again");
|
||||
await paseo(["daemon", "stop", "--home", paseoHome]);
|
||||
await expectClosed(kept.pid, "stop without a supervisor");
|
||||
assert.equal(workerRecordCount(), 0, "no worker record is left");
|
||||
console.log("✓ closed\n");
|
||||
|
||||
console.log("Test 4: restart --kill-terminals closes the terminal");
|
||||
await paseo(["daemon", "start"]);
|
||||
const killedByRestart = await openTerminal("restart-kill");
|
||||
trackedPids.push(killedByRestart.pid);
|
||||
await paseo(["daemon", "restart", "--home", paseoHome, "--kill-terminals"]);
|
||||
await expectClosed(killedByRestart.pid, "restart --kill-terminals");
|
||||
assert.deepEqual(await terminalIds(), [], "no terminal is listed after the kill restart");
|
||||
console.log("✓ closed\n");
|
||||
|
||||
console.log("Test 5: daemon stop closes the terminal and its worker");
|
||||
const killedByStop = await openTerminal("stop-kill");
|
||||
trackedPids.push(killedByStop.pid);
|
||||
await paseo(["daemon", "stop", "--home", paseoHome]);
|
||||
await expectClosed(killedByStop.pid, "stop");
|
||||
assert.equal(workerRecordCount(), 0, "no worker record is left");
|
||||
console.log("✓ closed\n");
|
||||
|
||||
console.log("Test 6: daemon stop --force closes the terminal");
|
||||
await paseo(["daemon", "start"]);
|
||||
const killedByForce = await openTerminal("force-kill");
|
||||
trackedPids.push(killedByForce.pid);
|
||||
await paseo(["daemon", "stop", "--home", paseoHome, "--force"]);
|
||||
await expectClosed(killedByForce.pid, "stop --force");
|
||||
assert.equal(workerRecordCount(), 0, "no worker record is left");
|
||||
console.log("✓ closed\n");
|
||||
|
||||
const endpoint = `127.0.0.1:${port}`;
|
||||
console.log("Test 7: restart --host --kill-terminals closes the terminal");
|
||||
await paseo(["daemon", "start"]);
|
||||
const killedByEndpointRestart = await openTerminal("endpoint-restart-kill");
|
||||
trackedPids.push(killedByEndpointRestart.pid);
|
||||
await paseo(["daemon", "restart", "--host", endpoint, "--kill-terminals"]);
|
||||
await expectClosed(killedByEndpointRestart.pid, "restart --host --kill-terminals");
|
||||
assert.deepEqual(await terminalIds(), [], "no terminal is listed after the kill restart");
|
||||
console.log("✓ closed and confirmed by the replacement\n");
|
||||
|
||||
console.log("Test 8: restart --host --kill-terminals reports a process cleanup could not end");
|
||||
const zombie = await startZombie();
|
||||
trackedPids.push(zombie.parentPid);
|
||||
await plantRecordNaming(zombie.zombiePid);
|
||||
const unconfirmed = await paseo(
|
||||
["daemon", "restart", "--host", endpoint, "--kill-terminals"],
|
||||
false,
|
||||
);
|
||||
assert.notEqual(unconfirmed.exitCode, 0, "the kill restart fails");
|
||||
const output = unconfirmed.stdout + unconfirmed.stderr;
|
||||
assert(output.includes("Terminal cleanup was not confirmed"), `names the failure:\n${output}`);
|
||||
assert(output.includes(`pid ${zombie.zombiePid}`), `names the surviving pid:\n${output}`);
|
||||
process.kill(zombie.parentPid, "SIGKILL");
|
||||
await expectClosed(zombie.zombiePid, "zombie reaped once its parent exits");
|
||||
await paseo(["daemon", "stop", "--home", paseoHome]);
|
||||
assert.equal(workerRecordCount(), 0, "the next stop retries the cleanup and retires the record");
|
||||
console.log("✓ reported, then retried by stop\n");
|
||||
|
||||
console.log("Test 9: restart --host --kill-terminals reports a worker record it cannot read");
|
||||
await paseo(["daemon", "start"]);
|
||||
// A kill cleanup cannot check what an unreadable record names, and leaves nothing else
|
||||
// behind, so the replacement daemon must still report it.
|
||||
const unreadablePath = join(paseoHome, "terminal-worker", `${await goneProcessPid()}.json`);
|
||||
writeFileSync(unreadablePath, "{ not json", { mode: 0o600 });
|
||||
const unreadable = await paseo(
|
||||
["daemon", "restart", "--host", endpoint, "--kill-terminals"],
|
||||
false,
|
||||
);
|
||||
assert.notEqual(unreadable.exitCode, 0, "the kill restart fails");
|
||||
const unreadableOutput = unreadable.stdout + unreadable.stderr;
|
||||
assert(unreadableOutput.includes(unreadablePath), `names the record:\n${unreadableOutput}`);
|
||||
assert(existsSync(unreadablePath), "the unreadable record is kept");
|
||||
rmSync(unreadablePath);
|
||||
await paseo(["daemon", "stop", "--home", paseoHome]);
|
||||
assert.equal(workerRecordCount(), 0, "no worker record is left");
|
||||
console.log("✓ reported with the record's path\n");
|
||||
} finally {
|
||||
await runLocalPaseo(["daemon", "stop", "--home", paseoHome, "--force"], env);
|
||||
for (const pid of trackedPids) {
|
||||
if (isAlive(pid)) process.kill(pid, "SIGKILL");
|
||||
}
|
||||
await rm(paseoHome, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
console.log("=== Daemon terminals survive restart passed ===");
|
||||
@@ -0,0 +1,527 @@
|
||||
#!/usr/bin/env npx tsx
|
||||
|
||||
/**
|
||||
* A terminal worker survives an upgrade of the installation that started it. Two disposable
|
||||
* installations A and B (built dist, real copies of @getpaseo/server, @getpaseo/cli and
|
||||
* node-pty, every other dependency a symlink into this repo) each get their own CLI shim, so
|
||||
* each daemon's PASEO_CLI is a distinct path:
|
||||
*
|
||||
* 1. A daemon from A runs its worker from a copy in $PASEO_HOME/terminal-worker/. After
|
||||
* `stop --keep-terminals` and moving A's node-pty, worker bundle and shell integration away,
|
||||
* a daemon from B adopts the same worker and opens a new, working terminal in it with B's
|
||||
* environment and CLI, while the first terminal keeps A's.
|
||||
* 2. Worker protocol versions (PASEO_TERMINAL_WORKER_PROTOCOL_OVERRIDE): a worker one version
|
||||
* old drains (keeps its terminal, new ones open in a second worker, it exits after its last
|
||||
* terminal and its copy is deleted); one two versions old is shut down; one from a newer
|
||||
* Paseo is left running, reported as terminal_worker_incompatible, and closed by
|
||||
* `daemon stop`.
|
||||
*
|
||||
* Every process it starts is stopped in `finally`.
|
||||
*/
|
||||
|
||||
import assert from "node:assert/strict";
|
||||
import { execFile } from "node:child_process";
|
||||
import {
|
||||
chmodSync,
|
||||
cpSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
renameSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { mkdtemp, rm } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { getAvailablePort } from "./helpers/network.ts";
|
||||
|
||||
if (process.platform === "win32") {
|
||||
console.log("Skipping terminal worker upgrade CLI test on Windows (POSIX shell terminals).");
|
||||
process.exit(0);
|
||||
}
|
||||
|
||||
console.log("=== Terminal worker survives an upgrade ===\n");
|
||||
|
||||
const repoRoot = join(import.meta.dirname, "..", "..", "..");
|
||||
const root = await mkdtemp(join(tmpdir(), "paseo-worker-upgrade-"));
|
||||
const paseoHome = join(root, "home");
|
||||
const cliHome = join(root, "os-home");
|
||||
const workDir = join(root, "work");
|
||||
for (const dir of [paseoHome, cliHome, workDir]) mkdirSync(dir, { recursive: true });
|
||||
const port = await getAvailablePort();
|
||||
const zsh = ["/bin/zsh", "/usr/bin/zsh"].find((candidate) => existsSync(candidate)) ?? null;
|
||||
|
||||
interface Installation {
|
||||
name: string;
|
||||
dir: string;
|
||||
version: string;
|
||||
shim: string;
|
||||
serverDir: string;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// Installation fixtures
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
function linkEntries(source: string, target: string, taken: Set<string>): void {
|
||||
if (!existsSync(source)) return;
|
||||
for (const name of readdirSync(source)) {
|
||||
if (name.startsWith(".")) continue;
|
||||
if (name.startsWith("@")) {
|
||||
mkdirSync(join(target, name), { recursive: true });
|
||||
for (const scoped of readdirSync(join(source, name))) {
|
||||
const key = `${name}/${scoped}`;
|
||||
if (taken.has(key)) continue;
|
||||
taken.add(key);
|
||||
symlinkSync(join(source, name, scoped), join(target, name, scoped));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (taken.has(name)) continue;
|
||||
taken.add(name);
|
||||
symlinkSync(join(source, name), join(target, name));
|
||||
}
|
||||
}
|
||||
|
||||
function createInstallation(name: string, version: string): Installation {
|
||||
const dir = join(root, `install-${name}`);
|
||||
const modules = join(dir, "node_modules");
|
||||
mkdirSync(join(modules, "@getpaseo"), { recursive: true });
|
||||
|
||||
// Real copies: the server (its version marks the installation), the CLI (so it resolves this
|
||||
// installation's server), and node-pty (so moving it away breaks only this installation).
|
||||
const serverDir = join(modules, "@getpaseo", "server");
|
||||
const serverPackage = JSON.parse(
|
||||
readFileSync(join(repoRoot, "packages/server/package.json"), "utf8"),
|
||||
) as Record<string, unknown>;
|
||||
mkdirSync(join(serverDir, "dist"), { recursive: true });
|
||||
writeFileSync(
|
||||
join(serverDir, "package.json"),
|
||||
`${JSON.stringify({ ...serverPackage, version }, null, 2)}\n`,
|
||||
);
|
||||
for (const part of ["server", "src", "scripts"]) {
|
||||
cpSync(join(repoRoot, "packages/server/dist", part), join(serverDir, "dist", part), {
|
||||
recursive: true,
|
||||
});
|
||||
}
|
||||
const cliDir = join(modules, "@getpaseo", "cli");
|
||||
mkdirSync(cliDir, { recursive: true });
|
||||
cpSync(join(repoRoot, "packages/cli/package.json"), join(cliDir, "package.json"));
|
||||
cpSync(join(repoRoot, "packages/cli/dist"), join(cliDir, "dist"), { recursive: true });
|
||||
cpSync(join(repoRoot, "packages/server/node_modules/node-pty"), join(modules, "node-pty"), {
|
||||
recursive: true,
|
||||
});
|
||||
|
||||
const taken = new Set(["@getpaseo/server", "@getpaseo/cli", "node-pty"]);
|
||||
for (const source of [
|
||||
"packages/server/node_modules",
|
||||
"packages/cli/node_modules",
|
||||
"node_modules",
|
||||
]) {
|
||||
linkEntries(join(repoRoot, source), modules, taken);
|
||||
}
|
||||
|
||||
const shim = join(dir, "bin", "paseo");
|
||||
mkdirSync(join(dir, "bin"));
|
||||
writeFileSync(
|
||||
shim,
|
||||
`#!/bin/sh\nexec "${process.execPath}" "${join(cliDir, "dist", "index.js")}" "$@"\n`,
|
||||
);
|
||||
chmodSync(shim, 0o755);
|
||||
return { name, dir, version, shim, serverDir };
|
||||
}
|
||||
|
||||
// The runtime files an upgrade replaces: node-pty, the worker bundle, the shell integration.
|
||||
function moveRuntimeAway(installation: Installation): () => void {
|
||||
const away = join(installation.dir, "moved-away");
|
||||
const moves = [
|
||||
join(installation.dir, "node_modules", "node-pty"),
|
||||
join(installation.serverDir, "dist", "server", "terminal-worker"),
|
||||
join(installation.serverDir, "dist", "server", "terminal", "shell-integration"),
|
||||
].map((from, index) => ({ from, to: join(away, String(index)) }));
|
||||
mkdirSync(away, { recursive: true });
|
||||
for (const move of moves) renameSync(move.from, move.to);
|
||||
return () => {
|
||||
for (const move of moves) renameSync(move.to, move.from);
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
|
||||
function isAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function waitFor<T>(
|
||||
probe: () => Promise<T | null | false> | T | null | false,
|
||||
message: () => string | Promise<string>,
|
||||
timeoutMs = 30_000,
|
||||
): Promise<T> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const value = await probe();
|
||||
if (value) return value;
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
}
|
||||
throw new Error(await message());
|
||||
}
|
||||
|
||||
interface RunResult {
|
||||
exitCode: number;
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
function run(
|
||||
installation: Installation,
|
||||
args: string[],
|
||||
extraEnv: Record<string, string> = {},
|
||||
): Promise<RunResult> {
|
||||
const env = {
|
||||
...Object.fromEntries(Object.entries(process.env).filter(([key]) => !key.startsWith("PASEO_"))),
|
||||
HOME: cliHome,
|
||||
USERPROFILE: cliHome,
|
||||
PASEO_HOME: paseoHome,
|
||||
PASEO_LOCAL_SPEECH_AUTO_DOWNLOAD: "0",
|
||||
PASEO_DICTATION_ENABLED: "0",
|
||||
PASEO_VOICE_MODE_ENABLED: "0",
|
||||
// Each daemon hands its terminals its own installation's CLI.
|
||||
PASEO_CLI: installation.shim,
|
||||
PASEO_TEST_INSTALL: installation.name,
|
||||
SHELL: "/bin/sh",
|
||||
...extraEnv,
|
||||
};
|
||||
return new Promise((resolve) => {
|
||||
execFile(installation.shim, args, { env, timeout: 120_000 }, (error, stdout, stderr) => {
|
||||
let exitCode = 0;
|
||||
if (error) exitCode = typeof error.code === "number" ? error.code : 1;
|
||||
resolve({ exitCode, stdout, stderr });
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async function paseo(
|
||||
installation: Installation,
|
||||
args: string[],
|
||||
extraEnv: Record<string, string> = {},
|
||||
): Promise<RunResult> {
|
||||
const result = await run(installation, args, extraEnv);
|
||||
assert.equal(
|
||||
result.exitCode,
|
||||
0,
|
||||
`paseo ${args.join(" ")} (${installation.name}) failed:\nstdout:\n${result.stdout}\nstderr:\n${result.stderr}`,
|
||||
);
|
||||
return result;
|
||||
}
|
||||
|
||||
async function json<T>(installation: Installation, args: string[]): Promise<T> {
|
||||
return JSON.parse((await paseo(installation, [...args, "--json"])).stdout) as T;
|
||||
}
|
||||
|
||||
async function capture(installation: Installation, terminalId: string): Promise<string> {
|
||||
const { lines } = await json<{ lines: string[] }>(installation, [
|
||||
"terminal",
|
||||
"capture",
|
||||
terminalId,
|
||||
]);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function type(installation: Installation, terminalId: string, line: string) {
|
||||
await paseo(installation, ["terminal", "send-keys", terminalId, "-l", line]);
|
||||
await paseo(installation, ["terminal", "send-keys", terminalId, "Enter"]);
|
||||
}
|
||||
|
||||
async function waitForScreen(installation: Installation, terminalId: string, text: string) {
|
||||
await waitFor(
|
||||
async () => (await capture(installation, terminalId)).includes(text),
|
||||
async () =>
|
||||
`terminal ${terminalId} never showed ${text}:\n${await capture(installation, terminalId)}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Opens a terminal, records its shell's pid, and prints a marker.
|
||||
async function openTerminal(
|
||||
installation: Installation,
|
||||
name: string,
|
||||
): Promise<{ id: string; pid: number }> {
|
||||
const terminal = await json<{ id: string }>(installation, [
|
||||
"terminal",
|
||||
"create",
|
||||
"--cwd",
|
||||
workDir,
|
||||
"--name",
|
||||
name,
|
||||
]);
|
||||
const pidFile = join(workDir, `${name}.pid`);
|
||||
await type(installation, terminal.id, `echo $$ > '${pidFile}'; echo MARKER-${name}`);
|
||||
const pid = await waitFor(
|
||||
() =>
|
||||
existsSync(pidFile) && readFileSync(pidFile, "utf8").trim()
|
||||
? Number(readFileSync(pidFile, "utf8"))
|
||||
: null,
|
||||
() => `terminal ${name} never wrote its pid`,
|
||||
);
|
||||
await waitForScreen(installation, terminal.id, `MARKER-${name}`);
|
||||
trackedPids.push(pid);
|
||||
return { id: terminal.id, pid };
|
||||
}
|
||||
|
||||
// The value of `$variable` in the terminal's shell, written to a file (a long path would wrap
|
||||
// on screen).
|
||||
async function echoVariable(
|
||||
installation: Installation,
|
||||
terminalId: string,
|
||||
label: string,
|
||||
variable: string,
|
||||
): Promise<string> {
|
||||
const valueFile = join(workDir, `${label}.value`);
|
||||
await type(
|
||||
installation,
|
||||
terminalId,
|
||||
`printf '%s' "$${variable}" > '${valueFile}.tmp' && mv '${valueFile}.tmp' '${valueFile}'`,
|
||||
);
|
||||
await waitFor(
|
||||
() => existsSync(valueFile),
|
||||
async () =>
|
||||
`terminal ${terminalId} never wrote ${label}:\n${await capture(installation, terminalId)}`,
|
||||
);
|
||||
return readFileSync(valueFile, "utf8");
|
||||
}
|
||||
|
||||
interface WorkerRecordFile {
|
||||
pid: number;
|
||||
protocolVersion: unknown;
|
||||
runtime?: string | null;
|
||||
terminals: Array<{ id: string; pid: number }>;
|
||||
}
|
||||
|
||||
function workerRecords(): WorkerRecordFile[] {
|
||||
const dir = join(paseoHome, "terminal-worker");
|
||||
if (!existsSync(dir)) return [];
|
||||
return readdirSync(dir)
|
||||
.filter((name) => /^\d+\.json$/.test(name))
|
||||
.map((name) => JSON.parse(readFileSync(join(dir, name), "utf8")) as WorkerRecordFile);
|
||||
}
|
||||
|
||||
function runtimeCopies(): string[] {
|
||||
const dir = join(paseoHome, "terminal-worker");
|
||||
if (!existsSync(dir)) return [];
|
||||
return readdirSync(dir).filter((name) =>
|
||||
existsSync(join(dir, name, "paseo-terminal-worker-runtime.json")),
|
||||
);
|
||||
}
|
||||
|
||||
function commandLineOf(pid: number): Promise<string> {
|
||||
return new Promise((resolve) => {
|
||||
execFile("ps", ["-ww", "-o", "command=", "-p", String(pid)], (_error, stdout) =>
|
||||
resolve(stdout.trim()),
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
async function terminalIds(installation: Installation): Promise<string[]> {
|
||||
return (await json<Array<{ id: string }>>(installation, ["terminal", "ls", "--all"])).map(
|
||||
(terminal) => terminal.id,
|
||||
);
|
||||
}
|
||||
|
||||
async function startDaemon(installation: Installation, extraEnv: Record<string, string> = {}) {
|
||||
await paseo(installation, ["daemon", "start"], extraEnv);
|
||||
}
|
||||
|
||||
async function stopDaemon(installation: Installation, keepTerminals = false) {
|
||||
await paseo(installation, [
|
||||
"daemon",
|
||||
"stop",
|
||||
"--home",
|
||||
paseoHome,
|
||||
...(keepTerminals ? ["--keep-terminals"] : []),
|
||||
]);
|
||||
}
|
||||
|
||||
// A terminal in a worker reporting `protocolVersion`, left running by `stop --keep-terminals`.
|
||||
async function leaveWorkerWithProtocol(
|
||||
installation: Installation,
|
||||
protocolVersion: number,
|
||||
name: string,
|
||||
): Promise<{ terminal: { id: string; pid: number }; workerPid: number; runtime: string }> {
|
||||
await startDaemon(installation, {
|
||||
PASEO_TERMINAL_WORKER_PROTOCOL_OVERRIDE: String(protocolVersion),
|
||||
});
|
||||
const terminal = await openTerminal(installation, name);
|
||||
const [worker] = workerRecords();
|
||||
assert(worker, "the daemon started a worker");
|
||||
assert.equal(worker.protocolVersion, protocolVersion, "the worker reports the override");
|
||||
await stopDaemon(installation, true);
|
||||
return { terminal, workerPid: worker.pid, runtime: worker.runtime! };
|
||||
}
|
||||
|
||||
const trackedPids: number[] = [];
|
||||
const a = createInstallation("A", "0.98.0");
|
||||
const b = createInstallation("B", "0.99.0");
|
||||
let restoreA: (() => void) | null = null;
|
||||
let passed = false;
|
||||
|
||||
try {
|
||||
for (const [field, value] of [
|
||||
["daemon.listen", `127.0.0.1:${port}`],
|
||||
["daemon.relay.enabled", "false"],
|
||||
["features.dictation.enabled", "false"],
|
||||
["features.voiceMode.enabled", "false"],
|
||||
] as const) {
|
||||
await paseo(a, ["daemon", "config", "set", field, value]);
|
||||
}
|
||||
|
||||
console.log("Test 1: a daemon from A runs its worker from a copy in the home");
|
||||
await startDaemon(a);
|
||||
const t1 = await openTerminal(a, "t1");
|
||||
const [firstWorker] = workerRecords();
|
||||
assert(firstWorker, "one worker runs");
|
||||
assert.match(firstWorker.runtime ?? "", /^0\.98\.0-[0-9a-f]{12}$/);
|
||||
const copyA = join(paseoHome, "terminal-worker", firstWorker.runtime!);
|
||||
assert(
|
||||
(await commandLineOf(firstWorker.pid)).includes(join(copyA, "terminal-worker.mjs")),
|
||||
"the worker runs the copy's bundle",
|
||||
);
|
||||
assert.equal(await echoVariable(a, t1.id, "CLI1", "PASEO_CLI"), a.shim, "T1 has A's CLI");
|
||||
console.log("✓ worker runs from", copyA, "\n");
|
||||
|
||||
console.log("Test 2: B adopts the worker after A's runtime files are gone");
|
||||
await stopDaemon(a, true);
|
||||
restoreA = moveRuntimeAway(a);
|
||||
await startDaemon(b, zsh ? { SHELL: zsh } : {});
|
||||
assert.deepEqual(
|
||||
workerRecords().map((record) => record.pid),
|
||||
[firstWorker.pid],
|
||||
"B adopted the same worker instead of starting one",
|
||||
);
|
||||
assert((await terminalIds(b)).includes(t1.id), "T1 is listed by B");
|
||||
const t2 = await openTerminal(b, "t2");
|
||||
assert.equal(await echoVariable(b, t2.id, "INSTALL2", "PASEO_TEST_INSTALL"), "B");
|
||||
assert.equal(await echoVariable(b, t2.id, "CLI2", "PASEO_CLI"), b.shim, "T2 has B's CLI");
|
||||
if (zsh) {
|
||||
// Creating a zsh terminal reads Paseo's integration from the worker's copy (A's is gone);
|
||||
// a missing file would have failed the create.
|
||||
const zshVersion = await echoVariable(b, t2.id, "ZSH", "ZSH_VERSION");
|
||||
assert(zshVersion.length > 0, "T2 runs zsh with Paseo's shell integration");
|
||||
} else {
|
||||
console.log(" (zsh is not installed: the shell-integration check is skipped)");
|
||||
}
|
||||
assert.equal(await echoVariable(b, t1.id, "CLI1AGAIN", "PASEO_CLI"), a.shim, "T1 keeps A's CLI");
|
||||
console.log("✓ same worker, T2 uses B's environment, T1 still works\n");
|
||||
|
||||
console.log("Test 3: daemon stop closes both terminals and deletes the copy");
|
||||
await stopDaemon(b);
|
||||
await waitFor(
|
||||
() => !isAlive(t1.pid) && !isAlive(t2.pid) && !isAlive(firstWorker.pid),
|
||||
() => "terminals or worker outlived daemon stop",
|
||||
);
|
||||
assert.deepEqual(workerRecords(), [], "no worker record is left");
|
||||
assert.deepEqual(runtimeCopies(), [], "no copy is left");
|
||||
restoreA();
|
||||
restoreA = null;
|
||||
console.log("✓ closed and cleaned up\n");
|
||||
|
||||
console.log("Test 4: a worker one protocol version old drains");
|
||||
const old = await leaveWorkerWithProtocol(a, 0, "t3");
|
||||
await startDaemon(b);
|
||||
const workers = workerRecords();
|
||||
assert.equal(workers.length, 2, "the old worker drains beside a new one");
|
||||
const current = workers.find((record) => record.pid !== old.workerPid)!;
|
||||
assert.equal(current.protocolVersion, 1);
|
||||
assert.match(current.runtime ?? "", /^0\.99\.0-/);
|
||||
assert((await terminalIds(b)).includes(old.terminal.id), "the old worker's terminal is listed");
|
||||
await type(b, old.terminal.id, "echo still-served");
|
||||
await waitForScreen(b, old.terminal.id, "\nstill-served");
|
||||
const t4 = await openTerminal(b, "t4");
|
||||
assert.deepEqual(
|
||||
workerRecords()
|
||||
.find((record) => record.pid === current.pid)!
|
||||
.terminals.map((terminal) => terminal.id),
|
||||
[t4.id],
|
||||
"new terminals open in the current worker",
|
||||
);
|
||||
await paseo(b, ["terminal", "kill", old.terminal.id]);
|
||||
await waitFor(
|
||||
() => !isAlive(old.workerPid),
|
||||
() => `the drained worker ${old.workerPid} did not exit`,
|
||||
);
|
||||
await waitFor(
|
||||
() => !runtimeCopies().includes(old.runtime),
|
||||
() => `the drained worker's copy ${old.runtime} was not deleted: ${runtimeCopies().join(", ")}`,
|
||||
);
|
||||
assert.deepEqual(runtimeCopies(), [current.runtime], "the current worker's copy stays");
|
||||
await type(b, t4.id, "echo after-drain");
|
||||
await waitForScreen(b, t4.id, "\nafter-drain");
|
||||
await stopDaemon(b);
|
||||
console.log("✓ drained, exited, copy deleted\n");
|
||||
|
||||
console.log("Test 5: a worker two protocol versions old is shut down");
|
||||
const obsolete = await leaveWorkerWithProtocol(a, -1, "t5");
|
||||
await startDaemon(b);
|
||||
await waitFor(
|
||||
() => !isAlive(obsolete.workerPid) && !isAlive(obsolete.terminal.pid),
|
||||
() => "the obsolete worker or its terminal survived",
|
||||
);
|
||||
assert(!(await terminalIds(b)).includes(obsolete.terminal.id), "its terminal is gone");
|
||||
assert.equal(workerRecords().length, 1, "one current worker runs");
|
||||
await stopDaemon(b);
|
||||
console.log("✓ shut down\n");
|
||||
|
||||
console.log("Test 6: a worker from a newer Paseo is left running and reported");
|
||||
const newer = await leaveWorkerWithProtocol(a, 2, "t6");
|
||||
await startDaemon(b);
|
||||
const listed = await run(b, ["terminal", "ls", "--all", "--json"]);
|
||||
assert.notEqual(listed.exitCode, 0, "terminal ls fails");
|
||||
assert.match(listed.stdout + listed.stderr, /held by a newer Paseo \(v0\.98\.0/);
|
||||
const created = await run(b, ["terminal", "create", "--cwd", workDir, "--json"]);
|
||||
assert.notEqual(created.exitCode, 0, "terminal create fails");
|
||||
assert.match(created.stdout + created.stderr, /held by a newer Paseo/);
|
||||
const status = await paseo(b, ["daemon", "status", "--home", paseoHome]);
|
||||
assert.match(status.stdout, /terminals: Terminals are held by a newer Paseo/);
|
||||
assert(isAlive(newer.terminal.pid), "its program keeps running");
|
||||
assert.deepEqual(
|
||||
workerRecords().map((record) => record.pid),
|
||||
[newer.workerPid],
|
||||
"no second worker starts",
|
||||
);
|
||||
await stopDaemon(b);
|
||||
await waitFor(
|
||||
() => !isAlive(newer.workerPid) && !isAlive(newer.terminal.pid),
|
||||
() => "daemon stop left the newer worker or its terminal running",
|
||||
);
|
||||
assert.deepEqual(workerRecords(), [], "no worker record is left");
|
||||
assert.deepEqual(runtimeCopies(), [], "no copy is left");
|
||||
console.log("✓ reported, untouched, closed by stop\n");
|
||||
passed = true;
|
||||
} finally {
|
||||
if (!passed) {
|
||||
for (const log of ["daemon.log", "terminal-worker.log"]) {
|
||||
const logPath = join(paseoHome, log);
|
||||
if (existsSync(logPath)) {
|
||||
console.error(`--- ${log} (last 60 lines) ---`);
|
||||
console.error(readFileSync(logPath, "utf8").split("\n").slice(-60).join("\n"));
|
||||
}
|
||||
}
|
||||
}
|
||||
restoreA?.();
|
||||
for (const installation of [b, a]) {
|
||||
await run(installation, ["daemon", "stop", "--home", paseoHome, "--force"]);
|
||||
}
|
||||
for (const pid of trackedPids) {
|
||||
if (isAlive(pid)) process.kill(pid, "SIGKILL");
|
||||
}
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
console.log("=== Terminal worker survives an upgrade passed ===");
|
||||
Reference in New Issue
Block a user