Compare commits

...
Author SHA1 Message Date
Thomas VanandClaude Opus 5.5 7f318bf7cf fix(codex): end an interrupted turn's shell commands (#7525)
`turn/interrupt` ends a Codex turn and leaves the shell commands it started running: a
command the user just stopped runs on to its own end, alongside the next turn.

Once Codex accepts the interrupt, the provider now ends each command the interrupted turn
started that has not exited, by process id, with `thread/backgroundTerminals/terminate`,
and a turn Paseo starts waits for those calls before its `turn/start`.

- Attribution: `thread/backgroundTerminals/list` carries no turn, so the provider records
  each root-thread `commandExecution` process id under the turn of its `item/started`. A
  `completed` item drops its id; a `failed` one keeps it, because Codex reports an
  interrupted turn's still-running command as failed. A turn that ends any way but
  `interrupted` forgets its ids, so what an earlier turn left running on purpose is never
  ended. At most 16 turns are remembered.
- A late command: the turn keeps its accepted interrupt, so a command whose `item/started`
  arrives after Codex accepted the interrupt (one queued behind an earlier notification's
  conversion, for instance) is ended as soon as it is seen. Each process is terminated
  once, and a turn start waits for every pending sweep.
- It never sends `thread/backgroundTerminals/clean`, which ends every terminal on the
  thread. It ends nothing after an already-idle interrupt, nothing on a subagent's thread,
  and nothing after an interrupt of a turn Codex never identified.
- Bounds: each terminate times out after 2 s; a failure is logged
  (`provider.codex.interrupt.terminal_terminate_failed`) and fails nothing. `startTurn`
  waits at most the interrupt's bound plus one terminate's plus 500 ms.

Tests (codex-interrupted-turn-terminals.test.ts) drive a real CodexAppServerAgentSession
against the scripted app-server in codex/test-utils/fake-app-server.ts: exactly the
interrupted turn's processes are terminated, by id, and `clean` is never sent; a turn
Codex starts before acknowledging the interrupt keeps its commands; the next `turn/start`
waits for the terminates; a command announced after the interrupt was accepted is
terminated once and the next turn waits for both sweeps, whichever answers first; a
completed command is forgotten and a failed one kept; only the latest sixteen accepted
interrupts are kept; a failing or unanswered terminate does not block the next turn. The
harness waits for the interrupted turn's `turn_canceled` before it starts the next turn,
as a caller does: notifications are handled asynchronously, so the interrupt's answer can
arrive before the turn has ended.

Fixes #7522


Claude-Session: https://claude.ai/code/session_01Ay568T2kmtotoh23Dtzi5K

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 19:58:12 +00:00
paseo-bot[bot] 04fe4be40f fix(claude): retain configured model metadata before a turn (#8465)
Rebuilding an idle session cleared its observed model and published null despite retaining an explicit model in its configuration. Fall back to that configuration until the next turn supplies an observed model.

Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 18:59:04 +00:00
paseo-bot[bot] 2d75cc80b2 test(server): cover routed permissions in one host journey (#8486)
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 18:34:57 +00:00
paseo-bot[bot]andDominic Anaeto bfac132920 Stop Git polling while workspace folders are missing (#8462)
* fix(server): stop polling git for a workspace whose folder is gone

When a workspace's folder disappears together with its project (the
repository deleted, a volume unmounted), reconciliation leaves the
workspace registered, and the daemon kept polling it: a failed git
command and a warning about once a minute, and a working-tree watcher
retry every minute or two, for as long as the daemon ran.

The degraded poll now checks that the workspace folder exists before
refreshing. A workspace whose folder is gone is suspended once: it is
detached from its repository and working-tree targets, its timers stop,
and one warning is logged. A worktree that shared a repository target
hands the repository's polling and background fetch to a surviving
worktree. Nothing is archived. A working-tree watch that an open diff
still holds skips its consumer notifications and watcher recovery while
its folder is missing.

While suspended, the workspace runs no git. A stat every 60 seconds, a
new subscription, a forced snapshot or a refresh request checks whether
the folder is back; when it is, the workspace reloads its git facts,
refreshes, and rebuilds its observation.

Fixes #7856

* fix(server): keep missing workspaces suspended during refresh

Admit Git refreshes only after the folder has returned. Cover missing-folder polling, explicit refreshes, and restoration with real Git and the Linux observer.

Co-authored-by: Dominic Anaeto <dominic.anaeto@gmail.com>

* fix(server): preserve workspace snapshots through suspension

Drop queued refreshes and withhold publication when a folder disappears during an active read. Wait for that read before restoring observation so its stale result cannot overwrite the recovered snapshot.

---------

Co-authored-by: Dominic Anaeto <dominic.anaeto@gmail.com>
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 18:02:26 +00:00
paseo-bot[bot]andDominic Anaeto 34f996b32a Preserve pending questions and approvals after peer messages (#8453)
* fix(server): keep a recipient's pending permission when another agent messages it

A message from another agent (send_agent_message with sourceAgentId) was
enqueued with clearPendingPermissions: true, so it superseded the
recipient's pending permission request and the request was denied before
a human saw it. Agent-sourced messages now leave pending permissions
alone. Typed messages and sourceless `paseo send` still clear them.

Fixes #7602

* test(server): verify peer messages preserve answerable questions

Exercise message delivery through the daemon and provider interfaces so the regression checks that the question survives and its answer reaches the provider. Retain the human-steer cancellation behavior.

Co-authored-by: Dominic Anaeto <dominic.anaeto@gmail.com>

* test(terminal): make runtime copy fixture independent of umask

Set the source helper permissions after creation so the copy test measures preservation instead of inheriting the shell file-creation mask. The existing test fails under umask 077 without this setup correction.

Fixes #8175

---------

Co-authored-by: Dominic Anaeto <dominic.anaeto@gmail.com>
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 17:11:42 +00:00
3e83f433da Import large Claude sessions without exceeding the string limit (#8457)
* fix(claude): read the session rename from the end of a large transcript

Importing a Claude session read the whole transcript into one UTF-8 string to
look for its `custom-title` record, so a transcript past V8's maximum string
length failed the import with `RangeError: Invalid string length` before the
incremental history reader ever ran. The scan now reads the file backwards a
chunk at a time, which keeps the original intent of not walking the transcript
and never holds it as one string. Records without the needle are not decoded, so
a transcript with no rename costs a byte search per chunk.

Closes #8451

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(app): wait for send readiness in composer recall

The previous prompt can appear and clear its draft before the send acknowledgement arrives. Wait for Send to be enabled before pressing Enter so recall setup does not silently leave the next prompt unsent.

Co-authored-by: qwist1233-cpu <qwist1233@gmail.com>

* fix(claude): scan large transcript records once

Retain unfinished record chunks and assemble each record once instead of copying and rescanning a growing buffer on every backward read. Cover import of a large record after the last explicit rename.

---------

Co-authored-by: qwist1233-cpu <qwist1233@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 15:56:42 +00:00
paseo-bot[bot] 674530152e test(server): cover filename downloads in one workspace journey (#8475)
Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com>
2026-11-01 15:31:08 +00:00
15 changed files with 1818 additions and 139 deletions
@@ -3,7 +3,7 @@ import {
composerLocator,
expectComposerDraft,
expectComposerVisible,
submitMessage,
fillComposerDraft,
typeIntoFocusedComposer,
} from "../support/helpers/composer";
import { openAgentRoute, seedMockAgentWorkspace } from "../support/helpers/mock-agent";
@@ -12,8 +12,14 @@ function userMessages(page: Page) {
return page.getByTestId("user-message");
}
async function sendDraftWhenReady(page: Page) {
await expect(page.getByRole("button", { name: "Send message", exact: true })).toBeEnabled();
await pressInComposer(page, "Enter");
}
async function sendAndWaitForMessage(page: Page, text: string, expectedCount: number) {
await submitMessage(page, text);
await fillComposerDraft(page, text);
await sendDraftWhenReady(page);
await expect(userMessages(page)).toHaveCount(expectedCount, { timeout: 30_000 });
await expectComposerDraft(page, "");
}
@@ -49,7 +55,7 @@ test.describe("Composer recall", () => {
await pressInComposer(page, "ArrowUp");
await expectComposerDraft(page, "add tests for login");
await pressInComposer(page, "Enter");
await sendDraftWhenReady(page);
await expect(userMessages(page)).toHaveCount(3, { timeout: 30_000 });
await expect(userMessages(page).last()).toContainText("add tests for login");
} finally {
@@ -1,3 +1,4 @@
import { constants as bufferConstants } from "node:buffer";
import * as fs from "node:fs/promises";
import * as os from "node:os";
import * as path from "node:path";
@@ -236,4 +237,64 @@ describe("importing a renamed Claude session", () => {
await expect(importSession(sessionId)).rejects.toThrow();
});
test("reads the rename from a transcript too large to hold as one string", async () => {
const sessionId = "oversized-transcript";
const projectDir = claudeProjectDirSync(SESSION_CWD, { configDir });
await fs.mkdir(projectDir, { recursive: true });
const transcript = path.join(projectDir, `${sessionId}.jsonl`);
// A hole stands in for the long session this used to fail on: the file only
// has to be longer than a string can be, and the rename sits at its end.
await fs.writeFile(transcript, "");
await fs.truncate(transcript, bufferConstants.MAX_STRING_LENGTH + 1);
await fs.appendFile(
transcript,
`\n${[
userRecord({ sessionId, content: "Trace the slow query" }),
{ type: "custom-title", customTitle: "Slow query hunt", sessionId },
]
.map((record) => JSON.stringify(record))
.join("\n")}\n`,
);
const imported = await importSession(sessionId);
expect(imported.config.title).toBe("Slow query hunt");
});
test("reads the rename from behind more records than one read covers", async () => {
const sessionId = "late-records-rename";
await writeSession({
sessionId,
records: [
userRecord({ sessionId, content: "Review this project" }),
{ type: "custom-title", customTitle: "Scrollback review", sessionId },
// One record longer than a single read, then enough records to leave the
// rename several reads back from the end of the file.
userRecord({ sessionId, content: "x".repeat(128 * 1024) }),
...Array.from({ length: 64 }, (_, index) =>
userRecord({ sessionId, content: `${"y".repeat(4 * 1024)} ${index}` }),
),
],
});
const imported = await importSession(sessionId);
expect(imported.config.title).toBe("Scrollback review");
});
test("imports a large record after the last rename without repeatedly copying it", async () => {
const sessionId = "large-record-after-rename";
await writeSession({
sessionId,
records: [
{ type: "custom-title", customTitle: "Large attachment review", sessionId },
userRecord({ sessionId, content: "x".repeat(96 * 1024 * 1024) }),
],
});
const imported = await importSession(sessionId);
expect(imported.config.title).toBe("Large attachment review");
});
});
@@ -0,0 +1,48 @@
import pino from "pino";
import { describe, expect, test } from "vitest";
import { ClaudeAgentClient } from "./agent.js";
describe("Claude runtime info before a turn", () => {
const client = new ClaudeAgentClient({ logger: pino({ level: "silent" }) });
test("reports the explicitly configured model before the first turn", async () => {
const session = await client.createSession({
provider: "claude",
cwd: process.cwd(),
model: "claude-haiku-5-5",
});
try {
expect(await session.getRuntimeInfo()).toMatchObject({ model: "claude-haiku-5-5" });
} finally {
await session.close();
}
});
test("reports the persisted configured model after rebuilding an idle session", async () => {
const session = await client.resumeSession({
provider: "claude",
sessionId: "runtime-model-before-turn",
metadata: { cwd: process.cwd(), model: "claude-haiku-5-5" },
});
try {
expect(await session.getRuntimeInfo()).toMatchObject({
sessionId: "runtime-model-before-turn",
model: "claude-haiku-5-5",
});
expect((await session.getRuntimeInfo()).model).toBe("claude-haiku-5-5");
expect((await session.getRuntimeInfo()).extra?.runtimeModel).toBeUndefined();
} finally {
await session.close();
}
});
test("leaves the model unknown when Claude will choose its default", async () => {
const session = await client.createSession({ provider: "claude", cwd: process.cwd() });
try {
expect((await session.getRuntimeInfo()).model).toBeNull();
} finally {
await session.close();
}
});
});
@@ -2460,7 +2460,7 @@ class ClaudeAgentSession implements AgentSession {
const info: AgentRuntimeInfo = {
provider: "claude",
sessionId: this.claudeSessionId,
model: this.lastOptionsModel,
model: this.lastOptionsModel ?? this.config.model ?? null,
modeId: this.currentMode ?? null,
...(this.lastRuntimeModel
? {
@@ -6853,40 +6853,79 @@ function normalizeClaudeSessionTitle(title: string | null): string | null {
}
const CLAUDE_CUSTOM_TITLE_TYPE = "custom-title";
const CLAUDE_CUSTOM_TITLE_NEEDLE = Buffer.from(`"${CLAUDE_CUSTOM_TITLE_TYPE}"`);
const CLAUDE_TITLE_SCAN_CHUNK_BYTES = 64 * 1024;
function readClaudeCustomTitleFromRecord(record: Buffer): string | null {
// The needle also matches a prompt that quotes it, so a record still has to
// parse as a `custom-title` entry carrying a usable name. Records without the
// needle are never decoded, so a long one costs a byte search and nothing more.
if (!record.includes(CLAUDE_CUSTOM_TITLE_NEEDLE)) {
return null;
}
let parsed: unknown;
try {
parsed = JSON.parse(record.toString("utf8").trim());
} catch {
return null;
}
const entry = toObjectRecord(parsed);
if (!entry || entry.isSidechain || entry.type !== CLAUDE_CUSTOM_TITLE_TYPE) {
return null;
}
return normalizeClaudeSessionTitle(
typeof entry.customTitle === "string" ? entry.customTitle : null,
);
}
/**
* The `custom-title` record is re-emitted every time the session is saved, so
* the current name is the last usable one in the file. Searching backwards finds it without
* walking the transcript, which matters because sessions reach tens of
* megabytes.
* the current name is the last usable one in the file. Reading backwards a chunk
* at a time finds it without walking the transcript, which matters because
* sessions reach tens of megabytes, and never holds the file as one string, so a
* transcript past V8's maximum string length still imports.
*/
function readLastClaudeCustomTitle(content: string): string | null {
const needle = `"${CLAUDE_CUSTOM_TITLE_TYPE}"`;
let at = content.lastIndexOf(needle);
while (at >= 0) {
const lineStart = content.lastIndexOf("\n", at) + 1;
const lineEnd = content.indexOf("\n", at);
const line = lineEnd === -1 ? content.slice(lineStart) : content.slice(lineStart, lineEnd);
let parsed: unknown;
try {
parsed = JSON.parse(line.trim());
} catch {
parsed = undefined;
}
const entry = toObjectRecord(parsed);
if (entry && !entry.isSidechain && entry.type === CLAUDE_CUSTOM_TITLE_TYPE) {
const title = normalizeClaudeSessionTitle(
typeof entry.customTitle === "string" ? entry.customTitle : null,
async function readLastClaudeCustomTitle(filePath: string): Promise<string | null> {
const handle = await fsPromises.open(filePath, "r");
try {
const stats = await handle.stat();
if (!stats.isFile()) {
// Opening a directory succeeds on Windows even though reading it does not,
// so reject the way reading the transcript as a file would.
throw Object.assign(
new Error(`EISDIR: illegal operation on a directory, read '${filePath}'`),
{ code: "EISDIR", path: filePath },
);
if (title) {
return title;
}
}
// The needle also matches a prompt that quotes it, and a record can carry no
// usable name, so keep looking backwards for one that does.
at = at <= 0 ? -1 : content.lastIndexOf(needle, at - 1);
let end = stats.size;
// Retain unfinished record pieces in reverse file order. Assemble each
// complete record once, so a large tool/image record costs linear work.
let pieces: Buffer[] = [];
while (end > 0) {
const length = Math.min(CLAUDE_TITLE_SCAN_CHUNK_BYTES, end);
const chunk = Buffer.alloc(length);
await handle.read(chunk, 0, length, end - length);
let recordEnd = chunk.length;
for (
let at = chunk.lastIndexOf(10);
at >= 0;
at = at === 0 ? -1 : chunk.lastIndexOf(10, at - 1)
) {
pieces.push(chunk.subarray(at + 1, recordEnd));
const record = Buffer.concat(pieces.toReversed());
const title = readClaudeCustomTitleFromRecord(record);
if (title) return title;
pieces = [];
recordEnd = at;
}
if (recordEnd > 0) pieces.push(chunk.subarray(0, recordEnd));
end -= length;
}
// The first record in the file has no newline in front of it.
return readClaudeCustomTitleFromRecord(Buffer.concat(pieces.toReversed()));
} finally {
await handle.close();
}
return null;
}
function isErrnoException(error: unknown): error is NodeJS.ErrnoException {
@@ -6898,9 +6937,8 @@ async function readClaudeRenamedSessionTitle(input: {
sessionId: string;
configDir: string;
}): Promise<string | null> {
let content: string;
try {
content = await fsPromises.readFile(await claudeTranscriptPath(input), "utf8");
return await readLastClaudeCustomTitle(await claudeTranscriptPath(input));
} catch (error) {
// A session with no transcript on disk has no rename to carry over. Every
// other read failure is a real fault and stays explicit.
@@ -6909,7 +6947,6 @@ async function readClaudeRenamedSessionTitle(input: {
}
throw error;
}
return readLastClaudeCustomTitle(content);
}
function normalizeImportablePromptPreview(text: string): string | null {
@@ -150,6 +150,15 @@ function isCodexAlreadyUnarchivedError(error: unknown, threadId: string): boolea
const TURN_START_TIMEOUT_MS = 90 * 1000;
const INTERRUPT_TIMEOUT_MS = 2_000;
// `turn/interrupt` ends a turn but leaves the shell commands it started running.
// Each terminate is bounded by the first constant. A turn start waits for the
// interrupted turn's commands to end, at most the interrupt's own bound plus one
// terminate's and a margin, so it gives up only when a request outlives its timeout.
const INTERRUPTED_TURN_TERMINATE_TIMEOUT_MS = 2_000;
const INTERRUPTED_TURN_CLEANUP_WAIT_MS =
INTERRUPT_TIMEOUT_MS + INTERRUPTED_TURN_TERMINATE_TIMEOUT_MS + 500;
// Turns whose commands are remembered; older turns are forgotten first.
const INTERRUPTED_TURN_TRACKED_TURNS = 16;
const CODEX_PROVIDER = "codex" as const;
// These native provider identities belong to Codex, not to an older custom
// Paseo provider. Never include them in a custom provider's legacy import scan.
@@ -3476,6 +3485,14 @@ export class CodexAppServerAgentSession implements AgentSession {
private pendingFileChangeOutputDeltas = new Map<string, string[]>();
private pendingAssistantMessageBoundary = false;
private terminalCommandByProcessId = new Map<string, string>();
// The root thread's still-running command process ids, keyed by the native turn that
// started them. Codex's `thread/backgroundTerminals/list` carries no turn, so
// `item/started` is the only record of which turn owns a process.
private terminalProcessIdsByTurnId = new Map<
string,
{ processIds: Set<string>; terminatedProcessIds: Set<string>; interruptAccepted: boolean }
>();
private interruptedTurnCleanup: Promise<void> | null = null;
private pendingUnlabeledTerminalInteractions = new Map<
string,
Array<{ callId: string; stdin: string | null }>
@@ -4431,6 +4448,9 @@ export class CodexAppServerAgentSession implements AgentSession {
hasDeveloperInstructions: turnStart.hasDeveloperInstructions,
hasCodexConfig: turnStart.hasCodexConfig,
});
// The interrupted turn's commands end before this turn begins, so none of them
// outlives it into the new turn.
await this.waitForInterruptedTurnCleanup();
if (pendingStart.cancelRequested) {
throw new Error("Codex turn start was interrupted before reaching Codex");
}
@@ -5036,15 +5056,27 @@ export class CodexAppServerAgentSession implements AgentSession {
if (!turnId || (foregroundTurnId && this.activeForegroundTurnId !== foregroundTurnId)) {
throw new Error("Cannot interrupt Codex before turn/started identifies the active turn");
}
const client = this.client;
const threadId = this.currentThreadId;
const interruptRequest = client.request(
"turn/interrupt",
{
threadId,
turnId,
},
INTERRUPT_TIMEOUT_MS,
);
// Once Codex accepts the interrupt, end the commands this turn started. Registered
// before the await, so a turn start racing this interrupt waits for it. An already-idle
// turn was not interrupted, and what it left running stays.
this.trackInterruptedTurnCleanup(
interruptRequest.then(
() => this.endInterruptedTurnProcesses(client, threadId, turnId),
() => undefined,
),
);
try {
await this.client.request(
"turn/interrupt",
{
threadId: this.currentThreadId,
turnId,
},
INTERRUPT_TIMEOUT_MS,
);
await interruptRequest;
} catch (error) {
if (!isCodexAlreadyIdleInterrupt(error)) {
throw error;
@@ -5085,6 +5117,137 @@ export class CodexAppServerAgentSession implements AgentSession {
return !this.client.hasPendingRequests();
}
// Remember which turn started a command, so an interrupt ends that turn's commands and
// no other turn's. Only the root thread's: a subagent's commands live on its own thread.
// A command that starts after its turn's interrupt was accepted is ended as soon as it
// is seen.
private recordTurnTerminalProcess(
parsed: Extract<ParsedCodexNotification, { kind: "item_started" | "item_completed" }>,
): void {
const processId = parsed.item.processId;
if (
parsed.source !== "item" ||
!parsed.turnId ||
!parsed.threadId ||
parsed.threadId !== this.currentThreadId ||
typeof processId !== "string" ||
!processId ||
normalizeCodexThreadItemType(
typeof parsed.item.type === "string" ? parsed.item.type : undefined,
) !== "commandExecution"
) {
return;
}
const record = this.terminalProcessIdsByTurnId.get(parsed.turnId);
if (parsed.kind === "item_completed") {
// Only a completed command has exited. A failed one can still be running:
// Codex reports an interrupted turn's live command as failed.
if (parsed.item.status === "completed") record?.processIds.delete(processId);
return;
}
const current = record ?? this.getTurnTerminalRecord(parsed.turnId);
current.processIds.add(processId);
if (current.interruptAccepted && this.client) {
this.trackInterruptedTurnCleanup(
this.endInterruptedTurnProcesses(this.client, parsed.threadId, parsed.turnId),
);
}
}
// An accepted interrupt needs a record even before any of its turn's commands is seen:
// an `item/started` can still be queued behind an earlier notification's conversion.
private getTurnTerminalRecord(turnId: string) {
const existing = this.terminalProcessIdsByTurnId.get(turnId);
if (existing) return existing;
const record = {
processIds: new Set<string>(),
terminatedProcessIds: new Set<string>(),
interruptAccepted: false,
};
this.terminalProcessIdsByTurnId.set(turnId, record);
for (const oldestTurnId of this.terminalProcessIdsByTurnId.keys()) {
if (this.terminalProcessIdsByTurnId.size <= INTERRUPTED_TURN_TRACKED_TURNS) break;
this.terminalProcessIdsByTurnId.delete(oldestTurnId);
}
return record;
}
// A later sweep joins the pending one, so a turn start waits for both.
private trackInterruptedTurnCleanup(cleanup: Promise<void>): void {
const tracked = Promise.all([this.interruptedTurnCleanup, cleanup])
.then(() => undefined)
.finally(() => {
if (this.interruptedTurnCleanup === tracked) this.interruptedTurnCleanup = null;
});
this.interruptedTurnCleanup = tracked;
}
// Terminate each process the interrupted turn started, by its own id, once each.
// `thread/backgroundTerminals/clean` is never used: it ends every terminal on the thread,
// an earlier turn's included. A process already gone answers `terminated: false`. Never
// rejects; failures are logged.
private async endInterruptedTurnProcesses(
client: CodexAppServerClient,
threadId: string | null,
turnId: string,
): Promise<void> {
if (!threadId) return;
const record = this.getTurnTerminalRecord(turnId);
record.interruptAccepted = true;
const processIds = [...record.processIds].filter(
(processId) => !record.terminatedProcessIds.has(processId),
);
for (const processId of processIds) record.terminatedProcessIds.add(processId);
const logContext = { agentId: this.agentId, provider: CODEX_PROVIDER, threadId, turnId };
await Promise.all(
processIds.map(async (processId) => {
try {
const response = await client.request(
"thread/backgroundTerminals/terminate",
{ threadId, processId },
INTERRUPTED_TURN_TERMINATE_TIMEOUT_MS,
);
this.logger.info(
{ ...logContext, processId, terminated: toObjectRecord(response)?.terminated ?? null },
"provider.codex.interrupt.terminal_terminated",
);
} catch (error) {
this.logger.warn(
{ ...logContext, processId, err: error },
"provider.codex.interrupt.terminal_terminate_failed",
);
}
}),
);
}
// A new turn starts only after every pending sweep of interrupted turns' commands has
// ended, or after the bound, whichever is first.
private async waitForInterruptedTurnCleanup(): Promise<void> {
if (!this.interruptedTurnCleanup) return;
let timer: ReturnType<typeof setTimeout> | undefined;
const timedOut = await Promise.race([
(async () => {
while (this.interruptedTurnCleanup) await this.interruptedTurnCleanup;
return false;
})(),
new Promise<boolean>((resolve) => {
timer = setTimeout(() => resolve(true), INTERRUPTED_TURN_CLEANUP_WAIT_MS);
}),
]);
clearTimeout(timer);
if (timedOut) {
this.logger.warn(
{
agentId: this.agentId,
provider: CODEX_PROVIDER,
waitMs: INTERRUPTED_TURN_CLEANUP_WAIT_MS,
},
"provider.codex.interrupt.terminal_cleanup_timed_out",
);
}
}
async close(): Promise<void> {
this.closed = true;
this.clearPendingPermissions();
@@ -6209,6 +6372,11 @@ export class CodexAppServerAgentSession implements AgentSession {
return;
}
this.completePendingRootCompactions();
// What a turn that was not interrupted leaves running, it left on purpose; no later
// interrupt may end it.
if (parsed.status !== "interrupted" && this.currentTurnId) {
this.terminalProcessIdsByTurnId.delete(this.currentTurnId);
}
if (parsed.status === "failed") {
this.emitEvent({
type: "turn_failed",
@@ -6613,6 +6781,7 @@ export class CodexAppServerAgentSession implements AgentSession {
private async handleItemCompletedNotification(
parsed: Extract<ParsedCodexNotification, { kind: "item_completed" }>,
): Promise<void> {
this.recordTurnTerminalProcess(parsed);
// Codex emits mirrored lifecycle notifications via both `codex/event/item_*`
// and canonical `item/*`. Render ordinary items only from the canonical
// channel, but accept a legacy-only child announcement so it can establish
@@ -6775,6 +6944,7 @@ export class CodexAppServerAgentSession implements AgentSession {
private async handleItemStartedNotification(
parsed: Extract<ParsedCodexNotification, { kind: "item_started" }>,
): Promise<void> {
this.recordTurnTerminalProcess(parsed);
if (shouldIgnoreMirroredLifecycleItem(parsed.source, parsed.item)) {
return;
}
@@ -0,0 +1,374 @@
import { afterEach, describe, expect, test, vi } from "vitest";
import type { AgentSession } from "../agent-sdk-types.js";
import { createTestLogger } from "../../../test-utils/test-logger.js";
import { CodexAppServerAgentSession } from "./codex-app-server-agent.js";
import {
createFakeCodexAppServer,
type FakeCodexAppServer,
waitForNextEvent,
waitForTimelineToolCall,
} from "./codex/test-utils/fake-app-server.js";
const THREAD = "thread-1";
const TERMINATE = "thread/backgroundTerminals/terminate";
type Handler = (params: unknown) => unknown;
function deferred<T = void>() {
let resolve!: (value: T) => void;
const promise = new Promise<T>((resolvePromise) => {
resolve = resolvePromise;
});
return { promise, resolve };
}
function paramsOf(value: unknown): Record<string, unknown> {
return value && typeof value === "object" ? (value as Record<string, unknown>) : {};
}
let sessions: AgentSession[] = [];
afterEach(async () => {
for (const session of sessions) await session.close().catch(() => undefined);
sessions = [];
});
/**
* A Codex app-server whose interrupt ends the turn the way Codex does: an `interrupted`
* turn completion, then the response. `terminate` answers per process.
*/
function createHarness(
options: {
terminate?: (processId: string) => unknown;
interruptResponse?: (completeTurn: () => void) => unknown;
} = {},
) {
const handlers: Record<string, Handler> = {
"turn/interrupt": () => {
const completeTurn = () =>
appServer.completeTurn({ threadId: THREAD, status: "interrupted" });
if (options.interruptResponse) return options.interruptResponse(completeTurn);
completeTurn();
return {};
},
[TERMINATE]: (params) =>
options.terminate?.(String(paramsOf(params).processId)) ?? { terminated: true },
};
const appServer: FakeCodexAppServer = createFakeCodexAppServer(handlers);
const session = new CodexAppServerAgentSession(
{
provider: "codex",
cwd: "/workspace/project",
modeId: "auto",
model: "gpt-5.4",
},
null,
createTestLogger(),
async () => appServer.child,
);
sessions.push(session);
function requests(method: string): Record<string, unknown>[] {
return appServer
.requests()
.filter((message) => message.method === method && "id" in message)
.map((message) => paramsOf(message.params));
}
function terminated(): string[] {
return requests(TERMINATE)
.map((params) => String(params.processId))
.sort();
}
// Starts a turn and lets Codex identify it as `nativeTurnId`.
async function startTurn(prompt: string, nativeTurnId: string): Promise<void> {
const before = requests("turn/start").length;
const started = session.startTurn(prompt);
await vi.waitFor(() => expect(requests("turn/start")).toHaveLength(before + 1), {
timeout: 5_000,
});
const turnStarted = waitForNextEvent(session, "turn_started");
appServer.startsTurn({ threadId: THREAD, turnId: nativeTurnId });
await started;
await turnStarted;
}
async function runsCommand(params: { turnId: string; processId: string }): Promise<void> {
const itemId = `exec-${params.processId}`;
const shown = waitForTimelineToolCall(session, itemId);
appServer.startsTerminalCommand({
threadId: THREAD,
turnId: params.turnId,
itemId,
processId: params.processId,
command: `sleep ${params.processId}`,
});
await shown;
}
async function completesTurn(): Promise<void> {
const completed = waitForNextEvent(session, "turn_completed");
appServer.completeTurn({ threadId: THREAD });
await completed;
}
// Interrupts the running turn and waits for the session to report it canceled, as a
// caller does before it starts the next turn: Codex notifications are handled in order
// but asynchronously, so the interrupt's answer can arrive before the turn has ended.
async function interrupts(): Promise<void> {
const canceled = waitForNextEvent(session, "turn_canceled");
await session.interrupt();
await canceled;
}
return {
appServer,
session,
requests,
terminated,
startTurn,
runsCommand,
completesTurn,
interrupts,
};
}
// Turn A leaves a server running on purpose and completes. Turn B starts two
// commands, and a subagent on its own thread starts a third, then B is interrupted.
async function interruptTurnB(harness: ReturnType<typeof createHarness>): Promise<void> {
await harness.startTurn("start a server", "native-A");
await harness.runsCommand({ turnId: "native-A", processId: "101" });
await harness.completesTurn();
await harness.startTurn("run two commands", "native-B");
await harness.runsCommand({ turnId: "native-B", processId: "201" });
// A subagent thread's command shows no row here; the next command's row shows it
// was read, because notifications are read in order.
harness.appServer.startsTerminalCommand({
threadId: "child-thread",
turnId: "native-B",
itemId: "exec-209",
processId: "209",
command: "sleep 209",
});
await harness.runsCommand({ turnId: "native-B", processId: "202" });
await harness.interrupts();
}
describe("an interrupted Codex turn's commands end before the next turn", () => {
test("terminates exactly the processes the interrupted turn started, by id, and never cleans", async () => {
const harness = createHarness();
await interruptTurnB(harness);
await harness.startTurn("correction", "native-C");
expect(harness.terminated()).toEqual(["201", "202"]);
expect(harness.requests(TERMINATE).every((params) => params.threadId === THREAD)).toBe(true);
// `clean` would end every terminal on the thread, an earlier turn's included.
expect(
harness.appServer
.requests()
.filter((message) => String(message.method).startsWith("thread/backgroundTerminals/"))
.map((message) => message.method),
).toEqual([TERMINATE, TERMINATE]);
harness.appServer.assertNoErrors();
});
test("leaves the next turn's commands running when Codex starts that turn before acknowledging the interrupt", async () => {
const acknowledge = deferred();
const harness = createHarness({
// Codex ends B, starts a turn of its own (a goal continuation), and only then
// answers the interrupt.
interruptResponse: (completeTurn) => {
completeTurn();
return acknowledge.promise.then(() => ({}));
},
});
await harness.startTurn("run two commands", "native-B");
await harness.runsCommand({ turnId: "native-B", processId: "201" });
await harness.runsCommand({ turnId: "native-B", processId: "202" });
const interrupting = harness.session.interrupt();
await vi.waitFor(() => expect(harness.requests("turn/interrupt")).toHaveLength(1));
const autonomousTurn = waitForNextEvent(harness.session, "turn_started");
harness.appServer.startsTurn({ threadId: THREAD, turnId: "native-C" });
await autonomousTurn;
await harness.runsCommand({ turnId: "native-C", processId: "301" });
acknowledge.resolve();
await interrupting;
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201", "202"]));
await new Promise((resolve) => setTimeout(resolve, 50));
expect(harness.terminated()).toEqual(["201", "202"]);
});
test("sends the next turn/start only after the terminates are answered", async () => {
const answer = deferred();
const harness = createHarness({
terminate: () => answer.promise.then(() => ({ terminated: true })),
});
await interruptTurnB(harness);
const next = harness.startTurn("correction", "native-C");
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201", "202"]));
await new Promise((resolve) => setTimeout(resolve, 200));
expect(harness.requests("turn/start")).toHaveLength(2);
answer.resolve();
await next;
expect(harness.requests("turn/start")).toHaveLength(3);
});
test.each(["initial", "late"] as const)(
"tracks both cleanup sweeps when the %s answer arrives first",
async (answeredFirst) => {
const first = deferred();
const second = deferred();
const harness = createHarness({
terminate: (processId) =>
(processId === "201" ? first : second).promise.then(() => ({ terminated: true })),
});
await harness.startTurn("run commands", "native-B");
await harness.runsCommand({ turnId: "native-B", processId: "201" });
await harness.interrupts();
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201"]));
const next = harness.startTurn("correction", "native-C");
await new Promise((resolve) => setTimeout(resolve, 100));
expect(harness.requests("turn/start")).toHaveLength(1);
// The next turn is already waiting when another process is announced.
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-B",
itemId: "late-202",
processId: "202",
command: "sleep 202",
});
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-B",
itemId: "duplicate-202",
processId: "202",
command: "sleep 202",
});
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201", "202"]));
(answeredFirst === "initial" ? first : second).resolve();
await new Promise((resolve) => setTimeout(resolve, 100));
expect(harness.requests("turn/start")).toHaveLength(1);
(answeredFirst === "initial" ? second : first).resolve();
await next;
expect(harness.terminated()).toEqual(["201", "202"]);
// Accepted state survives the completed sweep and a newer foreground turn.
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-B",
itemId: "late-203",
processId: "203",
command: "sleep 203",
});
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201", "202", "203"]));
harness.appServer.assertNoErrors();
},
);
test("keeps the first sweep pending when a late sweep answers before the next start", async () => {
const first = deferred();
const harness = createHarness({
terminate: (processId) =>
processId === "201"
? first.promise.then(() => ({ terminated: true }))
: { terminated: true },
});
await harness.startTurn("run commands", "native-B");
await harness.runsCommand({ turnId: "native-B", processId: "201" });
await harness.interrupts();
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-B",
itemId: "late-202",
processId: "202",
command: "sleep 202",
});
await vi.waitFor(() => expect(harness.terminated()).toEqual(["201", "202"]));
const next = harness.startTurn("correction", "native-C");
await new Promise((resolve) => setTimeout(resolve, 100));
expect(harness.requests("turn/start")).toHaveLength(1);
first.resolve();
await next;
harness.appServer.assertNoErrors();
});
test("forgets completed commands but retains failed commands at interrupt", async () => {
const harness = createHarness();
await harness.startTurn("run commands", "native-B");
for (const processId of ["201", "202"]) {
await harness.runsCommand({ turnId: "native-B", processId });
harness.appServer.child.stdout.write(
JSON.stringify({
method: "item/completed",
params: {
threadId: THREAD,
turnId: "native-B",
item: {
id: `exec-${processId}`,
type: "commandExecution",
processId,
command: `sleep ${processId}`,
status: processId === "201" ? "completed" : "failed",
exitCode: processId === "201" ? 0 : null,
},
},
}) + "\n",
);
}
// A following command's row is a barrier for both completion handlers.
await harness.runsCommand({ turnId: "native-B", processId: "203" });
await harness.interrupts();
await harness.startTurn("correction", "native-C");
expect(harness.terminated()).toEqual(["202", "203"]);
harness.appServer.assertNoErrors();
});
test("retains accepted interrupts for only the latest sixteen turns", async () => {
const harness = createHarness();
for (let index = 0; index < 17; index++) {
await harness.startTurn("run a command", `native-${index}`);
await harness.interrupts();
}
await harness.startTurn("correction", "native-current");
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-0",
itemId: "evicted-201",
processId: "201",
command: "sleep 201",
});
harness.appServer.startsTerminalCommand({
threadId: THREAD,
turnId: "native-16",
itemId: "retained-202",
processId: "202",
command: "sleep 202",
});
await harness.runsCommand({ turnId: "native-current", processId: "301" });
expect(harness.terminated()).toEqual(["202"]);
harness.appServer.assertNoErrors();
});
test("a failing or unanswered terminate does not block or fail the next turn", async () => {
const harness = createHarness({
// 201 is refused while the next turn waits; 202 is never answered.
terminate: (processId) =>
processId === "201"
? new Promise((resolve) => setTimeout(resolve, 300)).then(() => ({
__jsonRpcError: { code: -32600, message: "invalid background terminal process id" },
}))
: new Promise(() => {}),
});
await interruptTurnB(harness);
const startedAt = Date.now();
await harness.startTurn("correction", "native-C");
expect(Date.now() - startedAt).toBeLessThan(5_000);
expect(harness.requests("turn/start")).toHaveLength(3);
expect(harness.terminated()).toEqual(["201", "202"]);
}, 15_000);
});
@@ -183,7 +183,7 @@ test("ordinary routed APIs isolate scopes and enforce one-hop, disable and origi
await paseo.dispose();
});
test("routed authority is bounded by the caller, with isolated simultaneous sessions", async () => {
test("routed authority isolates simultaneous sessions and preserves ordinary permission RPC errors", async () => {
const b = await daemon();
const a = await daemon();
const owner = await api(a);
@@ -210,33 +210,16 @@ test("routed authority is bounded by the caller, with isolated simultaneous sess
code: "permission_denied",
});
expect((await owner.paseo.hosts.list()).hosts).toHaveLength(1);
await remote.dispose();
});
test("SDK host clients preserve ordinary permission RPC errors", async () => {
const b = await daemon();
const a = await daemon();
const owner = await api(a);
await owner.paseo.hosts.add({ connection: `127.0.0.1:${b.port}` });
const serverId = (await online(owner.paseo)).hosts[0]!.serverId!;
const clients = [a, b].map((host) =>
createHostClient(`127.0.0.1:${host.port}`, {
clientId: crypto.randomUUID(),
sessionPermissions: ["daemon.read", "workspace.read"],
reconnect: { enabled: false },
}),
);
for (const client of clients) {
cleanup.push(() => client.close());
await client.connect();
}
const origin = createPaseoApi(clients[0]!);
const direct = createPaseoApi(clients[1]!);
const remote = await origin.hosts.client(serverId);
const directClient = createHostClient(`127.0.0.1:${b.port}`, {
clientId: crypto.randomUUID(),
sessionPermissions: ["daemon.read", "workspace.read"],
reconnect: { enabled: false },
});
await directClient.connect();
const direct = createPaseoApi(directClient);
cleanup.push(async () => {
await remote.dispose();
await direct.dispose();
await origin.dispose();
await directClient.close();
});
const input = { source: { kind: "directory" as const, path: b.staticDir } };
const directError = (await direct.workspaces.create(input).catch((error: unknown) => error)) as {
@@ -247,6 +230,7 @@ test("SDK host clients preserve ordinary permission RPC errors", async () => {
};
expect(directError.code).toBe("access_denied");
expect(remoteError.code).toBe(directError.code);
await remote.dispose();
});
test("distinct origin sessions cannot collide through a caller-selected route scope, including terminal binary output", async () => {
@@ -102,53 +102,52 @@ describe("daemon E2E", () => {
},
);
test.each([
{ fileName: "download.txt", disposition: 'attachment; filename="download.txt"' },
{
fileName: "café.txt",
disposition: "attachment; filename=\"caf?.txt\"; filename*=UTF-8''caf%C3%A9.txt",
},
{
fileName: "中文报告 (最终版).txt",
disposition:
"attachment; filename=\"???? (???).txt\"; filename*=UTF-8''%E4%B8%AD%E6%96%87%E6%8A%A5%E5%91%8A%20%28%E6%9C%80%E7%BB%88%E7%89%88%29.txt",
},
{
fileName: "報告書.md",
disposition:
"attachment; filename=\"???.md\"; filename*=UTF-8''%E5%A0%B1%E5%91%8A%E6%9B%B8.md",
},
{
fileName: "보고서.txt",
disposition:
"attachment; filename=\"???.txt\"; filename*=UTF-8''%EB%B3%B4%EA%B3%A0%EC%84%9C.txt",
},
{
fileName: "report-📄.txt",
disposition:
"attachment; filename=\"report-??.txt\"; filename*=UTF-8''report-%F0%9F%93%84.txt",
},
])(
"issues token over WS and downloads $fileName via HTTP",
async ({ fileName, disposition }) => {
const cwd = tmpCwd();
try {
test("downloads ASCII and Unicode filenames through one workspace", async () => {
const cwd = tmpCwd();
try {
const agent = await ctx.client.createAgent({
provider: "codex",
model: CODEX_TEST_MODEL,
thinkingOptionId: CODEX_TEST_THINKING_OPTION_ID,
cwd,
title: "Download Token Test Agent",
});
expect(agent.id).toMatch(
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/,
);
for (const { fileName, disposition } of [
{ fileName: "download.txt", disposition: 'attachment; filename="download.txt"' },
{
fileName: "café.txt",
disposition: "attachment; filename=\"caf?.txt\"; filename*=UTF-8''caf%C3%A9.txt",
},
{
fileName: "中文报告 (最终版).txt",
disposition:
"attachment; filename=\"???? (???).txt\"; filename*=UTF-8''%E4%B8%AD%E6%96%87%E6%8A%A5%E5%91%8A%20%28%E6%9C%80%E7%BB%88%E7%89%88%29.txt",
},
{
fileName: "報告書.md",
disposition:
"attachment; filename=\"???.md\"; filename*=UTF-8''%E5%A0%B1%E5%91%8A%E6%9B%B8.md",
},
{
fileName: "보고서.txt",
disposition:
"attachment; filename=\"???.txt\"; filename*=UTF-8''%EB%B3%B4%EA%B3%A0%EC%84%9C.txt",
},
{
fileName: "report-📄.txt",
disposition:
"attachment; filename=\"report-??.txt\"; filename*=UTF-8''report-%F0%9F%93%84.txt",
},
]) {
const filePath = path.join(cwd, fileName);
const fileContents = "download test payload";
const fileContents = `download test payload: ${fileName}`;
writeFileSync(filePath, fileContents, "utf-8");
const agent = await ctx.client.createAgent({
provider: "codex",
model: CODEX_TEST_MODEL,
thinkingOptionId: CODEX_TEST_THINKING_OPTION_ID,
cwd,
title: "Download Token Test Agent",
});
expect(agent.id).toMatch(
/^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/,
);
const tokenResponse = await ctx.client.requestDownloadToken(cwd, fileName);
expect(tokenResponse.error).toBeNull();
@@ -167,12 +166,11 @@ describe("daemon E2E", () => {
const body = await response.text();
expect(body).toBe(fileContents);
} finally {
rmSync(cwd, { recursive: true, force: true });
}
},
60000,
);
} finally {
rmSync(cwd, { recursive: true, force: true });
}
}, 60000);
test("rejects invalid token", async () => {
const response = await fetch(
@@ -0,0 +1,129 @@
import { mkdir, mkdtemp, rm } from "node:fs/promises";
import path from "node:path";
import type { FormInfo } from "@opencode/client";
import { expect, test } from "vitest";
import { createTestLogger } from "../../test-utils/test-logger.js";
import { OpenCodeV2AgentClient } from "../agent/providers/opencode/v2/agent.js";
import { V2Harness } from "../agent/providers/opencode/test-utils/v2-harness.js";
import { createTestAgentClients } from "../test-utils/fake-agent-client.js";
import { createTestPaseoDaemon } from "../test-utils/paseo-daemon.js";
import { DaemonClient } from "../test-utils/daemon-client.js";
async function createQuestionHarness() {
const artifacts = path.resolve(".artifacts");
await mkdir(artifacts, { recursive: true });
const root = await mkdtemp(path.join(artifacts, "peer-permissions-"));
const runtime = new V2Harness();
runtime.autoComplete = false;
runtime.api.model.list = async () => ({ location: runtime.info.location, data: [] });
runtime.api.provider.list = async () => ({
location: runtime.info.location,
data: [{ id: "test", name: "Test", activation: "enabled", package: "test" }],
});
let form: FormInfo | null = null;
const answers: Array<Record<string, unknown>> = [];
runtime.api.session.form.list = async () => (form ? [form] : []);
runtime.api.session.form.cancel = async () => {
form = null;
runtime.finishExecution("interrupted");
};
runtime.api.session.form.reply = async ({ answer }) => {
answers.push(answer);
form = null;
runtime.finishExecution();
};
// A steer is admitted to the existing execution, rather than starting another one.
runtime.prompt = async (input) => {
runtime.prompts.push(input.text);
if (runtime.prompts.length === 1) runtime.startExecution();
};
const provider = new OpenCodeV2AgentClient({
logger: createTestLogger(),
runtime: runtime.runtime,
});
const daemon = await createTestPaseoDaemon({
agentClients: { ...createTestAgentClients(), opencode: provider },
paseoHomeRoot: path.join(root, "daemon"),
staticDir: path.join(root, "static"),
cleanup: false,
});
const client = new DaemonClient({ url: `ws://127.0.0.1:${daemon.port}/ws` });
try {
await client.connect();
await client.fetchAgents({ subscribe: {} });
const recipient = await client.createAgent({
provider: "opencode",
cwd: root,
title: "Waiting for a human answer",
});
const sender = await client.createAgent({ provider: "claude", cwd: root });
await client.sendMessage(recipient.id, "Ask which color to use");
await expect.poll(() => runtime.prompts).toHaveLength(1);
form = {
id: "color-question",
sessionID: "session",
title: "Color",
fields: [{ key: "color", title: "Color", type: "string" }],
};
runtime.push({ id: "question-created", created: 2, type: "form.created", data: { form } });
await client.waitForAgentUpsert(recipient.id, (agent) =>
(agent.pendingPermissions ?? []).some((request) => request.id === "color-question"),
);
return {
client,
recipient,
sender,
answers,
async close() {
await client.close();
await daemon.close();
await rm(root, { recursive: true, force: true });
},
};
} catch (error) {
await client.close();
await daemon.close();
await rm(root, { recursive: true, force: true });
throw error;
}
}
test("an agent-sourced steer leaves the question pending and answerable", async () => {
const { client, recipient, sender, answers, close } = await createQuestionHarness();
try {
await client.sendAgentMessage(recipient.id, "Peer status check", {
sourceAgentId: sender.id,
});
// The public wait includes admitted message delivery before reporting permission status.
const waiting = await client.waitForFinish(recipient.id);
expect(waiting.status).toBe("permission");
expect(waiting.final?.pendingPermissions).toMatchObject([
{ id: "color-question", kind: "question" },
]);
await client.respondToPermission(recipient.id, "color-question", {
behavior: "allow",
updatedInput: { answers: { Color: "Blue" } },
});
await client.waitForAgentUpsert(
recipient.id,
(agent) => (agent.pendingPermissions ?? []).length === 0 && agent.status === "idle",
);
expect(answers).toEqual([{ color: "Blue" }]);
} finally {
await close();
}
});
test("a human steer still dismisses the question", async () => {
const { client, recipient, answers, close } = await createQuestionHarness();
try {
await client.sendAgentMessage(recipient.id, "Use red instead", { activeTurnBehavior: "steer" });
const finished = await client.waitForFinish(recipient.id);
expect(finished.status).toBe("idle");
expect(finished.final?.pendingPermissions ?? []).toEqual([]);
expect(answers).toEqual([]);
} finally {
await close();
}
});
+1 -1
View File
@@ -8318,7 +8318,7 @@ export class Session {
},
activeTurnBehavior: msg.activeTurnBehavior ?? "steer",
sessionMode: msg.sessionMode,
clearPendingPermissions: true,
clearPendingPermissions: false,
});
deliveryStatus = "queued";
queued();
@@ -1,3 +1,4 @@
import { flushGitCommandTrace } from "../utils/git-command-trace.js";
import { createWorkspaceRuntimeStore } from "./workspace-runtime-store/index.js";
import { mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, writeFileSync } from "node:fs";
import { execFileSync } from "node:child_process";
@@ -730,3 +731,103 @@ test("Git event identity retains a symlink's literal target before canonicalizin
rmSync(root, { recursive: true, force: true });
}
});
test.skipIf(process.platform !== "linux")(
"missing folders stop degraded Git polling and explicit refreshes, then recover",
async () => {
const artifacts = path.resolve(".artifacts");
mkdirSync(artifacts, { recursive: true });
const root = mkdtempSync(path.join(artifacts, "workspace-git-missing-"));
const cwd = path.join(root, "repo");
const initializeRepo = (branch: string) => {
mkdirSync(cwd);
execFileSync("git", ["init", "-b", branch], { cwd, stdio: "pipe" });
writeFileSync(path.join(cwd, "README.md"), "fixture\n");
execFileSync("git", ["add", "."], { cwd, stdio: "pipe" });
execFileSync(
"git",
[
"-c",
"user.name=Test",
"-c",
"user.email=test@example.invalid",
"-c",
"commit.gpgsign=false",
"commit",
"-m",
"fixture",
],
{ cwd, stdio: "pipe" },
);
};
initializeRepo("main");
// Exceed the real Linux observer's directory limit to enter degraded polling.
// No watcher errors or Git responses are injected.
for (let index = 0; index < 5100; index++) {
mkdirSync(path.join(cwd, `generated-${index}`));
}
const trace = path.join(root, "git.jsonl");
writeFileSync(trace, "");
const previousTrace = process.env.PASEO_GIT_TRACE_FILE;
process.env.PASEO_GIT_TRACE_FILE = trace;
let degraded = false;
let suspended = false;
const logger = pino(
{ level: "warn" },
{
write(line) {
const message = JSON.parse(line).msg;
if (message === "Failed to start working tree watcher; using degraded polling")
degraded = true;
if (
message ===
"Workspace folder no longer exists; suspended git observation until it is back"
)
suspended = true;
},
},
);
const service = new WorkspaceGitServiceImpl({ logger, paseoHome: path.join(root, "home") });
const subscription = service.registerWorkspace({ cwd }, () => {});
const commandCount = async () => {
await flushGitCommandTrace();
return readFileSync(trace, "utf8").trim().split("\n").filter(Boolean).length;
};
try {
await expect
.poll(() => degraded && service.getMetrics().workspaceRefreshInFlightCount === 0, {
timeout: 15_000,
})
.toBe(true);
rmSync(cwd, { recursive: true, force: true });
await expect.poll(() => suspended, { timeout: 15_000 }).toBe(true);
await expect.poll(() => service.getMetrics().workspaceRefreshInFlightCount).toBe(0);
const commandsWhileMissing = await commandCount();
await service.getSnapshot(cwd, {
force: true,
includeForge: false,
reason: "refresh-button",
});
await service.refresh(cwd);
expect(await commandCount()).toBe(commandsWhileMissing);
await new Promise((resolve) => setTimeout(resolve, 16_000));
expect(await commandCount()).toBe(commandsWhileMissing);
initializeRepo("restored");
const snapshot = await service.getSnapshot(cwd, {
force: true,
includeForge: false,
reason: "refresh-button",
});
expect(snapshot.git).toMatchObject({ isGit: true, currentBranch: "restored" });
await expect.poll(() => service.getMetrics().repositoryWorkspaceLinkCount).toBe(1);
} finally {
subscription.unsubscribe();
await service.dispose();
if (previousTrace === undefined) delete process.env.PASEO_GIT_TRACE_FILE;
else process.env.PASEO_GIT_TRACE_FILE = previousTrace;
rmSync(root, { recursive: true, force: true });
}
},
60_000,
);
@@ -244,6 +244,7 @@ function createService(
filterEvents: (events) => events,
verify: vi.fn(async () => {}),
})),
workspaceFolderExists: vi.fn(async () => true),
...overrides,
} as never,
});
@@ -1,5 +1,7 @@
import { createWorkspaceRuntimeStore } from "./workspace-runtime-store/index.js";
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
import { existsSync } from "node:fs";
import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path, { join } from "node:path";
import type pino from "pino";
@@ -10,11 +12,14 @@ import type {
PullRequestStatusResult,
} from "../utils/checkout-git.js";
import {
BACKGROUND_GIT_FETCH_INTERVAL_MS,
WORKSPACE_GIT_OBSERVATION_REENSURE_INTERVAL_MS,
WORKSPACE_GIT_OBSERVATION_SETUP_CONCURRENCY,
WORKSPACE_GIT_REFRESH_CONCURRENCY,
WORKSPACE_GIT_WATCHER_SUBSCRIBE_TIMEOUT_MS,
WorkspaceGitServiceImpl,
type WorkspaceGitRuntimeSnapshot,
workspaceFolderExists,
} from "./workspace-git-service.js";
const REPO_CWD = path.resolve("/tmp/repo");
@@ -23,6 +28,7 @@ function createLogger() {
const logger = {
child: () => logger,
debug: vi.fn(),
info: vi.fn(),
warn: vi.fn(),
};
return logger;
@@ -251,7 +257,7 @@ function createGitHubServiceStub(): ForgeService {
interface CreateServiceTestOptions {
subscribe?: ReturnType<typeof vi.fn>;
getCheckoutStatus?: ReturnType<typeof vi.fn>;
getCheckoutSnapshotFacts?: ReturnType<typeof vi.fn>;
getCheckoutSnapshotFacts?: (cwd: string) => Promise<CheckoutSnapshotFacts>;
getCheckoutShortstat?: ReturnType<typeof vi.fn>;
getCheckoutWorktreeState?: ReturnType<typeof vi.fn>;
getPullRequestStatus?: ReturnType<typeof vi.fn>;
@@ -261,6 +267,7 @@ interface CreateServiceTestOptions {
runGitFetch?: ReturnType<typeof vi.fn>;
runGitCommand?: ReturnType<typeof vi.fn>;
getWorkspaceGitSelfHealPhaseMs?: (cwd: string) => number;
workspaceFolderExists?: (cwd: string) => Promise<boolean>;
now?: () => Date;
}
@@ -294,6 +301,7 @@ function buildDefaultTestServiceDeps() {
signal: null,
})),
getWorkspaceGitSelfHealPhaseMs: vi.fn(() => 30_000),
workspaceFolderExists: vi.fn(async () => true),
createWatcherLivenessCanary: vi.fn(() => ({
path: "",
filterEvents: (events) => events,
@@ -324,6 +332,145 @@ function createService(options?: CreateServiceTestOptions) {
});
}
type WatchCallback = (error: Error | null, events: Array<{ path: string; type: "update" }>) => void;
interface FolderBackedGitOptions {
facts?: (cwd: string) => CheckoutSnapshotFacts;
unwatchableDirectories?: string[];
heldWatch?: { directory: string; release: Promise<void> };
}
function createFolderBackedGit(options?: FolderBackedGitOptions) {
const watchCallbacks = new Map<string, WatchCallback>();
const subscribe = vi.fn(async (directory: string, callback: WatchCallback) => {
if (!existsSync(directory) || options?.unwatchableDirectories?.includes(directory)) {
throw new Error(`Cannot watch ${directory}`);
}
if (options?.heldWatch?.directory === directory) {
await options.heldWatch.release;
}
watchCallbacks.set(directory, callback);
return createAsyncSubscription();
});
const getCheckoutSnapshotFacts = vi.fn(
async (cwd: string): Promise<CheckoutSnapshotFacts> =>
existsSync(cwd) ? (options?.facts ?? createCheckoutSnapshotFacts)(cwd) : { isGit: false },
);
const getCheckoutStatus = vi.fn(async (cwd: string) =>
existsSync(cwd) ? createCheckoutStatus(cwd) : ({ isGit: false } as const),
);
const getCheckoutShortstat = vi.fn(async (_cwd: string) => ({ additions: 1, deletions: 0 }));
const getCheckoutWorktreeState = vi.fn(async (_cwd: string) => ({
isDirty: false,
diffStat: { additions: 1, deletions: 0 },
}));
const runGitCommand = vi.fn(async (args: string[], runOptions: { cwd: string }) => ({
stdout: args[0] === "rev-parse" ? `${runOptions.cwd}\n` : "",
stderr: "",
truncated: false,
exitCode: 0,
signal: null,
}));
const runGitFetch = vi.fn(async (_cwd: string) => ({ changes: [], error: null }));
const callsIn = (fake: ReturnType<typeof vi.fn>, cwd: string) =>
fake.mock.calls.filter(([calledCwd]) => calledCwd === cwd).length;
return {
deps: {
subscribe,
getCheckoutSnapshotFacts,
getCheckoutStatus,
getCheckoutShortstat,
getCheckoutWorktreeState,
runGitCommand,
runGitFetch,
workspaceFolderExists: async (cwd: string) => existsSync(cwd),
now: () => new Date(),
},
watchCallbackFor(directory: string): WatchCallback {
const callback = watchCallbacks.get(directory);
if (!callback) {
throw new Error(`No watcher was started for ${directory}`);
}
return callback;
},
watchAttemptsFor: (directory: string) => callsIn(subscribe, directory),
pollsOf: (cwd: string) => callsIn(getCheckoutShortstat, cwd),
fetchesIn: (cwd: string) => callsIn(runGitFetch, cwd),
gitCommandsIn(cwd: string): number {
const cwdCommands = [
getCheckoutSnapshotFacts,
getCheckoutStatus,
getCheckoutShortstat,
getCheckoutWorktreeState,
runGitFetch,
].reduce((count, fake) => count + callsIn(fake, cwd), 0);
const commandsRunInCwd = runGitCommand.mock.calls.filter(
([, runOptions]) => runOptions.cwd === cwd,
).length;
return cwdCommands + commandsRunInCwd;
},
};
}
function linkedWorktreeFacts(gitCommonDir: string) {
return (cwd: string): CheckoutSnapshotFacts => ({
...createCheckoutSnapshotFacts(cwd),
absoluteGitDir: join(gitCommonDir, "worktrees", path.basename(cwd)),
gitCommonDir,
});
}
const SETTLE_ATTEMPTS = 1_000;
async function settle(assertion: () => void): Promise<void> {
let lastFailure: unknown = null;
for (let attempt = 0; attempt < SETTLE_ATTEMPTS; attempt += 1) {
try {
assertion();
return;
} catch (failure) {
lastFailure = failure;
}
await vi.advanceTimersByTimeAsync(0);
}
throw lastFailure;
}
async function waitForRefreshesToSettle(service: WorkspaceGitServiceImpl): Promise<void> {
await settle(() => {
expect(service.getMetrics().workspaceRefreshInFlightCount).toBe(0);
});
}
async function waitForObservationToSettle(
service: WorkspaceGitServiceImpl,
repositoryWorkspaceLinkCount: number,
): Promise<void> {
await settle(() => {
const metrics = service.getMetrics();
expect(metrics.repositoryWorkspaceLinkCount).toBe(repositoryWorkspaceLinkCount);
expect(metrics.workspaceObservationSetupInFlightCount).toBe(0);
expect(metrics.workspaceRefreshInFlightCount).toBe(0);
});
}
async function advanceThroughDegradedPolls(service: WorkspaceGitServiceImpl): Promise<void> {
for (const delayMs of [5_000, 10_000, 20_000, 40_000, 60_000, 60_000]) {
await vi.advanceTimersByTimeAsync(delayMs);
await waitForRefreshesToSettle(service);
}
}
async function advanceThroughWorkingTreeRecoveryAttempts(
service: WorkspaceGitServiceImpl,
): Promise<void> {
for (const delayMs of [30_000, 60_000, 120_000]) {
await vi.advanceTimersByTimeAsync(delayMs);
await waitForRefreshesToSettle(service);
}
}
describe("WorkspaceGitServiceImpl", () => {
beforeEach(() => {
vi.useFakeTimers();
@@ -1553,3 +1700,460 @@ describe("WorkspaceGitServiceImpl", () => {
service.dispose();
});
});
describe("WorkspaceGitServiceImpl with workspace folders that no longer exist", () => {
const temporaryFolders: string[] = [];
async function createTemporaryFolder(): Promise<string> {
const folder = await mkdtemp(join(os.tmpdir(), "paseo-workspace-git-"));
temporaryFolders.push(folder);
return folder;
}
beforeEach(() => {
vi.useFakeTimers();
});
afterEach(async () => {
vi.useRealTimers();
await Promise.all(
temporaryFolders.splice(0).map((folder) => rm(folder, { recursive: true, force: true })),
);
});
test("polling stops for a workspace whose folder no longer exists", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const service = createService(git.deps);
const subscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
const gitCommandsAtDeletion = git.gitCommandsIn(workspace);
await advanceThroughDegradedPolls(service);
expect(git.gitCommandsIn(workspace)).toBe(gitCommandsAtDeletion);
expect(service.getMetrics()).toMatchObject({
repositoryTargetCount: 0,
workingTreeWatchTargetCount: 0,
});
expect(vi.getTimerCount()).toBe(1);
subscription.unsubscribe();
await service.dispose();
});
test("a deleted worktree leaves its shared repository polling and fetch to the surviving worktree", async () => {
const gitCommonDir = join(await createTemporaryFolder(), ".git");
const deletedWorktree = await createTemporaryFolder();
const survivingWorktree = await createTemporaryFolder();
const git = createFolderBackedGit({ facts: linkedWorktreeFacts(gitCommonDir) });
let survivingWorktreeEdits = 0;
git.deps.getCheckoutShortstat.mockImplementation(async (cwd: string) => ({
additions: cwd === survivingWorktree ? ++survivingWorktreeEdits : 1,
deletions: 0,
}));
const service = createService(git.deps);
const deletedSubscription = service.registerWorkspace({ cwd: deletedWorktree }, vi.fn());
await waitForObservationToSettle(service, 1);
const survivingSubscription = service.registerWorkspace({ cwd: survivingWorktree }, vi.fn());
await waitForObservationToSettle(service, 2);
await rm(deletedWorktree, { recursive: true, force: true });
await vi.advanceTimersByTimeAsync(60_000);
await waitForRefreshesToSettle(service);
const deletedWorktreeCommands = git.gitCommandsIn(deletedWorktree);
for (let tick = 0; tick < 3; tick += 1) {
const survivingWorktreePolls = git.pollsOf(survivingWorktree);
await vi.advanceTimersByTimeAsync(5_000);
await settle(() => {
expect(git.pollsOf(survivingWorktree)).toBe(survivingWorktreePolls + 1);
});
}
expect(git.gitCommandsIn(deletedWorktree)).toBe(deletedWorktreeCommands);
const fetchesBefore = git.deps.runGitFetch.mock.calls.length;
await vi.advanceTimersByTimeAsync(2 * BACKGROUND_GIT_FETCH_INTERVAL_MS);
expect(git.deps.runGitFetch.mock.calls.slice(fetchesBefore).map(([cwd]) => cwd)).toEqual([
survivingWorktree,
survivingWorktree,
]);
expect(git.gitCommandsIn(deletedWorktree)).toBe(deletedWorktreeCommands);
deletedSubscription.unsubscribe();
survivingSubscription.unsubscribe();
await service.dispose();
});
test("deleting every worktree of a repository stops its polling and background fetch", async () => {
const gitCommonDir = join(await createTemporaryFolder(), ".git");
const firstWorktree = await createTemporaryFolder();
const secondWorktree = await createTemporaryFolder();
const git = createFolderBackedGit({ facts: linkedWorktreeFacts(gitCommonDir) });
const service = createService(git.deps);
const firstSubscription = service.registerWorkspace({ cwd: firstWorktree }, vi.fn());
await waitForObservationToSettle(service, 1);
const secondSubscription = service.registerWorkspace({ cwd: secondWorktree }, vi.fn());
await waitForObservationToSettle(service, 2);
await rm(firstWorktree, { recursive: true, force: true });
await rm(secondWorktree, { recursive: true, force: true });
const commandsAtDeletion = [
git.gitCommandsIn(firstWorktree),
git.gitCommandsIn(secondWorktree),
];
await advanceThroughDegradedPolls(service);
expect([git.gitCommandsIn(firstWorktree), git.gitCommandsIn(secondWorktree)]).toEqual(
commandsAtDeletion,
);
expect(service.getMetrics()).toMatchObject({
repositoryTargetCount: 0,
workingTreeWatchTargetCount: 0,
});
expect(vi.getTimerCount()).toBe(2);
firstSubscription.unsubscribe();
secondSubscription.unsubscribe();
await service.dispose();
});
test("working tree fallback polling and watcher recovery stop for a workspace whose folder no longer exists", async () => {
const workspace = await createTemporaryFolder();
await mkdir(join(workspace, ".git"));
const git = createFolderBackedGit();
const service = createService(git.deps);
const subscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
git.watchCallbackFor(workspace)(new Error("watched directory was deleted"), []);
await vi.advanceTimersByTimeAsync(1_000);
await waitForRefreshesToSettle(service);
const commandsAfterWatcherError = git.gitCommandsIn(workspace);
const watchAttemptsAfterWatcherError = git.watchAttemptsFor(workspace);
await advanceThroughDegradedPolls(service);
expect(git.gitCommandsIn(workspace)).toBe(commandsAfterWatcherError);
expect(git.watchAttemptsFor(workspace)).toBe(watchAttemptsAfterWatcherError);
expect(vi.getTimerCount()).toBe(1);
subscription.unsubscribe();
await service.dispose();
});
test("a workspace whose folder comes back resumes polling on its next subscription", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const service = createService(git.deps);
const firstSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
const commandsWhileMissing = git.gitCommandsIn(workspace);
await advanceThroughDegradedPolls(service);
expect(git.gitCommandsIn(workspace)).toBe(commandsWhileMissing);
await mkdir(workspace);
const secondSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
const pollsAfterResume = git.pollsOf(workspace);
await vi.advanceTimersByTimeAsync(5_000);
await waitForRefreshesToSettle(service);
expect(git.pollsOf(workspace)).toBe(pollsAfterResume + 1);
firstSubscription.unsubscribe();
secondSubscription.unsubscribe();
await service.dispose();
});
test("a suspended workspace costs no git while its folder is missing, even when a refresh is requested", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const service = createService(git.deps);
const subscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
await advanceThroughDegradedPolls(service);
const commandsWhileMissing = git.gitCommandsIn(workspace);
for (let probe = 0; probe < 10; probe += 1) {
await vi.advanceTimersByTimeAsync(WORKSPACE_GIT_OBSERVATION_REENSURE_INTERVAL_MS);
await waitForRefreshesToSettle(service);
}
service.scheduleRefreshForCwd(workspace);
service.onWorkspaceStateMayHaveChanged(workspace);
await vi.advanceTimersByTimeAsync(5_000);
await waitForRefreshesToSettle(service);
expect(git.gitCommandsIn(workspace)).toBe(commandsWhileMissing);
expect(vi.getTimerCount()).toBe(1);
subscription.unsubscribe();
await service.dispose();
});
test("a suspended worktree resumes polling and its repository fetch once its folder is back, without a new subscription", async () => {
const gitCommonDir = join(await createTemporaryFolder(), ".git");
const worktree = await createTemporaryFolder();
const git = createFolderBackedGit({ facts: linkedWorktreeFacts(gitCommonDir) });
const service = createService(git.deps);
const subscription = service.registerWorkspace({ cwd: worktree }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(worktree, { recursive: true, force: true });
await advanceThroughDegradedPolls(service);
await waitForObservationToSettle(service, 0);
await mkdir(worktree);
const fetchesBeforeRestore = git.fetchesIn(worktree);
await vi.advanceTimersByTimeAsync(WORKSPACE_GIT_OBSERVATION_REENSURE_INTERVAL_MS);
await waitForObservationToSettle(service, 1);
expect(git.fetchesIn(worktree)).toBe(fetchesBeforeRestore + 1);
const pollsAfterRestore = git.pollsOf(worktree);
await vi.advanceTimersByTimeAsync(5_000);
await waitForRefreshesToSettle(service);
expect(git.pollsOf(worktree)).toBe(pollsAfterRestore + 1);
subscription.unsubscribe();
await service.dispose();
});
test("a forced snapshot resumes a suspended workspace whose folder is back", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const service = createService(git.deps);
const subscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
await advanceThroughDegradedPolls(service);
await waitForObservationToSettle(service, 0);
await mkdir(workspace);
const snapshot = await service.getSnapshot(workspace, {
force: true,
reason: "refresh-button",
});
await waitForObservationToSettle(service, 1);
expect(snapshot.git.isGit).toBe(true);
subscription.unsubscribe();
await service.dispose();
});
test("a resumed workspace rebuilds observation from fresh facts, not the non-git facts cached while its folder was gone", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const service = createService(git.deps);
const firstSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
await rm(workspace, { recursive: true, force: true });
service.onWorkspaceStateMayHaveChanged(workspace);
await vi.advanceTimersByTimeAsync(1_000);
await waitForRefreshesToSettle(service);
expect((await service.getSnapshot(workspace)).git.isGit).toBe(false);
await advanceThroughDegradedPolls(service);
await waitForObservationToSettle(service, 0);
await mkdir(workspace);
const secondSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
expect((await service.getSnapshot(workspace)).git.isGit).toBe(true);
firstSubscription.unsubscribe();
secondSubscription.unsubscribe();
await service.dispose();
});
test("resuming waits for an observation setup that was still running when the folder vanished", async () => {
const workspace = await createTemporaryFolder();
const gitDir = join(workspace, ".git");
await mkdir(gitDir);
const metadataWatchRelease = createDeferred<void>();
const git = createFolderBackedGit({
unwatchableDirectories: [workspace],
heldWatch: { directory: gitDir, release: metadataWatchRelease.promise },
});
const service = createService(git.deps);
const firstSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await settle(() => {
expect(git.watchAttemptsFor(gitDir)).toBe(1);
});
await rm(workspace, { recursive: true, force: true });
await vi.advanceTimersByTimeAsync(5_000);
await settle(() => {
expect(service.getMetrics()).toMatchObject({
repositoryTargetCount: 0,
workingTreeWatchTargetCount: 0,
});
});
await mkdir(workspace);
const secondSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
metadataWatchRelease.resolve();
await waitForObservationToSettle(service, 1);
expect(service.getMetrics().workingTreeWatchTargetCount).toBe(1);
firstSubscription.unsubscribe();
secondSubscription.unsubscribe();
await service.dispose();
});
test("an open working tree watch neither notifies nor recovers while its folder is missing, and picks up once it is back", async () => {
const workspace = await createTemporaryFolder();
await mkdir(join(workspace, ".git"));
const git = createFolderBackedGit();
const service = createService(git.deps);
const workspaceSubscription = service.registerWorkspace({ cwd: workspace }, vi.fn());
await waitForObservationToSettle(service, 1);
const diffListener = vi.fn();
const diffWatch = await service.requestWorkingTreeWatch(workspace, diffListener);
await rm(workspace, { recursive: true, force: true });
git.watchCallbackFor(workspace)(new Error("watched directory was deleted"), []);
await vi.advanceTimersByTimeAsync(1_000);
await waitForRefreshesToSettle(service);
const commandsAfterWatcherError = git.gitCommandsIn(workspace);
const notificationsAfterWatcherError = diffListener.mock.calls.length;
const watchAttemptsAfterWatcherError = git.watchAttemptsFor(workspace);
await advanceThroughWorkingTreeRecoveryAttempts(service);
expect(git.gitCommandsIn(workspace)).toBe(commandsAfterWatcherError);
expect(diffListener.mock.calls.length).toBe(notificationsAfterWatcherError);
expect(git.watchAttemptsFor(workspace)).toBe(watchAttemptsAfterWatcherError);
expect(service.getMetrics().workingTreeWatchTargetCount).toBe(1);
await mkdir(join(workspace, ".git"), { recursive: true });
await vi.advanceTimersByTimeAsync(60_000);
await waitForRefreshesToSettle(service);
expect(diffListener.mock.calls.length).toBe(notificationsAfterWatcherError + 1);
await vi.advanceTimersByTimeAsync(240_000);
await waitForRefreshesToSettle(service);
expect(git.watchAttemptsFor(workspace)).toBe(watchAttemptsAfterWatcherError + 1);
const notificationsAfterResubscribe = diffListener.mock.calls.length;
await vi.advanceTimersByTimeAsync(120_000);
await waitForRefreshesToSettle(service);
expect(diffListener.mock.calls.length).toBe(notificationsAfterResubscribe);
diffWatch.unsubscribe();
workspaceSubscription.unsubscribe();
await service.dispose();
});
test("a refresh crossing suspension preserves the cached snapshot and drops queued work", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const started = createDeferred<void>();
const release = createDeferred<void>();
let holdNextRead = false;
let factsReads = 0;
const service = createService({
...git.deps,
getCheckoutSnapshotFacts: async (cwd: string) => {
factsReads++;
if (holdNextRead) {
holdNextRead = false;
started.resolve();
await release.promise;
}
return git.deps.getCheckoutSnapshotFacts(cwd);
},
});
const updates: WorkspaceGitRuntimeSnapshot[] = [];
const subscription = service.registerWorkspace({ cwd: workspace }, (snapshot) =>
updates.push(snapshot),
);
await waitForObservationToSettle(service, 1);
const cached = await service.getSnapshot(workspace);
holdNextRead = true;
const active = service.getSnapshot(workspace, {
force: true,
includeForge: false,
reason: "overlapping-refresh",
});
await started.promise;
const queued = service.getSnapshot(workspace, {
force: true,
includeForge: true,
reason: "queued-refresh",
});
await rm(workspace, { recursive: true, force: true });
await vi.advanceTimersByTimeAsync(5_000);
await settle(() => expect(service.getMetrics().repositoryWorkspaceLinkCount).toBe(0));
const readsAtSuspension = factsReads;
const updatesAtSuspension = updates.length;
release.resolve();
expect(await active).toEqual(cached);
expect(await queued).toEqual(cached);
expect(await service.getSnapshot(workspace)).toEqual(cached);
expect(updates).toHaveLength(updatesAtSuspension);
expect(factsReads).toBe(readsAtSuspension);
subscription.unsubscribe();
await service.dispose();
});
test("restoration settles a suspended refresh before reloading facts", async () => {
const workspace = await createTemporaryFolder();
const git = createFolderBackedGit();
const staleFacts = createDeferred<CheckoutSnapshotFacts>();
const started = createDeferred<void>();
let holdNextRead = false;
const service = createService({
...git.deps,
getCheckoutSnapshotFacts: async (cwd: string) => {
if (holdNextRead) {
holdNextRead = false;
started.resolve();
return staleFacts.promise;
}
return git.deps.getCheckoutSnapshotFacts(cwd);
},
});
const updates: WorkspaceGitRuntimeSnapshot[] = [];
const subscription = service.registerWorkspace({ cwd: workspace }, (snapshot) =>
updates.push(snapshot),
);
await waitForObservationToSettle(service, 1);
const cached = await service.getSnapshot(workspace);
holdNextRead = true;
const active = service.getSnapshot(workspace, {
force: true,
includeForge: false,
reason: "overlapping-refresh",
});
await started.promise;
await rm(workspace, { recursive: true, force: true });
await vi.advanceTimersByTimeAsync(5_000);
await settle(() => expect(service.getMetrics().repositoryWorkspaceLinkCount).toBe(0));
const updatesAtSuspension = updates.length;
await mkdir(workspace);
const restored = service.getSnapshot(workspace, {
force: true,
includeForge: false,
reason: "folder-returned",
});
await vi.advanceTimersByTimeAsync(0);
staleFacts.resolve({ isGit: false });
expect(await active).toEqual(cached);
expect((await restored).git.isGit).toBe(true);
expect(updates.slice(updatesAtSuspension).every((snapshot) => snapshot.git.isGit)).toBe(true);
await waitForObservationToSettle(service, 1);
subscription.unsubscribe();
await service.dispose();
});
test("workspaceFolderExists reports a missing folder and a path under a file as missing", async () => {
const folder = await createTemporaryFolder();
const file = join(folder, "file.txt");
await writeFile(file, "");
expect(await workspaceFolderExists(folder)).toBe(true);
expect(await workspaceFolderExists(join(folder, "missing"))).toBe(false);
expect(await workspaceFolderExists(join(file, "nested"))).toBe(false);
});
});
@@ -1,5 +1,5 @@
import { createHash } from "node:crypto";
import { readFile } from "node:fs/promises";
import { readFile, stat } from "node:fs/promises";
import { basename, dirname, join, resolve } from "node:path";
import { LRUCache } from "lru-cache";
import { CheckoutDiffCache } from "./checkout-diff-cache.js";
@@ -74,7 +74,7 @@ import { checkoutLiteFromGitSnapshot } from "./workspace-registry-model.js";
import { createWatcherLivenessCanary } from "./watcher-liveness-canary.js";
const WORKSPACE_GIT_WATCH_DEBOUNCE_MS = 1_000;
const BACKGROUND_GIT_FETCH_INTERVAL_MS = 180_000;
export const BACKGROUND_GIT_FETCH_INTERVAL_MS = 180_000;
const FETCH_METADATA_ECHO_TTL_MS = 5_000;
export const WORKSPACE_GIT_OBSERVATION_REENSURE_INTERVAL_MS = 60_000;
const FORGE_PR_STATUS_POLL_FAST_INTERVAL_MS = 20_000;
@@ -384,6 +384,7 @@ interface WorkspaceGitServiceDependencies {
runGitCommand: typeof runGitCommand;
getWorkspaceGitSelfHealPhaseMs: typeof getWorkspaceGitObservationReensurePhaseMs;
createWatcherLivenessCanary: typeof createWatcherLivenessCanary;
workspaceFolderExists: (cwd: string) => Promise<boolean>;
now: () => Date;
}
@@ -435,6 +436,8 @@ interface WorkspaceGitTarget {
repoGitRoot: string | null;
observationSetupPromise: Promise<void> | null;
observationSetupComplete: boolean;
folderMissing: boolean;
folderRestorePromise: Promise<void> | null;
closed: boolean;
}
@@ -571,10 +574,21 @@ function buildDefaultWorkspaceGitServiceDeps(
runGitCommand,
getWorkspaceGitSelfHealPhaseMs: getWorkspaceGitObservationReensurePhaseMs,
createWatcherLivenessCanary,
workspaceFolderExists,
now: () => new Date(),
};
}
export async function workspaceFolderExists(cwd: string): Promise<boolean> {
try {
await stat(cwd);
return true;
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
return code !== "ENOENT" && code !== "ENOTDIR";
}
}
function resolveWorkspaceGitServiceDeps(
subscribe: SubscribeToFileChanges,
deps: Partial<WorkspaceGitServiceDependencies> | undefined,
@@ -660,6 +674,9 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
const cwd = resolve(params.cwd);
const target = this.ensureWorkspaceTarget(cwd);
target.listeners.add(listener);
if (target.folderMissing) {
void this.resumeWorkspaceIfFolderRestored(target);
}
if (target.listeners.size === 1) {
this.startWorkspaceSubscriptionTimers(target);
}
@@ -751,6 +768,9 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
if (!request.force && target.latestSnapshot) {
return target.latestSnapshot;
}
if (target.folderMissing) {
await this.resumeWorkspaceIfFolderRestored(target);
}
return this.requestWorkspaceSnapshot(target, request);
}
@@ -972,6 +992,9 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
this.assertNotDisposed();
cwd = resolve(cwd);
const target = this.ensureWorkspaceTarget(cwd);
if (target.folderMissing) {
await this.resumeWorkspaceIfFolderRestored(target);
}
await this.refreshWorkspaceTarget(target, {
force: false,
refreshStructure: true,
@@ -1005,9 +1028,14 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
this.assertNotDisposed();
cwd = resolve(cwd);
const target = this.workspaceTargets.get(cwd);
if (target) {
this.scheduleWorkspaceRefresh(target, { queueIfBusy: false });
if (!target) {
return;
}
if (target.folderMissing) {
void this.resumeWorkspaceIfFolderRestored(target);
return;
}
this.scheduleWorkspaceRefresh(target, { queueIfBusy: false });
}
onWorkspaceStateMayHaveChanged(cwd: string): void {
@@ -1018,6 +1046,10 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
return;
}
this.invalidateForge(normalizedCwd);
if (target.folderMissing) {
void this.resumeWorkspaceIfFolderRestored(target);
return;
}
this.scheduleWorkspaceRefresh(target, {
force: true,
includeForge: true,
@@ -1227,6 +1259,8 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
repoGitRoot: null,
observationSetupPromise: null,
observationSetupComplete: false,
folderMissing: false,
folderRestorePromise: null,
closed: false,
};
@@ -1358,6 +1392,7 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
private isActiveObservedWorkspaceTarget(target: WorkspaceGitTarget): boolean {
return (
!target.closed &&
!target.folderMissing &&
target.listeners.size > 0 &&
this.workspaceTargets.get(target.cwd) === target
);
@@ -1612,11 +1647,110 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
workspaceTarget: WorkspaceGitTarget,
request: WorkspaceGitRefreshRequest,
): Promise<boolean> {
if (await this.suspendWorkspaceIfFolderMissing(workspaceTarget)) {
return false;
}
const fingerprintBefore = workspaceTarget.latestFingerprint;
await this.refreshWorkspaceTarget(workspaceTarget, request);
return workspaceTarget.latestFingerprint !== fingerprintBefore;
}
private async suspendWorkspaceIfFolderMissing(target: WorkspaceGitTarget): Promise<boolean> {
if (target.folderMissing) {
return true;
}
if (await this.deps.workspaceFolderExists(target.cwd)) {
return false;
}
this.suspendWorkspaceWithMissingFolder(target);
return true;
}
private suspendWorkspaceWithMissingFolder(target: WorkspaceGitTarget): void {
if (target.folderMissing || target.closed || this.workspaceTargets.get(target.cwd) !== target) {
return;
}
target.folderMissing = true;
if (target.refreshState.status === "in-flight") {
target.refreshState.queued = null;
}
this.detachWorkspaceFromRepoTarget(target);
target.repoGitRoot = null;
this.stopWorkspaceObservation(target);
target.observationSetupComplete = false;
this.scheduleMissingFolderProbe(target);
this.logger.warn(
{ cwd: target.cwd },
"Workspace folder no longer exists; suspended git observation until it is back",
);
}
private isSuspendedWorkspaceTarget(target: WorkspaceGitTarget): boolean {
return (
target.folderMissing && !target.closed && this.workspaceTargets.get(target.cwd) === target
);
}
private scheduleMissingFolderProbe(target: WorkspaceGitTarget): void {
if (
!this.isSuspendedWorkspaceTarget(target) ||
target.observationReensureTimer ||
target.folderRestorePromise
) {
return;
}
target.observationReensureTimer = setTimeout(() => {
target.observationReensureTimer = null;
void this.resumeWorkspaceIfFolderRestored(target);
}, WORKSPACE_GIT_OBSERVATION_REENSURE_INTERVAL_MS);
}
private resumeWorkspaceIfFolderRestored(target: WorkspaceGitTarget): Promise<void> {
if (!target.folderMissing) {
return Promise.resolve();
}
target.folderRestorePromise ??= this.resumeWorkspaceWithRestoredFolder(target).finally(() => {
target.folderRestorePromise = null;
this.scheduleMissingFolderProbe(target);
});
return target.folderRestorePromise;
}
private async resumeWorkspaceWithRestoredFolder(target: WorkspaceGitTarget): Promise<void> {
if (!(await this.deps.workspaceFolderExists(target.cwd))) {
return;
}
await target.observationSetupPromise;
if (target.refreshState.status === "in-flight") {
await target.refreshState.promise.catch(() => undefined);
}
if (!this.isSuspendedWorkspaceTarget(target)) {
return;
}
target.latestFacts = null;
target.folderMissing = false;
await this.refreshWorkspaceTarget(target, {
force: false,
refreshStructure: true,
refreshWorktree: true,
includeForge: false,
reason: "workspace-folder-restored",
queueIfBusy: true,
movedRemoteRefs: new Set(),
});
if (target.closed || this.workspaceTargets.get(target.cwd) !== target) {
return;
}
target.observationSetupComplete = false;
if (target.observationReensureTimer) {
clearTimeout(target.observationReensureTimer);
target.observationReensureTimer = null;
}
this.logger.info({ cwd: target.cwd }, "Workspace folder is back; resumed git observation");
this.startWorkspaceSubscriptionTimers(target);
this.scheduleWorkspaceObservationSetup(target);
}
private startWorkingTreeWatchFallback(
target: WorkingTreeWatchTarget,
reason: WorkingTreeWatchFallbackReason,
@@ -1654,7 +1788,9 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
return changed;
}),
);
this.notifyWorkingTreeConsumers(target);
if (await this.deps.workspaceFolderExists(target.watchPath)) {
this.notifyWorkingTreeConsumers(target);
}
if (!target.closed && (target.subscription === null || target.repoRoot === null)) {
target.fallbackPollQuietTickCount = changes.some(Boolean)
? 0
@@ -1731,6 +1867,10 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
if (target.closed || target.subscription) {
return;
}
if (!(await this.deps.workspaceFolderExists(target.watchPath))) {
this.scheduleWorkingTreeWatchRecovery(target);
return;
}
await this.refreshWorkingTreeIgnoredDirectories(target);
if (target.closed || target.subscription) {
return;
@@ -2786,7 +2926,7 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
}
private updateForgePrStatusPollForTarget(target: WorkspaceGitTarget): void {
if (target.listeners.size === 0) {
if (!this.isActiveObservedWorkspaceTarget(target)) {
this.stopForgePrStatusPollForTarget(target);
return;
}
@@ -3049,6 +3189,9 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
target: WorkspaceGitTarget,
request: WorkspaceGitRefreshRequest,
): Promise<WorkspaceGitRuntimeSnapshot> {
if (target.folderMissing) {
return Promise.resolve(target.latestSnapshot ?? buildNotGitSnapshot(target.cwd));
}
if (target.refreshState.status === "in-flight") {
const active = target.refreshState.request;
const addsWork =
@@ -3191,11 +3334,19 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
try {
const runRefreshGitCommand = createRunGitCommand(`workspace-refresh:${request.reason}`);
const admittedSnapshot = await this.workspaceRefreshLimit(() => {
if (target.closed || this.workspaceTargets.get(target.cwd) !== target) {
if (
target.closed ||
target.folderMissing ||
this.workspaceTargets.get(target.cwd) !== target
) {
return null;
}
return this.refreshSnapshot(target, request, runRefreshGitCommand);
});
if (target.folderMissing) {
failure = null;
break;
}
if (!admittedSnapshot) {
break;
}
@@ -3209,6 +3360,10 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
failure = { error };
}
if (target.folderMissing) {
failure = null;
break;
}
if (this.disposed || target.closed || this.workspaceTargets.get(target.cwd) !== target) {
break;
}
@@ -3504,7 +3659,7 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
snapshot: WorkspaceGitRuntimeSnapshot,
options?: { forceEmit?: boolean },
): void {
if (target.closed || this.workspaceTargets.get(target.cwd) !== target) {
if (target.closed || target.folderMissing || this.workspaceTargets.get(target.cwd) !== target) {
return;
}
target.latestSnapshot = snapshot;
@@ -3660,21 +3815,25 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
}
private removeWorkspaceTarget(target: WorkspaceGitTarget): void {
if (target.repoGitRoot) {
const repoTarget = this.repoTargets.get(target.repoGitRoot);
repoTarget?.workspaceKeys.delete(target.cwd);
if (repoTarget && repoTarget.workspaceKeys.size === 0) {
this.closeRepoTarget(repoTarget);
this.repoTargets.delete(target.repoGitRoot);
} else if (repoTarget?.cwd === target.cwd) {
repoTarget.cwd = repoTarget.workspaceKeys.values().next().value ?? repoTarget.cwd;
}
}
this.detachWorkspaceFromRepoTarget(target);
this.closeWorkspaceTarget(target);
this.workspaceTargets.delete(target.cwd);
}
private detachWorkspaceFromRepoTarget(target: WorkspaceGitTarget): void {
if (!target.repoGitRoot) {
return;
}
const repoTarget = this.repoTargets.get(target.repoGitRoot);
repoTarget?.workspaceKeys.delete(target.cwd);
if (repoTarget && repoTarget.workspaceKeys.size === 0) {
this.closeRepoTarget(repoTarget);
this.repoTargets.delete(target.repoGitRoot);
} else if (repoTarget?.cwd === target.cwd) {
repoTarget.cwd = repoTarget.workspaceKeys.values().next().value ?? repoTarget.cwd;
}
}
private removeWorkingTreeWatchListener(cwd: string, listener: () => void): void {
const target = this.workingTreeWatchTargets.get(cwd);
if (!target) {
@@ -3719,6 +3878,11 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
private closeWorkspaceTarget(target: WorkspaceGitTarget): void {
target.closed = true;
this.stopWorkspaceObservation(target);
target.listeners.clear();
}
private stopWorkspaceObservation(target: WorkspaceGitTarget): void {
if (target.workingTreeWatchTarget) {
this.removeWorkspaceWorkingTreeLink(target.workingTreeWatchTarget, target.cwd);
target.workingTreeWatchTarget = null;
@@ -3727,12 +3891,12 @@ export class WorkspaceGitServiceImpl implements WorkspaceGitService {
clearTimeout(target.debounceTimer);
target.debounceTimer = null;
}
target.pendingDebounceRequest = null;
if (target.observationReensureTimer) {
clearTimeout(target.observationReensureTimer);
target.observationReensureTimer = null;
}
this.stopForgePrStatusPollForTarget(target);
target.listeners.clear();
}
private closeWorkingTreeWatchTarget(target: WorkingTreeWatchTarget): void {
@@ -1,5 +1,6 @@
import { afterEach, describe, expect, it } from "vitest";
import {
chmodSync,
existsSync,
mkdirSync,
mkdtempSync,
@@ -54,6 +55,7 @@ function makeSources(bundleSource = "export {};\n"): {
mkdirSync(native, { recursive: true });
writeFileSync(path.join(native, "pty.node"), "native");
writeFileSync(path.join(native, "spawn-helper"), "#!/bin/sh\n", { mode: 0o755 });
chmodSync(path.join(native, "spawn-helper"), 0o755);
// Another platform's build is not copied.
mkdirSync(path.join(nodePtyDirectory, "prebuilds", "other-platform"));
return { bundleDirectory, nodePtyDirectory };