diff --git a/docs/agent-lifecycle.md b/docs/agent-lifecycle.md index 8662ee3287..d0d057949f 100644 --- a/docs/agent-lifecycle.md +++ b/docs/agent-lifecycle.md @@ -197,10 +197,11 @@ Claude Code announces subagent lifecycle on the SDK stream (`task_started` / `ta - **Not every announced task belongs in the track.** Task subagents announce as `local_agent` and workflows as `local_workflow`; a backgrounded shell announces as `local_bash` with the same `tool_use_id` shape, and ambient housekeeping sets `skip_transcript`. The Claude provider normalizes a workflow to a generic provider-subagent descriptor titled `Workflow`, using Claude's summary as its description and timeline opener. Shared storage, protocol, and UI do not distinguish it from another provider subagent. - **A task that was never declared gets no descriptor, by any route.** Filtered tasks still emit `task_notification`s carrying a `tool_use_id`, and still emit frames carrying `parent_tool_use_id`. Attributing either produces a descriptor with no identity and a defaulted `running` status — a nameless row that never finishes. Status, presentation updates, and sidechain frames all route through the declaration table. -- **Task ids are session-scoped, not turn-scoped.** Cancelling a turn must not clear the routing table: a backgrounded child settles after the interrupt and needs its descriptor to still exist. Cancellation instead terminalizes the declared children that were running in the foreground, and a later `task_notification` is free to correct that guess. Backgrounded children are identified by `task_updated.patch.is_backgrounded`. +- **Task ids are session-scoped, not turn-scoped.** Cancelling a turn must not clear the routing table: a backgrounded child settles after the interrupt and needs its descriptor to still exist. Cancellation instead terminalizes the declared children that were running in the foreground, and a later `task_notification` is free to correct that guess. Backgrounded children are identified by `is_backgrounded` on `task_started` (set at spawn; a resumed subagent is always background) or, when moved there later, on `task_updated.patch`. - **A resumed task can be announced again with a new `tool_use_id`.** The first Task tool id remains the canonical descriptor and later ids are routing aliases for the same session-scoped task. The resumed prompt is added to that child timeline. - **Effort is only reachable through hooks.** It appears nowhere on the message stream at any depth, and the level Paseo requests is not necessarily the level that runs — a model that does not support it is silently downgraded. A hook firing inside a subagent reports the active post-downgrade level next to its `agent_id`, which is the same id `task_started` calls `task_id`. - **Backgrounded subagents emit no frames carrying `parent_tool_use_id` at all.** Everything keyed off that field sees nothing for one; they are visible only because the task protocol announces them. +- **Only `system/init` and `result` bracket a main-session turn.** Background subagents emit neither on the main stream, while their task frames, their own background jobs' `task_notification`, and `rate_limit_event` arrive with no `parent_tool_use_id`, the same shape as main-session work. `session_state_changed` stays `running` for as long as any background subagent lives (Claude Code 2.1.280), so it cannot open a turn; its `idle` can close one. - **Nested ownership comes from the launching sidechain, not `spawn_depth` alone.** A sidechain's Agent or Bash tool call records the direct owner of that `tool_use_id`; the following `task_started` inherits it. This routes a grandchild descriptor and child-owned background notifications without relying on labels or flattening them into the managed parent. - **On replay, `/subagents/` holds every descendant beside the root.** Resolve the tree one proven generation at a time: the root transcript admits direct children, then each admitted sidechain transcript admits its children by `toolUseId`. `spawnDepth` orders candidates but does not establish ownership. Unresolved sidecars remain excluded as ambient or unrelated work. - **Replay `totalTokens` is a context-size reading, not cumulative spend.** Claude Code finalizes a subagent by summing the _last_ assistant message's usage block and shipping that as `usage.total_tokens`. Summing per-entry usage instead multiplies the cached prefix by the turn count and reports a number several times larger than the live path. diff --git a/docs/providers.md b/docs/providers.md index c5f37d4161..4609246922 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -103,7 +103,7 @@ OpenCode owns user message IDs. Do not pass Paseo-generated IDs to OpenCode prom Active-turn steering is an optional `AgentSession.steerActiveTurn` operation. The manager owns admission against its exact foreground turn, canonical user-message creation, echo reconciliation, and falls back to the normal interrupt-and-replace path only when the adapter reports `unavailable`. An adapter error leaves the steer's fate ambiguous and must surface without an interrupt or retry. Codex calls `turn/steer` with the native expected turn and Paseo client user-message ID. Claude pushes an admitted steer into the exact active SDK query input; isolated control commands remain unavailable. OpenCode calls `session/prompt_async` with an OpenCode-generated message ID; the server queues the prompt while busy and the next LLM call in the same Paseo turn includes it. Pi sends its native `steer` RPC, which queues the message for delivery after the in-flight assistant turn's tool calls. Slash-command inputs report `unavailable` because pi rejects extension commands on the steer path, and echo identity is correlated by message text because pi's steer RPC takes no message ID. A missing session reports `unavailable` and uses the normal interrupt fallback. -A steering adapter also owes its interrupt: stopping a turn must discard the steers the provider has not read yet, or one of them resumes the turn the user just stopped. Codex clears pending input when it aborts a turn; Claude does not, so its adapter cancels the SDK messages it queued before calling `query.interrupt()`. Pi requires `clear_queue` before `abort`; older binaries without that RPC retain their native queue behavior until the pi compatibility floor reaches 0.84.4. +A steering adapter also owes its interrupt: stopping a turn must discard the steers the provider has not read yet, or one of them resumes the turn the user just stopped. Codex clears pending input when it aborts a turn; Claude does not, so its adapter cancels the SDK messages it queued before calling `query.interrupt()`. It calls `query.interrupt()` only while a main-session turn is in flight (between `system/init` and that turn's `result`): with the main session idle, an interrupt reaches only background subagents and no result follows it, so the adapter withdraws its unstarted messages and stops there. The adapter also declares `perTaskStopAffordance`, because without it the CLI kills every background subagent on interrupt. Pi requires `clear_queue` before `abort`; older binaries without that RPC retain their native queue behavior until the pi compatibility floor reaches 0.84.4. `SteerActiveTurnOptions.clearPendingPermissions` makes permission release part of the provider contract. A provider that accepts such a steer queues it first, denies permissions blocking its delivery, and stops once the steer is read. Steers without the flag leave permissions open. A denied plan remains in the timeline because the pending card was the only other copy of its text. diff --git a/packages/app/src/runtime/host-runtime.test.ts b/packages/app/src/runtime/host-runtime.test.ts index 0954b0e422..996a47d498 100644 --- a/packages/app/src/runtime/host-runtime.test.ts +++ b/packages/app/src/runtime/host-runtime.test.ts @@ -3047,6 +3047,34 @@ describe("HostRuntimeStore", () => { useSessionStore.getState().clearSession(host.serverId); }); + it("drains a queued message as a steer so a turn the agent just started is not interrupted", async () => { + const host = makeHost({ serverId: "srv_steer_queue_drain" }); + const fakeClient = new FakeDaemonClient(); + const store = new HostRuntimeStore({ + deps: { + createClient: () => fakeClient as unknown as DaemonClient, + connectToDaemon: async () => ({ + client: fakeClient as unknown as DaemonClient, + serverId: host.serverId, + hostname: null, + }), + getClientId: async () => "cid_steer_queue_drain", + }, + }); + const sessionStore = useSessionStore.getState(); + sessionStore.initializeSession(host.serverId, fakeClient as unknown as DaemonClient, 1); + sessionStore.setQueuedMessages( + host.serverId, + new Map([["agent", [{ id: "queued", text: "next task", attachments: [] }]]]), + ); + + store.drainQueuedAgentMessage(host.serverId, "agent"); + await fakeClient.waitForSentMessages(1); + + expect(fakeClient.sentAgentMessages[0]?.[2]?.activeTurnBehavior).toBe("steer"); + sessionStore.clearSession(host.serverId); + }); + it("uses legacy GitHub attachments when draining a queue for an old daemon", async () => { const host = makeHost({ serverId: "srv_legacy_queue_attachment" }); const fakeClient = new FakeDaemonClient(); diff --git a/packages/app/src/runtime/host-runtime.ts b/packages/app/src/runtime/host-runtime.ts index de4b40f883..2ecbe60448 100644 --- a/packages/app/src/runtime/host-runtime.ts +++ b/packages/app/src/runtime/host-runtime.ts @@ -2426,6 +2426,11 @@ export class HostRuntimeStore { }), encodeImages, submission: createMessageSubmissionWriter(serverId), + // The queue drains when the agent goes idle, but the agent can start a turn of its own + // in that same moment (a wake-up after background work). Steering folds the message + // into that turn instead of interrupting it; with no turn running, the daemon starts + // a normal one. + activeTurnBehavior: "steer", }); }, }) diff --git a/packages/server/src/server/agent/providers/claude/agent.env.test.ts b/packages/server/src/server/agent/providers/claude/agent.env.test.ts index c99687e438..0d2748fd62 100644 --- a/packages/server/src/server/agent/providers/claude/agent.env.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.env.test.ts @@ -31,6 +31,7 @@ function createQueryMock(events: unknown[]): Query { describe("Claude SDK env", () => { test("forwards launch-context env through Claude process env", async () => { let capturedEnv: Record | undefined; + let capturedPerTaskStopAffordance: boolean | undefined; const launchContext: AgentLaunchContext = { env: { PASEO_AGENT_ID: "00000000-0000-4000-8000-000000000201", @@ -39,6 +40,7 @@ describe("Claude SDK env", () => { }; const queryFactory = vi.fn(({ options }: ClaudeQueryInput) => { capturedEnv = options.env; + capturedPerTaskStopAffordance = options.perTaskStopAffordance; return createQueryMock([ { type: "system", @@ -91,6 +93,10 @@ describe("Claude SDK env", () => { expect(capturedEnv?.MCP_TIMEOUT).toBe("claude-startup-timeout"); expect(capturedEnv?.MCP_TOOL_TIMEOUT).toBe("claude-tool-timeout"); expect(session.usageSession?.()?.env).toBe(capturedEnv); + // Paseo reads session_state_changed to know when an autonomous turn is over. + expect(capturedEnv?.CLAUDE_CODE_EMIT_SESSION_STATE_EVENTS).toBe("1"); + // Without it, Stop and replace kill every background subagent along with the turn. + expect(capturedPerTaskStopAffordance).toBe(true); } finally { await session.close(); expect(session.usageSession?.()).toBeNull(); diff --git a/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts b/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts index dada35945c..02a09ba44b 100644 --- a/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts @@ -91,6 +91,17 @@ function buildUsage() { }; } +/** Claude opening a turn of its own. Every main-session turn starts with one. */ +function buildTurnInit(sessionId: string) { + return { + type: "system", + subtype: "init", + session_id: sessionId, + permissionMode: "default", + model: "opus", + }; +} + function buildSuccessResult(sessionId: string) { return { type: "result", @@ -153,13 +164,7 @@ function createScriptedQuery(params: { }, } satisfies ScriptedQuery; - scriptedQuery.emit({ - type: "system", - subtype: "init", - session_id: params.sessionId, - permissionMode: "default", - model: "opus", - }); + scriptedQuery.emit(buildTurnInit(params.sessionId)); void (async () => { for await (const prompt of params.prompt) { @@ -252,6 +257,21 @@ async function waitFor( } } +/** + * Paseo interrupts Claude only while a main-session turn is in flight, which it learns from the + * init. The pump asks for its next frame only after routing the last one, so a second next() call + * proves the init was read. + */ +async function waitForInitRouted( + query: () => { next: ReturnType } | null | undefined, +) { + await waitFor(() => (query()?.next.mock.calls.length ?? 0) >= 2); +} + +function buildCommandLifecycle(commandUuid: string | null | undefined, state: string) { + return { type: "command_lifecycle", command_uuid: commandUuid, state }; +} + afterEach(() => { queryFactory.mockReset(); }); @@ -282,6 +302,7 @@ test("interrupt only calls query.interrupt and leaves the query open", async () const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); await waitFor(() => queries[0]?.prompts.length === 1); + await waitForInitRouted(() => queries[0]); await session.interrupt(); await waitFor(() => queries[0]?.interrupt.mock.calls.length === 1); @@ -322,6 +343,7 @@ async function startSteeredTurn(sessionId: string): Promise<{ throw new Error("Expected the original Claude turn to start"); } await waitFor(() => query?.prompts.length === 1); + await waitForInitRouted(() => query); const steered = await session.steerActiveTurn!("queued steer", { expectedTurnId: start.value.turnId, @@ -371,11 +393,8 @@ test("interrupt still stops the turn when Claude has already dequeued the steer" test("a steer Claude has already read is no longer discardable on interrupt", async () => { const { session, query, turn } = await startSteeredTurn("queued-steer-completed-session"); - query()?.emit({ - type: "command_lifecycle", - command_uuid: query()?.prompts[1]?.uuid, - state: "completed", - }); + query()?.emit(buildCommandLifecycle(query()?.prompts[0]?.uuid, "started")); + query()?.emit(buildCommandLifecycle(query()?.prompts[1]?.uuid, "completed")); // Frames are translated in order, so the marker landing proves the lifecycle frame was read. query()?.emit({ type: "assistant", message: { content: "STEER_READ" } }); await consumeUntil(turn, (event) => collectAssistantText([event]).includes("STEER_READ")); @@ -388,6 +407,23 @@ test("a steer Claude has already read is no longer discardable on interrupt", as await session.close(); }); +test("a steer Claude cancelled is no longer withdrawn on interrupt", async () => { + const { session, query, turn } = await startSteeredTurn("queued-steer-cancelled-session"); + + query()?.emit(buildCommandLifecycle(query()?.prompts[0]?.uuid, "started")); + // "cancelled" is terminal: API errors, interrupts and "now" pre-emption all end a message. + query()?.emit(buildCommandLifecycle(query()?.prompts[1]?.uuid, "cancelled")); + query()?.emit({ type: "assistant", message: { content: "STEER_CANCELLED" } }); + await consumeUntil(turn, (event) => collectAssistantText([event]).includes("STEER_CANCELLED")); + + await session.interrupt(); + await waitFor(() => query()?.interrupt.mock.calls.length === 1); + + expect(query()?.cancelAsyncMessage).not.toHaveBeenCalled(); + await collectUntilTerminal(turn); + await session.close(); +}); + function buildStoppedTaskNotification(sessionId: string) { return { type: "system", @@ -473,6 +509,7 @@ async function startInterruptedToolTurn(sessionId: string): Promise<{ const turn = streamSession(session, "run the slow tool"); await turn.next(); await waitFor(() => query?.prompts.length === 1); + await waitForInitRouted(() => query); await session.interrupt(); await collectUntilTerminal(turn); @@ -523,6 +560,7 @@ test("Claude can still wake into an autonomous turn once the interrupted request query()?.emit(buildStoppedTaskNotification(sessionId)); query()?.emit(buildAbortedResult(sessionId)); + query()?.emit(buildTurnInit(sessionId)); query()?.emit({ type: "assistant", message: { content: "AUTONOMOUS_WAKE_RESPONSE" }, @@ -579,6 +617,7 @@ test("reuses the existing query after interrupt before starting the next prompt" const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); await waitFor(() => queries[0]?.prompts.length === 1); + await waitForInitRouted(() => queries[0]); await session.interrupt(); await collectUntilTerminal(firstTurn); @@ -652,6 +691,7 @@ test("recovers when the query pump sees a single interrupt abort before the next const output = createAsyncQueue>(); const prompts: PromptRecord[] = []; let throwAbortOnNext = false; + let queryRef: Pick | null = null; queryFactory.mockImplementation(({ prompt }: { prompt: AsyncIterable }) => { const scriptedQuery = { @@ -685,13 +725,8 @@ test("recovers when the query pump sees a single interrupt abort before the next }, } satisfies ScriptedQuery; - scriptedQuery.emit({ - type: "system", - subtype: "init", - session_id: "interrupt-abort-recovery-session", - permissionMode: "default", - model: "opus", - }); + queryRef = scriptedQuery; + scriptedQuery.emit(buildTurnInit("interrupt-abort-recovery-session")); void (async () => { for await (const promptMessage of prompt) { @@ -730,6 +765,7 @@ test("recovers when the query pump sees a single interrupt abort before the next const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); + await waitForInitRouted(() => queryRef); await session.interrupt(); await collectUntilTerminal(firstTurn); @@ -768,6 +804,7 @@ test("stale abort result after replacement start does not poison the new foregro const firstTurn = streamSession(session, "first prompt"); const firstStarted = await firstTurn.next(); await waitFor(() => queryRef?.prompts.length === 1); + await waitForInitRouted(() => queryRef); await session.interrupt(); const firstTurnEvents = [firstStarted.value!, ...(await collectUntilTerminal(firstTurn))]; @@ -812,7 +849,7 @@ test("stale abort result after replacement start does not poison the new foregro await session.close(); }); -test("creates an autonomous live turn when assistant output arrives without a foreground run", async () => { +test("creates an autonomous live turn when Claude starts a turn without a foreground run", async () => { const logger = createTestLogger(); let queryRef: ScriptedQuery | null = null; @@ -848,6 +885,7 @@ test("creates an autonomous live turn when assistant output arrives without a fo await collectUntilTerminal(streamSession(session, "seed prompt")); const subscribedEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-live-session")); queryRef?.emit({ type: "assistant", message: { content: "AUTONOMOUS_WAKE_RESPONSE" }, @@ -916,6 +954,7 @@ test("steers an autonomous turn through its existing query without restarting it await collectUntilTerminal(streamSession(session, "seed prompt")); const autonomousEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-steer-session")); queryRef?.emit({ type: "assistant", message: { content: "AUTONOMOUS_RESPONSE" }, @@ -999,6 +1038,7 @@ test("auto-completes an open autonomous turn when a foreground prompt starts", a await collectUntilTerminal(streamSession(session, "seed prompt")); const subscribedEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-handoff-session")); queryRef?.emit({ type: "assistant", message: { content: "BACKGROUND_ONLY_RESPONSE" }, diff --git a/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts b/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts index 747a466c32..5393926520 100644 --- a/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts @@ -486,6 +486,7 @@ test("interruptActiveTurn only interrupts the active query without info logs", a } | null; input: { end: () => void } | null; queryRestartNeeded: boolean; + mainTurnInFlight: boolean; interruptActiveTurn: () => Promise; } = asInternals(session); const interrupt = vi.fn(async () => undefined); @@ -498,6 +499,7 @@ test("interruptActiveTurn only interrupts the active query without info logs", a }; internal.input = { end }; internal.queryRestartNeeded = false; + internal.mainTurnInFlight = true; try { await internal.interruptActiveTurn(); @@ -1191,6 +1193,7 @@ test("reuses one autonomous run for unbound stream_event bursts with no foregrou const queryMock = createBaseQueryMock(vi.fn(async () => ({ done: true, value: undefined }))); internal.turnState = "idle"; + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); await internal.routeSdkMessageFromPump( { type: "stream_event", @@ -1233,6 +1236,468 @@ test("reuses one autonomous run for unbound stream_event bursts with no foregrou await session.close(); }); +interface SessionStateInternals { + turnState: "idle" | "foreground" | "autonomous"; + routeSdkMessageFromPump: ( + message: Record, + activeQuery: QueryMock, + ) => Promise; + autonomousTurn: { id: string } | null; + activeForegroundTurnId: string | null; +} + +function sessionState(state: "idle" | "running" | "requires_action") { + return { + type: "system", + subtype: "session_state_changed", + state, + uuid: `state-${state}`, + session_id: "sess-1", + }; +} + +/** Claude opening a turn of its own. Every main-session turn starts with one. */ +function claudeTurnInit() { + return { + type: "system", + subtype: "init", + session_id: "sess-1", + permissionMode: "default", + model: "opus", + }; +} + +function successResult() { + return { type: "result", subtype: "success", usage: buildUsage(), total_cost_usd: 0 }; +} + +function commandLifecycle( + commandUuid: string, + state: "queued" | "started" | "completed" | "cancelled", +) { + return { type: "command_lifecycle", command_uuid: commandUuid, state }; +} + +function unboundTextDelta(text: string) { + return { + type: "stream_event", + parent_tool_use_id: null, + event: { type: "content_block_delta", delta: { type: "text_delta", text } }, + }; +} + +// What a background subagent emits on the main stream with no parent tool use id, so by frame +// shape alone each looks like main-session work. Verified on the wire: none arrives inside a turn +// Claude opened, and no result follows any of them. +const helperTaskStarted = { + type: "system", + subtype: "task_started", + task_id: "helper-task", + tool_use_id: "toolu_helper", + task_type: "local_agent", + subagent_type: "general-purpose", + description: "Background helper", + is_backgrounded: true, + uuid: "uuid-task-started", + session_id: "sess-1", +}; +const helperTaskProgress = { + type: "system", + subtype: "task_progress", + task_id: "helper-task", + tool_use_id: "toolu_helper", + description: "Background helper", + usage: { total_tokens: 1200, tool_uses: 3, duration_ms: 900 }, + last_tool_name: "Bash", + uuid: "uuid-task-progress", + session_id: "sess-1", +}; +// A subagent's own background Bash job completing. +const subagentOwnedTaskNotification = { + type: "system", + subtype: "task_notification", + task_id: "bg-job-of-subagent", + tool_use_id: "toolu_subagent_bash", + status: "completed", + summary: "Background command completed", + output_file: "/tmp/bg-job-of-subagent.output", + uuid: "uuid-task-notification", + session_id: "sess-1", +}; +const helperCompleted = { + type: "system", + subtype: "task_notification", + task_id: "helper-task", + tool_use_id: "toolu_helper", + status: "completed", + summary: "Background helper finished", + output_file: "/tmp/helper-task.output", + uuid: "uuid-helper-completed", + session_id: "sess-1", +}; +const rateLimitEvent = { + type: "rate_limit_event", + rate_limit_info: { status: "allowed" }, + uuid: "uuid-rate-limit", + session_id: "sess-1", +}; + +function createPendingQueryMock(): QueryMock { + return createBaseQueryMock(vi.fn(() => new Promise(() => {}))); +} + +test("a main-session frame opens no turn until Claude starts one", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + // Main-session output only ever arrives inside a turn Claude opened with system/init, so a + // stray unbound frame is not the start of one. + await internal.routeSdkMessageFromPump(unboundTextDelta("stray"), queryMock); + expect(internal.autonomousTurn).toBeNull(); + + // The wake-up case autonomous turns exist for: a completed background subagent makes the main + // session start a turn of its own, with no prompt from the user. + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(unboundTextDelta("picking this up"), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + + await session.close(); +}); + +test("a background subagent's frames never open a turn while the main session is idle", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + // Claude Code 2.1.280 reports "running" for as long as any background subagent is alive. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + for (const frame of [ + helperTaskStarted, + helperTaskProgress, + subagentOwnedTaskNotification, + rateLimitEvent, + helperCompleted, + ]) { + await internal.routeSdkMessageFromPump(frame, queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + } + + // The next user message starts its own foreground turn rather than landing on a phantom one. + const { turnId } = await session.startTurn("PING"); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("foreground"); + + await session.close(); +}); + +test("Claude waking up after a helper finishes gets an autonomous turn that its result closes", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // The helper's completion arrives outside any turn. Claude then starts one to react to it. + await internal.routeSdkMessageFromPump(helperTaskStarted, queryMock); + await internal.routeSdkMessageFromPump(helperCompleted, queryMock); + expect(internal.autonomousTurn).toBeNull(); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + expect(internal.turnState).toBe("autonomous"); + + await internal.routeSdkMessageFromPump(unboundTextDelta("the helper is done"), queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_started")).toHaveLength(1); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(1); + + await session.close(); +}); + +test("a steer folded into a turn keeps one turn, and a turn Claude starts after it gets its own", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + const { turnId } = await session.startTurn("hello"); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + // A message pushed mid-turn is folded in at the next tool boundary: Claude acknowledges it, but + // starts no new turn. + await internal.routeSdkMessageFromPump(commandLifecycle("folded-steer", "queued"), queryMock); + await internal.routeSdkMessageFromPump(commandLifecycle("folded-steer", "started"), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(internal.turnState).toBe("idle"); + + // A message that arrived while Claude wrote its final answer runs as a turn of its own. + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-2"); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_started")).toHaveLength(2); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(2); + + await session.close(); +}); + +test("session state never opens a turn", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // "running" covers background subagents too, so it says nothing about the main session. + for (const state of ["idle", "running", "requires_action", "idle"] as const) { + await internal.routeSdkMessageFromPump(sessionState(state), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + } + // State frames are bookkeeping, never timeline content. + expect(events).toEqual([]); + + await session.close(); +}); + +test("an idle session state closes an autonomous turn that no result closed", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + // Safety net: Claude says nothing at all is running any more. + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(1); + + await session.close(); +}); + +test("session state never opens or closes a foreground turn", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + const { turnId } = await session.startTurn("hello"); + + // The send's own running, then an idle that trails the previous turn and lands late. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("foreground"); + + await internal.routeSdkMessageFromPump( + { type: "result", subtype: "success", usage: buildUsage(), total_cost_usd: 0 }, + queryMock, + ); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(internal.turnState).toBe("idle"); + + await session.close(); +}); + +interface InterruptInternals extends SessionStateInternals { + pendingInterruptAbort: boolean; +} + +type InterruptibleQueryMock = QueryMock & { cancelAsyncMessage: ReturnType }; + +function createInterruptibleQueryMock(): InterruptibleQueryMock { + return Object.assign(createPendingQueryMock(), { + cancelAsyncMessage: vi.fn(async () => true), + }); +} + +/** The uuid Paseo stamped on the first message it pushed into the SDK input. */ +async function readFirstPromptUuid(): Promise { + const input = sdkQueryFactory.mock.calls[0]?.[0] as { prompt: AsyncIterable }; + return await createPromptUuidReader(input.prompt)(); +} + +function abortedResult() { + return { + type: "result", + subtype: "error_during_execution", + errors: [], + usage: buildUsage(), + total_cost_usd: 0, + }; +} + +test("an interrupt before Claude starts the message withdraws it instead of interrupting", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await session.startTurn("hello"); + // Claude has not acknowledged the message yet: no init, no command_lifecycle "started". + await session.interrupt(); + + await vi.waitFor(() => expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(await readFirstPromptUuid()); + expect(queryMock.interrupt).not.toHaveBeenCalled(); + // No result follows an interrupt that never happened, so no flag may wait for one. + expect(internal.pendingInterruptAbort).toBe(false); + expect(events.filter((event) => event.type === "turn_canceled")).toHaveLength(1); + + await session.close(); +}); + +test("an interrupt after Claude took the message but before it started stops it", async () => { + const queryMock = createInterruptibleQueryMock(); + // Claude has dequeued the message for its next turn, so withdrawing it is a no-op. + queryMock.cancelAsyncMessage.mockImplementation(async () => false); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + + await session.startTurn("hello"); + // No command_lifecycle "started" or init has reached Paseo yet. + await session.interrupt(); + + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(await readFirstPromptUuid()); + expect(internal.pendingInterruptAbort).toBe(true); + + await session.close(); +}); + +test("a prompt sent while a stopped prompt is being withdrawn is not withdrawn too", async () => { + const queryMock = createInterruptibleQueryMock(); + let settleWithdrawal: (cancelled: boolean) => void = () => {}; + queryMock.cancelAsyncMessage.mockImplementationOnce( + () => new Promise((resolve) => (settleWithdrawal = resolve)), + ); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + + await session.startTurn("first"); + const firstUuid = await readFirstPromptUuid(); + await session.interrupt(); + await vi.waitFor(() => expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1)); + + await session.startTurn("second"); + settleWithdrawal(false); + + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(firstUuid); + + await session.close(); +}); + +test("an interrupt while only background subagents run never reaches Claude", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + + await session.startTurn("delegate to a helper"); + const promptUuid = await readFirstPromptUuid(); + await internal.routeSdkMessageFromPump(commandLifecycle(promptUuid ?? "", "started"), queryMock); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(helperTaskStarted, queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + // The main session is done; its helper keeps working. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + await internal.routeSdkMessageFromPump(helperTaskProgress, queryMock); + + // Without perTaskStopAffordance, an interrupt here kills every running helper. + await session.interrupt(); + + expect(queryMock.interrupt).not.toHaveBeenCalled(); + expect(queryMock.cancelAsyncMessage).not.toHaveBeenCalled(); + expect(internal.pendingInterruptAbort).toBe(false); + + await session.close(); +}); + +test("an interrupt during a main turn interrupts Claude once and drops the aborted result", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await session.startTurn("long task"); + const promptUuid = await readFirstPromptUuid(); + await internal.routeSdkMessageFromPump(commandLifecycle(promptUuid ?? "", "started"), queryMock); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + + await session.interrupt(); + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + // Claude already started the message, so there is nothing to withdraw. + expect(queryMock.cancelAsyncMessage).not.toHaveBeenCalled(); + expect(internal.pendingInterruptAbort).toBe(true); + + // A replacement starts before the aborted turn reports its result. + const { turnId } = await session.startTurn("replacement"); + await internal.routeSdkMessageFromPump(abortedResult(), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.pendingInterruptAbort).toBe(false); + expect(events.filter((event) => event.type === "turn_failed")).toHaveLength(0); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(queryMock.interrupt).toHaveBeenCalledTimes(1); + + await session.close(); +}); + +test("a new turn clears an interrupt flag no result will consume", async () => { + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // An interrupt that landed just after its turn had ended: Claude emits no result for it. + internal.pendingInterruptAbort = true; + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.pendingInterruptAbort).toBe(false); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + // This turn's own failure is reported, not swallowed as the interrupted turn's leftover. + await internal.routeSdkMessageFromPump( + { ...abortedResult(), errors: ["API Error: 500 upstream failure"] }, + queryMock, + ); + expect(events.filter((event) => event.type === "turn_failed")).toHaveLength(1); + expect(internal.autonomousTurn).toBeNull(); + + await session.close(); +}); + test("tracks run lifecycle transitions for success, error, and interrupt", async () => { const session = await createSession(); let streamCase: "success" | "error" | "interrupt" = "success"; diff --git a/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts b/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts index d4fc0bcc0c..1223bbc541 100644 --- a/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts @@ -78,7 +78,7 @@ describe("subagent lifecycle across an interrupt", () => { queryFactory.mockReset(); }); - async function startInterruptibleTurn() { + async function startInterruptibleTurn(taskStarted: Record = {}) { const channel = buildOpenQueryMock(); queryFactory.mockImplementation(() => channel.query); const session = await new ClaudeAgentClient({ @@ -107,6 +107,7 @@ describe("subagent lifecycle across an interrupt", () => { task_type: "local_agent", subagent_type: "general-purpose", description: "Reply with banana", + ...taskStarted, }); await vi.waitFor(() => expect(subagentEvents.length).toBeGreaterThan(0)); @@ -169,6 +170,31 @@ describe("subagent lifecycle across an interrupt", () => { await session.close(); }); + test("a child spawned in the background is not canceled with the turn that spawned it", async () => { + // Claude announces a background spawn on task_started itself and sends no task_updated patch. + const { channel, session, subagentEvents } = await startInterruptibleTurn({ + is_backgrounded: true, + }); + + await session.interrupt(); + channel.push({ + type: "system", + subtype: "task_notification", + task_id: TASK_ID, + tool_use_id: SUBAGENT_ID, + status: "completed", + }); + + await vi.waitFor(() => + expect(subagentEvents).toContainEqual( + expect.objectContaining({ id: SUBAGENT_ID, status: "completed" }), + ), + ); + expect(statuses(subagentEvents, SUBAGENT_ID)).toEqual(["running", "completed"]); + + await session.close(); + }); + test("a foreground child is canceled with the turn that spawned it", async () => { const { session, subagentEvents } = await startInterruptibleTurn(); diff --git a/packages/server/src/server/agent/providers/claude/agent.test.ts b/packages/server/src/server/agent/providers/claude/agent.test.ts index d88d8105cf..2d970c49de 100644 --- a/packages/server/src/server/agent/providers/claude/agent.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.test.ts @@ -3160,10 +3160,15 @@ describe("ClaudeAgentSession context window usage", () => { }); test("a compaction abandoned in an autonomous turn does not suppress the next marker", async () => { - // Trailing output after the foreground result opens an autonomous turn, which starts - // compacting and is then ended by the next foreground turn, never reaching a boundary. + // Claude starts a turn of its own after the foreground result, which starts compacting and is + // then ended by the next foreground turn, never reaching a boundary. const session = await createSessionForTurns([ - [createSuccessResult(), createMessageStartEvent(), createCompactingStatus()], + [ + createSuccessResult(), + createInitMessage(), + createMessageStartEvent(), + createCompactingStatus(), + ], [createCompactingStatus(), createSuccessResult()], ]); diff --git a/packages/server/src/server/agent/providers/claude/agent.ts b/packages/server/src/server/agent/providers/claude/agent.ts index ddac56a0eb..d505689a95 100644 --- a/packages/server/src/server/agent/providers/claude/agent.ts +++ b/packages/server/src/server/agent/providers/claude/agent.ts @@ -452,6 +452,15 @@ function errorToMessageString(error: unknown): string { return ""; } +/** + * Claude opens every main-session turn with system/init and closes it with exactly one result: + * a prompt, a wake-up after background work, a slash command, an API error. Background subagents + * never emit either on the main stream. + */ +function isClaudeTurnInit(message: SDKMessage): boolean { + return message.type === "system" && message.subtype === "init"; +} + function firstStringField( input: Record, primaryKey: string, @@ -2070,11 +2079,12 @@ class ClaudeAgentSession implements AgentSession { private activeForegroundQuery: Query | null = null; private activeForegroundInput: AsyncMessageInput | null = null; /** - * Steers pushed into the live SDK input that Claude may not have read yet. Interrupting the turn - * has to discard them, or the SDK dequeues one and resumes the turn we just stopped. SDK user - * UUIDs are provider-private; never let them escape the adapter boundary. + * Messages pushed into the live SDK input (steers, and a foreground turn's own prompt) whose + * command_lifecycle has not reached "started". Interrupting has to withdraw them, or the SDK + * dequeues one and resumes the turn we just stopped. SDK user UUIDs are provider-private; never + * let them escape the adapter boundary. */ - private readonly queuedSteerUuids = new Set(); + private readonly unstartedMessageUuids = new Set(); /** Human steers whose text has not reached Claude yet and therefore supersede blocking cards. */ private readonly permissionClearingSteerUuids = new Set(); private claudeSessionId: string | null; @@ -2126,6 +2136,12 @@ class ClaudeAgentSession implements AgentSession { private queryPumpPromise: Promise | null = null; private queryRestartNeeded = false; private pendingInterruptAbort = false; + /** + * Whether Claude's main session is inside one of its own turns: set by system/init and by a + * pushed message's command_lifecycle "started" (/compact works for seconds before its init), + * cleared by the turn's result or an idle session state. See observeMainTurnBracket. + */ + private mainTurnInFlight = false; private foregroundHasVisibleActivity = false; private activeTurnHasAssistantText = false; private readonly contextUsage: ClaudeContextUsageState; @@ -2272,6 +2288,7 @@ class ClaudeAgentSession implements AgentSession { this.contextUsage.beginTurn(); this.transitionTurnState("foreground", "foreground turn started"); this.clearRecentStderr(); + if (sdkUserMessageId) this.unstartedMessageUuids.add(sdkUserMessageId); let cancelIssued = false; const requestCancel = () => { @@ -2282,13 +2299,18 @@ class ClaudeAgentSession implements AgentSession { if (this.cancelCurrentTurn === requestCancel) { this.cancelCurrentTurn = null; } + // Whether Claude has this message in hand. A late idle from the previous turn can clear + // mainTurnInFlight after Claude started it, so the message's own lifecycle counts too. + const claudeStartedTurn = + this.mainTurnInFlight || + (sdkUserMessageId !== null && !this.unstartedMessageUuids.has(sdkUserMessageId)); this.rejectAllPendingPermissions(new Error("Permission request canceled")); this.finishForegroundTurn({ type: "turn_canceled", provider: "claude", reason: "Interrupted", }); - void this.interruptActiveTurn().catch((error) => { + void this.interruptActiveTurn(claudeStartedTurn).catch((error) => { this.logger.warn({ err: error }, "Failed to interrupt during cancel"); }); }; @@ -2298,6 +2320,11 @@ class ClaudeAgentSession implements AgentSession { try { await this.ensureQuery(); + if (cancelIssued) { + // Stopped while Claude was still starting up: withdrawn by never sending it. + if (sdkUserMessageId) this.unstartedMessageUuids.delete(sdkUserMessageId); + return { turnId }; + } if (!this.input) { throw new Error("Claude session input stream not initialized"); } @@ -2311,6 +2338,7 @@ class ClaudeAgentSession implements AgentSession { } }, 0); } catch (error) { + if (sdkUserMessageId) this.unstartedMessageUuids.delete(sdkUserMessageId); this.finishForegroundTurn( this.buildTurnFailedEvent(error instanceof Error ? error.message : "Claude stream failed"), ); @@ -2359,7 +2387,7 @@ class ClaudeAgentSession implements AgentSession { clearPendingPermissions: boolean, ): void { const uuid = message.uuid; - if (uuid) this.queuedSteerUuids.add(uuid); + if (uuid) this.unstartedMessageUuids.add(uuid); if (uuid && clearPendingPermissions) { this.permissionClearingSteerUuids.add(uuid); } @@ -2370,7 +2398,7 @@ class ClaudeAgentSession implements AgentSession { } } catch (error) { if (uuid) { - this.queuedSteerUuids.delete(uuid); + this.unstartedMessageUuids.delete(uuid); this.permissionClearingSteerUuids.delete(uuid); } throw error; @@ -3182,6 +3210,8 @@ class ClaudeAgentSession implements AgentSession { const options = await this.buildOptions(launchMode); this.logger.debug({ options: summarizeClaudeOptionsForLog(options) }, "claude query"); this.input = input; + // A fresh Claude process has no turn of its own in flight. + this.mainTurnInFlight = false; this.query = claudeQuery( { prompt: input.iterable, options }, { @@ -3301,7 +3331,12 @@ class ClaudeAgentSession implements AgentSession { return createProviderEnv({ baseEnv: process.env, runtimeSettings: this.runtimeSettings, - overlays: [this.launchEnv], + overlays: [ + this.launchEnv, + // Makes the CLI announce session_state_changed, whose "idle" is the safety net that closes + // an autonomous turn (see consumeSessionStateChange). + { CLAUDE_CODE_EMIT_SESSION_STATE_EVENTS: "1" }, + ], }); } @@ -3368,6 +3403,10 @@ class ClaudeAgentSession implements AgentSession { ...settingsOptions, // Provider subagent panes render the child's nested transcript. forwardSubagentText: true, + // Stop and replace abort only the main turn, like Esc in Claude Code, and background + // subagents keep running. Without this declaration an interrupt kills every one of them. + // Claude stops a helper itself with its TaskStop tool. + perTaskStopAffordance: true, hooks: this.buildSubagentEffortHooks(), ...(this.persistSession === undefined ? {} : { persistSession: this.persistSession }), env: sdkEnv, @@ -3898,35 +3937,67 @@ class ClaudeAgentSession implements AgentSession { return false; } - private isAssistantishMessage(message: SDKMessage): boolean { - return ( - message.type === "assistant" || - message.type === "stream_event" || - message.type === "tool_progress" || - (message.type === "system" && message.subtype === "task_notification") - ); + /** + * Claude's account of whether anything in the session is working, which is not the same as the + * main session working: Claude Code 2.1.280 stays "running" for as long as any background + * subagent is alive. So "running" never opens a turn. + * + * "idle" means nothing at all is running, so it closes an autonomous turn whatever opened it. + * It never closes a foreground turn: an idle can trail the previous turn and land just after a + * new send, and it must not end that send. + * + * Returns true when the frame was a state change, which is bookkeeping and never reaches the + * timeline. + */ + private consumeSessionStateChange(message: SDKMessage): boolean { + if (message.type !== "system" || message.subtype !== "session_state_changed") { + return false; + } + if (message.state === "idle") { + this.mainTurnInFlight = false; + this.completeAutonomousTurn(); + } + return true; + } + + /** Tracks mainTurnInFlight from Claude's own turn brackets (isClaudeTurnInit). */ + private observeMainTurnBracket(message: SDKMessage): void { + if (message.type === "result") { + this.mainTurnInFlight = false; + return; + } + if (isClaudeTurnInit(message)) { + this.mainTurnInFlight = true; + // A new turn has begun, so the interrupted one is over. An interrupt that landed after its + // turn had already ended gets no result, and the leftover flag would swallow this turn's. + this.pendingInterruptAbort = false; + return; + } + if (readClaudeCommandLifecycle(message)?.state === "started") { + this.mainTurnInFlight = true; + } } /** - * Claude keeps talking about the request it was told to kill — the notification for the tool it - * just stopped, trailing assistant output. That is not new work, so it must not open a turn: the - * only message that would close that turn is the result shouldSuppressStaleResult drops, leaving - * the agent stuck reporting "running" forever. + * Only the main session's own turn start opens an autonomous turn. Everything a background + * subagent emits (its task frames, its own background jobs' notifications, rate limit events) + * arrives outside one, and no result would ever close a turn it opened. */ - private shouldStartAutonomousTurn(message: SDKMessage): boolean { - if (this.activeForegroundTurnId || this.pendingInterruptAbort) { - return false; - } - return this.isAssistantishMessage(message); + private opensAutonomousTurn(message: SDKMessage): boolean { + return isClaudeTurnInit(message) && !this.activeForegroundTurnId; } private async routeSdkMessageFromPump(message: SDKMessage): Promise { + if (this.consumeSessionStateChange(message)) { + return; + } + this.observeMainTurnBracket(message); if (this.shouldSuppressStaleResult(message)) { return; } const isForeground = Boolean(this.activeForegroundTurnId); - if (this.shouldStartAutonomousTurn(message)) { + if (this.opensAutonomousTurn(message)) { this.startAutonomousTurn(); } if (!isForeground && !this.autonomousTurn && message.type === "result") { @@ -4068,7 +4139,13 @@ class ClaudeAgentSession implements AgentSession { return true; } - private async interruptActiveTurn(): Promise { + /** + * Interrupts Claude only when a turn of its main session is running. With nothing running there + * is nothing to stop but our own queued messages, so those are withdrawn instead: an interrupt + * would reach only background helpers (and kill them on a CLI that ignores + * perTaskStopAffordance), and no result would follow to consume pendingInterruptAbort. + */ + private async interruptActiveTurn(claudeStartedTurn = this.mainTurnInFlight): Promise { const queryToInterrupt = this.query; if (!queryToInterrupt || typeof queryToInterrupt.interrupt !== "function") { this.logger.trace( @@ -4082,8 +4159,16 @@ class ClaudeAgentSession implements AgentSession { ); return; } - this.pendingInterruptAbort = true; - await this.discardQueuedSteers(queryToInterrupt); + if (claudeStartedTurn) { + this.pendingInterruptAbort = true; + await this.withdrawUnstartedMessages(queryToInterrupt); + } else { + // Claude can dequeue a message for its next turn before announcing it started; withdrawing + // it is then a no-op, and only an interrupt stops it. Withdraw once: a prompt sent while + // this withdrawal was in flight belongs to the next turn. + if (await this.withdrawUnstartedMessages(queryToInterrupt)) return; + this.pendingInterruptAbort = true; + } try { await this.awaitWithTimeout( queryToInterrupt.interrupt(), @@ -4095,14 +4180,15 @@ class ClaudeAgentSession implements AgentSession { } /** - * Interrupt means interrupt: a steer Claude never read dies with the turn instead of resuming it. - * A steer already dequeued cannot be recalled, and does not need to be — the interrupt kills it. + * Interrupt means interrupt: a message Claude never started dies with the turn instead of + * resuming it. One already dequeued cannot be recalled, and does not need to be, since the + * interrupt kills it. Returns whether Claude confirmed every message withdrawn. */ - private async discardQueuedSteers(query: Query): Promise { - const uuids = [...this.queuedSteerUuids]; - this.queuedSteerUuids.clear(); + private async withdrawUnstartedMessages(query: Query): Promise { + const uuids = [...this.unstartedMessageUuids]; + this.unstartedMessageUuids.clear(); this.permissionClearingSteerUuids.clear(); - if (uuids.length === 0) return; + if (uuids.length === 0) return true; // The SDK runtime supports this, but its public Query type has not caught up. Keep the // compatibility escape hatch inside the Claude adapter. const cancelAsyncMessage = ( @@ -4110,14 +4196,17 @@ class ClaudeAgentSession implements AgentSession { cancelAsyncMessage?: (uuid: string) => Promise; } ).cancelAsyncMessage; - if (!cancelAsyncMessage) return; + if (!cancelAsyncMessage) return false; + let withdrewAll = true; for (const uuid of uuids) { try { - await cancelAsyncMessage.call(query, uuid); + if (!(await cancelAsyncMessage.call(query, uuid))) withdrewAll = false; } catch (error) { - this.logger.warn({ err: error }, "Failed to discard a queued Claude steer"); + withdrewAll = false; + this.logger.warn({ err: error }, "Failed to withdraw a queued Claude message"); } } + return withdrewAll; } private translateMessageToEvents( @@ -4166,7 +4255,7 @@ class ClaudeAgentSession implements AgentSession { } } - this.forgetReadSteer(message); + this.forgetStartedMessage(message); switch (message.type) { case "system": @@ -4200,11 +4289,14 @@ class ClaudeAgentSession implements AgentSession { return events; } - /** Once Claude has read a steer there is nothing left to discard on interrupt. */ - private forgetReadSteer(message: unknown): void { + /** + * Once Claude has started a message, or cancelled it, there is nothing left to withdraw on + * interrupt. + */ + private forgetStartedMessage(message: unknown): void { const lifecycle = readClaudeCommandLifecycle(message); if (!lifecycle || lifecycle.state === "queued") return; - this.queuedSteerUuids.delete(lifecycle.commandUuid); + this.unstartedMessageUuids.delete(lifecycle.commandUuid); this.permissionClearingSteerUuids.delete(lifecycle.commandUuid); } @@ -6519,7 +6611,8 @@ function extractClaudeUserText(messageRaw: unknown): string | null { interface ClaudeCommandLifecycle { commandUuid: string; - state: "queued" | "started" | "completed"; + /** "cancelled" is terminal too: API errors, interrupts and "now" pre-emption end a message. */ + state: "queued" | "started" | "completed" | "cancelled"; } /** Runtime-only Claude frames are validated here because the SDK's public union omits them. */ @@ -6528,7 +6621,7 @@ function readClaudeCommandLifecycle(message: unknown): ClaudeCommandLifecycle | if (record?.type !== "command_lifecycle") return null; if ( typeof record.command_uuid !== "string" || - !["queued", "started", "completed"].includes(String(record.state)) + !["queued", "started", "completed", "cancelled"].includes(String(record.state)) ) { return null; } diff --git a/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts b/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts index 1eca4cedc2..b0bfc3d755 100644 --- a/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts +++ b/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts @@ -511,6 +511,33 @@ describe("ClaudeTaskProtocolSource", () => { expect(source.cancelRunningForegroundTasks()).toEqual([]); }); + it("does not cancel a subagent announced as backgrounded at spawn", () => { + // Claude sets is_backgrounded on task_started for a child spawned in the background, and + // sends no task_updated patch for it. + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ is_backgrounded: true })); + + expect(source.cancelRunningForegroundTasks()).toEqual([]); + }); + + it("does not cancel a resumed subagent, which Claude always runs in the background", () => { + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ tool_use_id: "toolu_original", is_backgrounded: false })); + source.observe(taskUpdated("completed")); + source.observe(taskStarted({ tool_use_id: "toolu_resumed", is_backgrounded: true })); + + expect(source.cancelRunningForegroundTasks()).toEqual([]); + }); + + it("still cancels a subagent announced in the foreground", () => { + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ is_backgrounded: false })); + + expect(source.cancelRunningForegroundTasks()).toEqual([ + { kind: "status", id: "toolu_01DgLoPMW9", status: "canceled" }, + ]); + }); + it("still routes a backgrounded subagent that settles after the interrupt", () => { // The headline case: interrupt, continue, and the child that was told to outlive the turn // reports completion later. Wiping the routing table on cancel drops this on the floor and diff --git a/packages/server/src/server/agent/providers/claude/subagents/live-source.ts b/packages/server/src/server/agent/providers/claude/subagents/live-source.ts index 1a44520366..2149fae139 100644 --- a/packages/server/src/server/agent/providers/claude/subagents/live-source.ts +++ b/packages/server/src/server/agent/providers/claude/subagents/live-source.ts @@ -16,7 +16,8 @@ import { * * Verified on the wire (Claude Code 2.1.220, Paseo's own query options): * - * task_started task_id, tool_use_id, description, subagent_type, task_type + * task_started task_id, tool_use_id, description, subagent_type, task_type, + * is_backgrounded * task_updated task_id, patch.status, patch.is_backgrounded * task_notification task_id, tool_use_id, status * @@ -39,6 +40,11 @@ interface TaskStartedMessage { task_type?: string; prompt?: string; skip_transcript?: boolean; + /** + * Set at spawn: true for a task registered in the background. A resumed subagent is always + * registered in the background. + */ + is_backgrounded?: boolean; } /** Task-tool subagents. Backgrounded shell commands announce as `local_bash`. */ @@ -71,8 +77,8 @@ interface TaskUpdatedMessage { task_id: string; /** * `is_backgrounded` is declared on `SDKTaskUpdatedMessage["patch"]`: it flips when a foreground - * task is backgrounded, which is the only signal that separates a child that dies with its turn - * from one that was explicitly told to outlive it. + * task is moved to the background after `task_started`. Together with `task_started`'s own + * flag it separates a child that dies with its turn from one told to outlive it. */ patch?: { status?: string; is_backgrounded?: boolean }; } @@ -171,8 +177,8 @@ export class ClaudeTaskProtocolSource { /** Workflow invocations already own a real Workflow card in the parent timeline. */ private readonly idsWithExistingParentToolCard = new Set(); /** - * Declared subagents that were moved to the background. They outlive the turn that spawned - * them, so a turn ending is not evidence that they stopped. + * Declared subagents running in the background, whether spawned there or moved there later. + * They outlive the turn that spawned them, so a turn ending is not evidence that they stopped. */ private readonly backgroundedIds = new Set(); /** Last status emitted per subagent, so a redundant announcement is not re-broadcast. */ @@ -331,6 +337,7 @@ export class ClaudeTaskProtocolSource { this.sawTaskStarted = true; const existingId = this.subagentIdByTaskId.get(message.task_id); + this.recordBackgrounded(existingId ?? id, message.is_backgrounded); if (existingId) { return this.observeExistingTaskStart(message, id, existingId); } @@ -414,14 +421,15 @@ export class ClaudeTaskProtocolSource { private observeTaskUpdated(message: TaskUpdatedMessage): SubagentObservation[] { const id = this.subagentIdByTaskId.get(message.task_id); - const backgrounded = message.patch?.is_backgrounded; - if (id && typeof backgrounded === "boolean") { - if (backgrounded) this.backgroundedIds.add(id); - else this.backgroundedIds.delete(id); - } + if (id) this.recordBackgrounded(id, message.patch?.is_backgrounded); return this.observeStatus(message.task_id, message.patch?.status); } + private recordBackgrounded(id: string, backgrounded: boolean | undefined): void { + if (backgrounded === true) this.backgroundedIds.add(id); + else if (backgrounded === false) this.backgroundedIds.delete(id); + } + private observeTaskNotification(message: TaskNotificationMessage): SubagentObservation[] { const observations = this.observeWorkflowResult(message); observations.push(...this.observeUsage(message.task_id, message.usage)); diff --git a/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts b/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts index 2471ff8c5a..5eea6dab24 100644 --- a/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts +++ b/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts @@ -304,6 +304,166 @@ async function waitForRunningClaudeSleep( ); } +// Explicit point-in-time polls of agent status, on top of (not instead of) the pushed +// agent_update events the collector already records. A phantom "running" flip that starts and +// ends between two polls would still be caught by the pushed-event scan the callers pair this +// with; polling here additionally proves the daemon answers fetchAgent with "idle" throughout, +// not just that it eventually pushes one. +async function sampleAgentStatuses( + client: DaemonClient, + agentId: string, + samples: number, + intervalMs: number, +): Promise { + const statuses: string[] = []; + for (let i = 0; i < samples; i += 1) { + const snapshot = await client.fetchAgent({ agentId }); + statuses.push(snapshot?.agent.status ?? "missing"); + if (i < samples - 1) { + await sleep(intervalMs); + } + } + return statuses; +} + +// NOTE: agent_stream events on the wire (AgentStreamEventPayloadSchema in +// packages/protocol/src/messages.ts) do not include a "provider_subagent" variant. A backgrounded +// subagent's descriptor updates are forwarded exclusively as separate "agent.provider_subagents.update" +// messages (see forwardProviderSubagentUpdate in session.ts), and that push channel's delivery is +// gated by capabilities this test's plain DaemonClient does not declare. list_provider_subagents is a +// plain request/response RPC unaffected by that gating, so status history below polls it directly +// instead of trying to reconstruct status transitions from the stream. +type ProviderSubagentDescriptor = Awaited< + ReturnType +>["subagents"][number]; + +interface ProviderSubagentPoll { + subagent: ProviderSubagentDescriptor; + // Every status observed while polling, in order, so a transient cancel cannot hide behind a + // later "completed" read even though the store's status field is not expected to un-cancel. + history: string[]; +} + +// Polls agent.provider_subagents.list until the backgrounded helper is registered at all. +// "Main's turn is running" (waitForAgentUpsert) fires on generic turn-start bookkeeping, which +// can be true before Claude has actually issued the Agent tool call; canceling that early +// cancels a turn with no helper launched yet, which is not what "Stop spares a helper" means to +// test. This confirms the helper actually exists before the caller acts on it. +async function waitForProviderSubagentToExist( + client: DaemonClient, + agentId: string, + timeoutMs = 60_000, +): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const { subagents } = await client.listProviderSubagents(agentId); + if (subagents[0]) { + return subagents[0]; + } + await sleep(300); + } + throw new Error( + `Timed out waiting for a background helper to be registered for agent ${agentId}`, + ); +} + +// Claude starts the wake-up turn a moment after the helper's completion lands (0.1 to 5 s in +// recorded traces), so poll for it instead of checking the instant the helper reads completed. +// The daemon announces an autonomous turn to subscribers as a pushed "running" agent_update, +// not as an agent_stream turn_started (that stream reaches the client that started a run). +async function waitForWakeTurn( + collector: ReturnType, + agentId: string, + fromIndex: number, + timeoutMs = 30_000, +): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const window = collector.messages.slice(fromIndex); + const woke = + getAgentStatuses(window, agentId).includes("running") || + window.some( + (message) => + message.type === "agent_stream" && + message.payload.agentId === agentId && + message.payload.event.type === "turn_started", + ); + if (woke) return true; + await sleep(500); + } + return false; +} + +// The wake-up turn runs briefly after it opens, so the final state is read once it settles. +async function waitForAgentStatus( + client: DaemonClient, + agentId: string, + status: string, + timeoutMs = 60_000, +): Promise { + const deadline = Date.now() + timeoutMs; + let last: string | undefined; + while (Date.now() < deadline) { + last = (await client.fetchAgent({ agentId }))?.agent.status; + if (last === status) return last; + await sleep(500); + } + return last; +} + +// Polls agent.provider_subagents.list until the single backgrounded helper leaves "running". +// Scenarios in this file always launch exactly one background helper, so subagents[0] is it. +async function waitForProviderSubagentTerminal( + client: DaemonClient, + agentId: string, + timeoutMs = 90_000, +): Promise { + const deadline = Date.now() + timeoutMs; + const history: string[] = []; + let lastSeen: ProviderSubagentDescriptor[] = []; + while (Date.now() < deadline) { + const { subagents } = await client.listProviderSubagents(agentId); + lastSeen = subagents; + const subagent = subagents[0]; + if (subagent) { + history.push(subagent.status); + if (subagent.status !== "running") { + return { subagent, history }; + } + } + await sleep(1_000); + } + throw new Error( + `Timed out waiting for the background helper to leave "running". history=${JSON.stringify(history)} lastSeen=${JSON.stringify(lastSeen)}`, + ); +} + +// Pulls the main agent's own assistant text via fetchAgentTimeline rather than the push +// collector. Assistant text delivery is paced (docs/agent-stream-performance.md), so checking +// the collector immediately after waitForFinish resolves can still miss the final text; the +// timeline fetch is authoritative and matches what the pre-existing tests in this file do. +async function fetchAssistantTexts( + client: DaemonClient, + agentId: string, + limit = 100, +): Promise { + const timeline = await client.fetchAgentTimeline(agentId, { limit }); + return timeline.entries + .filter((entry) => entry.item.type === "assistant_message") + .map((entry) => (entry.item as Extract).text); +} + +async function fetchSubagentAssistantTexts( + client: DaemonClient, + agentId: string, + subagentId: string, +): Promise { + const timeline = await client.fetchProviderSubagentTimeline(agentId, subagentId, { limit: 50 }); + return timeline.rows + .filter((row) => row.item.type === "assistant_message") + .map((row) => (row.item as Extract).text); +} + describe("daemon E2E (real claude) - send message during tool call", () => { let canRun = false; interface SteeringResources { @@ -504,6 +664,141 @@ describe("daemon E2E (real claude) - send message during tool call", () => { } }, 210_000); + test("a background subagent keeps running after a message is sent while it works", async () => { + // The scenario the rest of this file explicitly rules out: every other test here tells Claude + // "Do not use a background task", so steering during a backgrounded subagent was never + // covered. A subagent streams on the parent's channel, so a parent that mistakes the child's + // frames for its own work reports "running" after it has already finished, and the message + // sent into that window is refused, or delivered by cancelling the turn and taking the + // subagent with it. Both outcomes are user-visible, and both are what this asserts against. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("Claude background-subagent test client was not created"); + await within("connect background-subagent test client", 15_000, client.connect()); + await within( + "subscribe background-subagent test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create background-subagent test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-background-survives-send", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + await within( + "launch one background subagent and end the turn", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor".`, + `prompt: "Use the Bash tool with run_in_background: true to run exactly: sleep 40.`, + `Wait for its completion notification, then reply exactly: CHILD_DONE."`, + "Immediately after launching it, reply with the single word LAUNCHED and end your turn.", + "Do not wait for it. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // The main session has ended its turn. It must read as idle even though the subagent it + // launched is still streaming on the shared channel. + const afterLaunch = await within( + "wait for the main session to go idle after launching the subagent", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect( + afterLaunch.status, + "a main session that finished its turn must not report busy because a subagent is running", + ).not.toBe("running"); + + // And a message sent in that window must be accepted rather than refused. + const followUpId = generateClientMessageId(); + await within( + "send a message to the idle main session while the subagent runs", + 30_000, + client.sendAgentMessage(agent.id, "Are you there? Reply exactly: MAIN_ALIVE.", { + messageId: followUpId, + }), + ); + await within( + "wait for the main session to answer", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + + const timeline = await within( + "fetch background-subagent timeline", + 15_000, + client.fetchAgentTimeline(agent.id, { limit: 200 }), + ); + const assistantText = timeline.entries + .filter((entry) => entry.item.type === "assistant_message") + .map( + (entry) => (entry.item as Extract).text, + ) + .join("\\n"); + expect( + assistantText, + "the main session must be reachable while its background subagent still runs", + ).toContain("MAIN_ALIVE"); + + // The subagent must have survived the send rather than being cancelled with the turn. Its + // status is read from agent.provider_subagents.list (see the note above + // waitForProviderSubagentToExist): agent_stream carries no provider_subagent events. + const { subagents } = await within( + "list provider subagents after the send", + 15_000, + client.listProviderSubagents(agent.id), + ); + expect(subagents.length, "the background subagent must be tracked").toBeGreaterThan(0); + expect( + subagents.map((subagent) => subagent.status), + "a backgrounded subagent must not be canceled by a message sent to the main session", + ).not.toContain("canceled"); + const { subagent, history } = await within( + "wait for the background subagent to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect(history, "no status poll may read canceled").not.toContain("canceled"); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "Claude background-subagent E2E cleanup failures").toEqual([]); + } + }, 300_000); + test("Stop cancels a queued Claude steer before it can resume the interrupted turn", async () => { const logger = pino({ level: "silent" }); const cwd = tmpCwd(); @@ -678,4 +973,430 @@ describe("daemon E2E (real claude) - send message during tool call", () => { rmSync(cwd, { recursive: true, force: true }); } }, 300_000); + + test("a helper's own nested background job does not phantom-run the parent or kill the helper", async () => { + // The exact reported bug: main backgrounds a helper, the helper itself starts its OWN + // background job (a grandchild from main's perspective) and then keeps working in the + // foreground. The grandchild's completion is bookkeeping that belongs to the helper, not to + // main, and must not make main phantom-report "running" or get treated as busy. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("nested background-job test client was not created"); + await within("connect nested background-job test client", 15_000, client.connect()); + await within( + "subscribe nested background-job test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create nested background-job test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-nested-background-job", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch a helper with its own nested background job and end the turn", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "nested bg survivor".`, + `prompt: "Use the Bash tool with run_in_background: true to run exactly:`, + `python3 -c 'import time; time.sleep(5)'.`, + `Immediately after starting it, without waiting for it, use the Bash tool again`, + `in the foreground to run exactly: python3 -c 'import time; time.sleep(50)'.`, + `After that foreground command finishes, reply exactly: CHILD_DONE."`, + "Immediately after launching it, reply with the single word LAUNCHED and end your turn.", + "Do not wait for it. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + const afterLaunch = await within( + "wait for the main session to go idle after launching the nested-background helper", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect(afterLaunch.status, "the main session must be idle once it replied LAUNCHED").toBe( + "idle", + ); + + // The helper's own nested background job (5s) finishes while the helper is still busy + // in its unrelated foreground sleep (50s). Sample for 30s, well past the nested job's + // completion even when the helper is slow to start it, + // both by polling and by scanning every pushed agent_update in the window, so a brief + // phantom flip cannot slip between polls. + const sampleWindowStart = resources.collector.messages.length; + const polledStatuses = await sampleAgentStatuses(client, agent.id, 11, 3_000); + const pushedStatuses = getAgentStatuses( + resources.collector.messages.slice(sampleWindowStart), + agent.id, + ); + const observedRunning = [...polledStatuses, ...pushedStatuses].filter( + (status) => status === "running", + ); + expect( + observedRunning, + `the parent must stay idle across the nested job's completion; polled=${JSON.stringify(polledStatuses)} pushed=${JSON.stringify(pushedStatuses)}`, + ).toEqual([]); + + // A message sent in that window must still be answered normally. + await within( + "send PONG request while the nested job settles", + 30_000, + client.sendMessage(agent.id, "Reply with exactly PONG.", { + messageId: generateClientMessageId(), + }), + ); + await within("wait for the PONG reply", 60_000, client.waitForFinish(agent.id, 50_000)); + // Pulled via fetchAgentTimeline, not the push collector: assistant text delivery is paced + // (docs/agent-stream-performance.md) and can trail waitForFinish's own completion signal, + // so the collector can still be missing the final text the instant waitForFinish resolves. + const pongTexts = await within( + "fetch timeline for the PONG reply", + 15_000, + fetchAssistantTexts(client, agent.id, 20), + ); + expect( + pongTexts.join("\n"), + "the agent must answer PONG while the helper is still running", + ).toContain("PONG"); + + const messagesBeforeWake = resources.collector.messages.length; + + // The helper itself must survive: never recorded canceled, and it eventually reports + // CHILD_DONE from its own foreground sleep finishing. + const { subagent, history } = await within( + "wait for the nested-background helper to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + + // The helper's completion should wake the main session with a fresh turn (the + // session-state-driven autonomous turn), separate from the PONG exchange above. + const wokeMainAgent = await waitForWakeTurn( + resources.collector, + agent.id, + messagesBeforeWake, + ); + if (!wokeMainAgent) { + const wakeWindowMessages = resources.collector.messages.slice(messagesBeforeWake); + const wakeWindowEventTypes = wakeWindowMessages + .filter( + (message) => message.type === "agent_stream" && message.payload.agentId === agent.id, + ) + .map((message) => + message.type === "agent_stream" ? message.payload.event.type : "unknown", + ); + const timeline = await client + .fetchAgentTimeline(agent.id, { limit: 100 }) + .catch(() => null); + throw new Error( + `the helper's completion must wake the main session with a new turn. ` + + `wakeWindowEventTypes=${JSON.stringify(wakeWindowEventTypes)} ` + + `recentTimeline=${JSON.stringify(timeline ? summarizeTimelineItems(timeline) : null)}`, + ); + } + + expect(await waitForAgentStatus(client, agent.id, "idle"), "the agent must end idle").toBe( + "idle", + ); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "nested background-job E2E cleanup failures").toEqual([]); + } + }, 300_000); + + test("the app's default steer survives a backgrounded helper while the main session works", async () => { + // appSettings.sendBehavior defaults to "steer" (packages/app/src/hooks/use-settings/storage.ts), + // so this is what the app actually sends when a user replies while an agent looks busy. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("default-steer background-helper test client was not created"); + await within("connect default-steer test client", 15_000, client.connect()); + await within( + "subscribe default-steer test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create default-steer test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-default-steer-background-helper", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch the background helper and start the main session's own foreground sleep", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor b".`, + `prompt: "Use the Bash tool in the foreground to run exactly:`, + `python3 -c 'import time; time.sleep(42)'. After it finishes, reply exactly: CHILD_DONE."`, + "Do not wait for that background helper to finish before continuing.", + "After launching it, use the Bash tool one more time yourself, WITHOUT run_in_background,", + "to run exactly: python3 -c 'import time; time.sleep(16)'.", + // Matching the proven pattern from the first test in this file: tell Claude to wait + // for a later message rather than relying on the sleep call itself to block, since a + // Bash call can resolve through Claude Code's own async task_notification path once a + // background subagent is already active instead of blocking the turn. + "Do not reply or finish your turn until a later user message arrives, no matter how", + "quickly that Bash command finishes. When that later message arrives, address it and", + "end your turn. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // Confirm the daemon considers main's own turn active before steering it. This does not + // wait for a specific "sleep 16" shell call to show as running: Claude Code 2.1.280 can + // report a slower Bash call through its own async task_notification tool instead of a + // classic blocking call once a background subagent is already active, so pinning to a + // literal running shell tool_call is not reliable here. Busy at the daemon level is what + // "while main is busy with its own sleep" actually needs. + await within( + "confirm main's own turn is active before steering", + 30_000, + client.waitForAgentUpsert(agent.id, (snapshot) => snapshot.status === "running", 25_000), + ); + // The running status precedes the Agent tool call, so a steer sent on it alone can reach + // Claude before the helper exists and prove nothing about a helper surviving it. + await within( + "confirm the background helper is registered before steering", + 60_000, + waitForProviderSubagentToExist(client, agent.id, 55_000), + ); + + await within( + "send the app's default steer while main is busy with its own sleep", + 30_000, + client.sendMessage(agent.id, "Reply with exactly PONG.", { + messageId: generateClientMessageId(), + activeTurnBehavior: "steer", + }), + ); + + const finish = await within( + "wait for the main session to finish after the steer", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect(finish.status, "the main session must end idle after the steer").toBe("idle"); + // Pulled via fetchAgentTimeline, not the push collector: see fetchAssistantTexts. + const postSteerTexts = await within( + "fetch timeline for the post-steer PONG reply", + 15_000, + fetchAssistantTexts(client, agent.id, 20), + ); + expect(postSteerTexts.join("\n"), "PONG must be answered after the steer").toContain("PONG"); + + const { subagent, history } = await within( + "wait for the background helper to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "default-steer background-helper E2E cleanup failures").toEqual([]); + } + }, 300_000); + + test("Stop cancels the main session's own turn but spares its background helper", async () => { + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("Stop-spares-helper test client was not created"); + await within("connect Stop-spares-helper test client", 15_000, client.connect()); + await within( + "subscribe Stop-spares-helper test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create Stop-spares-helper test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-stop-spares-helper", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch the background helper and start the main session's own foreground sleep", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor c".`, + `prompt: "Use the Bash tool in the foreground to run exactly:`, + `python3 -c 'import time; time.sleep(43)'. After it finishes, reply exactly: CHILD_DONE."`, + "Do not wait for that background helper to finish before continuing.", + "After launching it, use the Bash tool one more time yourself, WITHOUT run_in_background,", + "to run exactly: python3 -c 'import time; time.sleep(23)'.", + "Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // Confirm the helper has actually been launched (registered) before Stop, not just that + // main's turn shows "running": that status can flip true on generic turn-start bookkeeping + // before Claude has issued the Agent tool call, and canceling that early cancels a turn + // with no helper to spare yet. + await within( + "confirm the background helper is registered before Stop", + 60_000, + waitForProviderSubagentToExist(client, agent.id, 55_000), + ); + + const messagesBeforeStop = resources.collector.messages.length; + await within("Stop the main session's own turn", 30_000, client.cancelAgent(agent.id)); + const afterStop = await within( + "wait for the agent to go idle after Stop", + 60_000, + client.waitForFinish(agent.id, 50_000), + ); + expect(afterStop.status, "Stop must leave the agent idle").toBe("idle"); + + // The helper must survive Stop: it keeps running and later reports CHILD_DONE, which + // wakes the main session with a fresh autonomous turn. + const { subagent, history } = await within( + "wait for the background helper to finish after Stop", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "Stop must not cancel or fail the background helper").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + + const wokeMainAgent = await waitForWakeTurn( + resources.collector, + agent.id, + messagesBeforeStop, + ); + expect( + wokeMainAgent, + "the helper's completion after Stop must wake the main session with a new turn", + ).toBe(true); + + expect(await waitForAgentStatus(client, agent.id, "idle"), "the agent must end idle").toBe( + "idle", + ); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "Stop-spares-helper E2E cleanup failures").toEqual([]); + } + }, 300_000); });