From 8ddeb79c8dbb8934244b35ba18472fb9074d8209 Mon Sep 17 00:00:00 2001 From: "paseo-bot[bot]" <266920839+paseo-bot[bot]@users.noreply.github.com> Date: Wed, 7 Oct 2026 14:08:48 +0200 Subject: [PATCH] Claude agents stay idle while background helpers work, and send or Stop no longer kills them (#6295) * fix(claude): open autonomous turns only when Claude starts a turn Paseo guessed from frame shape whether the main session had started work on its own, and only a result could close the turn that guess opened. A background subagent's task frames, its own background jobs' task_notification and rate limit events all arrive with no parent tool use id, so they looked like main-session work and opened a turn no result would ever close. The agent then reported "running" while its main session sat idle, and the app's message queue never drained. Claude brackets every main-session turn itself: system/init opens it and exactly one result closes it, whether a prompt, a wake-up after background work, a slash command or an API error started it. Background subagents never emit either on the main stream. An autonomous turn now opens only on that init, and only when no foreground turn is open. The frame heuristic is removed. - Paseo launches the CLI with CLAUDE_CODE_EMIT_SESSION_STATE_EVENTS=1, added as an overlay in buildSdkEnv so the SDK and the usage probe share one env object. An "idle" session state closes an autonomous turn as a safety net, never a foreground one. "running" opens nothing: Claude Code 2.1.280 stays "running" for as long as any background subagent is alive. State frames never reach the timeline. - An init clears a leftover pendingInterruptAbort. A new turn has begun, so the interrupted one is over, and a stale flag would swallow the new turn's result. Tests that opened autonomous turns from bare frames now send the init Claude sends. New coverage: helper frames never open a turn while the main session is idle, a wake-up turn opens on init and closes on its result, a folded steer keeps one turn while a turn Claude starts after the result gets its own, an idle closes a turn no result closed, session state never opens a turn, and the env flag reaches the SDK launch. docs/agent-lifecycle.md records that only system/init and result bracket a main-session turn, since the code cannot tell a reader why frame shape fails. Co-Authored-By: Claude Opus 5.5 * fix(claude): interrupt Claude only while its main turn runs, and spare helpers Every Stop, and every send that replaces a running turn (paseo send, MCP send_agent_prompt, voice, the app's queue drain, Send now), ends in query.interrupt(). Without the SDK's perTaskStopAffordance declaration that call kills every running background subagent, even when the main session is idle and there is nothing of ours to stop. - Declare perTaskStopAffordance, so an interrupt aborts only the main turn, like Esc in Claude Code. Claude stops a helper itself with its TaskStop tool. - Track whether a main-session turn is in flight from Claude's own brackets: set by system/init and by a pushed message's command_lifecycle "started" (/compact works for seconds before its init), cleared by the result or an idle session state. - Call query.interrupt() and set pendingInterruptAbort only when that turn is in flight, or the cancelled turn's own message has already been started. Otherwise withdraw our not-yet-started messages with cancel_async_message instead: an idle interrupt reaches only helpers, and with no result to follow it the flag would linger. - Track the foreground prompt with the queued steers until Claude starts it, so Stop withdraws a message still waiting in the queue. A turn stopped while the Claude process is still starting never sends it. - Accept command_lifecycle "cancelled" as terminal, so a cancelled steer no longer waits to be withdrawn or keeps denying permission cards. Interrupt tests now wait for the pump to read the init before stopping, since that is when Paseo learns a turn is in flight. New coverage: a stop before Claude starts the message withdraws it without interrupting, a stop while only helpers run never reaches Claude, a stop mid-turn interrupts once and drops the aborted result, a new turn clears a flag no result will consume, the SDK option is passed, and a cancelled steer is not withdrawn again. docs/providers.md described the old unconditional interrupt; it now states when the Claude adapter interrupts and why it declares the affordance. Co-Authored-By: Claude Opus 5.5 * fix(claude): keep a helper spawned in the background out of a turn's cancel When a turn is canceled, Paseo terminalizes the subagents that were running in its foreground and skips the backgrounded ones, which outlive it. It only learned a child was backgrounded from task_updated's patch.is_backgrounded, which Claude sends when a foreground task moves to the background later. A child spawned in the background carries the flag on task_started instead, and a resumed subagent always does, so the track marked those helpers "canceled" while they kept working. Read is_backgrounded from task_started as well. docs/agent-lifecycle.md said task_updated was the only signal; it now names both. Co-Authored-By: Claude Opus 5.5 * fix(app): drain queued messages as steers The composer queue drains when an agent goes idle. It sent each message with no activeTurnBehavior, which the daemon treats as "interrupt": if the agent had just started a turn of its own (a Claude wake-up after background work finished), the drained message stopped that turn and replaced it. Send drained messages with activeTurnBehavior "steer". A drain that races such a turn folds into it; with no turn running the daemon starts a normal one, as before. "steer" is an existing protocol value, so older daemons need nothing new. Co-Authored-By: Claude Opus 5.5 * test(claude): cover background helpers against real Claude Every other test in this file tells Claude "Do not use a background task", so a message sent or a Stop pressed while a backgrounded subagent runs was never exercised. Four real-Claude scenarios cover it: - "a background subagent keeps running after a message is sent while it works": one helper, the main turn ends, the parent is not reported running, a message in that window is answered, and the helper is never read as canceled and completes. - "a helper's own nested background job does not phantom-run the parent or kill the helper": a helper starts its own background Bash job, then keeps working in the foreground. The parent stays idle across the nested job's completion (sampled for 30 s around a 5 s job), a message sent in that window is answered, and the helper reports CHILD_DONE. - "the app's default steer survives a backgrounded helper while the main session works": the app's default send behavior (steer) does not kill a helper when it interrupts the main session's own turn. - "Stop cancels the main session's own turn but spares its background helper": the same for Stop. The wake-up turn Claude starts after a helper finishes lands a moment after the helper reads completed (0.1 to 5 s in recorded traces), and the daemon announces it as a pushed "running" agent_update, not as a turn_started stream event. The checks poll for it and read the final idle once the wake-up settles. waitForFinish can return a stale "idle" right after a send, and the push stream can miss a message's final text when it resolves, so the tests read fetchAgentTimeline like the rest of the file. Helper status comes from agent.provider_subagents.list, since agent_stream carries no provider_subagent events. The scenarios need the previous commits together. Run against the frame heuristic this branch removes, the nested-job test fails: the parent flips to running once the nested job finishes and stays there. Co-Authored-By: Claude Opus 5.5 * fix(claude): interrupt a stopped prompt Claude has already taken Stop before Claude announces a prompt started withdraws it with cancel_async_message instead of interrupting. Claude can dequeue a message for its next turn before announcing it, and withdrawing it is then a no-op that reports cancelled: false, so the stopped prompt kept running. Interrupt whenever a message could not be withdrawn. Co-authored-by: Isak Persson * fix(claude): withdraw a stopped prompt only once When the withdrawal of a stopped prompt failed, the interrupt path withdrew a second time, which could cancel a prompt sent while the first withdrawal was in flight. Withdraw once and interrupt on failure. The real-Claude steer scenario now waits for the background helper to exist before steering, so it proves the helper survives the steer. Co-authored-by: Isak Persson --------- Co-authored-by: Isak Persson Co-authored-by: Claude Opus 5.5 Co-authored-by: paseo-bot[bot] <266920839+paseo-bot[bot]@users.noreply.github.com> --- docs/agent-lifecycle.md | 3 +- docs/providers.md | 2 +- packages/app/src/runtime/host-runtime.test.ts | 28 + packages/app/src/runtime/host-runtime.ts | 5 + .../agent/providers/claude/agent.env.test.ts | 6 + ...agent.interrupt-restart-regression.test.ts | 80 +- .../providers/claude/agent.redesign.test.ts | 465 +++++++++++ .../claude/agent.subagent-interrupt.test.ts | 28 +- .../agent/providers/claude/agent.test.ts | 11 +- .../server/agent/providers/claude/agent.ts | 179 +++-- .../claude/subagents/live-source.test.ts | 27 + .../providers/claude/subagents/live-source.ts | 28 +- ...d-during-tool-call-claude.real.e2e.test.ts | 721 ++++++++++++++++++ 13 files changed, 1504 insertions(+), 79 deletions(-) diff --git a/docs/agent-lifecycle.md b/docs/agent-lifecycle.md index 8662ee3287..d0d057949f 100644 --- a/docs/agent-lifecycle.md +++ b/docs/agent-lifecycle.md @@ -197,10 +197,11 @@ Claude Code announces subagent lifecycle on the SDK stream (`task_started` / `ta - **Not every announced task belongs in the track.** Task subagents announce as `local_agent` and workflows as `local_workflow`; a backgrounded shell announces as `local_bash` with the same `tool_use_id` shape, and ambient housekeeping sets `skip_transcript`. The Claude provider normalizes a workflow to a generic provider-subagent descriptor titled `Workflow`, using Claude's summary as its description and timeline opener. Shared storage, protocol, and UI do not distinguish it from another provider subagent. - **A task that was never declared gets no descriptor, by any route.** Filtered tasks still emit `task_notification`s carrying a `tool_use_id`, and still emit frames carrying `parent_tool_use_id`. Attributing either produces a descriptor with no identity and a defaulted `running` status — a nameless row that never finishes. Status, presentation updates, and sidechain frames all route through the declaration table. -- **Task ids are session-scoped, not turn-scoped.** Cancelling a turn must not clear the routing table: a backgrounded child settles after the interrupt and needs its descriptor to still exist. Cancellation instead terminalizes the declared children that were running in the foreground, and a later `task_notification` is free to correct that guess. Backgrounded children are identified by `task_updated.patch.is_backgrounded`. +- **Task ids are session-scoped, not turn-scoped.** Cancelling a turn must not clear the routing table: a backgrounded child settles after the interrupt and needs its descriptor to still exist. Cancellation instead terminalizes the declared children that were running in the foreground, and a later `task_notification` is free to correct that guess. Backgrounded children are identified by `is_backgrounded` on `task_started` (set at spawn; a resumed subagent is always background) or, when moved there later, on `task_updated.patch`. - **A resumed task can be announced again with a new `tool_use_id`.** The first Task tool id remains the canonical descriptor and later ids are routing aliases for the same session-scoped task. The resumed prompt is added to that child timeline. - **Effort is only reachable through hooks.** It appears nowhere on the message stream at any depth, and the level Paseo requests is not necessarily the level that runs — a model that does not support it is silently downgraded. A hook firing inside a subagent reports the active post-downgrade level next to its `agent_id`, which is the same id `task_started` calls `task_id`. - **Backgrounded subagents emit no frames carrying `parent_tool_use_id` at all.** Everything keyed off that field sees nothing for one; they are visible only because the task protocol announces them. +- **Only `system/init` and `result` bracket a main-session turn.** Background subagents emit neither on the main stream, while their task frames, their own background jobs' `task_notification`, and `rate_limit_event` arrive with no `parent_tool_use_id`, the same shape as main-session work. `session_state_changed` stays `running` for as long as any background subagent lives (Claude Code 2.1.280), so it cannot open a turn; its `idle` can close one. - **Nested ownership comes from the launching sidechain, not `spawn_depth` alone.** A sidechain's Agent or Bash tool call records the direct owner of that `tool_use_id`; the following `task_started` inherits it. This routes a grandchild descriptor and child-owned background notifications without relying on labels or flattening them into the managed parent. - **On replay, `/subagents/` holds every descendant beside the root.** Resolve the tree one proven generation at a time: the root transcript admits direct children, then each admitted sidechain transcript admits its children by `toolUseId`. `spawnDepth` orders candidates but does not establish ownership. Unresolved sidecars remain excluded as ambient or unrelated work. - **Replay `totalTokens` is a context-size reading, not cumulative spend.** Claude Code finalizes a subagent by summing the _last_ assistant message's usage block and shipping that as `usage.total_tokens`. Summing per-entry usage instead multiplies the cached prefix by the turn count and reports a number several times larger than the live path. diff --git a/docs/providers.md b/docs/providers.md index c5f37d4161..4609246922 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -103,7 +103,7 @@ OpenCode owns user message IDs. Do not pass Paseo-generated IDs to OpenCode prom Active-turn steering is an optional `AgentSession.steerActiveTurn` operation. The manager owns admission against its exact foreground turn, canonical user-message creation, echo reconciliation, and falls back to the normal interrupt-and-replace path only when the adapter reports `unavailable`. An adapter error leaves the steer's fate ambiguous and must surface without an interrupt or retry. Codex calls `turn/steer` with the native expected turn and Paseo client user-message ID. Claude pushes an admitted steer into the exact active SDK query input; isolated control commands remain unavailable. OpenCode calls `session/prompt_async` with an OpenCode-generated message ID; the server queues the prompt while busy and the next LLM call in the same Paseo turn includes it. Pi sends its native `steer` RPC, which queues the message for delivery after the in-flight assistant turn's tool calls. Slash-command inputs report `unavailable` because pi rejects extension commands on the steer path, and echo identity is correlated by message text because pi's steer RPC takes no message ID. A missing session reports `unavailable` and uses the normal interrupt fallback. -A steering adapter also owes its interrupt: stopping a turn must discard the steers the provider has not read yet, or one of them resumes the turn the user just stopped. Codex clears pending input when it aborts a turn; Claude does not, so its adapter cancels the SDK messages it queued before calling `query.interrupt()`. Pi requires `clear_queue` before `abort`; older binaries without that RPC retain their native queue behavior until the pi compatibility floor reaches 0.84.4. +A steering adapter also owes its interrupt: stopping a turn must discard the steers the provider has not read yet, or one of them resumes the turn the user just stopped. Codex clears pending input when it aborts a turn; Claude does not, so its adapter cancels the SDK messages it queued before calling `query.interrupt()`. It calls `query.interrupt()` only while a main-session turn is in flight (between `system/init` and that turn's `result`): with the main session idle, an interrupt reaches only background subagents and no result follows it, so the adapter withdraws its unstarted messages and stops there. The adapter also declares `perTaskStopAffordance`, because without it the CLI kills every background subagent on interrupt. Pi requires `clear_queue` before `abort`; older binaries without that RPC retain their native queue behavior until the pi compatibility floor reaches 0.84.4. `SteerActiveTurnOptions.clearPendingPermissions` makes permission release part of the provider contract. A provider that accepts such a steer queues it first, denies permissions blocking its delivery, and stops once the steer is read. Steers without the flag leave permissions open. A denied plan remains in the timeline because the pending card was the only other copy of its text. diff --git a/packages/app/src/runtime/host-runtime.test.ts b/packages/app/src/runtime/host-runtime.test.ts index 0954b0e422..996a47d498 100644 --- a/packages/app/src/runtime/host-runtime.test.ts +++ b/packages/app/src/runtime/host-runtime.test.ts @@ -3047,6 +3047,34 @@ describe("HostRuntimeStore", () => { useSessionStore.getState().clearSession(host.serverId); }); + it("drains a queued message as a steer so a turn the agent just started is not interrupted", async () => { + const host = makeHost({ serverId: "srv_steer_queue_drain" }); + const fakeClient = new FakeDaemonClient(); + const store = new HostRuntimeStore({ + deps: { + createClient: () => fakeClient as unknown as DaemonClient, + connectToDaemon: async () => ({ + client: fakeClient as unknown as DaemonClient, + serverId: host.serverId, + hostname: null, + }), + getClientId: async () => "cid_steer_queue_drain", + }, + }); + const sessionStore = useSessionStore.getState(); + sessionStore.initializeSession(host.serverId, fakeClient as unknown as DaemonClient, 1); + sessionStore.setQueuedMessages( + host.serverId, + new Map([["agent", [{ id: "queued", text: "next task", attachments: [] }]]]), + ); + + store.drainQueuedAgentMessage(host.serverId, "agent"); + await fakeClient.waitForSentMessages(1); + + expect(fakeClient.sentAgentMessages[0]?.[2]?.activeTurnBehavior).toBe("steer"); + sessionStore.clearSession(host.serverId); + }); + it("uses legacy GitHub attachments when draining a queue for an old daemon", async () => { const host = makeHost({ serverId: "srv_legacy_queue_attachment" }); const fakeClient = new FakeDaemonClient(); diff --git a/packages/app/src/runtime/host-runtime.ts b/packages/app/src/runtime/host-runtime.ts index de4b40f883..2ecbe60448 100644 --- a/packages/app/src/runtime/host-runtime.ts +++ b/packages/app/src/runtime/host-runtime.ts @@ -2426,6 +2426,11 @@ export class HostRuntimeStore { }), encodeImages, submission: createMessageSubmissionWriter(serverId), + // The queue drains when the agent goes idle, but the agent can start a turn of its own + // in that same moment (a wake-up after background work). Steering folds the message + // into that turn instead of interrupting it; with no turn running, the daemon starts + // a normal one. + activeTurnBehavior: "steer", }); }, }) diff --git a/packages/server/src/server/agent/providers/claude/agent.env.test.ts b/packages/server/src/server/agent/providers/claude/agent.env.test.ts index c99687e438..0d2748fd62 100644 --- a/packages/server/src/server/agent/providers/claude/agent.env.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.env.test.ts @@ -31,6 +31,7 @@ function createQueryMock(events: unknown[]): Query { describe("Claude SDK env", () => { test("forwards launch-context env through Claude process env", async () => { let capturedEnv: Record | undefined; + let capturedPerTaskStopAffordance: boolean | undefined; const launchContext: AgentLaunchContext = { env: { PASEO_AGENT_ID: "00000000-0000-4000-8000-000000000201", @@ -39,6 +40,7 @@ describe("Claude SDK env", () => { }; const queryFactory = vi.fn(({ options }: ClaudeQueryInput) => { capturedEnv = options.env; + capturedPerTaskStopAffordance = options.perTaskStopAffordance; return createQueryMock([ { type: "system", @@ -91,6 +93,10 @@ describe("Claude SDK env", () => { expect(capturedEnv?.MCP_TIMEOUT).toBe("claude-startup-timeout"); expect(capturedEnv?.MCP_TOOL_TIMEOUT).toBe("claude-tool-timeout"); expect(session.usageSession?.()?.env).toBe(capturedEnv); + // Paseo reads session_state_changed to know when an autonomous turn is over. + expect(capturedEnv?.CLAUDE_CODE_EMIT_SESSION_STATE_EVENTS).toBe("1"); + // Without it, Stop and replace kill every background subagent along with the turn. + expect(capturedPerTaskStopAffordance).toBe(true); } finally { await session.close(); expect(session.usageSession?.()).toBeNull(); diff --git a/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts b/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts index dada35945c..02a09ba44b 100644 --- a/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.interrupt-restart-regression.test.ts @@ -91,6 +91,17 @@ function buildUsage() { }; } +/** Claude opening a turn of its own. Every main-session turn starts with one. */ +function buildTurnInit(sessionId: string) { + return { + type: "system", + subtype: "init", + session_id: sessionId, + permissionMode: "default", + model: "opus", + }; +} + function buildSuccessResult(sessionId: string) { return { type: "result", @@ -153,13 +164,7 @@ function createScriptedQuery(params: { }, } satisfies ScriptedQuery; - scriptedQuery.emit({ - type: "system", - subtype: "init", - session_id: params.sessionId, - permissionMode: "default", - model: "opus", - }); + scriptedQuery.emit(buildTurnInit(params.sessionId)); void (async () => { for await (const prompt of params.prompt) { @@ -252,6 +257,21 @@ async function waitFor( } } +/** + * Paseo interrupts Claude only while a main-session turn is in flight, which it learns from the + * init. The pump asks for its next frame only after routing the last one, so a second next() call + * proves the init was read. + */ +async function waitForInitRouted( + query: () => { next: ReturnType } | null | undefined, +) { + await waitFor(() => (query()?.next.mock.calls.length ?? 0) >= 2); +} + +function buildCommandLifecycle(commandUuid: string | null | undefined, state: string) { + return { type: "command_lifecycle", command_uuid: commandUuid, state }; +} + afterEach(() => { queryFactory.mockReset(); }); @@ -282,6 +302,7 @@ test("interrupt only calls query.interrupt and leaves the query open", async () const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); await waitFor(() => queries[0]?.prompts.length === 1); + await waitForInitRouted(() => queries[0]); await session.interrupt(); await waitFor(() => queries[0]?.interrupt.mock.calls.length === 1); @@ -322,6 +343,7 @@ async function startSteeredTurn(sessionId: string): Promise<{ throw new Error("Expected the original Claude turn to start"); } await waitFor(() => query?.prompts.length === 1); + await waitForInitRouted(() => query); const steered = await session.steerActiveTurn!("queued steer", { expectedTurnId: start.value.turnId, @@ -371,11 +393,8 @@ test("interrupt still stops the turn when Claude has already dequeued the steer" test("a steer Claude has already read is no longer discardable on interrupt", async () => { const { session, query, turn } = await startSteeredTurn("queued-steer-completed-session"); - query()?.emit({ - type: "command_lifecycle", - command_uuid: query()?.prompts[1]?.uuid, - state: "completed", - }); + query()?.emit(buildCommandLifecycle(query()?.prompts[0]?.uuid, "started")); + query()?.emit(buildCommandLifecycle(query()?.prompts[1]?.uuid, "completed")); // Frames are translated in order, so the marker landing proves the lifecycle frame was read. query()?.emit({ type: "assistant", message: { content: "STEER_READ" } }); await consumeUntil(turn, (event) => collectAssistantText([event]).includes("STEER_READ")); @@ -388,6 +407,23 @@ test("a steer Claude has already read is no longer discardable on interrupt", as await session.close(); }); +test("a steer Claude cancelled is no longer withdrawn on interrupt", async () => { + const { session, query, turn } = await startSteeredTurn("queued-steer-cancelled-session"); + + query()?.emit(buildCommandLifecycle(query()?.prompts[0]?.uuid, "started")); + // "cancelled" is terminal: API errors, interrupts and "now" pre-emption all end a message. + query()?.emit(buildCommandLifecycle(query()?.prompts[1]?.uuid, "cancelled")); + query()?.emit({ type: "assistant", message: { content: "STEER_CANCELLED" } }); + await consumeUntil(turn, (event) => collectAssistantText([event]).includes("STEER_CANCELLED")); + + await session.interrupt(); + await waitFor(() => query()?.interrupt.mock.calls.length === 1); + + expect(query()?.cancelAsyncMessage).not.toHaveBeenCalled(); + await collectUntilTerminal(turn); + await session.close(); +}); + function buildStoppedTaskNotification(sessionId: string) { return { type: "system", @@ -473,6 +509,7 @@ async function startInterruptedToolTurn(sessionId: string): Promise<{ const turn = streamSession(session, "run the slow tool"); await turn.next(); await waitFor(() => query?.prompts.length === 1); + await waitForInitRouted(() => query); await session.interrupt(); await collectUntilTerminal(turn); @@ -523,6 +560,7 @@ test("Claude can still wake into an autonomous turn once the interrupted request query()?.emit(buildStoppedTaskNotification(sessionId)); query()?.emit(buildAbortedResult(sessionId)); + query()?.emit(buildTurnInit(sessionId)); query()?.emit({ type: "assistant", message: { content: "AUTONOMOUS_WAKE_RESPONSE" }, @@ -579,6 +617,7 @@ test("reuses the existing query after interrupt before starting the next prompt" const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); await waitFor(() => queries[0]?.prompts.length === 1); + await waitForInitRouted(() => queries[0]); await session.interrupt(); await collectUntilTerminal(firstTurn); @@ -652,6 +691,7 @@ test("recovers when the query pump sees a single interrupt abort before the next const output = createAsyncQueue>(); const prompts: PromptRecord[] = []; let throwAbortOnNext = false; + let queryRef: Pick | null = null; queryFactory.mockImplementation(({ prompt }: { prompt: AsyncIterable }) => { const scriptedQuery = { @@ -685,13 +725,8 @@ test("recovers when the query pump sees a single interrupt abort before the next }, } satisfies ScriptedQuery; - scriptedQuery.emit({ - type: "system", - subtype: "init", - session_id: "interrupt-abort-recovery-session", - permissionMode: "default", - model: "opus", - }); + queryRef = scriptedQuery; + scriptedQuery.emit(buildTurnInit("interrupt-abort-recovery-session")); void (async () => { for await (const promptMessage of prompt) { @@ -730,6 +765,7 @@ test("recovers when the query pump sees a single interrupt abort before the next const firstTurn = streamSession(session, "first prompt"); await firstTurn.next(); + await waitForInitRouted(() => queryRef); await session.interrupt(); await collectUntilTerminal(firstTurn); @@ -768,6 +804,7 @@ test("stale abort result after replacement start does not poison the new foregro const firstTurn = streamSession(session, "first prompt"); const firstStarted = await firstTurn.next(); await waitFor(() => queryRef?.prompts.length === 1); + await waitForInitRouted(() => queryRef); await session.interrupt(); const firstTurnEvents = [firstStarted.value!, ...(await collectUntilTerminal(firstTurn))]; @@ -812,7 +849,7 @@ test("stale abort result after replacement start does not poison the new foregro await session.close(); }); -test("creates an autonomous live turn when assistant output arrives without a foreground run", async () => { +test("creates an autonomous live turn when Claude starts a turn without a foreground run", async () => { const logger = createTestLogger(); let queryRef: ScriptedQuery | null = null; @@ -848,6 +885,7 @@ test("creates an autonomous live turn when assistant output arrives without a fo await collectUntilTerminal(streamSession(session, "seed prompt")); const subscribedEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-live-session")); queryRef?.emit({ type: "assistant", message: { content: "AUTONOMOUS_WAKE_RESPONSE" }, @@ -916,6 +954,7 @@ test("steers an autonomous turn through its existing query without restarting it await collectUntilTerminal(streamSession(session, "seed prompt")); const autonomousEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-steer-session")); queryRef?.emit({ type: "assistant", message: { content: "AUTONOMOUS_RESPONSE" }, @@ -999,6 +1038,7 @@ test("auto-completes an open autonomous turn when a foreground prompt starts", a await collectUntilTerminal(streamSession(session, "seed prompt")); const subscribedEvents = subscribeToEvents(session); + queryRef?.emit(buildTurnInit("autonomous-handoff-session")); queryRef?.emit({ type: "assistant", message: { content: "BACKGROUND_ONLY_RESPONSE" }, diff --git a/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts b/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts index 747a466c32..5393926520 100644 --- a/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.redesign.test.ts @@ -486,6 +486,7 @@ test("interruptActiveTurn only interrupts the active query without info logs", a } | null; input: { end: () => void } | null; queryRestartNeeded: boolean; + mainTurnInFlight: boolean; interruptActiveTurn: () => Promise; } = asInternals(session); const interrupt = vi.fn(async () => undefined); @@ -498,6 +499,7 @@ test("interruptActiveTurn only interrupts the active query without info logs", a }; internal.input = { end }; internal.queryRestartNeeded = false; + internal.mainTurnInFlight = true; try { await internal.interruptActiveTurn(); @@ -1191,6 +1193,7 @@ test("reuses one autonomous run for unbound stream_event bursts with no foregrou const queryMock = createBaseQueryMock(vi.fn(async () => ({ done: true, value: undefined }))); internal.turnState = "idle"; + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); await internal.routeSdkMessageFromPump( { type: "stream_event", @@ -1233,6 +1236,468 @@ test("reuses one autonomous run for unbound stream_event bursts with no foregrou await session.close(); }); +interface SessionStateInternals { + turnState: "idle" | "foreground" | "autonomous"; + routeSdkMessageFromPump: ( + message: Record, + activeQuery: QueryMock, + ) => Promise; + autonomousTurn: { id: string } | null; + activeForegroundTurnId: string | null; +} + +function sessionState(state: "idle" | "running" | "requires_action") { + return { + type: "system", + subtype: "session_state_changed", + state, + uuid: `state-${state}`, + session_id: "sess-1", + }; +} + +/** Claude opening a turn of its own. Every main-session turn starts with one. */ +function claudeTurnInit() { + return { + type: "system", + subtype: "init", + session_id: "sess-1", + permissionMode: "default", + model: "opus", + }; +} + +function successResult() { + return { type: "result", subtype: "success", usage: buildUsage(), total_cost_usd: 0 }; +} + +function commandLifecycle( + commandUuid: string, + state: "queued" | "started" | "completed" | "cancelled", +) { + return { type: "command_lifecycle", command_uuid: commandUuid, state }; +} + +function unboundTextDelta(text: string) { + return { + type: "stream_event", + parent_tool_use_id: null, + event: { type: "content_block_delta", delta: { type: "text_delta", text } }, + }; +} + +// What a background subagent emits on the main stream with no parent tool use id, so by frame +// shape alone each looks like main-session work. Verified on the wire: none arrives inside a turn +// Claude opened, and no result follows any of them. +const helperTaskStarted = { + type: "system", + subtype: "task_started", + task_id: "helper-task", + tool_use_id: "toolu_helper", + task_type: "local_agent", + subagent_type: "general-purpose", + description: "Background helper", + is_backgrounded: true, + uuid: "uuid-task-started", + session_id: "sess-1", +}; +const helperTaskProgress = { + type: "system", + subtype: "task_progress", + task_id: "helper-task", + tool_use_id: "toolu_helper", + description: "Background helper", + usage: { total_tokens: 1200, tool_uses: 3, duration_ms: 900 }, + last_tool_name: "Bash", + uuid: "uuid-task-progress", + session_id: "sess-1", +}; +// A subagent's own background Bash job completing. +const subagentOwnedTaskNotification = { + type: "system", + subtype: "task_notification", + task_id: "bg-job-of-subagent", + tool_use_id: "toolu_subagent_bash", + status: "completed", + summary: "Background command completed", + output_file: "/tmp/bg-job-of-subagent.output", + uuid: "uuid-task-notification", + session_id: "sess-1", +}; +const helperCompleted = { + type: "system", + subtype: "task_notification", + task_id: "helper-task", + tool_use_id: "toolu_helper", + status: "completed", + summary: "Background helper finished", + output_file: "/tmp/helper-task.output", + uuid: "uuid-helper-completed", + session_id: "sess-1", +}; +const rateLimitEvent = { + type: "rate_limit_event", + rate_limit_info: { status: "allowed" }, + uuid: "uuid-rate-limit", + session_id: "sess-1", +}; + +function createPendingQueryMock(): QueryMock { + return createBaseQueryMock(vi.fn(() => new Promise(() => {}))); +} + +test("a main-session frame opens no turn until Claude starts one", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + // Main-session output only ever arrives inside a turn Claude opened with system/init, so a + // stray unbound frame is not the start of one. + await internal.routeSdkMessageFromPump(unboundTextDelta("stray"), queryMock); + expect(internal.autonomousTurn).toBeNull(); + + // The wake-up case autonomous turns exist for: a completed background subagent makes the main + // session start a turn of its own, with no prompt from the user. + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(unboundTextDelta("picking this up"), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + + await session.close(); +}); + +test("a background subagent's frames never open a turn while the main session is idle", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + // Claude Code 2.1.280 reports "running" for as long as any background subagent is alive. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + for (const frame of [ + helperTaskStarted, + helperTaskProgress, + subagentOwnedTaskNotification, + rateLimitEvent, + helperCompleted, + ]) { + await internal.routeSdkMessageFromPump(frame, queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + } + + // The next user message starts its own foreground turn rather than landing on a phantom one. + const { turnId } = await session.startTurn("PING"); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("foreground"); + + await session.close(); +}); + +test("Claude waking up after a helper finishes gets an autonomous turn that its result closes", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // The helper's completion arrives outside any turn. Claude then starts one to react to it. + await internal.routeSdkMessageFromPump(helperTaskStarted, queryMock); + await internal.routeSdkMessageFromPump(helperCompleted, queryMock); + expect(internal.autonomousTurn).toBeNull(); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + expect(internal.turnState).toBe("autonomous"); + + await internal.routeSdkMessageFromPump(unboundTextDelta("the helper is done"), queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_started")).toHaveLength(1); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(1); + + await session.close(); +}); + +test("a steer folded into a turn keeps one turn, and a turn Claude starts after it gets its own", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + const { turnId } = await session.startTurn("hello"); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + // A message pushed mid-turn is folded in at the next tool boundary: Claude acknowledges it, but + // starts no new turn. + await internal.routeSdkMessageFromPump(commandLifecycle("folded-steer", "queued"), queryMock); + await internal.routeSdkMessageFromPump(commandLifecycle("folded-steer", "started"), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(internal.turnState).toBe("idle"); + + // A message that arrived while Claude wrote its final answer runs as a turn of its own. + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-2"); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_started")).toHaveLength(2); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(2); + + await session.close(); +}); + +test("session state never opens a turn", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // "running" covers background subagents too, so it says nothing about the main session. + for (const state of ["idle", "running", "requires_action", "idle"] as const) { + await internal.routeSdkMessageFromPump(sessionState(state), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + } + // State frames are bookkeeping, never timeline content. + expect(events).toEqual([]); + + await session.close(); +}); + +test("an idle session state closes an autonomous turn that no result closed", async () => { + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + // Safety net: Claude says nothing at all is running any more. + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("idle"); + expect(events.filter((event) => event.type === "turn_completed")).toHaveLength(1); + + await session.close(); +}); + +test("session state never opens or closes a foreground turn", async () => { + sdkQueryFactory.mockImplementation(() => createPendingQueryMock()); + const session = await createSession(); + const internal: SessionStateInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + const { turnId } = await session.startTurn("hello"); + + // The send's own running, then an idle that trails the previous turn and lands late. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + await internal.routeSdkMessageFromPump(sessionState("idle"), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.autonomousTurn).toBeNull(); + expect(internal.turnState).toBe("foreground"); + + await internal.routeSdkMessageFromPump( + { type: "result", subtype: "success", usage: buildUsage(), total_cost_usd: 0 }, + queryMock, + ); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(internal.turnState).toBe("idle"); + + await session.close(); +}); + +interface InterruptInternals extends SessionStateInternals { + pendingInterruptAbort: boolean; +} + +type InterruptibleQueryMock = QueryMock & { cancelAsyncMessage: ReturnType }; + +function createInterruptibleQueryMock(): InterruptibleQueryMock { + return Object.assign(createPendingQueryMock(), { + cancelAsyncMessage: vi.fn(async () => true), + }); +} + +/** The uuid Paseo stamped on the first message it pushed into the SDK input. */ +async function readFirstPromptUuid(): Promise { + const input = sdkQueryFactory.mock.calls[0]?.[0] as { prompt: AsyncIterable }; + return await createPromptUuidReader(input.prompt)(); +} + +function abortedResult() { + return { + type: "result", + subtype: "error_during_execution", + errors: [], + usage: buildUsage(), + total_cost_usd: 0, + }; +} + +test("an interrupt before Claude starts the message withdraws it instead of interrupting", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await session.startTurn("hello"); + // Claude has not acknowledged the message yet: no init, no command_lifecycle "started". + await session.interrupt(); + + await vi.waitFor(() => expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(await readFirstPromptUuid()); + expect(queryMock.interrupt).not.toHaveBeenCalled(); + // No result follows an interrupt that never happened, so no flag may wait for one. + expect(internal.pendingInterruptAbort).toBe(false); + expect(events.filter((event) => event.type === "turn_canceled")).toHaveLength(1); + + await session.close(); +}); + +test("an interrupt after Claude took the message but before it started stops it", async () => { + const queryMock = createInterruptibleQueryMock(); + // Claude has dequeued the message for its next turn, so withdrawing it is a no-op. + queryMock.cancelAsyncMessage.mockImplementation(async () => false); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + + await session.startTurn("hello"); + // No command_lifecycle "started" or init has reached Paseo yet. + await session.interrupt(); + + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(await readFirstPromptUuid()); + expect(internal.pendingInterruptAbort).toBe(true); + + await session.close(); +}); + +test("a prompt sent while a stopped prompt is being withdrawn is not withdrawn too", async () => { + const queryMock = createInterruptibleQueryMock(); + let settleWithdrawal: (cancelled: boolean) => void = () => {}; + queryMock.cancelAsyncMessage.mockImplementationOnce( + () => new Promise((resolve) => (settleWithdrawal = resolve)), + ); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + + await session.startTurn("first"); + const firstUuid = await readFirstPromptUuid(); + await session.interrupt(); + await vi.waitFor(() => expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1)); + + await session.startTurn("second"); + settleWithdrawal(false); + + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledTimes(1); + expect(queryMock.cancelAsyncMessage).toHaveBeenCalledWith(firstUuid); + + await session.close(); +}); + +test("an interrupt while only background subagents run never reaches Claude", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + + await session.startTurn("delegate to a helper"); + const promptUuid = await readFirstPromptUuid(); + await internal.routeSdkMessageFromPump(commandLifecycle(promptUuid ?? "", "started"), queryMock); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(helperTaskStarted, queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + // The main session is done; its helper keeps working. + await internal.routeSdkMessageFromPump(sessionState("running"), queryMock); + await internal.routeSdkMessageFromPump(helperTaskProgress, queryMock); + + // Without perTaskStopAffordance, an interrupt here kills every running helper. + await session.interrupt(); + + expect(queryMock.interrupt).not.toHaveBeenCalled(); + expect(queryMock.cancelAsyncMessage).not.toHaveBeenCalled(); + expect(internal.pendingInterruptAbort).toBe(false); + + await session.close(); +}); + +test("an interrupt during a main turn interrupts Claude once and drops the aborted result", async () => { + const queryMock = createInterruptibleQueryMock(); + sdkQueryFactory.mockImplementation(() => queryMock); + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + await session.startTurn("long task"); + const promptUuid = await readFirstPromptUuid(); + await internal.routeSdkMessageFromPump(commandLifecycle(promptUuid ?? "", "started"), queryMock); + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + + await session.interrupt(); + await vi.waitFor(() => expect(queryMock.interrupt).toHaveBeenCalledTimes(1)); + // Claude already started the message, so there is nothing to withdraw. + expect(queryMock.cancelAsyncMessage).not.toHaveBeenCalled(); + expect(internal.pendingInterruptAbort).toBe(true); + + // A replacement starts before the aborted turn reports its result. + const { turnId } = await session.startTurn("replacement"); + await internal.routeSdkMessageFromPump(abortedResult(), queryMock); + expect(internal.activeForegroundTurnId).toBe(turnId); + expect(internal.pendingInterruptAbort).toBe(false); + expect(events.filter((event) => event.type === "turn_failed")).toHaveLength(0); + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + await internal.routeSdkMessageFromPump(successResult(), queryMock); + expect(internal.activeForegroundTurnId).toBeNull(); + expect(queryMock.interrupt).toHaveBeenCalledTimes(1); + + await session.close(); +}); + +test("a new turn clears an interrupt flag no result will consume", async () => { + const session = await createSession(); + const internal: InterruptInternals = asInternals(session); + const queryMock = createPendingQueryMock(); + const events: AgentStreamEvent[] = []; + session.subscribe((event) => events.push(event)); + + // An interrupt that landed just after its turn had ended: Claude emits no result for it. + internal.pendingInterruptAbort = true; + + await internal.routeSdkMessageFromPump(claudeTurnInit(), queryMock); + expect(internal.pendingInterruptAbort).toBe(false); + expect(internal.autonomousTurn?.id).toBe("autonomous-turn-1"); + + // This turn's own failure is reported, not swallowed as the interrupted turn's leftover. + await internal.routeSdkMessageFromPump( + { ...abortedResult(), errors: ["API Error: 500 upstream failure"] }, + queryMock, + ); + expect(events.filter((event) => event.type === "turn_failed")).toHaveLength(1); + expect(internal.autonomousTurn).toBeNull(); + + await session.close(); +}); + test("tracks run lifecycle transitions for success, error, and interrupt", async () => { const session = await createSession(); let streamCase: "success" | "error" | "interrupt" = "success"; diff --git a/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts b/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts index d4fc0bcc0c..1223bbc541 100644 --- a/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.subagent-interrupt.test.ts @@ -78,7 +78,7 @@ describe("subagent lifecycle across an interrupt", () => { queryFactory.mockReset(); }); - async function startInterruptibleTurn() { + async function startInterruptibleTurn(taskStarted: Record = {}) { const channel = buildOpenQueryMock(); queryFactory.mockImplementation(() => channel.query); const session = await new ClaudeAgentClient({ @@ -107,6 +107,7 @@ describe("subagent lifecycle across an interrupt", () => { task_type: "local_agent", subagent_type: "general-purpose", description: "Reply with banana", + ...taskStarted, }); await vi.waitFor(() => expect(subagentEvents.length).toBeGreaterThan(0)); @@ -169,6 +170,31 @@ describe("subagent lifecycle across an interrupt", () => { await session.close(); }); + test("a child spawned in the background is not canceled with the turn that spawned it", async () => { + // Claude announces a background spawn on task_started itself and sends no task_updated patch. + const { channel, session, subagentEvents } = await startInterruptibleTurn({ + is_backgrounded: true, + }); + + await session.interrupt(); + channel.push({ + type: "system", + subtype: "task_notification", + task_id: TASK_ID, + tool_use_id: SUBAGENT_ID, + status: "completed", + }); + + await vi.waitFor(() => + expect(subagentEvents).toContainEqual( + expect.objectContaining({ id: SUBAGENT_ID, status: "completed" }), + ), + ); + expect(statuses(subagentEvents, SUBAGENT_ID)).toEqual(["running", "completed"]); + + await session.close(); + }); + test("a foreground child is canceled with the turn that spawned it", async () => { const { session, subagentEvents } = await startInterruptibleTurn(); diff --git a/packages/server/src/server/agent/providers/claude/agent.test.ts b/packages/server/src/server/agent/providers/claude/agent.test.ts index d88d8105cf..2d970c49de 100644 --- a/packages/server/src/server/agent/providers/claude/agent.test.ts +++ b/packages/server/src/server/agent/providers/claude/agent.test.ts @@ -3160,10 +3160,15 @@ describe("ClaudeAgentSession context window usage", () => { }); test("a compaction abandoned in an autonomous turn does not suppress the next marker", async () => { - // Trailing output after the foreground result opens an autonomous turn, which starts - // compacting and is then ended by the next foreground turn, never reaching a boundary. + // Claude starts a turn of its own after the foreground result, which starts compacting and is + // then ended by the next foreground turn, never reaching a boundary. const session = await createSessionForTurns([ - [createSuccessResult(), createMessageStartEvent(), createCompactingStatus()], + [ + createSuccessResult(), + createInitMessage(), + createMessageStartEvent(), + createCompactingStatus(), + ], [createCompactingStatus(), createSuccessResult()], ]); diff --git a/packages/server/src/server/agent/providers/claude/agent.ts b/packages/server/src/server/agent/providers/claude/agent.ts index ddac56a0eb..d505689a95 100644 --- a/packages/server/src/server/agent/providers/claude/agent.ts +++ b/packages/server/src/server/agent/providers/claude/agent.ts @@ -452,6 +452,15 @@ function errorToMessageString(error: unknown): string { return ""; } +/** + * Claude opens every main-session turn with system/init and closes it with exactly one result: + * a prompt, a wake-up after background work, a slash command, an API error. Background subagents + * never emit either on the main stream. + */ +function isClaudeTurnInit(message: SDKMessage): boolean { + return message.type === "system" && message.subtype === "init"; +} + function firstStringField( input: Record, primaryKey: string, @@ -2070,11 +2079,12 @@ class ClaudeAgentSession implements AgentSession { private activeForegroundQuery: Query | null = null; private activeForegroundInput: AsyncMessageInput | null = null; /** - * Steers pushed into the live SDK input that Claude may not have read yet. Interrupting the turn - * has to discard them, or the SDK dequeues one and resumes the turn we just stopped. SDK user - * UUIDs are provider-private; never let them escape the adapter boundary. + * Messages pushed into the live SDK input (steers, and a foreground turn's own prompt) whose + * command_lifecycle has not reached "started". Interrupting has to withdraw them, or the SDK + * dequeues one and resumes the turn we just stopped. SDK user UUIDs are provider-private; never + * let them escape the adapter boundary. */ - private readonly queuedSteerUuids = new Set(); + private readonly unstartedMessageUuids = new Set(); /** Human steers whose text has not reached Claude yet and therefore supersede blocking cards. */ private readonly permissionClearingSteerUuids = new Set(); private claudeSessionId: string | null; @@ -2126,6 +2136,12 @@ class ClaudeAgentSession implements AgentSession { private queryPumpPromise: Promise | null = null; private queryRestartNeeded = false; private pendingInterruptAbort = false; + /** + * Whether Claude's main session is inside one of its own turns: set by system/init and by a + * pushed message's command_lifecycle "started" (/compact works for seconds before its init), + * cleared by the turn's result or an idle session state. See observeMainTurnBracket. + */ + private mainTurnInFlight = false; private foregroundHasVisibleActivity = false; private activeTurnHasAssistantText = false; private readonly contextUsage: ClaudeContextUsageState; @@ -2272,6 +2288,7 @@ class ClaudeAgentSession implements AgentSession { this.contextUsage.beginTurn(); this.transitionTurnState("foreground", "foreground turn started"); this.clearRecentStderr(); + if (sdkUserMessageId) this.unstartedMessageUuids.add(sdkUserMessageId); let cancelIssued = false; const requestCancel = () => { @@ -2282,13 +2299,18 @@ class ClaudeAgentSession implements AgentSession { if (this.cancelCurrentTurn === requestCancel) { this.cancelCurrentTurn = null; } + // Whether Claude has this message in hand. A late idle from the previous turn can clear + // mainTurnInFlight after Claude started it, so the message's own lifecycle counts too. + const claudeStartedTurn = + this.mainTurnInFlight || + (sdkUserMessageId !== null && !this.unstartedMessageUuids.has(sdkUserMessageId)); this.rejectAllPendingPermissions(new Error("Permission request canceled")); this.finishForegroundTurn({ type: "turn_canceled", provider: "claude", reason: "Interrupted", }); - void this.interruptActiveTurn().catch((error) => { + void this.interruptActiveTurn(claudeStartedTurn).catch((error) => { this.logger.warn({ err: error }, "Failed to interrupt during cancel"); }); }; @@ -2298,6 +2320,11 @@ class ClaudeAgentSession implements AgentSession { try { await this.ensureQuery(); + if (cancelIssued) { + // Stopped while Claude was still starting up: withdrawn by never sending it. + if (sdkUserMessageId) this.unstartedMessageUuids.delete(sdkUserMessageId); + return { turnId }; + } if (!this.input) { throw new Error("Claude session input stream not initialized"); } @@ -2311,6 +2338,7 @@ class ClaudeAgentSession implements AgentSession { } }, 0); } catch (error) { + if (sdkUserMessageId) this.unstartedMessageUuids.delete(sdkUserMessageId); this.finishForegroundTurn( this.buildTurnFailedEvent(error instanceof Error ? error.message : "Claude stream failed"), ); @@ -2359,7 +2387,7 @@ class ClaudeAgentSession implements AgentSession { clearPendingPermissions: boolean, ): void { const uuid = message.uuid; - if (uuid) this.queuedSteerUuids.add(uuid); + if (uuid) this.unstartedMessageUuids.add(uuid); if (uuid && clearPendingPermissions) { this.permissionClearingSteerUuids.add(uuid); } @@ -2370,7 +2398,7 @@ class ClaudeAgentSession implements AgentSession { } } catch (error) { if (uuid) { - this.queuedSteerUuids.delete(uuid); + this.unstartedMessageUuids.delete(uuid); this.permissionClearingSteerUuids.delete(uuid); } throw error; @@ -3182,6 +3210,8 @@ class ClaudeAgentSession implements AgentSession { const options = await this.buildOptions(launchMode); this.logger.debug({ options: summarizeClaudeOptionsForLog(options) }, "claude query"); this.input = input; + // A fresh Claude process has no turn of its own in flight. + this.mainTurnInFlight = false; this.query = claudeQuery( { prompt: input.iterable, options }, { @@ -3301,7 +3331,12 @@ class ClaudeAgentSession implements AgentSession { return createProviderEnv({ baseEnv: process.env, runtimeSettings: this.runtimeSettings, - overlays: [this.launchEnv], + overlays: [ + this.launchEnv, + // Makes the CLI announce session_state_changed, whose "idle" is the safety net that closes + // an autonomous turn (see consumeSessionStateChange). + { CLAUDE_CODE_EMIT_SESSION_STATE_EVENTS: "1" }, + ], }); } @@ -3368,6 +3403,10 @@ class ClaudeAgentSession implements AgentSession { ...settingsOptions, // Provider subagent panes render the child's nested transcript. forwardSubagentText: true, + // Stop and replace abort only the main turn, like Esc in Claude Code, and background + // subagents keep running. Without this declaration an interrupt kills every one of them. + // Claude stops a helper itself with its TaskStop tool. + perTaskStopAffordance: true, hooks: this.buildSubagentEffortHooks(), ...(this.persistSession === undefined ? {} : { persistSession: this.persistSession }), env: sdkEnv, @@ -3898,35 +3937,67 @@ class ClaudeAgentSession implements AgentSession { return false; } - private isAssistantishMessage(message: SDKMessage): boolean { - return ( - message.type === "assistant" || - message.type === "stream_event" || - message.type === "tool_progress" || - (message.type === "system" && message.subtype === "task_notification") - ); + /** + * Claude's account of whether anything in the session is working, which is not the same as the + * main session working: Claude Code 2.1.280 stays "running" for as long as any background + * subagent is alive. So "running" never opens a turn. + * + * "idle" means nothing at all is running, so it closes an autonomous turn whatever opened it. + * It never closes a foreground turn: an idle can trail the previous turn and land just after a + * new send, and it must not end that send. + * + * Returns true when the frame was a state change, which is bookkeeping and never reaches the + * timeline. + */ + private consumeSessionStateChange(message: SDKMessage): boolean { + if (message.type !== "system" || message.subtype !== "session_state_changed") { + return false; + } + if (message.state === "idle") { + this.mainTurnInFlight = false; + this.completeAutonomousTurn(); + } + return true; + } + + /** Tracks mainTurnInFlight from Claude's own turn brackets (isClaudeTurnInit). */ + private observeMainTurnBracket(message: SDKMessage): void { + if (message.type === "result") { + this.mainTurnInFlight = false; + return; + } + if (isClaudeTurnInit(message)) { + this.mainTurnInFlight = true; + // A new turn has begun, so the interrupted one is over. An interrupt that landed after its + // turn had already ended gets no result, and the leftover flag would swallow this turn's. + this.pendingInterruptAbort = false; + return; + } + if (readClaudeCommandLifecycle(message)?.state === "started") { + this.mainTurnInFlight = true; + } } /** - * Claude keeps talking about the request it was told to kill — the notification for the tool it - * just stopped, trailing assistant output. That is not new work, so it must not open a turn: the - * only message that would close that turn is the result shouldSuppressStaleResult drops, leaving - * the agent stuck reporting "running" forever. + * Only the main session's own turn start opens an autonomous turn. Everything a background + * subagent emits (its task frames, its own background jobs' notifications, rate limit events) + * arrives outside one, and no result would ever close a turn it opened. */ - private shouldStartAutonomousTurn(message: SDKMessage): boolean { - if (this.activeForegroundTurnId || this.pendingInterruptAbort) { - return false; - } - return this.isAssistantishMessage(message); + private opensAutonomousTurn(message: SDKMessage): boolean { + return isClaudeTurnInit(message) && !this.activeForegroundTurnId; } private async routeSdkMessageFromPump(message: SDKMessage): Promise { + if (this.consumeSessionStateChange(message)) { + return; + } + this.observeMainTurnBracket(message); if (this.shouldSuppressStaleResult(message)) { return; } const isForeground = Boolean(this.activeForegroundTurnId); - if (this.shouldStartAutonomousTurn(message)) { + if (this.opensAutonomousTurn(message)) { this.startAutonomousTurn(); } if (!isForeground && !this.autonomousTurn && message.type === "result") { @@ -4068,7 +4139,13 @@ class ClaudeAgentSession implements AgentSession { return true; } - private async interruptActiveTurn(): Promise { + /** + * Interrupts Claude only when a turn of its main session is running. With nothing running there + * is nothing to stop but our own queued messages, so those are withdrawn instead: an interrupt + * would reach only background helpers (and kill them on a CLI that ignores + * perTaskStopAffordance), and no result would follow to consume pendingInterruptAbort. + */ + private async interruptActiveTurn(claudeStartedTurn = this.mainTurnInFlight): Promise { const queryToInterrupt = this.query; if (!queryToInterrupt || typeof queryToInterrupt.interrupt !== "function") { this.logger.trace( @@ -4082,8 +4159,16 @@ class ClaudeAgentSession implements AgentSession { ); return; } - this.pendingInterruptAbort = true; - await this.discardQueuedSteers(queryToInterrupt); + if (claudeStartedTurn) { + this.pendingInterruptAbort = true; + await this.withdrawUnstartedMessages(queryToInterrupt); + } else { + // Claude can dequeue a message for its next turn before announcing it started; withdrawing + // it is then a no-op, and only an interrupt stops it. Withdraw once: a prompt sent while + // this withdrawal was in flight belongs to the next turn. + if (await this.withdrawUnstartedMessages(queryToInterrupt)) return; + this.pendingInterruptAbort = true; + } try { await this.awaitWithTimeout( queryToInterrupt.interrupt(), @@ -4095,14 +4180,15 @@ class ClaudeAgentSession implements AgentSession { } /** - * Interrupt means interrupt: a steer Claude never read dies with the turn instead of resuming it. - * A steer already dequeued cannot be recalled, and does not need to be — the interrupt kills it. + * Interrupt means interrupt: a message Claude never started dies with the turn instead of + * resuming it. One already dequeued cannot be recalled, and does not need to be, since the + * interrupt kills it. Returns whether Claude confirmed every message withdrawn. */ - private async discardQueuedSteers(query: Query): Promise { - const uuids = [...this.queuedSteerUuids]; - this.queuedSteerUuids.clear(); + private async withdrawUnstartedMessages(query: Query): Promise { + const uuids = [...this.unstartedMessageUuids]; + this.unstartedMessageUuids.clear(); this.permissionClearingSteerUuids.clear(); - if (uuids.length === 0) return; + if (uuids.length === 0) return true; // The SDK runtime supports this, but its public Query type has not caught up. Keep the // compatibility escape hatch inside the Claude adapter. const cancelAsyncMessage = ( @@ -4110,14 +4196,17 @@ class ClaudeAgentSession implements AgentSession { cancelAsyncMessage?: (uuid: string) => Promise; } ).cancelAsyncMessage; - if (!cancelAsyncMessage) return; + if (!cancelAsyncMessage) return false; + let withdrewAll = true; for (const uuid of uuids) { try { - await cancelAsyncMessage.call(query, uuid); + if (!(await cancelAsyncMessage.call(query, uuid))) withdrewAll = false; } catch (error) { - this.logger.warn({ err: error }, "Failed to discard a queued Claude steer"); + withdrewAll = false; + this.logger.warn({ err: error }, "Failed to withdraw a queued Claude message"); } } + return withdrewAll; } private translateMessageToEvents( @@ -4166,7 +4255,7 @@ class ClaudeAgentSession implements AgentSession { } } - this.forgetReadSteer(message); + this.forgetStartedMessage(message); switch (message.type) { case "system": @@ -4200,11 +4289,14 @@ class ClaudeAgentSession implements AgentSession { return events; } - /** Once Claude has read a steer there is nothing left to discard on interrupt. */ - private forgetReadSteer(message: unknown): void { + /** + * Once Claude has started a message, or cancelled it, there is nothing left to withdraw on + * interrupt. + */ + private forgetStartedMessage(message: unknown): void { const lifecycle = readClaudeCommandLifecycle(message); if (!lifecycle || lifecycle.state === "queued") return; - this.queuedSteerUuids.delete(lifecycle.commandUuid); + this.unstartedMessageUuids.delete(lifecycle.commandUuid); this.permissionClearingSteerUuids.delete(lifecycle.commandUuid); } @@ -6519,7 +6611,8 @@ function extractClaudeUserText(messageRaw: unknown): string | null { interface ClaudeCommandLifecycle { commandUuid: string; - state: "queued" | "started" | "completed"; + /** "cancelled" is terminal too: API errors, interrupts and "now" pre-emption end a message. */ + state: "queued" | "started" | "completed" | "cancelled"; } /** Runtime-only Claude frames are validated here because the SDK's public union omits them. */ @@ -6528,7 +6621,7 @@ function readClaudeCommandLifecycle(message: unknown): ClaudeCommandLifecycle | if (record?.type !== "command_lifecycle") return null; if ( typeof record.command_uuid !== "string" || - !["queued", "started", "completed"].includes(String(record.state)) + !["queued", "started", "completed", "cancelled"].includes(String(record.state)) ) { return null; } diff --git a/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts b/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts index 1eca4cedc2..b0bfc3d755 100644 --- a/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts +++ b/packages/server/src/server/agent/providers/claude/subagents/live-source.test.ts @@ -511,6 +511,33 @@ describe("ClaudeTaskProtocolSource", () => { expect(source.cancelRunningForegroundTasks()).toEqual([]); }); + it("does not cancel a subagent announced as backgrounded at spawn", () => { + // Claude sets is_backgrounded on task_started for a child spawned in the background, and + // sends no task_updated patch for it. + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ is_backgrounded: true })); + + expect(source.cancelRunningForegroundTasks()).toEqual([]); + }); + + it("does not cancel a resumed subagent, which Claude always runs in the background", () => { + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ tool_use_id: "toolu_original", is_backgrounded: false })); + source.observe(taskUpdated("completed")); + source.observe(taskStarted({ tool_use_id: "toolu_resumed", is_backgrounded: true })); + + expect(source.cancelRunningForegroundTasks()).toEqual([]); + }); + + it("still cancels a subagent announced in the foreground", () => { + const source = new ClaudeTaskProtocolSource(); + source.observe(taskStarted({ is_backgrounded: false })); + + expect(source.cancelRunningForegroundTasks()).toEqual([ + { kind: "status", id: "toolu_01DgLoPMW9", status: "canceled" }, + ]); + }); + it("still routes a backgrounded subagent that settles after the interrupt", () => { // The headline case: interrupt, continue, and the child that was told to outlive the turn // reports completion later. Wiping the routing table on cancel drops this on the floor and diff --git a/packages/server/src/server/agent/providers/claude/subagents/live-source.ts b/packages/server/src/server/agent/providers/claude/subagents/live-source.ts index 1a44520366..2149fae139 100644 --- a/packages/server/src/server/agent/providers/claude/subagents/live-source.ts +++ b/packages/server/src/server/agent/providers/claude/subagents/live-source.ts @@ -16,7 +16,8 @@ import { * * Verified on the wire (Claude Code 2.1.220, Paseo's own query options): * - * task_started task_id, tool_use_id, description, subagent_type, task_type + * task_started task_id, tool_use_id, description, subagent_type, task_type, + * is_backgrounded * task_updated task_id, patch.status, patch.is_backgrounded * task_notification task_id, tool_use_id, status * @@ -39,6 +40,11 @@ interface TaskStartedMessage { task_type?: string; prompt?: string; skip_transcript?: boolean; + /** + * Set at spawn: true for a task registered in the background. A resumed subagent is always + * registered in the background. + */ + is_backgrounded?: boolean; } /** Task-tool subagents. Backgrounded shell commands announce as `local_bash`. */ @@ -71,8 +77,8 @@ interface TaskUpdatedMessage { task_id: string; /** * `is_backgrounded` is declared on `SDKTaskUpdatedMessage["patch"]`: it flips when a foreground - * task is backgrounded, which is the only signal that separates a child that dies with its turn - * from one that was explicitly told to outlive it. + * task is moved to the background after `task_started`. Together with `task_started`'s own + * flag it separates a child that dies with its turn from one told to outlive it. */ patch?: { status?: string; is_backgrounded?: boolean }; } @@ -171,8 +177,8 @@ export class ClaudeTaskProtocolSource { /** Workflow invocations already own a real Workflow card in the parent timeline. */ private readonly idsWithExistingParentToolCard = new Set(); /** - * Declared subagents that were moved to the background. They outlive the turn that spawned - * them, so a turn ending is not evidence that they stopped. + * Declared subagents running in the background, whether spawned there or moved there later. + * They outlive the turn that spawned them, so a turn ending is not evidence that they stopped. */ private readonly backgroundedIds = new Set(); /** Last status emitted per subagent, so a redundant announcement is not re-broadcast. */ @@ -331,6 +337,7 @@ export class ClaudeTaskProtocolSource { this.sawTaskStarted = true; const existingId = this.subagentIdByTaskId.get(message.task_id); + this.recordBackgrounded(existingId ?? id, message.is_backgrounded); if (existingId) { return this.observeExistingTaskStart(message, id, existingId); } @@ -414,14 +421,15 @@ export class ClaudeTaskProtocolSource { private observeTaskUpdated(message: TaskUpdatedMessage): SubagentObservation[] { const id = this.subagentIdByTaskId.get(message.task_id); - const backgrounded = message.patch?.is_backgrounded; - if (id && typeof backgrounded === "boolean") { - if (backgrounded) this.backgroundedIds.add(id); - else this.backgroundedIds.delete(id); - } + if (id) this.recordBackgrounded(id, message.patch?.is_backgrounded); return this.observeStatus(message.task_id, message.patch?.status); } + private recordBackgrounded(id: string, backgrounded: boolean | undefined): void { + if (backgrounded === true) this.backgroundedIds.add(id); + else if (backgrounded === false) this.backgroundedIds.delete(id); + } + private observeTaskNotification(message: TaskNotificationMessage): SubagentObservation[] { const observations = this.observeWorkflowResult(message); observations.push(...this.observeUsage(message.task_id, message.usage)); diff --git a/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts b/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts index 2471ff8c5a..5eea6dab24 100644 --- a/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts +++ b/packages/server/src/server/daemon-e2e/send-during-tool-call-claude.real.e2e.test.ts @@ -304,6 +304,166 @@ async function waitForRunningClaudeSleep( ); } +// Explicit point-in-time polls of agent status, on top of (not instead of) the pushed +// agent_update events the collector already records. A phantom "running" flip that starts and +// ends between two polls would still be caught by the pushed-event scan the callers pair this +// with; polling here additionally proves the daemon answers fetchAgent with "idle" throughout, +// not just that it eventually pushes one. +async function sampleAgentStatuses( + client: DaemonClient, + agentId: string, + samples: number, + intervalMs: number, +): Promise { + const statuses: string[] = []; + for (let i = 0; i < samples; i += 1) { + const snapshot = await client.fetchAgent({ agentId }); + statuses.push(snapshot?.agent.status ?? "missing"); + if (i < samples - 1) { + await sleep(intervalMs); + } + } + return statuses; +} + +// NOTE: agent_stream events on the wire (AgentStreamEventPayloadSchema in +// packages/protocol/src/messages.ts) do not include a "provider_subagent" variant. A backgrounded +// subagent's descriptor updates are forwarded exclusively as separate "agent.provider_subagents.update" +// messages (see forwardProviderSubagentUpdate in session.ts), and that push channel's delivery is +// gated by capabilities this test's plain DaemonClient does not declare. list_provider_subagents is a +// plain request/response RPC unaffected by that gating, so status history below polls it directly +// instead of trying to reconstruct status transitions from the stream. +type ProviderSubagentDescriptor = Awaited< + ReturnType +>["subagents"][number]; + +interface ProviderSubagentPoll { + subagent: ProviderSubagentDescriptor; + // Every status observed while polling, in order, so a transient cancel cannot hide behind a + // later "completed" read even though the store's status field is not expected to un-cancel. + history: string[]; +} + +// Polls agent.provider_subagents.list until the backgrounded helper is registered at all. +// "Main's turn is running" (waitForAgentUpsert) fires on generic turn-start bookkeeping, which +// can be true before Claude has actually issued the Agent tool call; canceling that early +// cancels a turn with no helper launched yet, which is not what "Stop spares a helper" means to +// test. This confirms the helper actually exists before the caller acts on it. +async function waitForProviderSubagentToExist( + client: DaemonClient, + agentId: string, + timeoutMs = 60_000, +): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const { subagents } = await client.listProviderSubagents(agentId); + if (subagents[0]) { + return subagents[0]; + } + await sleep(300); + } + throw new Error( + `Timed out waiting for a background helper to be registered for agent ${agentId}`, + ); +} + +// Claude starts the wake-up turn a moment after the helper's completion lands (0.1 to 5 s in +// recorded traces), so poll for it instead of checking the instant the helper reads completed. +// The daemon announces an autonomous turn to subscribers as a pushed "running" agent_update, +// not as an agent_stream turn_started (that stream reaches the client that started a run). +async function waitForWakeTurn( + collector: ReturnType, + agentId: string, + fromIndex: number, + timeoutMs = 30_000, +): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + const window = collector.messages.slice(fromIndex); + const woke = + getAgentStatuses(window, agentId).includes("running") || + window.some( + (message) => + message.type === "agent_stream" && + message.payload.agentId === agentId && + message.payload.event.type === "turn_started", + ); + if (woke) return true; + await sleep(500); + } + return false; +} + +// The wake-up turn runs briefly after it opens, so the final state is read once it settles. +async function waitForAgentStatus( + client: DaemonClient, + agentId: string, + status: string, + timeoutMs = 60_000, +): Promise { + const deadline = Date.now() + timeoutMs; + let last: string | undefined; + while (Date.now() < deadline) { + last = (await client.fetchAgent({ agentId }))?.agent.status; + if (last === status) return last; + await sleep(500); + } + return last; +} + +// Polls agent.provider_subagents.list until the single backgrounded helper leaves "running". +// Scenarios in this file always launch exactly one background helper, so subagents[0] is it. +async function waitForProviderSubagentTerminal( + client: DaemonClient, + agentId: string, + timeoutMs = 90_000, +): Promise { + const deadline = Date.now() + timeoutMs; + const history: string[] = []; + let lastSeen: ProviderSubagentDescriptor[] = []; + while (Date.now() < deadline) { + const { subagents } = await client.listProviderSubagents(agentId); + lastSeen = subagents; + const subagent = subagents[0]; + if (subagent) { + history.push(subagent.status); + if (subagent.status !== "running") { + return { subagent, history }; + } + } + await sleep(1_000); + } + throw new Error( + `Timed out waiting for the background helper to leave "running". history=${JSON.stringify(history)} lastSeen=${JSON.stringify(lastSeen)}`, + ); +} + +// Pulls the main agent's own assistant text via fetchAgentTimeline rather than the push +// collector. Assistant text delivery is paced (docs/agent-stream-performance.md), so checking +// the collector immediately after waitForFinish resolves can still miss the final text; the +// timeline fetch is authoritative and matches what the pre-existing tests in this file do. +async function fetchAssistantTexts( + client: DaemonClient, + agentId: string, + limit = 100, +): Promise { + const timeline = await client.fetchAgentTimeline(agentId, { limit }); + return timeline.entries + .filter((entry) => entry.item.type === "assistant_message") + .map((entry) => (entry.item as Extract).text); +} + +async function fetchSubagentAssistantTexts( + client: DaemonClient, + agentId: string, + subagentId: string, +): Promise { + const timeline = await client.fetchProviderSubagentTimeline(agentId, subagentId, { limit: 50 }); + return timeline.rows + .filter((row) => row.item.type === "assistant_message") + .map((row) => (row.item as Extract).text); +} + describe("daemon E2E (real claude) - send message during tool call", () => { let canRun = false; interface SteeringResources { @@ -504,6 +664,141 @@ describe("daemon E2E (real claude) - send message during tool call", () => { } }, 210_000); + test("a background subagent keeps running after a message is sent while it works", async () => { + // The scenario the rest of this file explicitly rules out: every other test here tells Claude + // "Do not use a background task", so steering during a backgrounded subagent was never + // covered. A subagent streams on the parent's channel, so a parent that mistakes the child's + // frames for its own work reports "running" after it has already finished, and the message + // sent into that window is refused, or delivered by cancelling the turn and taking the + // subagent with it. Both outcomes are user-visible, and both are what this asserts against. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("Claude background-subagent test client was not created"); + await within("connect background-subagent test client", 15_000, client.connect()); + await within( + "subscribe background-subagent test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create background-subagent test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-background-survives-send", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + await within( + "launch one background subagent and end the turn", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor".`, + `prompt: "Use the Bash tool with run_in_background: true to run exactly: sleep 40.`, + `Wait for its completion notification, then reply exactly: CHILD_DONE."`, + "Immediately after launching it, reply with the single word LAUNCHED and end your turn.", + "Do not wait for it. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // The main session has ended its turn. It must read as idle even though the subagent it + // launched is still streaming on the shared channel. + const afterLaunch = await within( + "wait for the main session to go idle after launching the subagent", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect( + afterLaunch.status, + "a main session that finished its turn must not report busy because a subagent is running", + ).not.toBe("running"); + + // And a message sent in that window must be accepted rather than refused. + const followUpId = generateClientMessageId(); + await within( + "send a message to the idle main session while the subagent runs", + 30_000, + client.sendAgentMessage(agent.id, "Are you there? Reply exactly: MAIN_ALIVE.", { + messageId: followUpId, + }), + ); + await within( + "wait for the main session to answer", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + + const timeline = await within( + "fetch background-subagent timeline", + 15_000, + client.fetchAgentTimeline(agent.id, { limit: 200 }), + ); + const assistantText = timeline.entries + .filter((entry) => entry.item.type === "assistant_message") + .map( + (entry) => (entry.item as Extract).text, + ) + .join("\\n"); + expect( + assistantText, + "the main session must be reachable while its background subagent still runs", + ).toContain("MAIN_ALIVE"); + + // The subagent must have survived the send rather than being cancelled with the turn. Its + // status is read from agent.provider_subagents.list (see the note above + // waitForProviderSubagentToExist): agent_stream carries no provider_subagent events. + const { subagents } = await within( + "list provider subagents after the send", + 15_000, + client.listProviderSubagents(agent.id), + ); + expect(subagents.length, "the background subagent must be tracked").toBeGreaterThan(0); + expect( + subagents.map((subagent) => subagent.status), + "a backgrounded subagent must not be canceled by a message sent to the main session", + ).not.toContain("canceled"); + const { subagent, history } = await within( + "wait for the background subagent to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect(history, "no status poll may read canceled").not.toContain("canceled"); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "Claude background-subagent E2E cleanup failures").toEqual([]); + } + }, 300_000); + test("Stop cancels a queued Claude steer before it can resume the interrupted turn", async () => { const logger = pino({ level: "silent" }); const cwd = tmpCwd(); @@ -678,4 +973,430 @@ describe("daemon E2E (real claude) - send message during tool call", () => { rmSync(cwd, { recursive: true, force: true }); } }, 300_000); + + test("a helper's own nested background job does not phantom-run the parent or kill the helper", async () => { + // The exact reported bug: main backgrounds a helper, the helper itself starts its OWN + // background job (a grandchild from main's perspective) and then keeps working in the + // foreground. The grandchild's completion is bookkeeping that belongs to the helper, not to + // main, and must not make main phantom-report "running" or get treated as busy. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("nested background-job test client was not created"); + await within("connect nested background-job test client", 15_000, client.connect()); + await within( + "subscribe nested background-job test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create nested background-job test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-nested-background-job", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch a helper with its own nested background job and end the turn", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "nested bg survivor".`, + `prompt: "Use the Bash tool with run_in_background: true to run exactly:`, + `python3 -c 'import time; time.sleep(5)'.`, + `Immediately after starting it, without waiting for it, use the Bash tool again`, + `in the foreground to run exactly: python3 -c 'import time; time.sleep(50)'.`, + `After that foreground command finishes, reply exactly: CHILD_DONE."`, + "Immediately after launching it, reply with the single word LAUNCHED and end your turn.", + "Do not wait for it. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + const afterLaunch = await within( + "wait for the main session to go idle after launching the nested-background helper", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect(afterLaunch.status, "the main session must be idle once it replied LAUNCHED").toBe( + "idle", + ); + + // The helper's own nested background job (5s) finishes while the helper is still busy + // in its unrelated foreground sleep (50s). Sample for 30s, well past the nested job's + // completion even when the helper is slow to start it, + // both by polling and by scanning every pushed agent_update in the window, so a brief + // phantom flip cannot slip between polls. + const sampleWindowStart = resources.collector.messages.length; + const polledStatuses = await sampleAgentStatuses(client, agent.id, 11, 3_000); + const pushedStatuses = getAgentStatuses( + resources.collector.messages.slice(sampleWindowStart), + agent.id, + ); + const observedRunning = [...polledStatuses, ...pushedStatuses].filter( + (status) => status === "running", + ); + expect( + observedRunning, + `the parent must stay idle across the nested job's completion; polled=${JSON.stringify(polledStatuses)} pushed=${JSON.stringify(pushedStatuses)}`, + ).toEqual([]); + + // A message sent in that window must still be answered normally. + await within( + "send PONG request while the nested job settles", + 30_000, + client.sendMessage(agent.id, "Reply with exactly PONG.", { + messageId: generateClientMessageId(), + }), + ); + await within("wait for the PONG reply", 60_000, client.waitForFinish(agent.id, 50_000)); + // Pulled via fetchAgentTimeline, not the push collector: assistant text delivery is paced + // (docs/agent-stream-performance.md) and can trail waitForFinish's own completion signal, + // so the collector can still be missing the final text the instant waitForFinish resolves. + const pongTexts = await within( + "fetch timeline for the PONG reply", + 15_000, + fetchAssistantTexts(client, agent.id, 20), + ); + expect( + pongTexts.join("\n"), + "the agent must answer PONG while the helper is still running", + ).toContain("PONG"); + + const messagesBeforeWake = resources.collector.messages.length; + + // The helper itself must survive: never recorded canceled, and it eventually reports + // CHILD_DONE from its own foreground sleep finishing. + const { subagent, history } = await within( + "wait for the nested-background helper to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + + // The helper's completion should wake the main session with a fresh turn (the + // session-state-driven autonomous turn), separate from the PONG exchange above. + const wokeMainAgent = await waitForWakeTurn( + resources.collector, + agent.id, + messagesBeforeWake, + ); + if (!wokeMainAgent) { + const wakeWindowMessages = resources.collector.messages.slice(messagesBeforeWake); + const wakeWindowEventTypes = wakeWindowMessages + .filter( + (message) => message.type === "agent_stream" && message.payload.agentId === agent.id, + ) + .map((message) => + message.type === "agent_stream" ? message.payload.event.type : "unknown", + ); + const timeline = await client + .fetchAgentTimeline(agent.id, { limit: 100 }) + .catch(() => null); + throw new Error( + `the helper's completion must wake the main session with a new turn. ` + + `wakeWindowEventTypes=${JSON.stringify(wakeWindowEventTypes)} ` + + `recentTimeline=${JSON.stringify(timeline ? summarizeTimelineItems(timeline) : null)}`, + ); + } + + expect(await waitForAgentStatus(client, agent.id, "idle"), "the agent must end idle").toBe( + "idle", + ); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "nested background-job E2E cleanup failures").toEqual([]); + } + }, 300_000); + + test("the app's default steer survives a backgrounded helper while the main session works", async () => { + // appSettings.sendBehavior defaults to "steer" (packages/app/src/hooks/use-settings/storage.ts), + // so this is what the app actually sends when a user replies while an agent looks busy. + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("default-steer background-helper test client was not created"); + await within("connect default-steer test client", 15_000, client.connect()); + await within( + "subscribe default-steer test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create default-steer test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-default-steer-background-helper", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch the background helper and start the main session's own foreground sleep", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor b".`, + `prompt: "Use the Bash tool in the foreground to run exactly:`, + `python3 -c 'import time; time.sleep(42)'. After it finishes, reply exactly: CHILD_DONE."`, + "Do not wait for that background helper to finish before continuing.", + "After launching it, use the Bash tool one more time yourself, WITHOUT run_in_background,", + "to run exactly: python3 -c 'import time; time.sleep(16)'.", + // Matching the proven pattern from the first test in this file: tell Claude to wait + // for a later message rather than relying on the sleep call itself to block, since a + // Bash call can resolve through Claude Code's own async task_notification path once a + // background subagent is already active instead of blocking the turn. + "Do not reply or finish your turn until a later user message arrives, no matter how", + "quickly that Bash command finishes. When that later message arrives, address it and", + "end your turn. Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // Confirm the daemon considers main's own turn active before steering it. This does not + // wait for a specific "sleep 16" shell call to show as running: Claude Code 2.1.280 can + // report a slower Bash call through its own async task_notification tool instead of a + // classic blocking call once a background subagent is already active, so pinning to a + // literal running shell tool_call is not reliable here. Busy at the daemon level is what + // "while main is busy with its own sleep" actually needs. + await within( + "confirm main's own turn is active before steering", + 30_000, + client.waitForAgentUpsert(agent.id, (snapshot) => snapshot.status === "running", 25_000), + ); + // The running status precedes the Agent tool call, so a steer sent on it alone can reach + // Claude before the helper exists and prove nothing about a helper surviving it. + await within( + "confirm the background helper is registered before steering", + 60_000, + waitForProviderSubagentToExist(client, agent.id, 55_000), + ); + + await within( + "send the app's default steer while main is busy with its own sleep", + 30_000, + client.sendMessage(agent.id, "Reply with exactly PONG.", { + messageId: generateClientMessageId(), + activeTurnBehavior: "steer", + }), + ); + + const finish = await within( + "wait for the main session to finish after the steer", + 120_000, + client.waitForFinish(agent.id, 110_000), + ); + expect(finish.status, "the main session must end idle after the steer").toBe("idle"); + // Pulled via fetchAgentTimeline, not the push collector: see fetchAssistantTexts. + const postSteerTexts = await within( + "fetch timeline for the post-steer PONG reply", + 15_000, + fetchAssistantTexts(client, agent.id, 20), + ); + expect(postSteerTexts.join("\n"), "PONG must be answered after the steer").toContain("PONG"); + + const { subagent, history } = await within( + "wait for the background helper to finish", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "the helper must complete, not be canceled or fail").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "default-steer background-helper E2E cleanup failures").toEqual([]); + } + }, 300_000); + + test("Stop cancels the main session's own turn but spares its background helper", async () => { + const logger = pino({ level: "silent" }); + const resources: SteeringResources = { + cwd: tmpCwd(), + daemon: null, + client: null, + collector: null, + }; + try { + resources.daemon = await createTestPaseoDaemon({ + agentClients: { claude: new ClaudeAgentClient({ logger }) }, + logger, + }); + resources.client = new DaemonClient({ url: `ws://127.0.0.1:${resources.daemon.port}/ws` }); + const { client, cwd } = resources; + if (!client) throw new Error("Stop-spares-helper test client was not created"); + await within("connect Stop-spares-helper test client", 15_000, client.connect()); + await within( + "subscribe Stop-spares-helper test client", + 15_000, + client.fetchAgents({ subscribe: {} }), + ); + const agent = await within( + "create Stop-spares-helper test agent", + 30_000, + client.createAgent({ + cwd: cwd ?? process.cwd(), + title: "claude-stop-spares-helper", + ...getRealProviderConfig("claude"), + }), + ); + resources.collector = createMessageCollector(client); + + await within( + "launch the background helper and start the main session's own foreground sleep", + 30_000, + client.sendAgentMessage( + agent.id, + [ + "Use the Agent tool exactly once with run_in_background: true.", + "subagent_type: general-purpose.", + `description: "bg survivor c".`, + `prompt: "Use the Bash tool in the foreground to run exactly:`, + `python3 -c 'import time; time.sleep(43)'. After it finishes, reply exactly: CHILD_DONE."`, + "Do not wait for that background helper to finish before continuing.", + "After launching it, use the Bash tool one more time yourself, WITHOUT run_in_background,", + "to run exactly: python3 -c 'import time; time.sleep(23)'.", + "Do not call any other tools.", + ].join(" "), + { messageId: generateClientMessageId() }, + ), + ); + + // Confirm the helper has actually been launched (registered) before Stop, not just that + // main's turn shows "running": that status can flip true on generic turn-start bookkeeping + // before Claude has issued the Agent tool call, and canceling that early cancels a turn + // with no helper to spare yet. + await within( + "confirm the background helper is registered before Stop", + 60_000, + waitForProviderSubagentToExist(client, agent.id, 55_000), + ); + + const messagesBeforeStop = resources.collector.messages.length; + await within("Stop the main session's own turn", 30_000, client.cancelAgent(agent.id)); + const afterStop = await within( + "wait for the agent to go idle after Stop", + 60_000, + client.waitForFinish(agent.id, 50_000), + ); + expect(afterStop.status, "Stop must leave the agent idle").toBe("idle"); + + // The helper must survive Stop: it keeps running and later reports CHILD_DONE, which + // wakes the main session with a fresh autonomous turn. + const { subagent, history } = await within( + "wait for the background helper to finish after Stop", + 90_000, + waitForProviderSubagentTerminal(client, agent.id, 80_000), + ); + expect(subagent.status, "Stop must not cancel or fail the background helper").toBe( + "completed", + ); + expect( + history, + `no provider subagent status poll may ever read canceled; history=${JSON.stringify(history)}`, + ).not.toContain("canceled"); + const subagentTexts = await fetchSubagentAssistantTexts(client, agent.id, subagent.id); + expect( + subagentTexts.join("\n"), + "the helper's own transcript must contain its final CHILD_DONE reply", + ).toContain("CHILD_DONE"); + + const wokeMainAgent = await waitForWakeTurn( + resources.collector, + agent.id, + messagesBeforeStop, + ); + expect( + wokeMainAgent, + "the helper's completion after Stop must wake the main session with a new turn", + ).toBe(true); + + expect(await waitForAgentStatus(client, agent.id, "idle"), "the agent must end idle").toBe( + "idle", + ); + } finally { + const cleanup = await Promise.allSettled([ + Promise.resolve(resources.collector?.unsubscribe()), + resources.client?.close() ?? Promise.resolve(), + resources.daemon?.close() ?? Promise.resolve(), + ]); + if (resources.cwd) rmSync(resources.cwd, { recursive: true, force: true }); + const failures = cleanup.flatMap((result) => + result.status === "rejected" ? [result.reason] : [], + ); + expect(failures, "Stop-spares-helper E2E cleanup failures").toEqual([]); + } + }, 300_000); });