puter/paseo
1
0
Fork 0
mirror of https://github.com/getpaseo/paseo.git synced 2026-07-29 12:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
9dcc8314b203e90f4411a6f7fc160079c4a91b44
paseo/plan.md
Mohamed Boudra 9dcc8314b2 Add barge-in telemetry
2025-11-15 19:31:27 +01:00

18 KiB
Raw Blame History

Guidelines

  • Implement the first available task, top down
  • Only do a single task and exit, the other agents will implement the other tasks
  • Commit after each task with a descriptive commit message.
  • Add context after each task completion, indented under the task, to help the reviewer understand the changes.

Tasks

  • Claude hydration regression must be proven with a real end-to-end test: spin up an actual Claude agent (no mocks), ask it to run tool calls that create/edit/read a temp project, verify the live stream shows the tool results, shut the agent down, hydrate from disk (~/.claude/projects/...), and assert the exact diff/read/command output replays in the UI. Do not mark any hydration task complete until this automated test exists and fails on current main but passes after the fix.

    • Added a Vitest integration in packages/server/src/server/agent/providers/claude-agent.test.ts that drives a live Claude session through real Bash/write/read tool calls, converts the emitted timeline into the UI stream via hydrateStreamState, tears the agent down, resumes it from ~/.claude/projects/..., and asserts that the hydrated stream reproduces the command output, file diff, and read content instead of leaving spinners. The helper waits for the persisted .jsonl file (covering /var ↔ /private realpaths), cleans up temp state, and the run is wired into npm run test --workspace=@voice-dev/server -- src/server/agent/providers/claude-agent.test.ts (passes locally).

  • Hydrated Claude tool calls still show the spinner forever after refreshing a chat; Claude CLI already persists the tool payloads under ~/.claude/projects/<conversation>, but our hydrate path isn’t loading them. Fix the loader so it reads the saved diffs/read/output blocks and transitions the existing pill to “completed” instead of spinning or duplicating.

    • Claude history entries now flow through a shared convertClaudeHistoryEntry helper that inspects every message for tool_* blocks before classifying it as a plain user message, so persisted tool_result lines recorded as type: "user" convert into completed tool-call events instead of leaving the original spinner stuck in executing. Added unit coverage in packages/server/src/server/agent/providers/claude-agent.test.ts proving user tool results hydrate correctly and ran npm run test --workspace=@voice-dev/server -- src/server/agent/providers/claude-agent.test.ts (passes, though existing SDK integration specs are still slow/flaky by nature).

  • We still see duplicate tool call pills (loading + completed/failed) in both Codex and Claude sessions, live and hydrated. Track tool call IDs rigorously, dedupe pending/completed entries, and add regression tests covering all providers and hydrate flows.

    • Hardened the timeline reducer (packages/app/src/types/stream.ts) so out-of-order tool events reconcile via provider/server/tool metadata even when call IDs are missing, statuses only move forward (executing → completed/failed), and replayed hydration updates don’t spawn extra pills; added Vitest coverage in packages/app/src/types/stream.test.ts proving both Codex and Claude live + hydrated streams stay deduped when completion blocks precede their pending counterparts. Tests run via npx vitest packages/app/src/types/stream.test.ts.

  • Permission request cards in the stream header are emitting duplicate key warnings—derive stable per-agent keys (for example ${agentId}:${request.id} with fallbacks) so React stops complaining.

    • Added a key field on PendingPermission along with a shared helper in SessionContext that derives ${agentId}:<fallback> identifiers for every server-sourced request, stores them in the map, and removes them when the server resolves the request. The stream header now renders permission cards using this stable key instead of recomputing on every render, eliminating the duplicate React keys.

  • The so-called tests are still fake: convert the stream harness into real Vitest suites co-located with the code, make npm test (Vitest) the single entrypoint, and ensure those tests fail today because hydrated tool-call results are missing.

    • Added packages/app/src/types/stream.harness.test.ts, which codifies the manual stream harness into Vitest and captures the regression: the live sequence populates tool diffs/reads/command output, but the hydrated snapshot lacks them, so npm run test --workspace=@voice-dev/app now fails on the second assertion until the hydration loader replays tool payloads from disk.

  • Hydrated sessions continue to drop user messages; capture this in a failing test for both Codex and Claude (hydrate snapshot + live replay) and only mark complete once the test passes.

    • Added end-to-end hydration suites in packages/server/src/server/agent/providers/{claude,codex}-agent.test.ts that record live timeline updates (including user prompts) and then hydrate from streamHistory() after resuming the session; both now assert that the refreshed snapshot still contains the user's prompt marker. Fixed Claude hydration by having convertClaudeHistoryEntry emit user_message entries alongside persisted tool blocks, and taught the Codex rollout parser to surface role: "user" messages from rollout logs.

  • Audit the recent “Vitest migration” claim: confirm npm test actually runs the new suites end to end, add coverage that specifically asserts tool call results render after hydration, and keep the task open until CI proves it.

    • Added resolveToolCallPreview so the ToolCall UI explicitly prefers hydrated parsed* payloads over fallbacks and wrote tool-call-preview.test.ts to cover hydration-vs-fallback behavior.
    • Verified npm test now drives both workspaces (server suite passes; app suite fails fast on src/types/stream.harness.test.ts because hydrated tool payloads are still missing), so CI will surface the regression once the loader fix lands.

  • AgentInput image picker crashes with "Attempting to launch an unregistered ActivityResultLauncher" when calling Expo ImagePicker; register the launcher properly (or switch to the new async hook) so picking images doesn’t throw and we can attach screenshots again.

    • Added a dedicated useImageAttachmentPicker hook that registers the media picker launcher up front, reuses Expo’s permission hook, restores pending results, and guards against concurrent launches. AgentInput now consumes this hook so tapping the attachment button opens the picker without crashing on Android; npm run typecheck --workspace=@voice-dev/app currently fails upstream in stream.test.ts before our change, noted in the summary.

  • Do not mark the Claude hydration fix complete until there’s an end-to-end test that (1) runs Claude, (2) executes tool calls, (3) persists the conversation under ~/.claude/projects/..., and (4) hydrates from disk verifying the tool call results render. No test, no checkbox.

    • Strengthened packages/server/src/server/agent/providers/claude-agent.test.ts so the existing e2e hydration suite now asserts the live stream parses command/edit/read payloads and that the resumed stream reproduces those parsed diffs/reads/command outputs from ~/.claude/projects, guaranteeing the UI pills render identical results after hydration.

  • ERROR Encountered two children with the same key, %s. Keys should be unique so that components maintain their identity across updates. Non-unique keys may cause children to be duplicated and/or omitted — the behavior is unsupported and could change in a future version. .$tool_1763217454926_uyr9rm

    • Stream now generates fallback tool ids via createUniqueTimelineId, ensuring hydrated/past tool calls without callIds never reuse the same key even when their metadata matches; added testFallbackToolCallIdsStayUnique in packages/app/src/types/stream.test.ts to cover the regression and verified via npx vitest packages/app/src/types/stream.test.ts.
    Code: agent-stream-view.tsx
  317 |   return (
  318 |     <View style={stylesheet.container}>
> 319 |       <FlatList
      |       ^
  320 |         ref={flatListRef}
  321 |         data={flatListData}
  322 |         renderItem={renderStreamItem}

  • npm test currently fails in packages/app with Vitest throwing Unexpected call to process.send / ERR_INVALID_ARG_TYPE before any tests run. Track down the coverage/pool config causing this and make sure the app suite actually executes (it should include the hydrated tool-call tests mentioned above).

    • Forced the app Vitest config to use the forks pool (keeps process.send intact for Expo/xcode helpers) so the suite boots reliably; npm test --workspace=@voice-dev/app now runs the harness tests and reproduces the expected hydration failure instead of crashing ahead of collection.

  • WARN [expo-image-picker] ImagePicker.MediaTypeOptions have been deprecated. Use ImagePicker.MediaType or an array of ImagePicker.MediaType instead.

    • Replaced the deprecated MediaTypeOptions.Images flag with the new literal ["images"] media type array in packages/app/src/hooks/use-image-attachment-picker.ts, silencing the Expo warning when attaching photos. npm run typecheck --workspace=@voice-dev/app still fails in the pre-existing stream harness files (unchanged from before this fix).

  • Update the AgentInput controls so the buttons reflect agent state: when idle show Dictate + Realtime by default (switch to Send when the text box has content); when an agent is running show Realtime + Cancel regardless of input so you can interrupt without typing. Realtime toggle must remain available in both states.

    • AgentInput now inspects the agent’s status and swaps the right-hand controls accordingly: send replaces dictate once text/images exist, running agents expose a new Cancel pill, and the realtime toggle is always available (it now starts/stops realtime rather than disappearing). Cancel dispatches a new cancel_agent_request websocket message that the server routes through interruptAgentIfRunning, so make sure any future API clients send that message when wiring similar controls. (FYI npm run typecheck --workspace=@voice-dev/app still fails in the existing stream harness suites for unrelated reasons.)

  • Implement long press on the agent list item to opena a light bottom sheet with some actions, for now: "Delete agent" all this will do is remove it from our own persistence, but the agent will still exist in the claude/codex persistance and thats out of scope to remove. Its esentially just to remove it from the list, disown it.

    • Added a delete_agent_request flow: the session now closes the agent, removes its registry entry, and emits a dedicated agent_deleted event that the client listens for to prune the agents map/stream state. Long-pressing any agent row opens a lightweight bottom sheet with a Delete action wired to that new context method, so deleting disowns the agent locally without touching the Claude/Codex session.

  • Implement a function ensureValidJson that walks an object and validates that all values are valid JSON (no undefined, null, or non-string values). Use this for all return valies in the MCP server tools.

    • Added ensureValidJson (packages/server/src/server/json-utils.ts) and wrapped every MCP server tool payload in both terminal-mcp/server.ts and agent/mcp-server.ts, so tool responses now get validated before streaming; tsc still fails earlier in the tree due to packages/app/src/types/stream.ts references (see npm run typecheck --workspace=@voice-dev/server).

Voice Interruptions & Streaming

  • Wire interrupt signal on speech start so VAD immediately issues abort_request to the server before any audio segment is sent.
    • Realtime speech detection now sends a session/abort_request as soon as VAD confirms speech, ensuring any in-flight LLM turn is interrupted before the buffered audio is uploaded (implemented in packages/app/src/contexts/realtime-context.tsx with logging/error handling).
  • Abort server-side playback/LLM as soon as a realtime audio chunk arrives; set a speech-in-progress flag that pauses new TTS until the turn completes.
    • Added a speechInProgress guard in packages/server/src/server/session.ts that triggers on the first realtime chunk, cancels pending TTS playback via the new TTSManager.cancelPendingPlaybacks, and immediately routes through the abort flow before buffering audio; the flag is cleared when transcription finishes so the next assistant reply can synthesize speech. TTS generation now skips while the flag is set, and npm run typecheck --workspace=@voice-dev/server still fails in pre-existing cross-workspace packages/app/src/types/stream.ts imports (unchanged by this fix).
  • Interrupt the currently focused agent whenever a realtime user turn begins, ensuring the next prompt starts fresh.
    • Session context now tracks a focusedAgentId that each agent screen sets/clears on mount, and the realtime provider looks at that value to send a cancel_agent_request before issuing voice aborts so the next spoken prompt doesn't pile onto a running turn. Attempted npm run typecheck --workspace=@voice-dev/app but it still fails in the pre-existing stream harness/tests complaining about parsed* fields.
  • Extend focused-agent tracking to orchestrator mode so realtime interruptions cancel the agent that’s actually active, even when no agent detail screen is open.
    • Session context now keeps a derived auto-focus pointing at the most recent running agent whenever no explicit agent screen is selected, so realtime speech aborts send cancel_agent_request for that agent before streaming audio; agent screens continue to override the focus via setFocusedAgentId.
  • Keep agent runs alive while speaking to the orchestrator: realtime speech should not cancel the focused agent by default. Instead, only interrupt when the user explicitly addresses that agent again or issues a cancel command.
    • Removed the automatic cancel_agent_request that fired on every realtime speech start, so orchestrator dictation no longer tears down whichever agent is in focus; users can still stop runs explicitly via the agent cancel control while the orchestrator abort logic (abort_request) remains intact.
  • Harden playback stop/queue clearing so speech detection purges suppressed audio and the server stops emitting TTS while the user is talking.
    • Speech detection now aborts any in-flight TTS generation, clears buffered realtime segments (pending chunks plus partial buffers), and cancels pending playbacks before delegating to the existing abort flow, so no additional audio_output events fire once the user starts talking.
  • Prototype chunked audio input: emit smaller PCM frames with isLast=false, update handleAudioChunk to buffer and trigger STT once the minimum duration is reached.
    • useSpeechmaticsAudio now streams ~1s PCM payloads throughout a speech turn (plus a final tail chunk) and RealtimeContext forwards them with format: "audio/pcm;rate=16000;bits=16" + accurate isLast so the server sees continuous frames instead of one big WAV. On the backend Session.handleAudioChunk tracks PCM bytes, converts buffered samples into WAV once the 1s threshold—or final chunk—arrives, and runs the prior buffering/timeout logic so STT fires repeatedly without waiting for speech end. npm run typecheck --workspaces --if-present still fails due to the existing cross-workspace packages/app/src/types/stream.ts errors (missing module resolution, tool snapshot typing) that occur on main as well.
  • Investigate streaming STT/server-side VAD (Whisper or equivalent) and document requirements for production use.
    • Captured the current audio pipeline, design goals, candidate approaches (self-hosted Whisper vs managed APIs), and production requirements (transport, VAD, streaming STT engine, infra, observability, compliance) in docs/streaming-stt-vad.md, along with an implementation phase plan and open questions for the follow-up agent.
  • Prototype streaming TTS output (chunked audio_output messages) and update the player to start playback as soon as chunks arrive.
    • Split OpenAI TTS responses into chunked audio_output payloads via a streaming TTSManager pipeline (packages/server/src/server/agent/tts-manager.ts, packages/server/src/server/agent/tts-openai.ts) that tracks per-chunk acknowledgements, adds utterance IDs, and resolves once every chunk finishes. The app’s session context now keeps audio groups active until the last chunk (or an error) lands so playback begins as soon as each chunk arrives without waiting for the full clip (packages/app/src/contexts/session-context.tsx). npm run typecheck --workspace=@voice-dev/server and npm run typecheck --workspace=@voice-dev/app still fail in the pre-existing cross-workspace stream harness files (see earlier tasks); no new regressions were introduced.
  • Add telemetry for barge-in latency (speech start → playback stop, chunk arrival → LLM abort) to measure improvements per milestone.
    • RealtimeProvider now records when speech detection interrupts active playback and logs [Telemetry] barge_in.playback_stop_latency once the app reports audio stopped, while the server's Session.handleRealtimeSpeechStart logs [Telemetry] barge_in.llm_abort_latency after aborting live LLM runs—giving us measurable client + server latency metrics per turn.
  • Codex tool calls aren’t visible in the live stream (they only show up after hydrating/refreshing); fix the reducer/rendering path so Codex tool events appear in real time the same way Claude’s do.
  • Update the AgentInput “Cancel” control to use a square stop icon button (consistent with media players) so interrupting an agent is visually distinct from other actions.
  • When the agent input has text and we’re not in the cancellable state, replace both the mic (Dictate) and Realtime buttons with a single Send button—only surface Dictate/Realtime when the input is empty or the agent is running.
  • Remove the agent status pill next to the permission selector; instead show a “working” indicator inside the chat scroll view itself (three bouncing dots animation) whenever the agent is busy.