Commit Graph

92 Commits

Author SHA1 Message Date
Mohamed Boudra
3483bd9afd fix: improve realtime controls layout and add agent status bar
- Change realtime controls to horizontal layout with fixed height (88px)
- Add orientation prop to VolumeMeter for horizontal/vertical modes
- Create AgentStatusBar component showing mode badge and agent status
- Add agent persistence system for managing agent state across restarts
- Fix bottom sheet dismiss handling to clear state properly
- Remove debug timer from realtime controls

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-24 11:17:05 +02:00
Mohamed Boudra
d24bf65fe9 refactor: improve UI consistency across message components and theme styles 2025-10-23 22:46:09 +02:00
Mohamed Boudra
29e6da8536 feat: add screenshot and settings for Claude, refactor agent stream and orchestrator views
- Introduced a new screenshot image for the app.
- Added local settings for Claude permissions.
- Refactored AgentStreamView and OrchestratorMessagesView to utilize new types for tool calls and permissions.
- Updated ToolCallBottomSheet to handle selected tool calls more efficiently.
- Enhanced type definitions for shared data structures related to tool calls and permissions.
2025-10-23 22:12:02 +02:00
Mohamed Boudra
a0f45f2cfc fix: correct typescript ref types for bottom sheet and audio player 2025-10-23 20:06:11 +02:00
Mohamed Boudra
e7306ab495 fix: update expo-two-way-audio dependency to @boudra/expo-two-way-audio 2025-10-23 20:03:57 +02:00
Mohamed Boudra
15320819a7 fix: migrate to @boudra/expo-two-way-audio and add tool call details bottom sheet 2025-10-23 20:01:52 +02:00
Mohamed Boudra
74fffcd8a3 refactor: optimize audio initialization and fix agent interactions
- Lazy initialize audio (on-demand instead of mount) in both player and Speechmatics hooks
- Add audio session teardown on stop for proper cleanup
- Implement agent audio recording with sendAgentAudio in AgentInputArea
- Fix Android navigation race condition in CreateAgentModal (navigate before closing)
- Add animations to global footer and agent screen (FadeIn/FadeOut)
- Auto-cancel stuck agents when sending new messages
- Fix permission request ID propagation from AgentManager to Session
- Apply theme background color via AppContainer wrapper

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-23 13:15:18 +02:00
Mohamed Boudra
10189514dd ```
refactor: centralize global footer and simplify agent screen input handling
```
2025-10-23 12:35:28 +02:00
Mohamed Boudra
027ae59ae7 ```
fix: stop audio playback when ending realtime session
```
2025-10-23 12:29:21 +02:00
Mohamed Boudra
91d691f31d ```
refactor: reset message IDs on new turn and add activity log handlers

- Reset assistant message and thought IDs when starting new agent turn
- Add comprehensive activity log handlers in session context
- Handle tool calls, results, and errors with proper state updates
- Add transcription and assistant chunk streaming handlers
- Improve audio playback with proper error handling and confirmation
- Use generateMessageId() for consistent ID generation
- Filter out transcription activity logs from chat
- Stop audio playback when real speech is detected
```
2025-10-23 12:22:42 +02:00
Mohamed Boudra
41be2d4096 ```
refactor: implement async agent creation flow with loading states

- Add loading states and request tracking to agent creation modal
- Move agent creation logic from parent to modal component
- Add automatic navigation to agent page after creation
- Implement requestId-based response matching for agent_created events
- Add audio output handling in session context
- Improve modal UX with disabled states during creation
```
2025-10-23 12:04:12 +02:00
Mohamed Boudra
542707ecdd feat: add bottom sheet modal for agent creation UI 2025-10-23 11:53:38 +02:00
Mohamed Boudra
85a0e46ba9 ```
feat: add recent paths suggestions and improve UX polish

- Add recent paths hook to persist and suggest working directories
- Update Android icon assets and simplify adaptive icon configuration
- Remove background styling from header icon buttons for cleaner look
- Export expandTilde utility for path normalization in agent creation
- Add horizontal scrollable chips for recent path suggestions
```
2025-10-23 11:26:36 +02:00
Mohamed Boudra
8177aa0c2a feat: add home screen with agent list and orchestrator navigation 2025-10-23 11:11:20 +02:00
Mohamed Boudra
a930c4156a docs: add Codex integration guide and improve agent creation UX 2025-10-23 09:22:11 +02:00
Mohamed Boudra
0893a9dc33 ```
refactor: improve agent notification types and fix activity curator

- Convert agent notifications to discriminated union (session/permission/status)
- Fix activity curator to properly handle typed session updates
- Serialize Date objects in AgentInfo for JSON compatibility
- Use activity curator for title generation context
- Reduce console.error to console.warn for non-critical WebSocket errors
- Improve spacing in agent stream views
- Delay title generation until 15+ updates available
```
2025-10-23 09:01:38 +02:00
Mohamed Boudra
c6cdc09a0b ```
refactor: deduplicate messages using stable IDs and add permission UI with gesture-based sidebar
```
2025-10-23 08:37:53 +02:00
Mohamed Boudra
30bf4d1b63 ```
feat: implement agent permission flow with gesture-based sidebar navigation

Add permission request/response system for agent tool calls with UI approval flow, implement swipeable agent sidebar with edge gesture detection, and enhance agent updates with stable message IDs for proper deduplication across reconnections.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
```
2025-10-23 08:36:42 +02:00
Mohamed Boudra
70bdbd74eb refactor: deduplicate user messages and sync agent history on reconnection 2025-10-22 22:38:15 +02:00
Mohamed Boudra
b86e5fbb3f Based on the changes, I can see this is a significant feature addition that adds the ability to create agents and communicate with them through both text and audio, with a modal UI for agent creation and mode switching.
```
feat: add agent creation modal and agent-specific messaging

- Add CreateAgentModal component for creating new agents with mode selection
- Implement agent-specific text and audio message routing
- Add mode selector modal with visual mode switching UI
- Introduce unified stream reducer pattern for agent activity
- Refactor AgentStreamView to use StreamItem[] instead of raw notifications
- Update ActiveProcesses to show orchestrator pill and support mode switching
- Add message types: send_agent_message, send_agent_audio, create_agent_request, set_agent_mode
- Implement agent audio transcription and message forwarding in session handler
- Add mode badge UI showing current agent mode
- Update tmux capture-pane to allowed commands

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
```
2025-10-22 22:06:00 +02:00
Mohamed Boudra
2c72d02cc7 docs: add orchestrator agent specification 2025-10-22 20:29:59 +02:00
Mohamed Boudra
eb0d04904e Based on the changes, here's the commit message:
```
refactor: convert AgentManager to global singleton across sessions

Move AgentManager initialization from per-session to single global instance. Agents now persist across sessions, with sessions subscribing/unsubscribing to agent updates rather than creating/destroying agents.
```
2025-10-22 20:08:40 +02:00
Mohamed Boudra
ce07b54920 ```
refactor: improve UI polish with empty state, spacing, and remove status messages
```
2025-10-22 19:42:13 +02:00
Mohamed Boudra
8b9561b379 ```
feat: add realtime mode drift protection and improve voice UX
```
2025-10-22 19:06:59 +02:00
Mohamed Boudra
015d0f0e27 feat: merge tool calls with their updates into single unified view
Instead of showing tool_call and tool_call_update as separate events,
now merge them by toolCallId into a single MergedToolCall that updates
as new information comes in.

**Changes:**
- Added MergedToolCall type with start/end timestamps
- Renamed groupTextChunks to groupActivities with tool call merging logic
- Track tool calls by ID and merge all updates into single entry
- Replace ToolCallItem with MergedToolCallItem component
- AgentStreamView now uses groupActivities for unified grouping

**User Experience:**
- Each tool call appears once and updates in place
- Status changes from pending → in_progress → completed
- Input/output merge as they become available
- Cleaner, less noisy activity view

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-22 18:17:53 +02:00
Mohamed Boudra
674977d6f8 feat: improve agent activity UI and fix status update propagation
**Agent Activity UI Improvements:**
- Create typed discriminated unions for SessionNotification updates
- Add AgentActivity component with specialized rendering for each update type
- Group consecutive text chunks into single messages (like main chat)
- Add collapsible tool call rendering with input/output details
- Add plan rendering with task status indicators
- Add fallback rendering for unknown activity types (badge + drawer)

**Agent Status Update Fix:**
- Fix agent killing not updating UI status
- Session now detects status change notifications and emits agent_status messages
- Agent manager delays agent deletion to allow status updates to propagate
- Frontend agent_status handler now receives proper updates

**Technical Details:**
- Created /packages/app/src/types/agent-activity.ts with typed session updates
- Created /packages/app/src/components/agent-activity.tsx with activity renderers
- Updated AgentStreamView to use grouped activities with useMemo
- Modified session.ts subscribeToAgent to emit agent_status on status changes
- Modified agent-manager.ts killAgent to delay deletion for status propagation

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-22 18:10:29 +02:00
Mohamed Boudra
e1e14b1473 feat: improve UI/UX with volume meter and refine input area styling
- Add VolumeMeter component for realtime audio visualization
- Refactor input area to show volume meter in realtime mode
- Update button styling and layout for cleaner interface
- Improve text input integration with better visual hierarchy
- Add volume state tracking to Speechmatics audio hook
- Add agent working directory validation on creation
- Update agent prompt with better orchestration guidelines

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-22 18:00:59 +02:00
Mohamed Boudra
de5aba4186 docs: update tmux commands formatting and add Android screenshot instructions 2025-10-22 16:50:07 +02:00
Mohamed Boudra
89641a18e7 feat: refactor app structure and components for improved organization and functionality
- Moved components and hooks into a new directory structure under `src/app` for better organization.
- Updated the main entry point to use `index.ts` and adjusted routing accordingly.
- Introduced new components such as `ActiveProcesses`, `AgentStreamView`, and `ConnectionStatus` to enhance the app's functionality.
- Removed deprecated components and hooks to streamline the codebase.
- Updated styles to align with the new theme structure and ensure consistency across the app.

This refactor aims to improve maintainability and scalability of the application.
2025-10-22 15:48:53 +02:00
Mohamed Boudra
2d1ab5bf79 feat: add Babel configuration and update styles to use new theme structure
- Introduced a Babel configuration file for the app.
- Updated various components and styles to utilize the new theme structure, replacing direct color references with palette-based colors for improved consistency and maintainability.
- Adjusted styles across multiple components to align with the new theme, enhancing the overall UI experience.
2025-10-22 15:00:38 +02:00
Mohamed Boudra
32d4944c58 feat: migrate from NativeWind to react-native-unistyles and add Speechmatics audio
🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-22 14:38:46 +02:00
Mohamed Boudra
c30ef34ef7 Based on the changes, here's the commit message:
```
fix(app): improve PCM audio volume calculation with proper signed conversion
```
2025-10-22 11:06:45 +02:00
Mohamed Boudra
37156b5b80 Prompt is too long 2025-10-22 10:51:17 +02:00
Mohamed Boudra
e646f410be Prompt is too long 2025-10-22 09:06:18 +02:00
Mohamed Boudra
8235d8d1b1 feat: add expo-build-properties plugin to set minSdkVersion 26
Using expo-build-properties plugin for proper minSdk configuration.
Required for react-native-vad which needs Android O (API 26) or higher.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 22:19:12 +02:00
Mohamed Boudra
931a38634d fix: increase minSdkVersion to 26 for react-native-vad compatibility
react-native-vad requires Android O (API 26) or higher due to
MethodHandle.invoke usage in ONNX runtime dependencies.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 22:14:12 +02:00
Mohamed Boudra
492017d139 fix: add Maven repository for react-native-vad dependency
Added custom Maven repository configuration to android/build.gradle
to resolve com.davoice:vad dependency from react-native-vad package libs.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 21:40:48 +02:00
Mohamed Boudra
0f1450381a feat: implement realtime audio with VAD for Android
- Install react-native-vad and expo-audio-studio for voice detection
- Create use-realtime-audio hook for continuous recording + VAD segmentation
- Add Android FOREGROUND_SERVICE permissions for background mic access
- Update UI components with Lucide icons and markdown rendering
- Add npm overrides to resolve peer dependency conflicts with Expo SDK 54
- Integrate realtime mode toggle with pulse animation for speech detection

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 21:33:14 +02:00
Mohamed Boudra
cc3dbb88cb feat: add sessionMode parameter to send_agent_prompt and currentModeId to activity
Allow setting session mode when sending prompts, and include current mode in activity output.

Changes:
- sendPrompt now accepts options object with maxWait and sessionMode
- If sessionMode specified, sets mode before sending prompt
- send_agent_prompt MCP tool exposes sessionMode parameter
- get_agent_activity now returns currentModeId in output
- Enables workflow: send_agent_prompt({ sessionMode: "plan", prompt: "..." })

🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 19:56:51 +02:00
Mohamed Boudra
6106e5f65f fix: simplify activity curator to chronological text format
Remove grouping and clever formatting. Just convert activity feed to chronological text.

Changes:
- Process updates in order (chronological)
- Buffer message/thought chunks and flush when needed
- Output tool calls and plans as they appear
- No grouping, no reorganizing, no sections
- Simple text format: messages, [Tool] status, [Plan] entries

🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 19:49:44 +02:00
Mohamed Boudra
13f8e31295 Merge branch 'feature/mobile-app' 2025-10-21 18:15:41 +02:00
Mohamed Boudra
85e4e9ddaa Prompt is too long 2025-10-21 18:15:05 +02:00
Mohamed Boudra
f2daa86d30 feat: add curated activity format for token-efficient agent output
Create activity curator that consolidates hundreds of tiny message chunks into clean, human-readable text. Saves tokens and improves readability dramatically.

Changes:
- Add activity-curator.ts with curateAgentActivity() function
- Consolidate agent_message_chunk fragments into complete messages
- Format tool calls with status, input/output in readable structure
- Display plans with status icons ( completed, 🔄 in_progress,  pending)
- Add priority indicators (🔴 high, 🟡 medium, 🟢 low)
- Update get_agent_activity to support format='curated' (default) or 'raw'
- Curated format returns clean markdown text instead of JSON array

Example: 363 raw updates consolidating to single clean message instead of 363 JSON objects with fragments like " it.", " fix", "ify what".

🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 16:59:59 +02:00
Mohamed Boudra
5b53f4a7ce refactor: make sendPrompt non-blocking by default with optional maxWait
Change sendPrompt to return immediately by default instead of blocking, allowing the orchestrator to continue while agents process in the background. Add optional maxWait parameter for cases where waiting is needed.

Changes:
- sendPrompt now returns { didComplete, stopReason? } immediately if no maxWait
- Background promise handles completion and updates agent status via protocol
- Optional maxWait parameter races completion vs timeout, returns didComplete status
- Update send_agent_prompt MCP tool to expose maxWait parameter
- Agent continues processing in background even if maxWait expires
- Orchestrator can poll get_agent_status or get_agent_activity to check progress

This prevents orchestrator from blocking on long-running agent tasks while maintaining protocol-based completion tracking.

🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 16:54:47 +02:00
Mohamed Boudra
cba8780bca fix: use ACP protocol stopReason for proper agent completion detection
Replace unreliable 2-second timeout heuristic with proper protocol-based completion detection using PromptResponse.stopReason from ACP SDK.

Changes:
- Capture and handle PromptResponse.stopReason in sendPrompt()
- Set agent status based on stop reason: end_turn → completed, refusal → failed, cancelled → ready
- Remove 2-second timeout completion heuristic (was causing drift)
- Update SessionStateMessageSchema to match AgentInfo null types

This ensures agent completion is detected via the protocol spec, not manual polling/timing.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 16:49:23 +02:00
Mohamed Boudra
d883436395 fix: replace undefined with null in ACP MCP tool responses
Replace all undefined values with null in agent MCP tool responses to prevent JSON serialization issues that break the MCP client.

Changes:
- Update AgentInfo interface to use `| null` instead of optional `?` syntax
- Convert undefined to null in listAgents(), getCurrentMode(), getAvailableModes()
- Update all Zod schemas from .optional() to .nullable()
- Fix tool responses in create_coding_agent and set_agent_mode

This ensures all MCP tool responses properly serialize optional fields as null instead of undefined, preventing client-side errors.

🤖 Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 16:34:11 +02:00
Mohamed Boudra
795c10e3c9 Merge remote-tracking branch 'origin/main' into feature/mobile-app 2025-10-21 15:53:49 +02:00
Mohamed Boudra
cf977968c6 refactor: implement proper ACP session modes and fix type safety
Replace client-side permission mode hack with proper ACP session modes:
- Add typed Client interface implementation to ACPClient
- Import proper types from ACP SDK (SessionNotification, RequestPermissionRequest, etc.)
- Fix critical sessionUpdate bug where params.update was undefined
- Replace PermissionMode with SessionMode from ACP spec
- Add getCurrentMode(), getAvailableModes(), setSessionMode() to AgentManager
- Update MCP tools: replace set_agent_permission_mode with set_agent_mode
- Add session_state message type to send live agents/commands to client
- Update UI to display session mode badges (ask/code/architect)
- Remove all permissionsMode references from tests

Agents now have ONE mode (session mode) that controls their behavior,
system prompts, and tool availability - not separate permission/session modes.

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 15:49:55 +02:00
Mohamed Boudra
ca43aae534 feat: initialize voice-mobile React Native app with NativeWind v5
- Created new Expo app with SDK 54+ and Expo Router
- Installed and configured NativeWind v5 for Tailwind styling
- Upgraded voice-assistant to React 19 for consistency
- Kept voice-mobile outside workspace to avoid Metro bundler conflicts
- Disabled React Compiler experimental feature for stability

Setup includes:
- postcss.config.mjs with Tailwind PostCSS plugin
- global.css with Tailwind and NativeWind imports
- metro.config.js with NativeWind wrapper
- lightningcss pinned to 1.30.1 for stability

Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 15:24:44 +02:00
Mohamed Boudra
4f2c2fc14a feat: add conversation management, persistence, and mobile app spec
Major enhancements to voice assistant with conversation persistence
and mobile app planning:

- Add conversation management with multiple active conversations
- Implement conversation persistence (save/load from disk)
- Add conversation selector UI for switching between conversations
- Enhance agent prompt with per-conversation tmux sessions
- Add mobile app technical specification (React Native + Expo)
- Improve UI with active processes view and agent stream
- Add ACP (Agent Control Protocol) server implementation

Mobile spec details:
- React Native with Expo SDK 54+
- NativeWind v5 for styling
- Share Zod schemas/types from server
- Android in-call experience (InCallManager)
- Direct WebSocket connection via Tailscale
- Same audio quality as web prototype

🤖 Generated with [Claude Code](https://claude.com/claude-code)
via [Happy](https://happy.engineering)

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
2025-10-21 14:50:07 +02:00