From 2c72d02cc745bb79e9eb03d0299373aabfd7380e Mon Sep 17 00:00:00 2001
From: Mohamed Boudra <boudra.moha@gmail.com>
Date: Wed, 22 Oct 2025 20:29:59 +0200
Subject: [PATCH] docs: add orchestrator agent specification

---
 ORCHESTRATOR-AGENT-SPEC.md | 181 +++++++++++++++++++++++++++++++++++++
 1 file changed, 181 insertions(+)
 create mode 100644 ORCHESTRATOR-AGENT-SPEC.md

diff --git a/ORCHESTRATOR-AGENT-SPEC.md b/ORCHESTRATOR-AGENT-SPEC.md
new file mode 100644
index 000000000..0fc9cf55f
--- /dev/null
+++ b/ORCHESTRATOR-AGENT-SPEC.md
@@ -0,0 +1,181 @@
+# Voice Dev: Orchestrator + Agent Chat Architecture Spec
+
+## Paradigm
+
+### Two Interaction Models
+
+**1. Orchestrator (Realtime Voice)**
+- Top-level conversational agent
+- Controls everything: creates agents, checks status, manages projects
+- Realtime voice mode with TTS
+- App-level session (not project-specific)
+
+**2. Agent Chat (Text + Voice Notes)**
+- Direct interaction with individual coding agents
+- Text messages + voice notes (transcribed to text)
+- No realtime, no TTS
+- Per-agent conversation history
+
+### Key Rules
+
+1. **Realtime always routes to orchestrator** - regardless of which view you're in
+2. **Non-realtime routes based on view**:
+   - Orchestrator view → orchestrator session
+   - Agent view → specific agent
+3. **Input area is identical everywhere** - same UI, different routing
+4. **TTS is implied by message type** - no flags needed
+
+## Message Architecture
+
+### Separation of Concerns
+
+**Realtime Messages (Orchestrator Only)**
+- `audio_chunk` (realtime) → triggers TTS response
+- Implies: speech-to-speech interaction
+- Server handles with full orchestrator Session
+
+**Non-Realtime Messages**
+- `user_text` (orchestrator) → text to orchestrator, no TTS
+- `user_audio_note` (orchestrator) → transcribe → text to orchestrator, no TTS
+- `send_agent_message` (agent) → text to agent via ACP, no TTS
+- `send_agent_audio` (agent) → transcribe → text to agent, no TTS
+
+**Outbound (Existing)**
+- Orchestrator: `assistant_chunk`, `activity_log`, `audio_output` (TTS)
+- Agents: `agent_update` (session notifications from ACP)
+
+## Message Types
+
+### New Inbound Types
+
+```typescript
+// To agent (text)
+{
+  type: "send_agent_message",
+  agentId: string,
+  text: string
+}
+
+// To agent (voice note)
+{
+  type: "send_agent_audio",
+  agentId: string,
+  audio: string, // base64
+  format: string,
+  isLast: boolean
+}
+
+// Separate audio_chunk into realtime context
+{
+  type: "realtime_audio_chunk", // rename from audio_chunk
+  audio: string,
+  format: string,
+  isLast: boolean
+}
+```
+
+### Existing Types (Clarified)
+
+```typescript
+// To orchestrator (text)
+{
+  type: "user_text",
+  text: string
+  // NO disableTTS flag - never has TTS in non-realtime context
+}
+```
+
+## UI Architecture
+
+### Unified Input Area
+
+**Same everywhere, routes differently:**
+- Text input
+- Voice note button (push-to-talk, sends when released)
+- Realtime toggle
+
+**Routing Logic:**
+```
+if (realtimeMode) {
+  → orchestrator (speech-to-speech)
+} else if (viewMode === "agent") {
+  → agent[activeAgentId] (text only)
+} else {
+  → orchestrator (text only)
+}
+```
+
+### Views
+
+**Orchestrator View (index.tsx)**
+- Current chat UI (unchanged)
+- Conversation with orchestrator
+- Realtime UI when enabled
+
+**Agent View (agent-stream-view.tsx)**
+- Enhanced to match orchestrator chat quality
+- User messages + agent responses (parsed from session updates)
+- Tool calls, thoughts, plans (inline activity)
+- Same input area (routed to agent)
+- Shows realtime UI if enabled (but messages still go to orchestrator)
+
+### Navigation
+
+- Header: connection status, conversation selector, settings, "New Agent" button
+- ActiveProcesses: horizontal list of agents (click to open agent view)
+- "Back to Chat" button when in agent view
+
+## Implementation Phases
+
+### Phase 1: Message Routing Foundation
+
+**Server**
+- Rename `audio_chunk` → `realtime_audio_chunk` for clarity
+- Add `send_agent_message` and `send_agent_audio` message types
+- Route non-realtime messages to `agentManager.sendPrompt()`
+- Handle audio transcription for agent audio notes
+
+**App**
+- Add message routing logic based on `viewMode` and `isRealtimeMode`
+- Update WebSocket to handle new message types
+- Add per-agent message state: `Map<agentId, MessageEntry[]>`
+
+### Phase 2: Agent Chat Interface
+
+**App**
+- Parse `agent_update` messages into chat messages
+  - Extract text chunks (user_message_chunk, agent_message_chunk)
+  - Extract thoughts (agent_thought_chunk)
+  - Merge tool calls with updates
+- Enhance agent-stream-view to chat quality:
+  - Render user messages
+  - Render agent responses (streaming)
+  - Render activity inline (thoughts, tools, plans)
+  - Remove its own input area (uses shared one)
+
+### Phase 3: Agent Creation
+
+**Server**
+- Ensure `createAgent` supports mode selection (already exists)
+
+**App**
+- Create agent creation modal
+  - Directory input/picker
+  - Mode selector dropdown
+  - Validate directory exists (optional)
+- "New Agent" button in header
+- Auto-switch to agent view after creation
+
+### Phase 4: Polish
+
+**App**
+- Ensure realtime UI shows in both views
+- Clear visual indication of routing (subtle badge/label?)
+- Smooth transitions between views
+- Handle edge cases (agent killed while viewing, etc.)
+
+## Open Questions
+
+1. Should we show a subtle indicator in the input area showing where messages will go?
+2. How to handle agent errors during chat (failed, killed)?
+3. Should agent creation modal remember last used directory?