mirror of
https://github.com/getpaseo/paseo.git
synced 2026-07-29 12:01:31 +00:00
b70c0cfc73
``` feat: migrate from OpenAI Realtime API to LiveKit for voice infrastructure Replace direct OpenAI Realtime API WebRTC connection with LiveKit Cloud infrastructure. Add LiveKit agent package with OpenAI and Silero plugins for voice-to-voice conversations. Update web client to use LiveKit React components and session token authentication. Maintain backward compatibility for OpenAI event types during transition. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com> ``` This commit message: - Uses `feat:` type since this is a new feature/major change - Describes the main change (migration from OpenAI to LiveKit) - Provides context in the body about what was added/changed - Mentions backward compatibility consideration - Includes the Claude Code attribution as requested
Voice Dev
Voice-controlled development environment powered by LiveKit and OpenAI Realtime API.
Packages
-
@voice-dev/agent- LiveKit agent with OpenAI Realtime API- Located in
packages/agent/ - Handles voice conversations via LiveKit
- Integrates OpenAI Realtime API
- Manages agent lifecycle
- Located in
-
@voice-dev/mcp-server- MCP server for terminal control- Located in
packages/mcp-server/ - Provides terminal session management via MCP protocol
- Can run as HTTP server or stdio mode
- Located in
-
@voice-dev/web- Next.js web interface- Located in
packages/web/ - Voice interaction via LiveKit Cloud
- Password authentication
- Agent activity transparency (debug panel)
- Real-time status display
- Located in
Quick Start
For detailed setup instructions, see QUICKSTART.md.
You need TWO terminals:
# Terminal 1: Start the LiveKit agent
npm run dev:agent
# Terminal 2: Start the web app
npm run dev
Then open http://localhost:3000 and click "Start Voice Chat".
For more info:
- QUICKSTART.md - Step-by-step setup guide
- MIGRATION.md - Details about LiveKit migration
Other Commands
# Install all dependencies
npm install
# Run MCP server in dev mode
npm run dev:mcp
# Build everything
npm run build
# Type check all packages
npm run typecheck
Development
LiveKit Agent
# Development
npm run dev:agent # Start agent in dev mode
# Production
npm run build:agent # Build TypeScript
npm run start:agent # Run built agent
Web App
# Development
npm run dev # Start Next.js dev server (port 3000)
# Production
npm run build:web # Build for production
npm run start # Start production server
MCP Server
# Development
npm run dev:mcp # Start with HTTP server and dev-password
# Production
npm run build:mcp # Build TypeScript
npm run start:mcp # Run built server
Environment Variables
Create .env.local in both packages/web/ and packages/agent/:
# OpenAI
OPENAI_API_KEY=sk-your-api-key-here
# LiveKit Cloud
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your-api-key
LIVEKIT_API_SECRET=your-api-secret
# Optional
AUTH_PASSWORD=your-secure-password-here # For web auth
MCP_SERVER_URL=https://your-mcp-server-url # For MCP integration
See packages/web/.env.local.example for template.
Project Structure
voice-dev/
├── packages/
│ ├── agent/ # LiveKit Agent Package
│ │ ├── src/
│ │ │ └── agent.ts # Agent implementation
│ │ ├── dist/ # Compiled output
│ │ ├── package.json
│ │ └── tsconfig.json
│ │
│ ├── mcp-server/ # MCP Server Package
│ │ ├── src/
│ │ │ ├── index.ts # CLI entry point
│ │ │ ├── http-server.ts # HTTP server mode
│ │ │ └── tmux.ts # Terminal operations
│ │ ├── build/ # Compiled output
│ │ ├── package.json
│ │ └── tsconfig.json
│ │
│ └── web/ # Web App Package
│ ├── app/
│ │ ├── api/ # API routes (token generation)
│ │ ├── components/ # React components
│ │ ├── hooks/ # Custom hooks (LiveKit)
│ │ ├── types/ # TypeScript types
│ │ └── voice-client.tsx # Main voice UI
│ ├── package.json
│ └── next.config.js
│
├── package.json # Root workspace config
├── tsconfig.base.json # Shared TypeScript config
├── README.md # This file
├── QUICKSTART.md # Setup guide
└── MIGRATION.md # LiveKit migration details
Features
LiveKit Agent
- OpenAI Realtime API integration
- Voice-to-voice conversation
- Low-latency audio processing
- Automatic room management
- System prompt customization
- Tool/function calling support
Web App
- Real-time voice interaction via LiveKit
- LiveKit React SDK components
- Live audio level visualization
- Device selection
- Mute/unmute controls
- Password authentication
- Agent activity transparency (debug log panel)
- Real-time agent status display
- Automatic reconnection
MCP Server
- Full terminal session control
- List sessions, windows, and panes
- Send keystrokes and text
- Create and kill sessions/windows/panes
- Capture pane output
- HTTP server mode with password auth
- stdio mode for Claude Desktop integration
Deployment
Web App (Vercel)
Deploy from the web package directory:
# Deploy to Vercel
cd ~/dev/voice-dev/packages/web
vercel --prod
# Set environment variables
vercel env add OPENAI_API_KEY production
vercel env add AUTH_PASSWORD production
vercel env add MCP_SERVER_URL production
MCP Server
The MCP server typically runs locally:
# Build and run
npm run build:mcp
npm run start:mcp -- --http --password your-password
Scripts Reference
Root Level
| Script | Description |
|---|---|
npm run dev |
Start web app dev server |
npm run dev:agent |
Start LiveKit agent dev server |
npm run dev:mcp |
Start MCP server dev server |
npm run build |
Build all packages |
npm run build:agent |
Build agent only |
npm run build:mcp |
Build MCP server only |
npm run build:web |
Build web app only |
npm run typecheck |
Type check all packages |
npm run start |
Start web app production server |
npm run start:agent |
Start agent production |
npm run start:mcp |
Start MCP server production |
Package Level
Run commands in specific packages:
# Run command in web package
npm run <script> --workspace=web
# Run command in mcp-server package
npm run <script> --workspace=mcp-server
Tech Stack
- Architecture: npm workspaces
- Web: Next.js 15, React 19, TypeScript, Tailwind CSS, LiveKit React SDK
- Agent: LiveKit Agents Framework, OpenAI Realtime API, TypeScript
- MCP Server: TypeScript, Express, MCP SDK
- Voice: LiveKit Cloud, OpenAI Realtime API, WebRTC
- Build: TypeScript compiler, Next.js build
Architecture
┌─────────────┐ ┌──────────────┐ ┌─────────────┐
│ Browser │◄───────►│ LiveKit Cloud│◄───────►│ Agent │
│ (React UI) │ WebRTC │ (Server) │ WebRTC │ (Node.js) │
└─────────────┘ └──────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ OpenAI │
│ Realtime API│
└─────────────┘
License
MIT