puter/paseo
1
0
Fork 0
mirror of https://github.com/getpaseo/paseo.git synced 2026-07-29 12:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0c3c61be94de6cef9623e1727a004eb0a376d22a
paseo/public-docs/voice.md
Mohamed Boudra f91806defe Use separate OpenAI endpoints for speech-to-text and text-to-speech (#1823) 
* feat(voice): configure OpenAI STT and TTS endpoints separately

Replace providers.openai.voice (a single apiKey/baseUrl shared by
speech-to-text and text-to-speech) with independent providers.openai.stt
and providers.openai.tts, each carrying its own apiKey/baseUrl. STT and
TTS now resolve fully independently, so they can point at different
OpenAI-compatible endpoints. The env equivalents OPENAI_VOICE_API_KEY /
OPENAI_VOICE_BASE_URL split into OPENAI_STT_* and OPENAI_TTS_*.

No backcompat: the voice key is removed and no longer read. Each feature
still falls back to providers.openai.apiKey/baseUrl, then OPENAI_API_KEY/
OPENAI_BASE_URL. Composer dictation resolves from the STT endpoint.

* fix(voice): keep daemon bootable and respect global OpenAI key

Two issues from review of the STT/TTS endpoint split:

- A config from an older release that still sets providers.openai.voice
  crashed daemon startup, because the strict schema rejects the now-unknown
  key. Strip it before parsing (alongside the existing local.autoDownload
  strip) so the daemon boots; the value is discarded, not migrated.
- An empty endpoint env var (e.g. a copied .env.example leaving
  OPENAI_STT_API_KEY= blank) shadowed the OPENAI_API_KEY fallback, so
  speech was reported as missing credentials despite a configured global
  key. firstDefined now skips empty/whitespace strings.

* fix(voice): isolate STT and TTS option parsing per endpoint

An STT-only OpenAI setup could be broken by a stale or invalid TTS env
var (e.g. a leftover TTS_VOICE/TTS_MODEL), because the single resolution
schema validated both endpoints' option groups before the per-endpoint
gate. Split into endpoint-key, STT-option, and TTS-option schemas and
parse each option group only when that endpoint has credentials, so an
unused endpoint's bad env can no longer take down the configured one.

* fix(voice): tag voice-config shim and update direct-daemon test callers

- Mark the providers.openai.voice strip with a COMPAT(openaiVoiceConfig)
  comment + removal date so it shows up in the back-compat cleanup
  inventory, per repo convention.
- Update the tests that build the daemon directly with a resolved OpenAI
  config (bootstrap smoke + the real-API voice/daemon e2e suites) to the
  new { stt, tts } shape; the old top-level { apiKey } is no longer read,
  and these files are excluded from typecheck so the break was silent.

* fix(voice): update voice-roundtrip debug script to new OpenAI config shape

Last direct daemon caller still passing the removed top-level
openai: { apiKey }; the debug script lives outside tsconfig.scripts.json
so the stale shape wasn't caught by typecheck. Use { stt, tts }.
2026-06-30 10:00:27 +02:00

130 lines
6.2 KiB
Markdown
Raw Blame History

---
title: Voice
description: Paseo voice architecture, local-first model execution, and provider configuration.
nav: Voice
order: 41
category: Configuration
---
# Voice
Paseo has first-class voice support for dictation and voice mode conversations with your coding environment.
## Philosophy
Voice is local-first. You can run speech fully on-device, or choose OpenAI for speech features. For voice reasoning/orchestration, Paseo reuses agent providers already installed and authenticated on your machine.
This keeps credentials and execution in your environment and avoids introducing a separate cloud-only voice stack.
## Architecture
- Speech I/O: STT and TTS providers per feature (`local` or `openai`)
- Local speech runtime: ONNX models executed on CPU by default
- Voice LLM orchestration: hidden agent session using your configured provider (`claude`, `codex`, or `opencode`)
- Tooling path: MCP stdio bridge for voice tools and agent control
## Local Speech
Local speech defaults to model IDs `parakeet-tdt-0.6b-v2-int8` (STT) and `kokoro-en-v0_19` (TTS, speaker 0 / voice 00).
Missing models are downloaded at daemon startup into `$PASEO_HOME/models/local-speech`. Downloads happen only for missing files.
### Local STT models and language support
| Model ID | Languages |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `parakeet-tdt-0.6b-v2-int8` | English only (default). Includes punctuation and capitalization. |
| `parakeet-tdt-0.6b-v3-int8` | 25 European languages, auto-detected: Bulgarian, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, German, Greek, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Ukrainian. |
**To use a non-English language, switch the local STT model to `parakeet-tdt-0.6b-v3-int8`.** v3 detects the spoken language automatically — there is no per-language setting for it. The `language` field below does **not** steer the local Parakeet model (v2 is English-only, v3 auto-detects); it only applies to the OpenAI STT provider.
```json
{
"version": 1,
"features": {
"dictation": {
"stt": { "provider": "local", "model": "parakeet-tdt-0.6b-v2-int8", "language": "en" }
},
"voiceMode": {
"llm": { "provider": "claude", "model": "haiku" },
"stt": { "provider": "local", "model": "parakeet-tdt-0.6b-v2-int8", "language": "en" },
"tts": { "provider": "local", "model": "kokoro-en-v0_19", "speakerId": 0 }
}
},
"providers": {
"local": {
"modelsDir": "~/.paseo/models/local-speech"
}
}
}
```
For multilingual local dictation, set the model to v3 — it auto-detects the language, so no `language` field is needed:
```json
{
"version": 1,
"features": {
"dictation": {
"stt": { "provider": "local", "model": "parakeet-tdt-0.6b-v3-int8" }
}
}
}
```
The `language` field applies only to the OpenAI STT provider: set `features.dictation.stt.language` for dictation and `features.voiceMode.stt.language` for voice mode. If voice language is omitted, Paseo uses the dictation language before falling back to `en`. It has no effect on the local Parakeet models.
## OpenAI Voice Option
You can switch dictation, voice STT, and voice TTS to OpenAI by setting provider fields to `openai` and providing OpenAI credentials.
```json
{
"version": 1,
"features": {
"dictation": { "stt": { "provider": "openai" } },
"voiceMode": {
"stt": { "provider": "openai" },
"tts": { "provider": "openai" }
}
},
"providers": {
"openai": {
"stt": {
"apiKey": "...",
"baseUrl": "https://api.openai.com/v1"
},
"tts": {
"apiKey": "...",
"baseUrl": "https://api.openai.com/v1"
}
}
}
}
```
`providers.openai.stt` covers dictation and voice mode speech-to-text, and `providers.openai.tts` covers voice mode text-to-speech. Because they resolve independently, you can point STT and TTS at different endpoints. Each falls back to `providers.openai.apiKey`/`baseUrl`, then `OPENAI_API_KEY`/`OPENAI_BASE_URL`, when unset. These settings configure only Paseo OpenAI speech traffic, without changing Codex or other OpenAI-backed tools.
Paseo uses these paths under the configured OpenAI base URL:
- dictation STT: `/v1/audio/transcriptions`
- voice mode STT: `/v1/audio/transcriptions`
- voice mode TTS: `/v1/audio/speech`
## Environment Variables
- `PASEO_VOICE_LLM_PROVIDER`, voice agent provider override
- `PASEO_DICTATION_STT_PROVIDER`, `PASEO_VOICE_STT_PROVIDER`, `PASEO_VOICE_TTS_PROVIDER`, speech provider selection (`local` or `openai`)
- `OPENAI_STT_API_KEY`, `OPENAI_STT_BASE_URL`, OpenAI speech-to-text endpoint (dictation + voice mode STT)
- `OPENAI_TTS_API_KEY`, `OPENAI_TTS_BASE_URL`, OpenAI text-to-speech endpoint (voice mode TTS)
- `PASEO_LOCAL_MODELS_DIR`, local model storage directory
- `PASEO_DICTATION_LOCAL_STT_MODEL`, local dictation STT model ID
- `PASEO_VOICE_LOCAL_STT_MODEL`, `PASEO_VOICE_LOCAL_TTS_MODEL`, local voice STT/TTS model IDs
- `PASEO_DICTATION_LANGUAGE`, dictation STT language (OpenAI STT only; ignored by local Parakeet)
- `PASEO_VOICE_LANGUAGE`, voice mode STT language; falls back to `PASEO_DICTATION_LANGUAGE` when unset (OpenAI STT only; ignored by local Parakeet)
- `PASEO_VOICE_LOCAL_TTS_SPEAKER_ID`, `PASEO_VOICE_LOCAL_TTS_SPEED`, optional local voice TTS tuning
## Operational Notes
Voice mode can launch and control agents. Treat voice prompts with the same care as direct agent instructions, especially when specifying working directories or destructive operations.
Reference in New Issue View Git Blame Copy Permalink