puter/paseo
1
0
Fork 0
mirror of https://github.com/getpaseo/paseo.git synced 2026-11-03 23:23:21 +00:00
Code Issues Packages Projects Releases Wiki Activity

Choose voice providers and local speech models from a Voice settings page (#7295)

Browse Source 
* feat(voice): add a Voice settings page with provider choice and a local model catalog

Each host gets a Voice page under host settings. Dictation and voice mode
each pick a provider (Local or OpenAI) and a model. Provider settings sit in
one group of collapsible rows: local speech-to-text and text-to-speech list
every catalog model with its size, languages and description before download,
and OpenAI takes a write-only API key and an optional base URL.

The local catalog grows from Parakeet/Kokoro to Whisper, SenseVoice, Dolphin,
GigaAM, Parakeet Japanese, multilingual Kokoro, Kitten and Piper voices. Each
entry declares its sherpa engine family, so the recognizer and TTS build the
matching model config and the downloader verifies the files that family needs.
Kitten v0.8 is left out: it crashes the pinned sherpa-onnx 1.12.28.

Voice mode is now opt-in. The default lived only in code (true), never in
config.json, so hosts that never toggled it find it off after upgrading.

OpenAI speech-to-text resolved one model for both features, with voice mode's
choice taking precedence, so dictation silently used voice mode's model. The
resolved config now carries a model per feature; STT_MODEL still overrides both.

Rows revealed under a collapsible settings row are a nested density level set
by the container: tighter padding, smaller text, and xs controls, with
controls taking their size from the row instead of each caller.

* test(app): pre-bundle the JSX dev runtime for browser tests

Compiling test JSX with the automatic runtime imports react/jsx-dev-runtime. Vite discovered it mid-run on a cold cache and reloaded the page, which failed the first browser test file to import it.

* refactor(voice): keep speech provider settings at each use site

Store credentials and endpoint options beside the speech feature that consumes them so saved page values cannot be shadowed by a separate provider block. Normalize existing config on load and mirror the inline shape on config reads and patches.

This is the config, resolver and wire phase; the provider sheet follows separately.

* fix(voice): preserve saved keys when resaving provider settings

Forms send the selected provider on each save but cannot read back its API key. Merge patches for the existing provider so these saves retain credentials; replace the site only when the provider changes.

* feat(voice): configure each speech use site from a provider sheet

The Voice page now lists Speech-to-text, Listening and Speaking as one-line
summaries. Tapping one opens a sheet with the provider choice and only that
provider's settings: the local catalog (size, languages and description
before download, download state per model) or the OpenAI key, base URL,
model and language or voice.

The sheet's form model sends only the fields the user changed, so re-saving
never clears a saved key. Fields an env var decides are read-only with a
"Set by" hint; the API key and base URL stay editable because a saved value
wins over the env fallback. Save failures show a toast and keep the sheet
open with the input.

E2E daemons pin speech providers through env vars to avoid model downloads,
so the browser spec covers the locked-provider path, the key fallback and
the failed save; provider switching is covered by the form model tests.

* refactor(voice): press settings and catalog rows through PressHighlight

Settings rows that lead somewhere and local model rows now use the standard
row press primitive, so native shows the finger-down flash the rest of the
app's rows have. Web hover and press styling are unchanged.

* fix(voice): preserve STT language for local providers

Language belongs to every STT use site. Preserve legacy local language settings through config loading and saving, resolve them without provider gating, and expose them in the provider form.
This commit is contained in:
Mohamed Boudra
2026-10-26 14:19:02 +07:00
committed by GitHub
parent 88bd1cf05a
commit 74a7e5803b
83 changed files with 6060 additions and 1086 deletions
Download Patch File Download Diff File Expand all files Collapse all files
packages/server/src/server/agent/stt-manager.test.ts
+16 -7
View File
@@ -1,9 +1,12 @@
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it } from "vitest";
import pino from "pino";
import { EventEmitter } from "node:events";
import { STTManager } from "./stt-manager.js";
import { PersistedConfigSchema } from "../persisted-config.js";
import { loadPersistedConfig } from "../persisted-config.js";
import { resolveSpeechConfig } from "../speech/speech-config-resolver.js";
import type {
SpeechToTextProvider,
@@ -97,12 +100,18 @@ class SequencedFakeStt implements SpeechToTextProvider {
describe("STTManager", () => {
function resolveVoiceLanguage(params: { env?: NodeJS.ProcessEnv; persisted?: unknown }): string {
const result = resolveSpeechConfig({
paseoHome: "/tmp/paseo-home",
env: params.env ?? ({} as NodeJS.ProcessEnv),
persisted: PersistedConfigSchema.parse(params.persisted ?? {}),
});
return result.speech.sttLanguages.voice;
const home = mkdtempSync(path.join(tmpdir(), "stt-language-"));
try {
writeFileSync(path.join(home, "config.json"), JSON.stringify(params.persisted ?? {}));
const result = resolveSpeechConfig({
paseoHome: home,
env: params.env ?? {},
persisted: loadPersistedConfig(home),
});
return result.speech.sttLanguages.voice;
} finally {
rmSync(home, { recursive: true, force: true });
}
}
async function transcribeWithResolvedVoiceLanguage(params: {
packages/server/src/server/authorization/operation-permissions.ts
+6
View File
@@ -182,6 +182,9 @@ const INBOUND_PERMISSION = {
set_agent_model_request: "workspace.write",
set_agent_thinking_request: "workspace.write",
set_daemon_config_request: "daemon.manage",
"speech.local_models.list.request": "daemon.read",
"speech.local_models.download.request": "daemon.manage",
"speech.local_models.delete.request": "daemon.manage",
set_voice_mode: "workspace.write",
shutdown_server_request: "daemon.manage",
start_workspace_script_request: "workspace.write",
@@ -417,6 +420,9 @@ const OUTBOUND_PERMISSION = {
set_agent_model_response: "workspace.write",
set_agent_thinking_response: "workspace.write",
set_daemon_config_response: "daemon.manage",
"speech.local_models.list.response": "daemon.read",
"speech.local_models.download.response": "daemon.manage",
"speech.local_models.delete.response": "daemon.manage",
set_voice_mode_response: "workspace.write",
start_workspace_script_response: "workspace.write",
stash_list_response: "workspace.read",
packages/server/src/server/bootstrap.ts
+19 -8
View File
@@ -138,6 +138,7 @@ import type { OpenAiSpeechProviderConfig } from "./speech/providers/openai/confi
import type { LocalSpeechProviderConfig } from "./speech/providers/local/config.js";
import type { RequestedSpeechProviders } from "./speech/speech-types.js";
import { createSpeechService } from "./speech/speech-runtime.js";
import { createSpeechSettingsOwner } from "./speech/speech-settings.js";
import { AgentManager } from "./agent/agent-manager.js";
import { AgentStorage } from "./agent/agent-storage.js";
import { attachAgentStoragePersistence } from "./persistence-hooks.js";
@@ -379,6 +380,8 @@ export interface PaseoSpeechSttLanguages {
export interface PaseoSpeechConfig {
providers: RequestedSpeechProviders;
// Where local models live, whether or not a feature currently uses the local provider.
localModelsDir: string;
sttLanguages?: PaseoSpeechSttLanguages;
local?: PaseoLocalSpeechConfig;
}
@@ -555,6 +558,10 @@ function createInitialRelayConfig(config: PaseoDaemonConfig): MutableDaemonConfi
};
}
function resolveReloadEnv(config: PaseoDaemonConfig): NodeJS.ProcessEnv {
return config.configReload?.env ?? process.env;
}
function createInitialMutableDaemonConfig(config: PaseoDaemonConfig): MutableDaemonConfig {
const providers = config.providerOverrides ?? {};
@@ -615,13 +622,24 @@ export async function createPaseoDaemon(
const elapsed = () => `${(performance.now() - bootstrapStart).toFixed(0)}ms`;
const daemonVersion = config.daemonVersion ?? resolveDaemonVersion(import.meta.url);
const initialMutableConfig = createInitialMutableDaemonConfig(config);
const speechService = createSpeechService({
logger,
openaiConfig: config.openai,
speechConfig: config.speech,
});
logger.info({ elapsed: elapsed() }, "Speech service created");
const daemonConfigStore = new DaemonConfigStore(config.paseoHome, initialMutableConfig, logger, {
relayEnabledMutable: config.relayEnabledMutable ?? true,
startupPersisted: config.configReload?.startupPersisted,
speech: createSpeechSettingsOwner({
paseoHome: config.paseoHome,
env: resolveReloadEnv(config),
speechService,
}),
reloadSource: {
resolve: (persisted) => {
const reloaded = resolveConfigFromPersisted(config.paseoHome, persisted, {
env: config.configReload?.env ?? process.env,
env: resolveReloadEnv(config),
cli: config.configReload?.cli,
relayEnabledFallback: config.configReload?.relayEnabledFallback,
});
@@ -1677,13 +1695,6 @@ export async function createPaseoDaemon(
logger.info({ route: agentMcpRoute, enabled: mcpEnabled }, "Agent MCP route mounted");
}
const speechService = createSpeechService({
logger,
openaiConfig: config.openai,
speechConfig: config.speech,
});
logger.info({ elapsed: elapsed() }, "Speech service created");
logger.info({ elapsed: elapsed() }, "Bootstrap complete, ready to start listening");
const start = async () => {
packages/server/src/server/config.test.ts
+6 -1
View File
@@ -115,9 +115,13 @@ describe("server config", () => {
name: "OpenAI speech providers",
providers: { dictation: "openai", voiceStt: "openai", voiceTts: "openai" },
expected: [
"features.dictation.stt.apiKey",
"features.dictation.stt.confidenceThreshold",
"features.dictation.stt.model",
"features.voiceMode.stt.apiKey",
"features.voiceMode.stt.confidenceThreshold",
"features.voiceMode.stt.model",
"features.voiceMode.tts.apiKey",
"features.voiceMode.tts.model",
"features.voiceMode.tts.voice",
],
@@ -126,8 +130,9 @@ describe("server config", () => {
name: "mixed local and OpenAI speech providers",
providers: { dictation: "local", voiceStt: "openai", voiceTts: "local" },
expected: [
"features.dictation.stt.confidenceThreshold",
"features.dictation.stt.model",
"features.voiceMode.stt.apiKey",
"features.voiceMode.stt.confidenceThreshold",
"features.voiceMode.stt.model",
"features.voiceMode.tts.model",
],
packages/server/src/server/config.ts
+8 -63
View File
@@ -22,7 +22,7 @@ import { ProviderOverrideSchema } from "./agent/provider-launch-config.js";
import { AgentProviderSchema } from "@getpaseo/protocol/provider-manifest";
import { hashDaemonPassword } from "./auth.js";
import { resolveSpeechConfig } from "./speech/speech-config-resolver.js";
import type { RequestedSpeechProviders } from "./speech/speech-types.js";
import { readSpeechSettings } from "./speech/speech-settings.js";
import { mergeHostnames, parseHostnamesEnv, type HostnamesConfig } from "./hostnames.js";
import { resolveGitProcessPolicy } from "../utils/git-process-scheduler.js";
@@ -614,7 +614,7 @@ export function resolveConfigFromPersisted(
persisted.agents?.providers as Record<string, unknown> | undefined,
);
const overrideControlledPaths = resolveOverrideControlledPaths(env, cli, speech.providers);
const overrideControlledPaths = resolveOverrideControlledPaths(env, cli, persisted);
return {
listen,
@@ -692,13 +692,13 @@ function parsePositiveGitOverride(value: string | undefined): boolean {
function resolveOverrideControlledPaths(
env: NodeJS.ProcessEnv,
cli: CliConfigOverrides | undefined,
speechProviders: RequestedSpeechProviders,
persisted: PersistedConfig,
): string[] {
return Array.from(
new Set([
...resolveDaemonOverrideControlledPaths(env, cli),
...resolveLogOverrideControlledPaths(env),
...resolveSpeechOverrideControlledPaths(env, speechProviders),
...resolveSpeechOverrideControlledPaths(env, persisted),
]),
).sort();
}
@@ -807,67 +807,12 @@ function resolveLogOverrideControlledPaths(env: NodeJS.ProcessEnv): string[] {
return paths;
}
function isEnabledSpeechProvider(
provider: RequestedSpeechProviders[keyof RequestedSpeechProviders],
expected: "local" | "openai",
): boolean {
return provider.enabled !== false && provider.provider === expected;
}
function resolveSpeechOverrideControlledPaths(
env: NodeJS.ProcessEnv,
providers: RequestedSpeechProviders,
persisted: PersistedConfig,
): string[] {
const paths: string[] = [];
const add = (envName: string, ...configPaths: string[]) => {
if (env[envName] !== undefined) paths.push(...configPaths);
};
add("PASEO_DICTATION_ENABLED", "features.dictation.enabled");
add("PASEO_DICTATION_STT_PROVIDER", "features.dictation.stt.provider");
if (
env.PASEO_DICTATION_LOCAL_STT_MODEL !== undefined &&
isEnabledSpeechProvider(providers.dictationStt, "local")
) {
paths.push("features.dictation.stt.model");
}
add("PASEO_DICTATION_LANGUAGE", "features.dictation.stt.language");
add("PASEO_VOICE_MODE_ENABLED", "features.voiceMode.enabled");
add("PASEO_VOICE_LLM_PROVIDER", "features.voiceMode.llm.provider");
add("PASEO_VOICE_STT_PROVIDER", "features.voiceMode.stt.provider");
if (
env.PASEO_VOICE_LOCAL_STT_MODEL !== undefined &&
isEnabledSpeechProvider(providers.voiceStt, "local")
) {
paths.push("features.voiceMode.stt.model");
}
add("PASEO_VOICE_LANGUAGE", "features.voiceMode.stt.language");
add("PASEO_VOICE_TURN_DETECTION_PROVIDER", "features.voiceMode.turnDetection.provider");
add("PASEO_VOICE_TTS_PROVIDER", "features.voiceMode.tts.provider");
if (
env.PASEO_VOICE_LOCAL_TTS_MODEL !== undefined &&
isEnabledSpeechProvider(providers.voiceTts, "local")
) {
paths.push("features.voiceMode.tts.model");
}
add("PASEO_VOICE_LOCAL_TTS_SPEAKER_ID", "features.voiceMode.tts.speakerId");
add("PASEO_VOICE_LOCAL_TTS_SPEED", "features.voiceMode.tts.speed");
add("PASEO_LOCAL_MODELS_DIR", "providers.local.modelsDir");
const openAiDictationStt = isEnabledSpeechProvider(providers.dictationStt, "openai");
const openAiVoiceStt = isEnabledSpeechProvider(providers.voiceStt, "openai");
if (env.STT_CONFIDENCE_THRESHOLD !== undefined && (openAiDictationStt || openAiVoiceStt)) {
paths.push("features.dictation.stt.confidenceThreshold");
}
if (env.STT_MODEL !== undefined) {
if (openAiDictationStt) paths.push("features.dictation.stt.model");
if (openAiVoiceStt) paths.push("features.voiceMode.stt.model");
}
if (isEnabledSpeechProvider(providers.voiceTts, "openai")) {
add("TTS_MODEL", "features.voiceMode.tts.model");
add("TTS_VOICE", "features.voiceMode.tts.voice");
}
if (env.PASEO_DICTATION_LANGUAGE !== undefined && env.PASEO_VOICE_LANGUAGE === undefined) {
paths.push("features.voiceMode.stt.language");
}
const paths = Object.keys(readSpeechSettings(persisted, env).envOverrides);
if (env.PASEO_VOICE_LLM_PROVIDER !== undefined) paths.push("features.voiceMode.llm.provider");
if (env.PASEO_LOCAL_MODELS_DIR !== undefined) paths.push("providers.local.modelsDir");
return paths;
}
packages/server/src/server/daemon-client.e2e.test.ts
+2 -2
View File
@@ -1318,9 +1318,9 @@ test("emits disabled voice capability reasons on fresh daemon startup", async ()
expect(voice).toBeTruthy();
expect(voice?.dictation.enabled).toBe(false);
expect(voice?.dictation.reason).toBe("Dictation is disabled in daemon config.");
expect(voice?.dictation.reason).toBe("Dictation is off. Turn it on in Settings > Voice.");
expect(voice?.voice.enabled).toBe(false);
expect(voice?.voice.reason).toBe("Realtime voice is disabled in daemon config.");
expect(voice?.voice.reason).toBe("Voice mode is off. Turn it on in Settings > Voice.");
} finally {
await client.close().catch(() => undefined);
await isolatedCtx.cleanup();
packages/server/src/server/daemon-config-store.test.ts
+238 -1
View File
@@ -1,4 +1,4 @@
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { afterEach, describe, expect, test } from "vitest";
@@ -6,6 +6,7 @@ import { afterEach, describe, expect, test } from "vitest";
import { DaemonConfigStore, applyMutableProviderConfigToOverrides } from "./daemon-config-store.js";
import { loadConfig } from "./config.js";
import { loadPersistedConfig } from "./persisted-config.js";
import { readSpeechSettings } from "./speech/speech-settings.js";
import type { PersistedConfig } from "./persisted-config.js";
import type { MutableDaemonConfig } from "@getpaseo/protocol/messages";
@@ -1284,3 +1285,239 @@ describe("DaemonConfigStore reload", () => {
});
});
});
describe("DaemonConfigStore speech settings", () => {
const tempDirs: string[] = [];
afterEach(() => {
for (const dir of tempDirs) rmSync(dir, { recursive: true, force: true });
});
function createSpeechStore(initialPersisted?: unknown, env: NodeJS.ProcessEnv = {}) {
const paseoHome = mkdtempSync(path.join(tmpdir(), "paseo-daemon-config-speech-"));
tempDirs.push(paseoHome);
if (initialPersisted) {
writeFileSync(
path.join(paseoHome, "config.json"),
`${JSON.stringify(initialPersisted, null, 2)}\n`,
);
}
const persisted = loadPersistedConfig(paseoHome);
const applied: PersistedConfig[] = [];
const speech = {
read: (next: PersistedConfig) => readSpeechSettings(next, env),
apply: (next: PersistedConfig) => applied.push(next),
};
const store = new DaemonConfigStore(paseoHome, reloadableConfig(persisted), undefined, {
speech,
reloadSource: {
resolve: (next) => ({ mutable: reloadableConfig(next), overrideControlledPaths: [] }),
},
});
return { paseoHome, store, applied, persisted };
}
test("reads, patches, and clears local STT language without losing it on other saves", () => {
const { paseoHome, store } = createSpeechStore({
features: { dictation: { stt: { language: "fr" } } },
});
expect(store.get().features?.dictation.stt).toEqual({ provider: "local", language: "fr" });
store.patch({ features: { voiceMode: { enabled: true } } });
expect(loadPersistedConfig(paseoHome).features?.dictation?.stt).toEqual({
provider: "local",
language: "fr",
});
const patched = store.patch({
features: { dictation: { stt: { provider: "local", language: "de" } } },
});
expect(patched.features?.dictation.stt).toEqual({ provider: "local", language: "de" });
store.patch({ features: { dictation: { stt: { language: null } } } });
expect(loadPersistedConfig(paseoHome).features?.dictation?.stt).toEqual({ provider: "local" });
});
test("reports language env overrides for both local STT sites", () => {
const { store } = createSpeechStore(
{},
{ PASEO_DICTATION_LANGUAGE: "fr", PASEO_VOICE_LANGUAGE: "de" },
);
expect(store.get().envOverrides).toMatchObject({
"features.dictation.stt.language": "PASEO_DICTATION_LANGUAGE",
"features.voiceMode.stt.language": "PASEO_VOICE_LANGUAGE",
});
});
test("keeps voice mode off until it is turned on", () => {
const { paseoHome, store } = createSpeechStore();
expect(store.get().features?.dictation.enabled).toBe(true);
expect(store.get().features?.voiceMode.enabled).toBe(false);
const result = store.patch({ features: { voiceMode: { enabled: true } } });
expect(result.features?.voiceMode.enabled).toBe(true);
expect(loadPersistedConfig(paseoHome).features?.voiceMode?.enabled).toBe(true);
});
test("stores the OpenAI key without ever returning it", () => {
const { paseoHome, store, applied } = createSpeechStore();
const result = store.patch({
features: { dictation: { stt: { provider: "openai", apiKey: " sk-test " } } },
});
expect(result.features?.dictation.stt).toEqual({ provider: "openai", hasApiKey: true });
expect(JSON.stringify(result)).not.toContain("sk-test");
expect(loadPersistedConfig(paseoHome).features?.dictation?.stt).toMatchObject({
apiKey: "sk-test",
});
expect(applied).toHaveLength(1);
});
test("applies a replaced key even though the visible settings do not change", () => {
const { store, applied } = createSpeechStore({
version: 1,
features: { dictation: { stt: { provider: "openai", apiKey: "sk-old" } } },
});
store.patch({ features: { dictation: { stt: { apiKey: "sk-new" } } } });
expect(applied.map((next) => next.features?.dictation?.stt)).toEqual([
{ provider: "openai", apiKey: "sk-new" },
]);
});
test("switching provider clears the previous provider's model", () => {
const { paseoHome, store } = createSpeechStore({
version: 1,
features: { dictation: { stt: { provider: "local", model: "parakeet-tdt-0.6b-v3-int8" } } },
});
const result = store.patch({ features: { dictation: { stt: { provider: "openai" } } } });
expect(result.features?.dictation.stt).toEqual({ provider: "openai", hasApiKey: false });
expect(loadPersistedConfig(paseoHome).features?.dictation?.stt).toEqual({
provider: "openai",
});
});
test("rejects a local model id outside the catalog without persisting it", () => {
const { paseoHome, store, applied } = createSpeechStore();
expect(() =>
store.patch({ features: { voiceMode: { tts: { provider: "local", model: "nope" } } } }),
).toThrow("Unknown local voice model: nope");
expect(loadPersistedConfig(paseoHome).features?.voiceMode).toBeUndefined();
expect(applied).toEqual([]);
});
test("reload applies hand-edited speech settings without a restart", () => {
const { paseoHome, store, applied, persisted } = createSpeechStore();
writeFileSync(
path.join(paseoHome, "config.json"),
`${JSON.stringify({ ...persisted, features: { voiceMode: { enabled: false } } }, null, 2)}\n`,
);
const result = store.reload();
expect(result.appliedPaths).toContain("features.voiceMode");
expect(result.restartRequiredPaths).toEqual([]);
expect(store.get().features?.voiceMode.enabled).toBe(false);
expect(applied).toHaveLength(1);
});
test("merges same-provider fields and replaces the whole variant when provider changes", () => {
const { paseoHome, store } = createSpeechStore();
store.patch({
features: {
voiceMode: {
stt: { provider: "openai", apiKey: "secret", baseUrl: "https://site/v1", model: "first" },
},
},
});
store.patch({ features: { voiceMode: { stt: { model: "second", baseUrl: null } } } });
expect(loadPersistedConfig(paseoHome).features?.voiceMode?.stt).toEqual({
provider: "openai",
apiKey: "secret",
model: "second",
});
store.patch({ features: { voiceMode: { stt: { provider: "openai", model: "third" } } } });
expect(store.get().features?.voiceMode.stt).toEqual({
provider: "openai",
model: "third",
hasApiKey: true,
});
store.patch({ features: { voiceMode: { stt: { apiKey: "new-secret" } } } });
store.patch({ features: { voiceMode: { stt: { provider: "local" } } } });
store.patch({ features: { voiceMode: { stt: { provider: "openai" } } } });
expect(store.get().features?.voiceMode.stt).toEqual({ provider: "openai", hasApiKey: false });
});
test("keeps a saved OpenAI key when a full variant patch omits the write-only key", () => {
const { paseoHome, store } = createSpeechStore({
features: { voiceMode: { stt: { provider: "openai", apiKey: "saved-secret" } } },
});
const result = store.patch({
features: { voiceMode: { stt: { provider: "openai", model: "whisper-1" } } },
});
expect(result.features?.voiceMode.stt).toEqual({
provider: "openai",
model: "whisper-1",
hasApiKey: true,
});
expect(loadPersistedConfig(paseoHome).features?.voiceMode?.stt).toEqual({
provider: "openai",
apiKey: "saved-secret",
model: "whisper-1",
});
expect(JSON.stringify(result)).not.toContain("saved-secret");
});
test("loads a legacy host, redacts its key and writes the normalized config after any save", () => {
const { paseoHome, store } = createSpeechStore({
providers: { openai: { apiKey: "legacy-secret", stt: { baseUrl: "https://old/v1" } } },
features: { voiceMode: { enabled: true, stt: { provider: "openai" } } },
});
expect(store.get().features?.voiceMode.stt).toEqual({
provider: "openai",
baseUrl: "https://old/v1",
hasApiKey: true,
});
expect(JSON.stringify(store.get())).not.toContain("legacy-secret");
store.patch({ autoArchiveAfterMerge: true });
const disk = JSON.parse(readFileSync(path.join(paseoHome, "config.json"), "utf8"));
expect(disk.providers).not.toHaveProperty("openai");
expect(disk.features.voiceMode.stt.apiKey).toBe("legacy-secret");
});
test("reports env overrides and removes fallback provenance after saving a key", () => {
const { store } = createSpeechStore(
{ features: { voiceMode: { stt: { provider: "openai", model: "configured" } } } },
{
PASEO_VOICE_STT_PROVIDER: "local",
OPENAI_API_KEY: "env-secret",
},
);
expect(store.get().features?.voiceMode.stt.provider).toBe("local");
expect(store.get().envOverrides).toEqual({
"features.voiceMode.stt.provider": "PASEO_VOICE_STT_PROVIDER",
});
const other = createSpeechStore(
{ features: { voiceMode: { stt: { provider: "openai" } } } },
{
OPENAI_API_KEY: "env-secret",
OPENAI_STT_BASE_URL: "https://env/v1",
STT_MODEL: "override",
},
).store;
expect(other.get().envOverrides).toEqual({
"features.voiceMode.stt.apiKey": "OPENAI_API_KEY",
"features.voiceMode.stt.baseUrl": "OPENAI_STT_BASE_URL",
"features.voiceMode.stt.model": "STT_MODEL",
});
expect(other.get().features?.voiceMode.stt.hasApiKey).toBe(false);
other.patch({
features: { voiceMode: { stt: { apiKey: "saved-secret", baseUrl: "https://saved/v1" } } },
});
expect(other.get().envOverrides).toEqual({ "features.voiceMode.stt.model": "STT_MODEL" });
expect(JSON.stringify(other.get())).not.toContain("secret");
});
});
packages/server/src/server/daemon-config-store.ts
+68 -20
View File
@@ -9,6 +9,8 @@ import {
MutableDaemonConfigPatchSchema,
} from "@getpaseo/protocol/messages";
import type { AgentSkillSelection } from "@getpaseo/protocol/messages";
import type { SpeechSettingsPatch } from "@getpaseo/protocol/speech/rpc-schemas";
import { mergeSpeechSettingsPatch, type SpeechSettingsOwner } from "./speech/speech-settings.js";
export type { MutableDaemonConfig, MutableDaemonConfigPatch } from "@getpaseo/protocol/messages";
@@ -32,8 +34,11 @@ interface SupportedMutableConfigPatch {
skills?: MutableDaemonConfig["skills"];
pluginsEnabled?: boolean;
plugins?: MutableDaemonConfig["plugins"];
features?: SpeechSettingsPatch;
}
const SPEECH_PERSISTED_PATHS = ["features.dictation", "features.voiceMode"];
interface LoggerLike {
child(bindings: Record<string, unknown>): LoggerLike;
info(...args: unknown[]): void;
@@ -296,6 +301,7 @@ function pickSupportedPatchFields(patch: MutableDaemonConfigPatch): SupportedMut
...(patch.agentProfiles !== undefined ? { agentProfiles: patch.agentProfiles } : {}),
...(patch.pluginsEnabled !== undefined ? { pluginsEnabled: patch.pluginsEnabled } : {}),
...(patch.plugins !== undefined ? { plugins: patch.plugins } : {}),
...(patch.features !== undefined ? { features: patch.features } : {}),
};
}
@@ -338,6 +344,7 @@ export class DaemonConfigStore {
private readonly relayEnabledMutable: boolean;
private readonly reloadSource: DaemonConfigReloadSource | undefined;
private readonly startupPersisted: PersistedConfig;
private readonly speechOwner: SpeechSettingsOwner | undefined;
private lastKnownPersisted: PersistedConfig;
constructor(
@@ -348,18 +355,21 @@ export class DaemonConfigStore {
relayEnabledMutable?: boolean;
reloadSource?: DaemonConfigReloadSource;
startupPersisted?: PersistedConfig;
speech?: SpeechSettingsOwner;
} = {},
) {
this.paseoHome = paseoHome;
this.logger = getLogger(logger);
this.relayEnabledMutable = options.relayEnabledMutable ?? true;
this.reloadSource = options.reloadSource;
this.speechOwner = options.speech;
this.startupPersisted = options.startupPersisted ?? loadPersistedConfig(paseoHome, this.logger);
this.lastKnownPersisted = this.startupPersisted;
this.current = MutableDaemonConfigSchema.parse({
...initial,
relay: initial.relay ?? { enabled: true },
...this.readSpeech(this.startupPersisted),
});
this.relayEnabledMutable = options.relayEnabledMutable ?? true;
this.reloadSource = options.reloadSource;
this.startupPersisted = options.startupPersisted ?? loadPersistedConfig(paseoHome, this.logger);
this.lastKnownPersisted = this.startupPersisted;
}
public get(): MutableDaemonConfig {
@@ -381,9 +391,11 @@ export class DaemonConfigStore {
"Relay is controlled by a daemon launch override. Remove PASEO_RELAY_ENABLED or the relay CLI flag before changing it here.",
);
}
const { removeProviders = [], ...configPatch } = parsedPatch;
const { removeProviders = [], ...patch } = parsedPatch;
const { features: speechPatch, ...configPatch } = patch;
const removedProviders = Array.from(new Set(removeProviders));
const merged = deepMerge(this.current, configPatch);
if (speechPatch !== undefined) Object.assign(merged, this.readPatchedSpeech(speechPatch));
if (parsedPatch.skills?.selection !== undefined) {
merged.skills = { selection: parsedPatch.skills.selection };
}
@@ -397,30 +409,51 @@ export class DaemonConfigStore {
const configChanged = !isEqualValue(this.current, next);
if (!configChanged && removedProviders.length === 0) {
// A replaced API key leaves the speech view unchanged, so a speech patch always persists.
if (!configChanged && removedProviders.length === 0 && speechPatch === undefined) {
return this.current;
}
const { previous: persistedBeforePatch, knownNext } = this.persistConfig(
configPatch,
patch,
removedProviders,
);
if (!configChanged) {
this.lastKnownPersisted = knownNext;
return this.current;
}
try {
this.applyReplacement(next, { removedProviders });
this.lastKnownPersisted = knownNext;
} catch (error) {
savePersistedConfig(this.paseoHome, persistedBeforePatch, this.logger);
throw error;
const speechChanged = this.speechPersistedChanged(this.lastKnownPersisted, knownNext);
if (configChanged) {
try {
this.applyReplacement(next, { removedProviders });
} catch (error) {
savePersistedConfig(this.paseoHome, persistedBeforePatch, this.logger);
throw error;
}
}
this.lastKnownPersisted = knownNext;
if (speechChanged) this.speechOwner?.apply(knownNext);
return this.current;
}
private readSpeech(
persisted: PersistedConfig,
): Pick<MutableDaemonConfig, "features" | "envOverrides"> {
return this.speechOwner ? this.speechOwner.read(persisted) : {};
}
private readPatchedSpeech(
speechPatch: SpeechSettingsPatch,
): Pick<MutableDaemonConfig, "features" | "envOverrides"> {
if (!this.speechOwner) {
throw new Error("Speech settings are unavailable for this daemon instance");
}
return this.speechOwner.read(mergeSpeechSettingsPatch(this.lastKnownPersisted, speechPatch));
}
private speechPersistedChanged(previous: PersistedConfig, next: PersistedConfig): boolean {
return diffPaths(previous, next).some((path) =>
SPEECH_PERSISTED_PATHS.some((owner) => pathBelongsTo(path, owner)),
);
}
public reload(): DaemonConfigReloadResult {
if (!this.reloadSource) {
throw new Error("Daemon config reload is unavailable for this daemon instance");
@@ -433,6 +466,7 @@ export class DaemonConfigStore {
const desired = MutableDaemonConfigSchema.parse({
...resolved.mutable,
plugins: this.current.plugins,
...this.readSpeech(persisted),
});
const changedSinceLastApply = diffPaths(this.lastKnownPersisted, persisted);
const overrideControlledPaths = compactOwnedPaths(
@@ -441,7 +475,7 @@ export class DaemonConfigStore {
),
resolved.overrideControlledPaths,
);
const appliedPaths = RELOADABLE_PATHS.filter((persistedPath) => {
const appliedPaths: string[] = RELOADABLE_PATHS.filter((persistedPath) => {
if (resolved.overrideControlledPaths.some((owner) => pathBelongsTo(persistedPath, owner))) {
return false;
}
@@ -458,6 +492,8 @@ export class DaemonConfigStore {
diffPaths(this.startupPersisted, persisted).filter((path) => {
if (path === "$schema" || path === "version") return false;
if (RELOADABLE_PATHS.some((owner) => pathBelongsTo(path, owner))) return false;
if (this.speechOwner && SPEECH_PERSISTED_PATHS.some((owner) => pathBelongsTo(path, owner)))
return false;
return !resolved.overrideControlledPaths.some((owner) => pathBelongsTo(path, owner));
}),
[],
@@ -466,8 +502,17 @@ export class DaemonConfigStore {
const removedProviders = Object.keys(this.current.providers).filter(
(provider) => !(provider in desired.providers),
);
const speechChanged = this.speechPersistedChanged(this.lastKnownPersisted, persisted);
this.applyReplacement(desired, { removedProviders });
this.lastKnownPersisted = persisted;
if (speechChanged && this.speechOwner) {
this.speechOwner.apply(persisted);
appliedPaths.push(
...SPEECH_PERSISTED_PATHS.filter((owner) =>
changedSinceLastApply.some((path) => pathBelongsTo(path, owner)),
),
);
}
return {
appliedPaths: [...appliedPaths].sort(),
@@ -604,8 +649,11 @@ function mergeMutablePatchIntoPersistedConfig(params: {
const { persisted, patch, removeProviders, persistRelayEnabled } = params;
const daemon = mergeMutableDaemonPatch(persisted.daemon, patch, persistRelayEnabled);
const agents = mergeMutableAgentPatch(persisted.agents, patch, removeProviders);
const withSpeech = patch.features
? mergeSpeechSettingsPatch(persisted, patch.features)
: persisted;
return {
...persisted,
...withSpeech,
...(patch.pluginsEnabled !== undefined ? { pluginsEnabled: patch.pluginsEnabled } : {}),
...(patch.plugins !== undefined ? { plugins: patch.plugins } : {}),
...(daemon ? { daemon } : { daemon: undefined }),
packages/server/src/server/dictation/dictation-stream-manager.test.ts
+16 -7
View File
@@ -1,9 +1,12 @@
import { mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { describe, expect, it, vi, beforeEach, afterEach } from "vitest";
import { EventEmitter } from "node:events";
import pino from "pino";
import { DictationStreamManager } from "./dictation-stream-manager.js";
import { PersistedConfigSchema } from "../persisted-config.js";
import { loadPersistedConfig } from "../persisted-config.js";
import { resolveSpeechConfig } from "../speech/speech-config-resolver.js";
import type {
SpeechToTextProvider,
@@ -131,12 +134,18 @@ describe("DictationStreamManager (provider-agnostic provider)", () => {
env?: NodeJS.ProcessEnv;
persisted?: unknown;
}): string {
const result = resolveSpeechConfig({
paseoHome: "/tmp/paseo-home",
env: params.env ?? ({} as NodeJS.ProcessEnv),
persisted: PersistedConfigSchema.parse(params.persisted ?? {}),
});
return result.speech.sttLanguages.dictation;
const home = mkdtempSync(path.join(tmpdir(), "stt-language-"));
try {
writeFileSync(path.join(home, "config.json"), JSON.stringify(params.persisted ?? {}));
const result = resolveSpeechConfig({
paseoHome: home,
env: params.env ?? {},
persisted: loadPersistedConfig(home),
});
return result.speech.sttLanguages.dictation;
} finally {
rmSync(home, { recursive: true, force: true });
}
}
async function startWithResolvedDictationLanguage(params: {
packages/server/src/server/persisted-config-edit.test.ts
+16
View File
@@ -43,3 +43,19 @@ test("configuration edits validate before writing and preserve unrelated setting
await rm(root, { recursive: true, force: true });
}
});
test("configuration paths edit fields inside speech provider variants", async () => {
const home = await mkdtemp(path.join(tmpdir(), "paseo-speech-edit-"));
try {
editPersistedConfig(home, "features.voiceMode.stt", { value: { provider: "openai" } });
editPersistedConfig(home, "features.voiceMode.stt.apiKey", { value: "site-secret" });
const config = readPersistedConfig(home);
expect(getPersistedConfigValue(config, "features.voiceMode.stt.apiKey")).toBe("site-secret");
editPersistedConfig(home, "features.voiceMode.stt.apiKey", { unset: true });
expect(
getPersistedConfigValue(readPersistedConfig(home), "features.voiceMode.stt.apiKey"),
).toBeUndefined();
} finally {
await rm(home, { recursive: true, force: true });
}
});
packages/server/src/server/persisted-config.test.ts
+193 -20
View File
@@ -1,3 +1,4 @@
import { resolveSpeechConfig } from "./speech/speech-config-resolver.js";
import {
chmodSync,
lstatSync,
@@ -149,26 +150,33 @@ describe("PersistedConfigSchema worktrees config", () => {
});
describe("PersistedConfigSchema provider credentials", () => {
test("accepts separate OpenAI STT and TTS credentials", () => {
test("accepts separate inline STT and TTS credentials", () => {
const parsed = PersistedConfigSchema.parse({
providers: {
openai: {
features: {
voiceMode: {
stt: {
provider: "openai",
apiKey: " stt-secret ",
baseUrl: " https://stt.example.com/v1 ",
},
tts: {
provider: "openai",
apiKey: " tts-secret ",
baseUrl: " https://tts.example.com/v1 ",
},
},
},
});
expect(parsed.providers?.openai?.stt?.apiKey).toBe("stt-secret");
expect(parsed.providers?.openai?.stt?.baseUrl).toBe("https://stt.example.com/v1");
expect(parsed.providers?.openai?.tts?.apiKey).toBe("tts-secret");
expect(parsed.providers?.openai?.tts?.baseUrl).toBe("https://tts.example.com/v1");
expect(parsed.features?.voiceMode?.stt).toEqual({
provider: "openai",
apiKey: "stt-secret",
baseUrl: "https://stt.example.com/v1",
});
expect(parsed.features?.voiceMode?.tts).toEqual({
provider: "openai",
apiKey: "tts-secret",
baseUrl: "https://tts.example.com/v1",
});
});
});
@@ -660,19 +668,21 @@ describe("PersistedConfigSchema voice mode config", () => {
features: {
dictation: {
stt: {
provider: "openai",
language: " fr ",
},
},
voiceMode: {
stt: {
provider: "openai",
language: " de ",
},
},
},
});
expect(parsed.features?.dictation?.stt?.language).toBe("fr");
expect(parsed.features?.voiceMode?.stt?.language).toBe("de");
expect(parsed.features?.dictation?.stt).toMatchObject({ language: "fr" });
expect(parsed.features?.voiceMode?.stt).toMatchObject({ language: "de" });
});
test("accepts a voice and model from a self-hosted OpenAI-compatible server", () => {
@@ -685,7 +695,7 @@ describe("PersistedConfigSchema voice mode config", () => {
});
expect(parsed.features?.voiceMode?.tts?.model).toBe("kokoro-v1:Q8_0");
expect(parsed.features?.voiceMode?.tts?.voice).toBe("aiden");
expect(parsed.features?.voiceMode?.tts).toMatchObject({ voice: "aiden" });
});
});
@@ -754,10 +764,7 @@ describe("loadPersistedConfig", () => {
const config = loadPersistedConfig(home);
expect(config.providers?.openai?.apiKey).toBe("global-key");
expect((config.providers?.openai as Record<string, unknown>)?.voice).toBeUndefined();
expect(config.providers?.openai?.stt).toBeUndefined();
expect(config.providers?.openai?.tts).toBeUndefined();
expect(config.providers).not.toHaveProperty("openai");
} finally {
rmSync(home, { recursive: true, force: true });
}
@@ -852,11 +859,7 @@ describe.skipIf(process.platform === "win32")("persisted config file permissions
const home = path.join(parent, "home");
try {
savePersistedConfig(home, {
providers: {
openai: {
apiKey: "secret",
},
},
features: { dictation: { stt: { provider: "openai", apiKey: "secret" } } },
});
expect(modeOf(home)).toBe(PRIVATE_DIRECTORY_MODE);
@@ -898,3 +901,173 @@ describe.skipIf(process.platform === "win32")(
});
},
);
describe("speech provider config normalization", () => {
test.each([undefined, "local"])(
"preserves local STT language on load and save (provider: %s)",
(provider) => {
const home = createTempHome();
try {
writeFileSync(
path.join(home, "config.json"),
JSON.stringify({
features: { dictation: { stt: { provider, language: "fr" } } },
}),
);
const loaded = loadPersistedConfig(home);
expect(loaded.features?.dictation?.stt).toEqual({ provider: "local", language: "fr" });
expect(
resolveSpeechConfig({ paseoHome: home, env: {}, persisted: loaded }).speech.sttLanguages
.dictation,
).toBe("fr");
savePersistedConfig(home, loaded);
expect(loadPersistedConfig(home).features?.dictation?.stt).toEqual({
provider: "local",
language: "fr",
});
expect(
JSON.parse(readFileSync(path.join(home, "config.json"), "utf8")).features.dictation.stt,
).toEqual({ provider: "local", language: "fr" });
} finally {
rmSync(home, { recursive: true, force: true });
}
},
);
test("loads and resolves legacy fields ignored by each site's provider", () => {
const home = mkdtempSync(path.join(tmpdir(), "speech-unused-options-"));
try {
const legacy = {
features: {
dictation: { stt: { provider: "local", language: "fr", confidenceThreshold: -1 } },
voiceMode: {
enabled: true,
tts: { provider: "local", voice: "alloy", speakerId: 3, speed: 1.2 },
},
},
};
writeFileSync(path.join(home, "config.json"), JSON.stringify(legacy));
const local = loadPersistedConfig(home);
expect(local.features?.dictation?.stt).toEqual({ provider: "local", language: "fr" });
expect(local.features?.voiceMode?.tts).toEqual({
provider: "local",
speakerId: 3,
speed: 1.2,
});
expect(
resolveSpeechConfig({ paseoHome: home, env: {}, persisted: local }).speech.local?.models,
).toMatchObject({ voiceTtsSpeakerId: 3, voiceTtsSpeed: 1.2 });
writeFileSync(
path.join(home, "config.json"),
JSON.stringify({
features: {
voiceMode: {
enabled: true,
tts: { provider: "openai", apiKey: "key", voice: "alloy", speakerId: 3, speed: 1.2 },
},
},
}),
);
const openai = loadPersistedConfig(home);
expect(openai.features?.voiceMode?.tts).toEqual({
provider: "openai",
apiKey: "key",
voice: "alloy",
});
expect(
resolveSpeechConfig({ paseoHome: home, env: {}, persisted: openai }).openai?.tts,
).toMatchObject({ apiKey: "key", voice: "alloy" });
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test("warns with the path, never the discarded legacy key", () => {
const home = mkdtempSync(path.join(tmpdir(), "speech-warning-"));
try {
writeFileSync(
path.join(home, "config.json"),
JSON.stringify({
providers: { openai: { apiKey: "discarded-secret" } },
features: { voiceMode: { enabled: true, tts: { provider: "local" } } },
}),
);
const warnings: unknown[][] = [];
const logger = {
child: () => logger,
info: () => {},
warn: (...args: unknown[]) => {
warnings.push(args);
},
};
const config = loadPersistedConfig(home, logger);
expect(config.providers).not.toHaveProperty("openai");
expect(warnings).toHaveLength(1);
expect(JSON.stringify(warnings)).toContain("providers.openai.apiKey");
expect(JSON.stringify(warnings)).toContain("no use site in config.json selects OpenAI");
const resolved = resolveSpeechConfig({
paseoHome: home,
persisted: config,
env: { PASEO_VOICE_TTS_PROVIDER: "openai" },
});
expect(resolved.speech.providers.voiceTts.provider).toBe("openai");
expect(resolved.openai).toBeUndefined();
expect(JSON.stringify(warnings)).not.toContain("discarded-secret");
} finally {
rmSync(home, { recursive: true, force: true });
}
});
test("folds legacy endpoint credentials into OpenAI sites on load and saves inline", () => {
const home = mkdtempSync(path.join(tmpdir(), "speech-normalize-"));
try {
writeFileSync(
path.join(home, "config.json"),
JSON.stringify({
providers: {
openai: {
apiKey: "global-key",
baseUrl: "https://global/v1",
stt: { apiKey: "stt-key" },
},
},
features: {
dictation: { stt: { model: "whisper-base", language: "en" } },
voiceMode: {
enabled: true,
stt: { provider: "OPENAI", baseUrl: "https://site/v1" },
tts: { provider: "openai" },
},
},
}),
);
const config = loadPersistedConfig(home);
expect(config.features?.dictation?.stt).toEqual({
provider: "local",
model: "whisper-base",
language: "en",
});
expect(config.features?.voiceMode?.stt).toEqual({
provider: "openai",
apiKey: "stt-key",
baseUrl: "https://site/v1",
});
expect(config.features?.voiceMode?.tts).toEqual({
provider: "openai",
apiKey: "global-key",
baseUrl: "https://global/v1",
});
expect(config.providers).not.toHaveProperty("openai");
const resolved = resolveSpeechConfig({ paseoHome: home, env: {}, persisted: config });
expect(resolved.openai?.stt?.apiKey).toBe("stt-key");
expect(resolved.speech.providers.voiceStt).toMatchObject({
provider: "openai",
enabled: true,
});
savePersistedConfig(home, config);
expect(JSON.parse(readFileSync(path.join(home, "config.json"), "utf8"))).toEqual(config);
} finally {
rmSync(home, { recursive: true, force: true });
}
});
});
packages/server/src/server/persisted-config.ts
+142 -82
View File
@@ -49,22 +49,6 @@ const LogConfigSchema = z
})
.strict();
const OpenAiSpeechEndpointSchema = z
.object({
apiKey: z.string().trim().min(1).optional(),
baseUrl: z.string().trim().min(1).optional(),
})
.strict();
const OpenAiProviderSchema = z
.object({
apiKey: z.string().min(1).optional(),
baseUrl: z.string().trim().min(1).optional(),
stt: OpenAiSpeechEndpointSchema.optional(),
tts: OpenAiSpeechEndpointSchema.optional(),
})
.strict();
const LocalSpeechProviderSchema = z
.object({
modelsDir: z.string().min(1).optional(),
@@ -73,7 +57,6 @@ const LocalSpeechProviderSchema = z
const ProvidersSchema = z
.object({
openai: OpenAiProviderSchema.optional(),
local: LocalSpeechProviderSchema.optional(),
})
.strict();
@@ -101,18 +84,54 @@ const SpeechProviderIdSchema = z
.toLowerCase()
.pipe(z.enum(["openai", "local"]));
const SpeechModelSchema = z.string().min(1).optional();
const OpenAiConnectionFields = {
apiKey: z.string().trim().min(1).optional(),
baseUrl: z.string().trim().min(1).optional(),
};
export const SpeechSttConfigSchema = z.discriminatedUnion("provider", [
z
.object({
provider: z.literal("local"),
model: SpeechModelSchema,
language: z.string().trim().min(1).optional(),
})
.strict(),
z
.object({
provider: z.literal("openai"),
...OpenAiConnectionFields,
model: SpeechModelSchema,
language: z.string().trim().min(1).optional(),
confidenceThreshold: z.number().optional(),
})
.strict(),
]);
export const SpeechTtsConfigSchema = z.discriminatedUnion("provider", [
z
.object({
provider: z.literal("local"),
model: SpeechModelSchema,
speakerId: z.number().int().optional(),
speed: z.number().optional(),
})
.strict(),
z
.object({
provider: z.literal("openai"),
...OpenAiConnectionFields,
model: SpeechModelSchema,
voice: z.string().trim().min(1).optional(),
})
.strict(),
]);
const FeatureDictationSchema = z
.object({
enabled: z.boolean().optional(),
stt: z
.object({
provider: SpeechProviderIdSchema.optional(),
model: z.string().min(1).optional(),
language: z.string().trim().min(1).optional(),
confidenceThreshold: z.number().optional(),
})
.strict()
.optional(),
stt: SpeechSttConfigSchema.optional(),
})
.strict();
@@ -126,31 +145,14 @@ const FeatureVoiceModeSchema = z
})
.strict()
.optional(),
stt: z
.object({
provider: SpeechProviderIdSchema.optional(),
model: z.string().min(1).optional(),
language: z.string().trim().min(1).optional(),
})
.strict()
.optional(),
stt: SpeechSttConfigSchema.optional(),
turnDetection: z
.object({
provider: SpeechProviderIdSchema.optional(),
})
.strict()
.optional(),
tts: z
.object({
provider: SpeechProviderIdSchema.optional(),
model: z.string().min(1).optional(),
// Free-form: self-hosted OpenAI-compatible servers expose their own voice names.
voice: z.string().trim().min(1).optional(),
speakerId: z.number().int().optional(),
speed: z.number().optional(),
})
.strict()
.optional(),
tts: SpeechTtsConfigSchema.optional(),
})
.strict();
@@ -387,6 +389,7 @@ const DEFAULT_PERSISTED_CONFIG = PersistedConfigSchema.parse({
interface LoggerLike {
child(bindings: Record<string, unknown>): LoggerLike;
info(...args: unknown[]): void;
warn?(...args: unknown[]): void;
}
function getConfigPath(paseoHome: string): string {
@@ -397,45 +400,96 @@ function getLogger(logger: LoggerLike | undefined): LoggerLike | undefined {
return logger?.child({ module: "config" });
}
// Removed config fields are stripped before parsing so the strict schema does not
// reject a config written by an older release. The stripped values are discarded,
// not migrated — there is no back-compat for the removed `providers.openai.voice`
// block (use `providers.openai.stt` / `providers.openai.tts`).
function stripRemovedConfigFields(parsed: unknown): unknown {
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
return parsed;
function isConfigRecord(value: unknown): value is Record<string, unknown> {
return value !== null && typeof value === "object" && !Array.isArray(value);
}
function configRecord(value: unknown): Record<string, unknown> | undefined {
return isConfigRecord(value) ? value : undefined;
}
// COMPAT(speechProviderConfig): added in v0.12.0, remove after 2027-04-26.
// Fold shared credentials once at the file boundary; consumers see only the strict
// per-site shape. Never include credential values in diagnostics.
function normalizeSpeechProviderConfig(parsed: unknown, logger?: LoggerLike): unknown {
if (!isConfigRecord(parsed)) return parsed;
const root = structuredClone(parsed);
const providers = configRecord(root.providers);
const legacy = configRecord(providers?.openai);
const features = configRecord(root.features);
const foldedKeys = new Set<string>();
for (const [feature, kind] of [
["dictation", "stt"],
["voiceMode", "stt"],
["voiceMode", "tts"],
] as const) {
const site = configRecord(configRecord(features?.[feature])?.[kind]);
if (site) normalizeSpeechSite(site, kind, legacy, foldedKeys);
}
const root = { ...(parsed as Record<string, unknown>) };
const providers = root.providers;
if (!providers || typeof providers !== "object" || Array.isArray(providers)) {
return root;
if (legacy) warnDiscardedSpeechKeys(legacy, foldedKeys, logger);
if (providers) {
delete providers.openai;
const local = configRecord(providers.local);
if (local) delete local.autoDownload;
}
const providersRecord = { ...(providers as Record<string, unknown>) };
const local = providersRecord.local;
if (local && typeof local === "object" && !Array.isArray(local)) {
const localRecord = { ...(local as Record<string, unknown>) };
delete localRecord.autoDownload;
providersRecord.local = localRecord;
}
const openai = providersRecord.openai;
if (openai && typeof openai === "object" && !Array.isArray(openai)) {
const openaiRecord = { ...(openai as Record<string, unknown>) };
// COMPAT(openaiVoiceConfig): added 2026-06-30, remove after 2026-12-30.
// Drop a `providers.openai.voice` block left by an older release so the strict
// schema doesn't reject it. The value is discarded, not migrated — there is no
// back-compat; configure `providers.openai.stt` / `providers.openai.tts` instead.
delete openaiRecord.voice;
providersRecord.openai = openaiRecord;
}
root.providers = providersRecord;
return root;
}
function normalizeSpeechSite(
site: Record<string, unknown>,
kind: "stt" | "tts",
legacy: Record<string, unknown> | undefined,
foldedKeys: Set<string>,
) {
site.provider =
typeof site.provider === "string"
? site.provider.trim().toLowerCase()
: (site.provider ?? "local");
if (kind === "stt" && site.provider === "local") {
delete site.confidenceThreshold;
}
if (kind === "tts") {
if (site.provider === "local") delete site.voice;
if (site.provider === "openai") {
delete site.speakerId;
delete site.speed;
}
}
if (site.provider !== "openai" || !legacy) return;
const endpoint = configRecord(legacy[kind]);
for (const field of ["apiKey", "baseUrl"] as const) {
if (site[field] !== undefined) continue;
if (endpoint?.[field] !== undefined) {
site[field] = endpoint[field];
foldedKeys.add(`providers.openai.${kind}.${field}`);
} else if (legacy[field] !== undefined) {
site[field] = legacy[field];
foldedKeys.add(`providers.openai.${field}`);
}
}
}
function warnDiscardedSpeechKeys(
legacy: Record<string, unknown>,
foldedKeys: Set<string>,
logger?: LoggerLike,
) {
const keys = [
["providers.openai.apiKey", legacy.apiKey],
...["stt", "tts"].map((kind) => [
`providers.openai.${kind}.apiKey`,
configRecord(legacy[kind])?.apiKey,
]),
] as const;
for (const [keyPath, key] of keys) {
if (key && typeof keyPath === "string" && !foldedKeys.has(keyPath)) {
const message = `Dropped ${keyPath}: no use site in config.json selects OpenAI; re-enter it when selecting OpenAI.`;
if (logger?.warn) logger.warn(message);
else console.warn(message);
}
}
}
export function loadPersistedConfig(paseoHome: string, logger?: LoggerLike): PersistedConfig {
const log = getLogger(logger);
const configPath = getConfigPath(paseoHome);
@@ -464,7 +518,7 @@ export function loadPersistedConfig(paseoHome: string, logger?: LoggerLike): Per
});
}
const config = parseConfigFile(configPath, raw);
const config = parseConfigFile(configPath, raw, log);
log?.info(`Loaded from ${configPath}`);
return config;
}
@@ -486,7 +540,7 @@ export function readPersistedConfig(
return parseConfigFile(configPath, raw);
}
function parseConfigFile(configPath: string, raw: string): PersistedConfig {
function parseConfigFile(configPath: string, raw: string, logger?: LoggerLike): PersistedConfig {
let parsed: unknown;
try {
parsed = parseConfigText(raw);
@@ -497,7 +551,7 @@ function parseConfigFile(configPath: string, raw: string): PersistedConfig {
});
}
const result = PersistedConfigSchema.safeParse(stripRemovedConfigFields(parsed));
const result = PersistedConfigSchema.safeParse(normalizeSpeechProviderConfig(parsed, logger));
if (!result.success) {
const issues = result.error.issues
.map((i) => ` - ${i.path.join(".")}: ${i.message}`)
@@ -525,6 +579,12 @@ function configPathParts(field: string): string[] {
schema instanceof z.ZodPipe
)
schema = (schema instanceof z.ZodPipe ? schema.in : schema.unwrap()) as z.ZodType;
if (schema instanceof z.ZodUnion) {
const variant = schema.options.find(
(option) => option instanceof z.ZodObject && Object.hasOwn(option.shape, part),
);
if (variant instanceof z.ZodObject) schema = variant;
}
if (!(schema instanceof z.ZodObject) || !Object.hasOwn(schema.shape, part)) {
throw new Error(
`Unknown configuration path: ${field}. Replace its containing object with JSON for dynamic keys.`,
packages/server/src/server/session.ts
+20 -2
View File
@@ -182,6 +182,10 @@ import {
import { ScheduleSession } from "./session/schedule/schedule-session.js";
import { ProviderCatalogSession } from "./session/provider/provider-catalog-session.js";
import { UsageSession } from "./session/usage/usage-session.js";
import {
SpeechModelsSession,
type SpeechModelsSessionOptions,
} from "./session/speech/speech-models-session.js";
import { WorkspaceFilesSession } from "./session/files/workspace-files-session.js";
import { AgentConfigSession } from "./session/agent-config/agent-config-session.js";
import { ProjectConfigSession } from "./session/project-config/project-config-session.js";
@@ -528,6 +532,7 @@ export interface SessionOptions {
stt: Resolvable<SpeechToTextProvider | null>;
sttLanguage?: string;
tts: Resolvable<TextToSpeechProvider | null>;
speechModels?: SpeechModelsSessionOptions["runtime"];
terminalManager: TerminalManager | null;
providerSnapshotManager: ProviderSnapshotManager;
hubExecutionAgents?: HubExecutionAgents;
@@ -811,6 +816,7 @@ export class Session {
private readonly scheduleSession: ScheduleSession;
private readonly providerCatalogSession: ProviderCatalogSession;
private readonly usageSession: UsageSession;
private readonly speechModelsSession: SpeechModelsSession;
private readonly workspaceFilesSession: WorkspaceFilesSession;
private readonly agentConfigSession: AgentConfigSession;
private readonly projectConfigSession: ProjectConfigSession;
@@ -861,6 +867,7 @@ export class Session {
stt,
sttLanguage,
tts,
speechModels,
terminalManager,
providerSnapshotManager,
serviceProxy,
@@ -1037,6 +1044,10 @@ export class Session {
runtime: pluginRuntime,
logger: this.sessionLogger,
});
this.speechModelsSession = new SpeechModelsSession({
emit: (msg) => this.emit(msg),
runtime: speechModels,
});
this.agentConfigSession = new AgentConfigSession({
host: {
emit: (msg) => this.emit(msg),
@@ -2421,8 +2432,7 @@ export class Session {
this.dispatchOrchestrationSkillsMessage(msg) ??
this.dispatchPluginDirectoryMessage(msg) ??
this.dispatchPluginMessage(msg) ??
this.dispatchTerminalMessage(msg) ??
this.dispatchScheduleMessage(msg) ??
this.dispatchHostServiceMessage(msg) ??
this.dispatchMiscMessage(msg);
if (promise) await promise;
}
@@ -2438,6 +2448,14 @@ export class Session {
);
}
private dispatchHostServiceMessage(msg: SessionInboundMessage): Promise<void> | undefined {
return (
this.dispatchTerminalMessage(msg) ??
this.dispatchScheduleMessage(msg) ??
this.speechModelsSession.dispatch(msg)
);
}
private dispatchWorkspaceLifecycleMessage(msg: SessionInboundMessage): Promise<void> | undefined {
return (
this.dispatchWorkspaceStateMessage(msg) ??
packages/server/src/server/session/speech/speech-models-session.test.ts
+106
View File
@@ -0,0 +1,106 @@
import { describe, expect, test } from "vitest";
import type { SessionOutboundMessage } from "../../messages.js";
import {
listLocalSpeechModels,
type LocalSpeechModelId,
} from "../../speech/providers/local/models.js";
import {
LocalSpeechModelInUseError,
type LocalSpeechModelStatus,
} from "../../speech/speech-runtime.js";
import { SpeechModelsSession } from "./speech-models-session.js";
const [firstModel] = listLocalSpeechModels();
function createSession(models: LocalSpeechModelStatus[], inUse: LocalSpeechModelId[] = []) {
const emitted: SessionOutboundMessage[] = [];
const downloads: LocalSpeechModelId[] = [];
const session = new SpeechModelsSession({
emit: (message) => emitted.push(message),
runtime: {
listLocalModels: async () => models,
downloadLocalModel: (modelId) => {
downloads.push(modelId);
},
deleteLocalModel: async (modelId) => {
if (inUse.includes(modelId)) throw new LocalSpeechModelInUseError(modelId);
},
},
});
return { session, emitted, downloads };
}
describe("SpeechModelsSession", () => {
test("lists catalog details and download progress before anything is installed", async () => {
const { session, emitted } = createSession([
{
spec: firstModel,
state: { status: "downloading", receivedBytes: 10, totalBytes: 40 },
inUse: true,
},
]);
await session.dispatch({ type: "speech.local_models.list.request", requestId: "r1" });
expect(emitted).toEqual([
{
type: "speech.local_models.list.response",
payload: {
requestId: "r1",
error: null,
models: [
{
id: firstModel.id,
kind: "stt",
label: firstModel.label,
description: firstModel.description,
languages: [...firstModel.languages],
downloadSizeBytes: firstModel.downloadSizeBytes,
isDefault: true,
inUse: true,
status: "downloading",
receivedBytes: 10,
totalBytes: 40,
},
],
},
},
]);
});
test("rejects a model id outside the catalog without starting a download", () => {
const { session, emitted, downloads } = createSession([]);
void session.dispatch({
type: "speech.local_models.download.request",
requestId: "r2",
modelId: "../../etc",
});
expect(downloads).toEqual([]);
expect(emitted).toEqual([
{
type: "speech.local_models.download.response",
payload: { requestId: "r2", error: "Unknown local speech model: ../../etc" },
},
]);
});
test("reports why an in-use model cannot be deleted", async () => {
const { session, emitted } = createSession([], [firstModel.id]);
await session.dispatch({
type: "speech.local_models.delete.request",
requestId: "r3",
modelId: firstModel.id,
});
expect(emitted).toEqual([
{
type: "speech.local_models.delete.response",
payload: { requestId: "r3", error: new LocalSpeechModelInUseError(firstModel.id).message },
},
]);
});
});
packages/server/src/server/session/speech/speech-models-session.ts
+124
View File
@@ -0,0 +1,124 @@
import type { LocalSpeechModel } from "@getpaseo/protocol/speech/rpc-schemas";
import type { SessionInboundMessage, SessionOutboundMessage } from "../../messages.js";
import { LocalSpeechModelIdSchema } from "../../speech/providers/local/models.js";
import type { LocalSpeechModelStatus, SpeechService } from "../../speech/speech-runtime.js";
type SpeechModelsRuntime = Pick<
SpeechService,
"listLocalModels" | "downloadLocalModel" | "deleteLocalModel"
>;
export interface SpeechModelsSessionOptions {
emit(message: SessionOutboundMessage): void;
// Absent when the daemon runs without speech.
runtime?: SpeechModelsRuntime | null;
}
export class SpeechModelsSession {
constructor(private readonly options: SpeechModelsSessionOptions) {}
// Handles the speech.local_models.* requests; returns undefined for any other message.
dispatch(msg: SessionInboundMessage): Promise<void> | undefined {
switch (msg.type) {
case "speech.local_models.list.request":
return this.handleList(msg);
case "speech.local_models.download.request":
this.handleDownload(msg);
return Promise.resolve();
case "speech.local_models.delete.request":
return this.handleDelete(msg);
default:
return undefined;
}
}
private async handleList(
msg: Extract<SessionInboundMessage, { type: "speech.local_models.list.request" }>,
): Promise<void> {
try {
const models = await this.requireRuntime().listLocalModels();
this.options.emit({
type: "speech.local_models.list.response",
payload: { requestId: msg.requestId, models: models.map(toWireModel), error: null },
});
} catch (error) {
this.options.emit({
type: "speech.local_models.list.response",
payload: { requestId: msg.requestId, models: [], error: describeError(error) },
});
}
}
private handleDownload(
msg: Extract<SessionInboundMessage, { type: "speech.local_models.download.request" }>,
): void {
try {
this.requireRuntime().downloadLocalModel(parseModelId(msg.modelId));
this.options.emit({
type: "speech.local_models.download.response",
payload: { requestId: msg.requestId, error: null },
});
} catch (error) {
this.options.emit({
type: "speech.local_models.download.response",
payload: { requestId: msg.requestId, error: describeError(error) },
});
}
}
private async handleDelete(
msg: Extract<SessionInboundMessage, { type: "speech.local_models.delete.request" }>,
): Promise<void> {
try {
await this.requireRuntime().deleteLocalModel(parseModelId(msg.modelId));
this.options.emit({
type: "speech.local_models.delete.response",
payload: { requestId: msg.requestId, error: null },
});
} catch (error) {
this.options.emit({
type: "speech.local_models.delete.response",
payload: { requestId: msg.requestId, error: describeError(error) },
});
}
}
private requireRuntime(): SpeechModelsRuntime {
if (!this.options.runtime) {
throw new Error("Speech is unavailable on this daemon");
}
return this.options.runtime;
}
}
function parseModelId(modelId: string) {
const parsed = LocalSpeechModelIdSchema.safeParse(modelId);
if (!parsed.success) {
throw new Error(`Unknown local speech model: ${modelId}`);
}
return parsed.data;
}
function toWireModel(model: LocalSpeechModelStatus): LocalSpeechModel {
const { spec, state, inUse } = model;
return {
id: spec.id,
kind: spec.kind === "tts" ? "tts" : "stt",
label: spec.label,
description: spec.description,
languages: [...spec.languages],
downloadSizeBytes: spec.downloadSizeBytes,
isDefault: spec.defaultFor !== undefined,
inUse,
status: state.status,
...(state.status === "downloading"
? { receivedBytes: state.receivedBytes, totalBytes: state.totalBytes }
: {}),
...(state.status === "failed" ? { error: state.error } : {}),
};
}
function describeError(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
packages/server/src/server/speech/providers/local/config.ts
+12 -10
View File
@@ -29,6 +29,7 @@ export interface LocalSpeechProviderConfig {
export interface ResolvedLocalSpeechConfig {
local: LocalSpeechProviderConfig | undefined;
modelsDir: string;
sttLanguages: LocalSpeechSttLanguageConfig;
}
@@ -68,12 +69,12 @@ const LocalSpeechResolutionSchema = z.object({
function persistedLocalFeatureModel(
provider: RequestedSpeechProviders[keyof RequestedSpeechProviders]["provider"],
enabled: boolean | undefined,
model: string | undefined,
site: { provider: string; model?: string } | undefined,
): string | undefined {
if (provider !== "local" || enabled === false) {
return undefined;
}
return model;
return site?.provider === "local" ? site.model : undefined;
}
function shouldIncludeLocalProviderConfig(params: {
@@ -143,6 +144,8 @@ function buildLocalSpeechResolutionInput(params: {
includeProviderConfig: boolean;
}): Record<string, unknown> {
const { paseoHome, env, persisted, providers, includeProviderConfig } = params;
const tts = persisted.features?.voiceMode?.tts;
const localTts = tts?.provider === "local" ? tts : undefined;
return {
includeProviderConfig,
modelsDir: firstDefinedValue<string>([
@@ -155,7 +158,7 @@ function buildLocalSpeechResolutionInput(params: {
persistedLocalFeatureModel(
providers.dictationStt.provider,
providers.dictationStt.enabled,
persisted.features?.dictation?.stt?.model,
persisted.features?.dictation?.stt,
),
DEFAULT_LOCAL_STT_MODEL,
]),
@@ -164,7 +167,7 @@ function buildLocalSpeechResolutionInput(params: {
persistedLocalFeatureModel(
providers.voiceStt.provider,
providers.voiceStt.enabled,
persisted.features?.voiceMode?.stt?.model,
persisted.features?.voiceMode?.stt,
),
DEFAULT_LOCAL_STT_MODEL,
]),
@@ -173,18 +176,18 @@ function buildLocalSpeechResolutionInput(params: {
persistedLocalFeatureModel(
providers.voiceTts.provider,
providers.voiceTts.enabled,
persisted.features?.voiceMode?.tts?.model,
persisted.features?.voiceMode?.tts,
),
DEFAULT_LOCAL_TTS_MODEL,
]),
...buildLocalSpeechLanguageResolutionInput({ env, persisted }),
voiceLocalTtsSpeakerId: firstDefinedValue<string | number>([
env.PASEO_VOICE_LOCAL_TTS_SPEAKER_ID,
persisted.features?.voiceMode?.tts?.speakerId,
localTts?.speakerId,
]),
voiceLocalTtsSpeed: firstDefinedValue<string | number>([
env.PASEO_VOICE_LOCAL_TTS_SPEED,
persisted.features?.voiceMode?.tts?.speed,
localTts?.speed,
]),
};
}
@@ -200,11 +203,10 @@ export function resolveLocalSpeechConfig(params: {
buildLocalSpeechResolutionInput({ ...params, includeProviderConfig }),
);
const resolvedVoiceTtsSpeakerId =
parsed.voiceLocalTtsSpeakerId ??
(parsed.voiceLocalTtsModel === "kokoro-en-v0_19" ? 0 : undefined);
const resolvedVoiceTtsSpeakerId = parsed.voiceLocalTtsSpeakerId ?? 0;
return {
modelsDir: parsed.modelsDir,
sttLanguages: {
dictation: parsed.dictationLanguage,
voice: parsed.voiceLanguage,
packages/server/src/server/speech/providers/local/model-library.ts
+137
View File
@@ -0,0 +1,137 @@
import type { Logger } from "pino";
import {
deleteSherpaOnnxModel,
ensureSherpaOnnxModel,
isSherpaOnnxModelInstalled,
} from "./sherpa/model-downloader.js";
import {
listLocalSpeechModels,
type LocalSpeechModelId,
type LocalSpeechModelSpec,
} from "./models.js";
export type LocalSpeechModelInstallState =
| { status: "installed" }
| { status: "not_installed" }
| { status: "downloading"; receivedBytes: number; totalBytes: number }
| { status: "extracting" }
| { status: "failed"; error: string };
export interface LocalSpeechModelEntry {
spec: LocalSpeechModelSpec;
state: LocalSpeechModelInstallState;
}
type InFlightState = Extract<
LocalSpeechModelInstallState,
{ status: "downloading" | "extracting" }
>;
interface InFlightDownload {
promise: Promise<string>;
state: InFlightState;
}
export class LocalSpeechModelBusyError extends Error {
constructor(public readonly modelId: LocalSpeechModelId) {
super(`${modelId} is downloading. Wait for the download to finish before deleting it.`);
this.name = "LocalSpeechModelBusyError";
}
}
// Owns the models directory: which catalog models are installed, which are downloading,
// and the last failure per model. Concurrent downloads of one model share a single transfer.
export class LocalSpeechModelLibrary {
private readonly inFlight = new Map<LocalSpeechModelId, InFlightDownload>();
private readonly failures = new Map<LocalSpeechModelId, string>();
private readonly abortController = new AbortController();
private readonly logger: Logger;
constructor(
private readonly modelsDir: string,
logger: Logger,
) {
this.logger = logger.child({ module: "speech", component: "local-model-library" });
}
getModelsDir(): string {
return this.modelsDir;
}
async list(): Promise<LocalSpeechModelEntry[]> {
return Promise.all(
listLocalSpeechModels().map(async (spec) => ({ spec, state: await this.getState(spec.id) })),
);
}
async listMissing(modelIds: LocalSpeechModelId[]): Promise<LocalSpeechModelId[]> {
const installed = await Promise.all(
modelIds.map((modelId) => isSherpaOnnxModelInstalled(this.modelsDir, modelId)),
);
return modelIds.filter((_modelId, index) => !installed[index]);
}
download(modelId: LocalSpeechModelId): Promise<string> {
const existing = this.inFlight.get(modelId);
if (existing) {
return existing.promise;
}
this.failures.delete(modelId);
const entry: InFlightDownload = {
state: { status: "downloading", receivedBytes: 0, totalBytes: 0 },
promise: Promise.resolve(""),
};
entry.promise = ensureSherpaOnnxModel({
modelsDir: this.modelsDir,
modelId,
logger: this.logger,
signal: this.abortController.signal,
onProgress: (progress) => {
entry.state =
progress.phase === "downloading"
? {
status: "downloading",
receivedBytes: progress.receivedBytes,
totalBytes: progress.totalBytes,
}
: { status: "extracting" };
},
})
.catch((error: unknown) => {
this.failures.set(modelId, error instanceof Error ? error.message : String(error));
throw error;
})
.finally(() => {
this.inFlight.delete(modelId);
});
this.inFlight.set(modelId, entry);
return entry.promise;
}
async delete(modelId: LocalSpeechModelId): Promise<void> {
if (this.inFlight.has(modelId)) {
throw new LocalSpeechModelBusyError(modelId);
}
this.failures.delete(modelId);
await deleteSherpaOnnxModel(this.modelsDir, modelId);
this.logger.info({ modelId }, "Deleted local speech model");
}
async stop(): Promise<void> {
this.abortController.abort();
await Promise.allSettled(Array.from(this.inFlight.values(), (entry) => entry.promise));
}
private async getState(modelId: LocalSpeechModelId): Promise<LocalSpeechModelInstallState> {
const inFlight = this.inFlight.get(modelId);
if (inFlight) {
return inFlight.state;
}
if (await isSherpaOnnxModelInstalled(this.modelsDir, modelId)) {
return { status: "installed" };
}
const failure = this.failures.get(modelId);
return failure ? { status: "failed", error: failure } : { status: "not_installed" };
}
}
packages/server/src/server/speech/providers/local/models.test.ts
+22 -11
View File
@@ -13,12 +13,8 @@ import path from "node:path";
import pino from "pino";
import { describe, expect, onTestFinished, test } from "vitest";
import {
ensureLocalSpeechModels,
getLocalSpeechModelDir,
listLocalSpeechModels,
listMissingLocalSpeechModels,
} from "./models.js";
import { LocalSpeechModelLibrary } from "./model-library.js";
import { getLocalSpeechModelDir, listLocalSpeechModels } from "./models.js";
const logger = pino({ level: "silent" });
const modelId = "parakeet-tdt-0.6b-v2-int8";
@@ -55,16 +51,31 @@ describe("local speech models", () => {
const encoder = path.join(modelDir, spec.requiredFiles[0]);
truncateSync(encoder, 1_000);
expect(await listMissingLocalSpeechModels({ modelsDir, modelIds: [modelId] })).toEqual([
modelId,
]);
const library = new LocalSpeechModelLibrary(modelsDir, logger);
expect(await library.listMissing([modelId])).toEqual([modelId]);
await ensureLocalSpeechModels({ modelsDir, modelIds: [modelId], logger });
await library.download(modelId);
expect(readFileSync(encoder, "utf8")).toBe(
`complete ${spec.requiredFiles[0]}`.padEnd(64_000, "."),
);
expect(existsSync(archive)).toBe(false);
expect(await listMissingLocalSpeechModels({ modelsDir, modelIds: [modelId] })).toEqual([]);
expect(await library.listMissing([modelId])).toEqual([]);
});
test("lists install state and deletes an installed model", async () => {
const { modelsDir } = setupModelsDirWithArchive();
const library = new LocalSpeechModelLibrary(modelsDir, logger);
const stateOf = async () =>
(await library.list()).find((entry) => entry.spec.id === modelId)?.state;
expect(await stateOf()).toEqual({ status: "not_installed" });
await library.download(modelId);
expect(await stateOf()).toEqual({ status: "installed" });
await library.delete(modelId);
expect(await stateOf()).toEqual({ status: "not_installed" });
expect(existsSync(getLocalSpeechModelDir(modelsDir, modelId))).toBe(false);
});
});
packages/server/src/server/speech/providers/local/models.ts
+3 -15
View File
@@ -1,11 +1,8 @@
import {
ensureSherpaOnnxModels,
getSherpaOnnxModelDir,
isSherpaOnnxModelInstalled,
} from "./sherpa/model-downloader.js";
import { ensureSherpaOnnxModels, getSherpaOnnxModelDir } from "./sherpa/model-downloader.js";
import {
DEFAULT_LOCAL_STT_MODEL,
DEFAULT_LOCAL_TTS_MODEL,
LocalSpeechModelIdSchema,
LocalSttModelIdSchema,
LocalTtsModelIdSchema,
listSherpaOnnxModels,
@@ -17,6 +14,7 @@ import {
export {
DEFAULT_LOCAL_STT_MODEL,
DEFAULT_LOCAL_TTS_MODEL,
LocalSpeechModelIdSchema,
LocalSttModelIdSchema,
LocalTtsModelIdSchema,
type LocalSpeechModelId,
@@ -34,16 +32,6 @@ export function getLocalSpeechModelDir(modelsDir: string, modelId: LocalSpeechMo
return getSherpaOnnxModelDir(modelsDir, modelId);
}
export async function listMissingLocalSpeechModels(options: {
modelsDir: string;
modelIds: LocalSpeechModelId[];
}): Promise<LocalSpeechModelId[]> {
const installed = await Promise.all(
options.modelIds.map((modelId) => isSherpaOnnxModelInstalled(options.modelsDir, modelId)),
);
return options.modelIds.filter((_modelId, index) => !installed[index]);
}
export async function ensureLocalSpeechModels(options: {
modelsDir: string;
modelIds: LocalSpeechModelId[];
packages/server/src/server/speech/providers/local/sherpa/model-catalog.ts
+466 -22
View File
@@ -4,43 +4,467 @@ export type SherpaOnnxModelKind = "stt-offline" | "tts";
type DefaultModelRole = "stt" | "tts";
interface SherpaOnnxCatalogEntry {
kind: SherpaOnnxModelKind;
// How sherpa-onnx loads a model, with file names relative to the extracted directory.
export type SttEngine =
| { family: "transducer"; encoder: string; decoder: string; joiner: string; tokens: string }
| { family: "whisper"; encoder: string; decoder: string; tokens: string }
| { family: "sense_voice"; model: string; tokens: string }
| { family: "dolphin"; model: string; tokens: string }
| { family: "nemo_ctc"; model: string; tokens: string };
export type TtsEngine =
| {
family: "kokoro";
model: string;
voices: string;
tokens: string;
dataDir: string;
// Multi-language Kokoro models read pronunciations from these instead of espeak alone.
lexicons?: readonly string[];
dictDir?: string;
ruleFsts?: readonly string[];
}
| { family: "kitten"; model: string; voices: string; tokens: string; dataDir: string }
| { family: "vits"; model: string; tokens: string; dataDir: string };
interface SherpaOnnxCatalogEntryBase {
label: string;
// One user-facing sentence shown before download.
description: string;
// BCP 47 language tags the model transcribes or speaks.
languages: readonly string[];
// Exact size of the release archive. The downloader rejects any other size.
downloadSizeBytes: number;
archiveUrl: string;
extractedDir: string;
requiredFiles: string[];
description: string;
defaultFor?: DefaultModelRole;
}
type SherpaOnnxCatalogEntry =
| (SherpaOnnxCatalogEntryBase & { kind: "stt-offline"; engine: SttEngine })
| (SherpaOnnxCatalogEntryBase & { kind: "tts"; engine: TtsEngine });
// Every sherpa-onnx release archive extracts to a directory named after the archive.
function releaseArchive(tag: "asr-models" | "tts-models", name: string) {
return {
archiveUrl: `https://github.com/k2-fsa/sherpa-onnx/releases/download/${tag}/${name}.tar.bz2`,
extractedDir: name,
};
}
const PARAKEET_INT8_ENGINE = {
family: "transducer",
encoder: "encoder.int8.onnx",
decoder: "decoder.int8.onnx",
joiner: "joiner.int8.onnx",
tokens: "tokens.txt",
} as const satisfies SttEngine;
function whisperEngine(size: string): SttEngine {
return {
family: "whisper",
encoder: `${size}-encoder.int8.onnx`,
decoder: `${size}-decoder.int8.onnx`,
tokens: `${size}-tokens.txt`,
};
}
function piperEngine(voice: string): TtsEngine {
return {
family: "vits",
model: `${voice}.onnx`,
tokens: "tokens.txt",
dataDir: "espeak-ng-data",
};
}
// Parakeet TDT v3 transcribes these 25 European languages and detects which one is spoken.
const PARAKEET_V3_LANGUAGES = [
"bg",
"hr",
"cs",
"da",
"nl",
"en",
"et",
"fi",
"fr",
"de",
"el",
"hu",
"it",
"lv",
"lt",
"mt",
"pl",
"pt",
"ro",
"sk",
"sl",
"es",
"sv",
"ru",
"uk",
] as const;
// The 99 languages multilingual Whisper models transcribe, detecting which one is spoken.
const WHISPER_LANGUAGES = [
"en",
"zh",
"de",
"es",
"ru",
"ko",
"fr",
"ja",
"pt",
"tr",
"pl",
"ca",
"nl",
"ar",
"sv",
"it",
"id",
"hi",
"fi",
"vi",
"he",
"uk",
"el",
"ms",
"cs",
"ro",
"da",
"hu",
"ta",
"no",
"th",
"ur",
"hr",
"bg",
"lt",
"la",
"mi",
"ml",
"cy",
"sk",
"te",
"fa",
"lv",
"bn",
"sr",
"az",
"sl",
"kn",
"et",
"mk",
"br",
"eu",
"is",
"hy",
"ne",
"mn",
"bs",
"kk",
"sq",
"sw",
"gl",
"mr",
"pa",
"si",
"km",
"sn",
"yo",
"so",
"af",
"oc",
"ka",
"be",
"tg",
"sd",
"gu",
"am",
"yi",
"lo",
"uz",
"fo",
"ht",
"ps",
"tk",
"nn",
"mt",
"sa",
"lb",
"my",
"bo",
"tl",
"mg",
"as",
"tt",
"haw",
"ln",
"ha",
"ba",
"jv",
"su",
] as const;
// Dolphin's 40 Eastern languages. It also covers 22 Chinese dialects under "zh".
const DOLPHIN_LANGUAGES = [
"zh",
"ja",
"th",
"ru",
"ko",
"id",
"vi",
"yue",
"hi",
"ur",
"ms",
"uz",
"ar",
"fa",
"bn",
"ta",
"te",
"ug",
"gu",
"my",
"tl",
"kk",
"or",
"ne",
"mn",
"km",
"jv",
"lo",
"si",
"fil",
"ps",
"pa",
"kab",
"ba",
"ks",
"tg",
"su",
"mr",
"ky",
"az",
] as const;
export const SHERPA_ONNX_MODEL_CATALOG = {
"parakeet-tdt-0.6b-v2-int8": {
kind: "stt-offline",
archiveUrl:
"https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8.tar.bz2",
extractedDir: "sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8",
requiredFiles: ["encoder.int8.onnx", "decoder.int8.onnx", "joiner.int8.onnx", "tokens.txt"],
description: "NVIDIA Parakeet TDT v2 (offline NeMo transducer, English).",
label: "Parakeet v2",
description: "Accurate English transcription that runs well on a laptop CPU.",
languages: ["en"],
downloadSizeBytes: 482_468_385,
...releaseArchive("asr-models", "sherpa-onnx-nemo-parakeet-tdt-0.6b-v2-int8"),
engine: PARAKEET_INT8_ENGINE,
defaultFor: "stt",
},
"parakeet-tdt-110m-en-int8": {
kind: "stt-offline",
label: "Parakeet Small",
description: "Small, fast English model with punctuation. A quick download for any laptop.",
languages: ["en"],
downloadSizeBytes: 108_035_095,
...releaseArchive("asr-models", "sherpa-onnx-nemo-parakeet_tdt_transducer_110m-en-36000-int8"),
engine: PARAKEET_INT8_ENGINE,
},
"parakeet-tdt-0.6b-v3-int8": {
kind: "stt-offline",
archiveUrl:
"https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8.tar.bz2",
extractedDir: "sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8",
requiredFiles: ["encoder.int8.onnx", "decoder.int8.onnx", "joiner.int8.onnx", "tokens.txt"],
description:
"NVIDIA Parakeet TDT v3 (offline NeMo transducer, 25 European languages, auto-detected).",
label: "Parakeet v3",
description: "Parakeet for 25 European languages, detecting which one you speak.",
languages: PARAKEET_V3_LANGUAGES,
downloadSizeBytes: 487_170_055,
...releaseArchive("asr-models", "sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8"),
engine: PARAKEET_INT8_ENGINE,
},
"whisper-tiny-en": {
kind: "stt-offline",
label: "Whisper Tiny (English)",
description: "Small English Whisper model. Quick to download, less accurate than Parakeet.",
languages: ["en"],
downloadSizeBytes: 118_071_777,
...releaseArchive("asr-models", "sherpa-onnx-whisper-tiny.en"),
engine: whisperEngine("tiny.en"),
},
"whisper-base": {
kind: "stt-offline",
label: "Whisper Base",
description: "Compact Whisper model for 99 languages, detecting which one you speak.",
languages: WHISPER_LANGUAGES,
downloadSizeBytes: 207_557_382,
...releaseArchive("asr-models", "sherpa-onnx-whisper-base"),
engine: whisperEngine("base"),
},
"whisper-turbo": {
kind: "stt-offline",
label: "Whisper Turbo",
description: "Whisper's most accurate model for 99 languages and Cantonese. Slower on a CPU.",
languages: [...WHISPER_LANGUAGES, "yue"],
downloadSizeBytes: 563_790_207,
...releaseArchive("asr-models", "sherpa-onnx-whisper-turbo"),
engine: whisperEngine("turbo"),
},
"sense-voice-int8": {
kind: "stt-offline",
label: "SenseVoice",
description: "Chinese, Cantonese, English, Japanese and Korean, detecting which one you speak.",
languages: ["zh", "yue", "en", "ja", "ko"],
downloadSizeBytes: 165_783_878,
...releaseArchive("asr-models", "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2025-09-09"),
engine: { family: "sense_voice", model: "model.int8.onnx", tokens: "tokens.txt" },
},
"dolphin-base-int8": {
kind: "stt-offline",
label: "Dolphin",
description: "Small model for 40 Asian and Middle Eastern languages and Chinese dialects.",
languages: DOLPHIN_LANGUAGES,
downloadSizeBytes: 80_671_385,
...releaseArchive("asr-models", "sherpa-onnx-dolphin-base-ctc-multi-lang-int8-2025-04-02"),
engine: { family: "dolphin", model: "model.int8.onnx", tokens: "tokens.txt" },
},
"giga-am-v3-ru": {
kind: "stt-offline",
label: "GigaAM v3 (Russian)",
description: "Russian transcription with punctuation.",
languages: ["ru"],
downloadSizeBytes: 170_197_019,
...releaseArchive(
"asr-models",
"sherpa-onnx-nemo-transducer-punct-giga-am-v3-russian-2025-12-16",
),
engine: {
family: "transducer",
encoder: "encoder.int8.onnx",
decoder: "decoder.onnx",
joiner: "joiner.onnx",
tokens: "tokens.txt",
},
},
"parakeet-tdt-ctc-0.6b-ja-int8": {
kind: "stt-offline",
label: "Parakeet (Japanese)",
description: "Japanese transcription with punctuation.",
languages: ["ja"],
downloadSizeBytes: 489_389_564,
...releaseArchive("asr-models", "sherpa-onnx-nemo-parakeet-tdt_ctc-0.6b-ja-35000-int8"),
engine: { family: "nemo_ctc", model: "model.int8.onnx", tokens: "tokens.txt" },
},
"kokoro-en-v0_19": {
kind: "tts",
archiveUrl:
"https://github.com/k2-fsa/sherpa-onnx/releases/download/tts-models/kokoro-en-v0_19.tar.bz2",
extractedDir: "kokoro-en-v0_19",
requiredFiles: ["model.onnx", "voices.bin", "tokens.txt", "espeak-ng-data"],
description: "Kokoro TTS (higher quality; larger).",
label: "Kokoro (English)",
description: "Natural-sounding English voice for voice mode replies.",
languages: ["en"],
downloadSizeBytes: 319_625_534,
...releaseArchive("tts-models", "kokoro-en-v0_19"),
engine: {
family: "kokoro",
model: "model.onnx",
voices: "voices.bin",
tokens: "tokens.txt",
dataDir: "espeak-ng-data",
},
defaultFor: "tts",
},
"kokoro-int8-en-v0_19": {
kind: "tts",
label: "Kokoro Lite (English)",
description: "The same Kokoro English voices, quantized to a third of the download.",
languages: ["en"],
downloadSizeBytes: 103_248_205,
...releaseArchive("tts-models", "kokoro-int8-en-v0_19"),
engine: {
family: "kokoro",
model: "model.int8.onnx",
voices: "voices.bin",
tokens: "tokens.txt",
dataDir: "espeak-ng-data",
},
},
"kokoro-int8-multi-lang-v1_1": {
kind: "tts",
label: "Kokoro (Chinese and English)",
description: "Kokoro voices that speak both Mandarin Chinese and English.",
languages: ["zh", "en"],
downloadSizeBytes: 147_031_220,
...releaseArchive("tts-models", "kokoro-int8-multi-lang-v1_1"),
engine: {
family: "kokoro",
model: "model.int8.onnx",
voices: "voices.bin",
tokens: "tokens.txt",
dataDir: "espeak-ng-data",
lexicons: ["lexicon-us-en.txt", "lexicon-zh.txt"],
dictDir: "dict",
ruleFsts: ["date-zh.fst", "phone-zh.fst", "number-zh.fst"],
},
},
// Kitten v0.8 needs a newer sherpa-onnx than we pin; v0.2 loads on 1.12.28.
"kitten-nano-en-v0_2-fp16": {
kind: "tts",
label: "Kitten Nano (English)",
description: "Tiny, very fast English voice.",
languages: ["en"],
downloadSizeBytes: 26_586_708,
...releaseArchive("tts-models", "kitten-nano-en-v0_2-fp16"),
engine: {
family: "kitten",
model: "model.fp16.onnx",
voices: "voices.bin",
tokens: "tokens.txt",
dataDir: "espeak-ng-data",
},
},
"piper-en-us-libritts-r": {
kind: "tts",
label: "Piper LibriTTS (US English)",
description: "Lightweight American English voice.",
languages: ["en-US"],
downloadSizeBytes: 23_398_348,
...releaseArchive("tts-models", "vits-piper-en_US-libritts_r-medium-int8"),
engine: piperEngine("en_US-libritts_r-medium"),
},
"piper-en-gb-alba": {
kind: "tts",
label: "Piper Alba (British English)",
description: "Lightweight British English voice.",
languages: ["en-GB"],
downloadSizeBytes: 21_104_326,
...releaseArchive("tts-models", "vits-piper-en_GB-alba-medium-int8"),
engine: piperEngine("en_GB-alba-medium"),
},
"piper-de-thorsten": {
kind: "tts",
label: "Piper Thorsten (German)",
description: "Lightweight German voice.",
languages: ["de"],
downloadSizeBytes: 20_949_833,
...releaseArchive("tts-models", "vits-piper-de_DE-thorsten-medium-int8"),
engine: piperEngine("de_DE-thorsten-medium"),
},
"piper-fr-siwis": {
kind: "tts",
label: "Piper Siwis (French)",
description: "Lightweight French voice.",
languages: ["fr"],
downloadSizeBytes: 20_914_888,
...releaseArchive("tts-models", "vits-piper-fr_FR-siwis-medium-int8"),
engine: piperEngine("fr_FR-siwis-medium"),
},
"piper-es-davefx": {
kind: "tts",
label: "Piper Davefx (Spanish)",
description: "Lightweight Spanish voice.",
languages: ["es"],
downloadSizeBytes: 21_171_632,
...releaseArchive("tts-models", "vits-piper-es_ES-davefx-medium-int8"),
engine: piperEngine("es_ES-davefx-medium"),
},
} as const satisfies Record<string, SherpaOnnxCatalogEntry>;
export type SherpaOnnxModelId = keyof typeof SHERPA_ONNX_MODEL_CATALOG;
@@ -97,24 +521,44 @@ function createModelIdSchema<T extends string>(modelIds: readonly T[]): z.ZodTyp
.transform((value) => value as T);
}
export const LocalSpeechModelIdSchema = createModelIdSchema(ALL_MODEL_IDS);
export const LocalSttModelIdSchema = createModelIdSchema(LOCAL_STT_MODEL_IDS);
export const LocalTtsModelIdSchema = createModelIdSchema(LOCAL_TTS_MODEL_IDS);
export type SherpaOnnxModelSpec = SherpaOnnxCatalogEntry & {
id: SherpaOnnxModelId;
// Every file and directory that must exist for the model to count as installed.
requiredFiles: string[];
};
export function listSherpaOnnxModels(): SherpaOnnxModelSpec[] {
return ALL_MODEL_IDS.map((id) => Object.assign({ id }, SHERPA_ONNX_MODEL_CATALOG[id]));
return ALL_MODEL_IDS.map(getSherpaOnnxModelSpec);
}
export function getSherpaOnnxModelSpec(id: SherpaOnnxModelId): SherpaOnnxModelSpec {
const spec = SHERPA_ONNX_MODEL_CATALOG[id];
const spec: SherpaOnnxCatalogEntry | undefined = SHERPA_ONNX_MODEL_CATALOG[id];
if (!spec) {
throw new Error(`Unknown local speech model id: ${id}`);
}
return {
id,
...spec,
requiredFiles: engineFiles(spec.engine),
};
}
// Every path an engine names, so the downloader can verify an install.
export function engineFiles(engine: SttEngine | TtsEngine): string[] {
return Object.entries(engine).flatMap(([key, value]) => {
if (key === "family") return [];
return typeof value === "string" ? [value] : (value as readonly string[]);
});
}
export function getLocalSttEngine(id: LocalSttModelId): SttEngine {
return SHERPA_ONNX_MODEL_CATALOG[id].engine;
}
export function getLocalTtsEngine(id: LocalTtsModelId): TtsEngine {
return SHERPA_ONNX_MODEL_CATALOG[id].engine;
}
packages/server/src/server/speech/providers/local/sherpa/model-downloader.ts
+28
View File
@@ -8,11 +8,16 @@ import type pino from "pino";
import { getSherpaOnnxModelSpec, type SherpaOnnxModelId } from "./model-catalog.js";
import { spawnProcess } from "../../../../../utils/spawn.js";
export type SherpaOnnxModelInstallProgress =
| { phase: "downloading"; receivedBytes: number; totalBytes: number }
| { phase: "extracting" };
export interface EnsureSherpaOnnxModelOptions {
modelsDir: string;
modelId: SherpaOnnxModelId;
logger: pino.Logger;
signal?: AbortSignal;
onProgress?: (progress: SherpaOnnxModelInstallProgress) => void;
}
export function getSherpaOnnxModelDir(modelsDir: string, modelId: SherpaOnnxModelId): string {
@@ -41,7 +46,9 @@ async function hasRequiredFiles(modelDir: string, requiredFiles: string[]): Prom
interface DownloadToFileOptions {
url: string;
outputPath: string;
expectedBytes: number;
signal?: AbortSignal;
onProgress?: (receivedBytes: number, totalBytes: number) => void;
}
async function downloadToFile(options: DownloadToFileOptions): Promise<void> {
@@ -60,6 +67,13 @@ async function downloadToFile(options: DownloadToFileOptions): Promise<void> {
// The fetch ReadableStream type is slightly different from what Readable.fromWeb expects
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const nodeStream = Readable.fromWeb(res.body as any);
const contentLength = Number(res.headers.get("content-length"));
const totalBytes = contentLength > 0 ? contentLength : options.expectedBytes;
let receivedBytes = 0;
nodeStream.on("data", (chunk: Buffer) => {
receivedBytes += chunk.length;
options.onProgress?.(receivedBytes, totalBytes);
});
try {
await pipeline(nodeStream, createWriteStream(tmpPath), { signal: options.signal });
@@ -157,9 +171,13 @@ export async function ensureSherpaOnnxModel(
await downloadToFile({
url: spec.archiveUrl,
outputPath: archivePath,
expectedBytes: spec.downloadSizeBytes,
signal: options.signal,
onProgress: (receivedBytes, totalBytes) =>
options.onProgress?.({ phase: "downloading", receivedBytes, totalBytes }),
});
}
options.onProgress?.({ phase: "extracting" });
logger.info(
{
@@ -213,6 +231,16 @@ export async function ensureSherpaOnnxModel(
}
}
export async function deleteSherpaOnnxModel(
modelsDir: string,
modelId: SherpaOnnxModelId,
): Promise<void> {
const { modelDir, archivePath, extractionDir } = getSherpaOnnxModelPaths(modelsDir, modelId);
await rm(modelDir, { recursive: true, force: true });
await rm(archivePath, { force: true });
await rm(extractionDir, { recursive: true, force: true });
}
export async function ensureSherpaOnnxModels(options: {
modelsDir: string;
modelIds: SherpaOnnxModelId[];
packages/server/src/server/speech/providers/local/sherpa/sherpa-offline-recognizer.ts
+41 -18
View File
@@ -1,6 +1,8 @@
import { existsSync } from "node:fs";
import path from "node:path";
import type pino from "pino";
import { engineFiles, type SttEngine } from "./model-catalog.js";
import { loadSherpaOnnxNode } from "./sherpa-onnx-node-loader.js";
function assertFileExists(filePath: string, label: string): void {
@@ -9,16 +11,43 @@ function assertFileExists(filePath: string, label: string): void {
}
}
export interface SherpaOfflineRecognizerModel {
kind: "nemo_transducer";
encoder: string;
decoder: string;
joiner: string;
tokens: string;
// Each engine family takes its own block under sherpa's modelConfig.
function buildModelConfig(modelDir: string, engine: SttEngine): Record<string, unknown> {
const file = (name: string) => path.join(modelDir, name);
switch (engine.family) {
case "transducer":
return {
transducer: {
encoder: file(engine.encoder),
decoder: file(engine.decoder),
joiner: file(engine.joiner),
},
modelType: "nemo_transducer",
};
case "whisper":
// An empty language makes Whisper detect the spoken language.
return {
whisper: {
encoder: file(engine.encoder),
decoder: file(engine.decoder),
language: "",
task: "transcribe",
},
};
case "sense_voice":
return {
senseVoice: { model: file(engine.model), language: "", useInverseTextNormalization: 1 },
};
case "dolphin":
return { dolphin: { model: file(engine.model) } };
case "nemo_ctc":
return { nemoCtc: { model: file(engine.model) } };
}
}
export interface SherpaOfflineRecognizerConfig {
model: SherpaOfflineRecognizerModel;
modelDir: string;
engine: SttEngine;
numThreads?: number;
provider?: "cpu";
debug?: 0 | 1;
@@ -54,10 +83,9 @@ export class SherpaOfflineRecognizerEngine {
component: "offline-recognizer",
});
assertFileExists(config.model.encoder, "offline encoder");
assertFileExists(config.model.decoder, "offline decoder");
assertFileExists(config.model.joiner, "offline joiner");
assertFileExists(config.model.tokens, "tokens");
for (const name of engineFiles(config.engine)) {
assertFileExists(path.join(config.modelDir, name), `${config.engine.family} model file`);
}
const sherpa = loadSherpaOnnxNode();
@@ -67,13 +95,8 @@ export class SherpaOfflineRecognizerEngine {
featureDim: config.featureDim ?? 80,
},
modelConfig: {
transducer: {
encoder: config.model.encoder,
decoder: config.model.decoder,
joiner: config.model.joiner,
},
tokens: config.model.tokens,
modelType: "nemo_transducer",
...buildModelConfig(config.modelDir, config.engine),
tokens: path.join(config.modelDir, config.engine.tokens),
numThreads: config.numThreads ?? 1,
provider: config.provider ?? "cpu",
debug: config.debug ?? 0,
packages/server/src/server/speech/providers/local/sherpa/sherpa-tts.test.ts
+44 -5
View File
@@ -3,6 +3,7 @@ import { mkdtempSync, mkdirSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import pino from "pino";
import { engineFiles, getLocalTtsEngine, type LocalTtsModelId } from "./model-catalog.js";
import { SherpaOnnxTTS } from "./sherpa-tts.js";
const directories: string[] = [];
@@ -11,17 +12,28 @@ afterEach(() => {
rmSync(directory, { recursive: true, force: true });
});
function createTts(numThreads?: number, beforeGenerate?: () => Promise<void>) {
function createTts(
numThreads?: number,
beforeGenerate?: () => Promise<void>,
modelId: LocalTtsModelId = "kokoro-en-v0_19",
) {
const modelDir = mkdtempSync(join(tmpdir(), "paseo-tts-"));
directories.push(modelDir);
for (const name of ["model.onnx", "voices.bin", "tokens.txt"])
writeFileSync(join(modelDir, name), "");
mkdirSync(join(modelDir, "espeak-ng-data"));
// Data directories have no extension; everything else is a model file.
for (const name of engineFiles(getLocalTtsEngine(modelId))) {
if (name.includes(".")) writeFileSync(join(modelDir, name), "");
else mkdirSync(join(modelDir, name));
}
let nativeConfig: unknown;
const requests: unknown[] = [];
const released: boolean[] = [];
const tts = new SherpaOnnxTTS(
{ preset: "kokoro-en-v0_19", modelDir, numThreads },
{
modelId,
modelDir,
engine: getLocalTtsEngine(modelId),
numThreads,
},
pino({ level: "silent" }),
() => ({
OfflineTts: class {
@@ -138,4 +150,31 @@ describe("SherpaOnnxTTS", () => {
expect(nativeConfig).not.toHaveProperty("provider");
},
);
it("loads Piper voices through the vits engine", () => {
const { nativeConfig } = createTts(undefined, undefined, "piper-de-thorsten");
expect(nativeConfig).toMatchObject({
model: {
vits: {
model: expect.stringMatching(/de_DE-thorsten-medium\.onnx$/),
tokens: expect.stringMatching(/tokens\.txt$/),
dataDir: expect.stringMatching(/espeak-ng-data$/),
},
},
});
expect(nativeConfig).not.toHaveProperty("model.kokoro");
});
it("gives multi-language Kokoro its lexicons, dictionary and number rules", () => {
const { nativeConfig } = createTts(undefined, undefined, "kokoro-int8-multi-lang-v1_1");
expect(nativeConfig).toMatchObject({
model: {
kokoro: {
lexicon: expect.stringMatching(/lexicon-us-en\.txt,.*lexicon-zh\.txt$/),
dictDir: expect.stringMatching(/dict$/),
},
},
ruleFsts: expect.stringMatching(/date-zh\.fst,.*phone-zh\.fst,.*number-zh\.fst$/),
});
});
});
packages/server/src/server/speech/providers/local/sherpa/sherpa-tts.ts
+57 -20
View File
@@ -1,16 +1,17 @@
import type pino from "pino";
import { Readable } from "node:stream";
import { existsSync } from "node:fs";
import path from "node:path";
import type { SpeechStreamResult, TextToSpeechProvider } from "../../../speech-provider.js";
import { chunkBuffer, float32ToPcm16le } from "../../../audio.js";
import { loadSherpaOnnxNode } from "./sherpa-onnx-node-loader.js";
export type SherpaTtsPreset = "kokoro-en-v0_19";
import { engineFiles, type LocalTtsModelId, type TtsEngine } from "./model-catalog.js";
export interface SherpaTtsConfig {
preset: SherpaTtsPreset;
modelId: LocalTtsModelId;
modelDir: string;
engine: TtsEngine;
speakerId?: number;
speed?: number;
lengthScale?: number;
@@ -23,6 +24,53 @@ function assertFileExists(filePath: string, label: string): void {
}
}
// Each engine family takes its own block under sherpa's TTS model config.
function buildTtsModelConfig(
modelDir: string,
engine: TtsEngine,
lengthScale: number,
): Record<string, unknown> {
const file = (name: string) => path.join(modelDir, name);
switch (engine.family) {
case "kokoro":
return {
kokoro: {
model: file(engine.model),
voices: file(engine.voices),
tokens: file(engine.tokens),
dataDir: file(engine.dataDir),
lexicon: (engine.lexicons ?? []).map(file).join(","),
dictDir: engine.dictDir ? file(engine.dictDir) : "",
lengthScale,
},
};
case "kitten":
return {
kitten: {
model: file(engine.model),
voices: file(engine.voices),
tokens: file(engine.tokens),
dataDir: file(engine.dataDir),
lengthScale,
},
};
case "vits":
return {
vits: {
model: file(engine.model),
tokens: file(engine.tokens),
dataDir: file(engine.dataDir),
lengthScale,
},
};
}
}
function ruleFsts(modelDir: string, engine: TtsEngine): string {
if (engine.family !== "kokoro" || !engine.ruleFsts) return "";
return engine.ruleFsts.map((name) => path.join(modelDir, name)).join(",");
}
interface SherpaOfflineTtsNative {
sampleRate?: number;
generateAsync: (args: {
@@ -59,39 +107,28 @@ export class SherpaOnnxTTS implements TextToSpeechProvider {
throw new Error("sherpa-onnx-node OfflineTts is unavailable");
}
const modelPath = `${config.modelDir}/model.onnx`;
const voicesPath = `${config.modelDir}/voices.bin`;
const tokensPath = `${config.modelDir}/tokens.txt`;
const dataDir = `${config.modelDir}/espeak-ng-data`;
assertFileExists(modelPath, "TTS model");
assertFileExists(voicesPath, "TTS voices");
assertFileExists(tokensPath, "TTS tokens");
assertFileExists(dataDir, "TTS espeak-ng dataDir");
for (const name of engineFiles(config.engine)) {
assertFileExists(path.join(config.modelDir, name), `${config.engine.family} TTS file`);
}
const modelConfig = {
// The native parser reads these under model, despite the upstream JS typedef.
numThreads: config.numThreads ?? 2,
provider: "cpu",
kokoro: {
model: modelPath,
voices: voicesPath,
tokens: tokensPath,
dataDir,
lengthScale: config.lengthScale ?? 1.0,
},
...buildTtsModelConfig(config.modelDir, config.engine, config.lengthScale ?? 1.0),
};
const offlineTtsConfig = {
model: modelConfig,
maxNumSentences: 1,
ruleFsts: ruleFsts(config.modelDir, config.engine),
};
this.tts = new (
sherpa as unknown as { OfflineTts: new (config: unknown) => SherpaOfflineTtsNative }
).OfflineTts(offlineTtsConfig);
this.logger.info(
{ preset: config.preset, modelDir: config.modelDir },
{ modelId: config.modelId, modelDir: config.modelDir },
"Sherpa offline TTS initialized",
);
}
packages/server/src/server/speech/providers/local/sherpa/speech-download.e2e.test.ts
+9 -8
View File
@@ -7,6 +7,7 @@ import pino from "pino";
import { ensureSherpaOnnxModels, getSherpaOnnxModelDir } from "./model-downloader.js";
import { createDaemonTestContext } from "../../../../test-utils/index.js";
import { parsePcm16MonoWav, wordSimilarity } from "../../../../test-utils/dictation-e2e.js";
import { getLocalSttEngine, getLocalTtsEngine } from "./model-catalog.js";
import { SherpaOnnxTTS } from "./sherpa-tts.js";
import { SherpaOfflineRecognizerEngine } from "./sherpa-offline-recognizer.js";
import { SherpaOnnxParakeetSTT } from "./sherpa-parakeet-stt.js";
@@ -192,7 +193,12 @@ downloadTest(
const ttsText = "This is a voice note.";
const ttsModelDir = getSherpaOnnxModelDir(modelsDir, "kokoro-en-v0_19");
const tts = new SherpaOnnxTTS(
{ preset: "kokoro-en-v0_19", modelDir: ttsModelDir, speakerId: 0 },
{
modelId: "kokoro-en-v0_19",
modelDir: ttsModelDir,
engine: getLocalTtsEngine("kokoro-en-v0_19"),
speakerId: 0,
},
logger,
);
const { stream, format: ttsFormat } = await tts.synthesizeSpeech(ttsText);
@@ -206,13 +212,8 @@ downloadTest(
const sttModelDir = getSherpaOnnxModelDir(modelsDir, "parakeet-tdt-0.6b-v2-int8");
const engine = new SherpaOfflineRecognizerEngine(
{
model: {
kind: "nemo_transducer",
encoder: `${sttModelDir}/encoder.int8.onnx`,
decoder: `${sttModelDir}/decoder.int8.onnx`,
joiner: `${sttModelDir}/joiner.int8.onnx`,
tokens: `${sttModelDir}/tokens.txt`,
},
modelDir: sttModelDir,
engine: getLocalSttEngine("parakeet-tdt-0.6b-v2-int8"),
numThreads: 2,
debug: 0,
},
packages/server/src/server/speech/providers/local/worker-process.ts
+5 -8
View File
@@ -3,6 +3,7 @@ import pino from "pino";
import type { StreamingTranscriptionSession } from "../../speech-provider.js";
import type { TurnDetectionSession } from "../../turn-detection-provider.js";
import { getLocalSpeechModelDir, type LocalSttModelId, type LocalTtsModelId } from "./models.js";
import { getLocalSttEngine, getLocalTtsEngine } from "./sherpa/model-catalog.js";
import { SherpaOfflineRecognizerEngine } from "./sherpa/sherpa-offline-recognizer.js";
import { SherpaOnnxParakeetSTT } from "./sherpa/sherpa-parakeet-stt.js";
import { SherpaParakeetRealtimeTranscriptionSession } from "./sherpa/sherpa-parakeet-realtime-session.js";
@@ -85,13 +86,8 @@ function getSttEngine(
const modelDir = getLocalSpeechModelDir(config.modelsDir, modelId);
const created = new SherpaOfflineRecognizerEngine(
{
model: {
kind: "nemo_transducer",
encoder: `${modelDir}/encoder.int8.onnx`,
decoder: `${modelDir}/decoder.int8.onnx`,
joiner: `${modelDir}/joiner.int8.onnx`,
tokens: `${modelDir}/tokens.txt`,
},
modelDir,
engine: getLocalSttEngine(modelId),
numThreads: 2,
debug: 0,
},
@@ -125,8 +121,9 @@ function getTtsProvider(config: LocalSpeechWorkerConfig): SherpaOnnxTTS {
const modelDir = getLocalSpeechModelDir(config.modelsDir, ttsModelId(config));
const created = new SherpaOnnxTTS(
{
preset: ttsModelId(config),
modelId: ttsModelId(config),
modelDir,
engine: getLocalTtsEngine(ttsModelId(config)),
speakerId: config.voiceTtsSpeakerId,
speed: config.voiceTtsSpeed,
},
packages/server/src/server/speech/providers/openai/config.test.ts
+115 -210
View File
@@ -1,5 +1,4 @@
import { describe, expect, test } from "vitest";
import { PersistedConfigSchema } from "../../../persisted-config.js";
import { resolveOpenAiSpeechConfig } from "./config.js";
import type { RequestedSpeechProviders } from "../../speech-types.js";
@@ -10,233 +9,139 @@ const ALL_OPENAI: RequestedSpeechProviders = {
voiceStt: { provider: "openai", explicit: true },
voiceTts: { provider: "openai", explicit: true },
};
function resolve(input: unknown, env: NodeJS.ProcessEnv = {}) {
return resolveOpenAiSpeechConfig({
persisted: PersistedConfigSchema.parse(input),
env,
providers: ALL_OPENAI,
});
}
describe("resolveOpenAiSpeechConfig", () => {
test("treats empty OPENAI_API_KEY as unset", () => {
const persisted = PersistedConfigSchema.parse({});
const env = {
OPENAI_API_KEY: "",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({
env,
persisted,
providers: {
...ALL_OPENAI,
dictationStt: { provider: "local", explicit: false },
voiceStt: { provider: "local", explicit: false },
voiceTts: { provider: "local", explicit: false },
test("treats empty credentials as unset", () => {
expect(resolve({}, { OPENAI_API_KEY: "", OPENAI_STT_API_KEY: " " })).toBeUndefined();
});
test("uses endpoint env before global env and ignores empty fallbacks", () => {
const result = resolve(
{},
{
OPENAI_API_KEY: " global ",
OPENAI_BASE_URL: " https://global/v1 ",
OPENAI_STT_API_KEY: " stt ",
OPENAI_STT_BASE_URL: " https://stt/v1 ",
OPENAI_TTS_API_KEY: " ",
OPENAI_TTS_BASE_URL: "",
},
);
expect(result?.dictationStt).toEqual({ apiKey: "stt", baseUrl: "https://stt/v1" });
expect(result?.stt).toEqual({ apiKey: "stt", baseUrl: "https://stt/v1" });
expect(result?.tts).toEqual({
apiKey: "global",
baseUrl: "https://global/v1",
model: "tts-1",
voice: "alloy",
responseFormat: "pcm",
});
expect(resolved).toBeUndefined();
});
test("applies trimmed OPENAI_API_KEY to both STT and TTS", () => {
const persisted = PersistedConfigSchema.parse({});
const env = {
OPENAI_API_KEY: " sk-test ",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("sk-test");
expect(resolved?.tts?.apiKey).toBe("sk-test");
});
test("resolves distinct endpoints for STT and TTS", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: {
stt: {
apiKey: "stt-key",
baseUrl: " https://stt.example.com/v1 ",
test("each site wins over endpoint and global env credentials and URL", () => {
const result = resolve(
{
features: {
dictation: {
stt: {
provider: "openai",
apiKey: "dictation",
baseUrl: "https://dictation/v1",
model: "dictation-model",
confidenceThreshold: -1,
},
},
tts: {
apiKey: "tts-key",
baseUrl: " https://tts.example.com/v1 ",
voiceMode: {
stt: {
provider: "openai",
apiKey: "voice",
baseUrl: "https://voice/v1",
model: "voice-model",
confidenceThreshold: -2,
},
tts: {
provider: "openai",
apiKey: "tts",
baseUrl: "https://tts/v1",
model: "kokoro-v1:Q8_0",
voice: "Aiden",
},
},
},
},
{
OPENAI_API_KEY: "global",
OPENAI_BASE_URL: "https://global/v1",
OPENAI_STT_API_KEY: "endpoint",
OPENAI_STT_BASE_URL: "https://endpoint/v1",
OPENAI_TTS_API_KEY: "endpoint",
OPENAI_TTS_BASE_URL: "https://endpoint/v1",
},
);
expect(result?.dictationStt).toEqual({
apiKey: "dictation",
baseUrl: "https://dictation/v1",
model: "dictation-model",
confidenceThreshold: -1,
});
const resolved = resolveOpenAiSpeechConfig({
env: {} as NodeJS.ProcessEnv,
persisted,
providers: ALL_OPENAI,
expect(result?.stt).toEqual({
apiKey: "voice",
baseUrl: "https://voice/v1",
model: "voice-model",
confidenceThreshold: -2,
});
expect(result?.tts).toEqual({
apiKey: "tts",
baseUrl: "https://tts/v1",
model: "kokoro-v1:Q8_0",
voice: "Aiden",
responseFormat: "pcm",
});
expect(resolved?.stt?.apiKey).toBe("stt-key");
expect(resolved?.stt?.baseUrl).toBe("https://stt.example.com/v1");
expect(resolved?.tts?.apiKey).toBe("tts-key");
expect(resolved?.tts?.baseUrl).toBe("https://tts.example.com/v1");
});
test("prefers nested STT/TTS config over env and global fallbacks", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: {
apiKey: "fallback-config-key",
baseUrl: "https://global-config.example.com/v1",
stt: { apiKey: "stt-config-key", baseUrl: " https://stt.example.com/v1 " },
tts: { apiKey: "tts-config-key", baseUrl: " https://tts.example.com/v1 " },
test("env overrides model, voice and threshold at every OpenAI site", () => {
const result = resolve(
{
features: {
dictation: { stt: { provider: "openai", model: "dictation", confidenceThreshold: -1 } },
voiceMode: {
stt: { provider: "openai", model: "voice", confidenceThreshold: -2 },
tts: { provider: "openai", model: "configured", voice: "Aiden" },
},
},
},
});
const env = {
OPENAI_API_KEY: "env-key",
OPENAI_STT_API_KEY: "stt-env-key",
OPENAI_STT_BASE_URL: "https://stt-env.example.com/v1",
OPENAI_TTS_API_KEY: "tts-env-key",
OPENAI_TTS_BASE_URL: "https://tts-env.example.com/v1",
OPENAI_BASE_URL: "https://env.example.com/v1",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("stt-config-key");
expect(resolved?.stt?.baseUrl).toBe("https://stt.example.com/v1");
expect(resolved?.tts?.apiKey).toBe("tts-config-key");
expect(resolved?.tts?.baseUrl).toBe("https://tts.example.com/v1");
});
test("uses STT/TTS env config when nested config is unset", () => {
const persisted = PersistedConfigSchema.parse({});
const env = {
OPENAI_API_KEY: "sk-test",
OPENAI_STT_API_KEY: "stt-env-key",
OPENAI_STT_BASE_URL: " https://stt-env.example.com/v1 ",
OPENAI_TTS_API_KEY: "tts-env-key",
OPENAI_TTS_BASE_URL: " https://tts-env.example.com/v1 ",
OPENAI_BASE_URL: "https://env.example.com/v1",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("stt-env-key");
expect(resolved?.stt?.baseUrl).toBe("https://stt-env.example.com/v1");
expect(resolved?.tts?.apiKey).toBe("tts-env-key");
expect(resolved?.tts?.baseUrl).toBe("https://tts-env.example.com/v1");
});
test("falls back to global OpenAI config for both STT and TTS", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: {
apiKey: "fallback-config-key",
baseUrl: " https://global-config.example.com/v1 ",
},
{
OPENAI_API_KEY: "key",
STT_MODEL: "override",
TTS_MODEL: "tts-override",
TTS_VOICE: "Nova",
STT_CONFIDENCE_THRESHOLD: "-3",
},
);
expect(result?.dictationStt).toEqual({
apiKey: "key",
model: "override",
confidenceThreshold: -3,
});
const env = {} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("fallback-config-key");
expect(resolved?.stt?.baseUrl).toBe("https://global-config.example.com/v1");
expect(resolved?.tts?.apiKey).toBe("fallback-config-key");
expect(resolved?.tts?.baseUrl).toBe("https://global-config.example.com/v1");
expect(result?.stt).toEqual({ apiKey: "key", model: "override", confidenceThreshold: -3 });
expect(result?.tts).toMatchObject({ model: "tts-override", voice: "Nova" });
});
test("falls back to global OpenAI env config when feature inputs are unset", () => {
const persisted = PersistedConfigSchema.parse({});
const env = {
OPENAI_API_KEY: "env-key",
OPENAI_BASE_URL: " https://env.example.com/v1 ",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("env-key");
expect(resolved?.stt?.baseUrl).toBe("https://env.example.com/v1");
expect(resolved?.tts?.apiKey).toBe("env-key");
expect(resolved?.tts?.baseUrl).toBe("https://env.example.com/v1");
test("only configures endpoints with credentials and ignores unused TTS options", () => {
const result = resolve(
{ features: { dictation: { stt: { provider: "openai", apiKey: "dictation" } } } },
{ TTS_VOICE: "arbitrary", TTS_MODEL: "arbitrary" },
);
expect(result).toEqual({ dictationStt: { apiKey: "dictation" } });
});
test("ignores empty endpoint env vars and falls back to OPENAI_API_KEY", () => {
const persisted = PersistedConfigSchema.parse({});
const env = {
OPENAI_API_KEY: "global-key",
OPENAI_STT_API_KEY: "",
OPENAI_STT_BASE_URL: " ",
OPENAI_TTS_API_KEY: "",
OPENAI_TTS_BASE_URL: "",
} as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("global-key");
expect(resolved?.tts?.apiKey).toBe("global-key");
});
test("omits TTS when only an STT key is configured", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: {
stt: { apiKey: "stt-only-key" },
},
},
test("does not borrow another site's key", () => {
const result = resolve({
features: { voiceMode: { stt: { provider: "openai", apiKey: "voice-only" } } },
});
const resolved = resolveOpenAiSpeechConfig({
env: {} as NodeJS.ProcessEnv,
persisted,
providers: ALL_OPENAI,
});
expect(resolved?.stt?.apiKey).toBe("stt-only-key");
expect(resolved?.tts).toBeUndefined();
});
test("resolves STT even when an unused TTS env var is invalid", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: {
stt: { apiKey: "stt-only-key" },
},
},
});
const env = { TTS_VOICE: "not-a-real-voice", TTS_MODEL: "bogus-model" } as NodeJS.ProcessEnv;
const resolved = resolveOpenAiSpeechConfig({ env, persisted, providers: ALL_OPENAI });
expect(resolved?.stt?.apiKey).toBe("stt-only-key");
expect(resolved?.tts).toBeUndefined();
});
test("sends the model and voice an OpenAI-compatible server serves, as written", () => {
const persisted = PersistedConfigSchema.parse({
providers: {
openai: { tts: { apiKey: "tts-key", baseUrl: "https://tts.example.com/v1" } },
},
features: {
voiceMode: { tts: { provider: "openai", model: "kokoro-v1:Q8_0", voice: "Aiden" } },
},
});
const resolved = resolveOpenAiSpeechConfig({
env: {} as NodeJS.ProcessEnv,
persisted,
providers: ALL_OPENAI,
});
expect(resolved?.tts?.model).toBe("kokoro-v1:Q8_0");
expect(resolved?.tts?.voice).toBe("Aiden");
});
test("accepts OpenAI's own gpt-4o-mini-tts model", () => {
const persisted = PersistedConfigSchema.parse({
providers: { openai: { tts: { apiKey: "tts-key" } } },
features: { voiceMode: { tts: { provider: "openai", model: "gpt-4o-mini-tts" } } },
});
const resolved = resolveOpenAiSpeechConfig({
env: {} as NodeJS.ProcessEnv,
persisted,
providers: ALL_OPENAI,
});
expect(resolved?.tts?.model).toBe("gpt-4o-mini-tts");
expect(resolved?.tts?.voice).toBe("alloy");
expect(result?.dictationStt).toBeUndefined();
expect(result?.stt).toEqual({ apiKey: "voice-only" });
});
});
packages/server/src/server/speech/providers/openai/config.ts
+50 -168
View File
@@ -6,147 +6,61 @@ import type { STTConfig } from "./stt.js";
import type { TTSConfig } from "./tts.js";
export const DEFAULT_OPENAI_TTS_MODEL = "tts-1";
export type OpenAiSttFeature = "dictation" | "voiceMode";
export interface OpenAiSpeechProviderConfig {
stt?: Partial<STTConfig> & { apiKey?: string };
tts?: Partial<TTSConfig> & { apiKey?: string };
dictationStt?: Partial<STTConfig>;
stt?: Partial<STTConfig>;
tts?: Partial<TTSConfig>;
}
const NumberLikeSchema = z.union([z.number(), z.string().trim().min(1)]);
const OptionalFiniteNumberSchema = z
.union([z.number(), z.string().trim().min(1)])
.pipe(z.coerce.number<string | number>().finite())
.optional();
const OptionalFiniteNumberSchema = NumberLikeSchema.pipe(
z.coerce.number<string | number>().finite(),
).optional();
const OptionalTrimmedStringSchema = z
.string()
.trim()
.optional()
.transform((value) => (value && value.length > 0 ? value : undefined));
// Endpoint credentials only — plain trimmed strings, so this never throws on a
// malformed value. The STT/TTS option groups parse separately and only for the
// endpoint that is actually configured, so a stale env var for an unused endpoint
// (e.g. a leftover TTS_VOICE in an STT-only setup) can't break the other one.
const OpenAiEndpointKeysSchema = z.object({
sttApiKey: OptionalTrimmedStringSchema,
sttBaseUrl: OptionalTrimmedStringSchema,
ttsApiKey: OptionalTrimmedStringSchema,
ttsBaseUrl: OptionalTrimmedStringSchema,
});
const OpenAiSttOptionsSchema = z.object({
sttConfidenceThreshold: OptionalFiniteNumberSchema,
sttModel: OptionalTrimmedStringSchema,
});
// Voice and model are free-form so a self-hosted server can use its own ids; the
// OpenAI catalog is only the default. Both are sent as written, because the
// server owns those ids and their case (rosters use ids like kokoro-v1:Q8_0).
const OpenAiTtsOptionsSchema = z.object({
ttsVoice: z.string().trim().default("alloy"),
ttsModel: z.string().trim().default(DEFAULT_OPENAI_TTS_MODEL),
});
function isOpenAiProviderActive(provider: { enabled?: boolean; provider: string }): boolean {
return provider.enabled !== false && provider.provider === "openai";
function firstString(...values: (string | undefined)[]): string | undefined {
return values.map((value) => value?.trim()).find((value) => Boolean(value));
}
function pickIfOpenAi<T>(
provider: { enabled?: boolean; provider: string },
value: T | undefined,
): T | undefined {
return isOpenAiProviderActive(provider) ? value : undefined;
}
type Features = NonNullable<PersistedConfig["features"]>;
type SttSite = NonNullable<NonNullable<Features["dictation"]>["stt"]>;
type TtsSite = NonNullable<NonNullable<Features["voiceMode"]>["tts"]>;
function firstDefined<T>(values: Array<T | null | undefined>): T | undefined {
for (const value of values) {
if (value === undefined || value === null) {
continue;
}
// Empty/whitespace env vars (e.g. a copied .env.example with OPENAI_STT_API_KEY=)
// must not shadow a later fallback such as OPENAI_API_KEY.
if (typeof value === "string" && value.trim().length === 0) {
continue;
}
return value;
}
return undefined;
}
function buildOpenAiSttInput(params: {
env: NodeJS.ProcessEnv;
persisted: PersistedConfig;
providers: RequestedSpeechProviders;
}): Record<string, unknown> {
const { env, persisted, providers } = params;
function resolveStt(
site: SttSite | undefined,
env: NodeJS.ProcessEnv,
): Partial<STTConfig> | undefined {
const config = site?.provider === "openai" ? site : undefined;
const apiKey = firstString(config?.apiKey, env.OPENAI_STT_API_KEY, env.OPENAI_API_KEY);
if (!apiKey) return undefined;
const baseUrl = firstString(config?.baseUrl, env.OPENAI_STT_BASE_URL, env.OPENAI_BASE_URL);
const model = firstString(env.STT_MODEL, config?.model);
const confidenceThreshold = OptionalFiniteNumberSchema.parse(
firstString(env.STT_CONFIDENCE_THRESHOLD) ?? config?.confidenceThreshold,
);
return {
sttConfidenceThreshold: firstDefined<string | number>([
env.STT_CONFIDENCE_THRESHOLD,
persisted.features?.dictation?.stt?.confidenceThreshold,
]),
sttModel: firstDefined<string>([
env.STT_MODEL,
pickIfOpenAi(providers.voiceStt, persisted.features?.voiceMode?.stt?.model),
pickIfOpenAi(providers.dictationStt, persisted.features?.dictation?.stt?.model),
]),
apiKey,
...(baseUrl ? { baseUrl } : {}),
...(model ? { model } : {}),
...(confidenceThreshold !== undefined ? { confidenceThreshold } : {}),
};
}
function buildOpenAiTtsInput(params: {
env: NodeJS.ProcessEnv;
persisted: PersistedConfig;
providers: RequestedSpeechProviders;
}): Record<string, unknown> {
const { env, persisted, providers } = params;
function resolveTts(
site: TtsSite | undefined,
env: NodeJS.ProcessEnv,
): Partial<TTSConfig> | undefined {
const config = site?.provider === "openai" ? site : undefined;
const apiKey = firstString(config?.apiKey, env.OPENAI_TTS_API_KEY, env.OPENAI_API_KEY);
if (!apiKey) return undefined;
const baseUrl = firstString(config?.baseUrl, env.OPENAI_TTS_BASE_URL, env.OPENAI_BASE_URL);
return {
ttsVoice: firstDefined<string>([
env.TTS_VOICE,
pickIfOpenAi(providers.voiceTts, persisted.features?.voiceMode?.tts?.voice),
"alloy",
]),
ttsModel: firstDefined<string>([
env.TTS_MODEL,
pickIfOpenAi(providers.voiceTts, persisted.features?.voiceMode?.tts?.model),
DEFAULT_OPENAI_TTS_MODEL,
]),
};
}
function buildOpenAiResolutionInput(params: {
env: NodeJS.ProcessEnv;
persisted: PersistedConfig;
providers: RequestedSpeechProviders;
}): Record<string, unknown> {
const { env } = params;
const openai = params.persisted.providers?.openai;
return {
sttApiKey: firstDefined<string>([
openai?.stt?.apiKey,
env.OPENAI_STT_API_KEY,
openai?.apiKey,
env.OPENAI_API_KEY,
]),
sttBaseUrl: firstDefined<string>([
openai?.stt?.baseUrl,
env.OPENAI_STT_BASE_URL,
openai?.baseUrl,
env.OPENAI_BASE_URL,
]),
ttsApiKey: firstDefined<string>([
openai?.tts?.apiKey,
env.OPENAI_TTS_API_KEY,
openai?.apiKey,
env.OPENAI_API_KEY,
]),
ttsBaseUrl: firstDefined<string>([
openai?.tts?.baseUrl,
env.OPENAI_TTS_BASE_URL,
openai?.baseUrl,
env.OPENAI_BASE_URL,
]),
...buildOpenAiSttInput(params),
...buildOpenAiTtsInput(params),
apiKey,
...(baseUrl ? { baseUrl } : {}),
model: firstString(env.TTS_MODEL, config?.model) ?? DEFAULT_OPENAI_TTS_MODEL,
voice: firstString(env.TTS_VOICE, config?.voice) ?? "alloy",
responseFormat: "pcm",
};
}
@@ -155,46 +69,14 @@ export function resolveOpenAiSpeechConfig(params: {
persisted: PersistedConfig;
providers: RequestedSpeechProviders;
}): OpenAiSpeechProviderConfig | undefined {
const input = buildOpenAiResolutionInput(params);
const keys = OpenAiEndpointKeysSchema.parse(input);
if (!keys.sttApiKey && !keys.ttsApiKey) {
return undefined;
}
const { env, persisted } = params;
const dictationStt = resolveStt(persisted.features?.dictation?.stt, env);
const stt = resolveStt(persisted.features?.voiceMode?.stt, env);
const tts = resolveTts(persisted.features?.voiceMode?.tts, env);
if (!dictationStt && !stt && !tts) return undefined;
return {
...(keys.sttApiKey ? { stt: buildSttConfig(keys.sttApiKey, keys.sttBaseUrl, input) } : {}),
...(keys.ttsApiKey ? { tts: buildTtsConfig(keys.ttsApiKey, keys.ttsBaseUrl, input) } : {}),
};
}
function buildSttConfig(
apiKey: string,
baseUrl: string | undefined,
input: Record<string, unknown>,
): OpenAiSpeechProviderConfig["stt"] {
const options = OpenAiSttOptionsSchema.parse(input);
return {
apiKey,
...(baseUrl ? { baseUrl } : {}),
...(options.sttConfidenceThreshold !== undefined
? { confidenceThreshold: options.sttConfidenceThreshold }
: {}),
...(options.sttModel ? { model: options.sttModel } : {}),
};
}
function buildTtsConfig(
apiKey: string,
baseUrl: string | undefined,
input: Record<string, unknown>,
): OpenAiSpeechProviderConfig["tts"] {
const options = OpenAiTtsOptionsSchema.parse(input);
return {
apiKey,
...(baseUrl ? { baseUrl } : {}),
voice: options.ttsVoice,
model: options.ttsModel,
responseFormat: "pcm",
...(dictationStt ? { dictationStt } : {}),
...(stt ? { stt } : {}),
...(tts ? { tts } : {}),
};
}
packages/server/src/server/speech/providers/openai/runtime.test.ts
+121 -1
View File
@@ -1,6 +1,12 @@
import { createServer, type Server } from "node:http";
import type { AddressInfo } from "node:net";
import pino from "pino";
import { describe, expect, test } from "vitest";
import { afterEach, describe, expect, test } from "vitest";
import { mergeSpeechSettingsPatch } from "../../speech-settings.js";
import { PersistedConfigSchema } from "../../../persisted-config.js";
import type { SpeechToTextProvider } from "../../speech-provider.js";
import { resolveOpenAiSpeechConfig } from "./config.js";
import { initializeOpenAiSpeechServices } from "./runtime.js";
import { OpenAISTT } from "./stt.js";
import { OpenAITTS } from "./tts.js";
@@ -16,6 +22,7 @@ describe("initializeOpenAiSpeechServices", () => {
},
openaiConfig: {
stt: { apiKey: "sk-test" },
dictationStt: { apiKey: "sk-test" },
tts: { apiKey: "sk-test" },
},
existing: {
@@ -32,3 +39,116 @@ describe("initializeOpenAiSpeechServices", () => {
expect(services.ttsService).toBeInstanceOf(OpenAITTS);
});
});
// An OpenAI-compatible transcription endpoint that records the model each request asks for.
async function startTranscriptionServer(): Promise<{
server: Server;
baseUrl: string;
models: string[];
keys: (string | undefined)[];
paths: (string | undefined)[];
}> {
const models: string[] = [];
const keys: (string | undefined)[] = [];
const paths: (string | undefined)[] = [];
const server = createServer((request, response) => {
let body = "";
request.on("data", (chunk: Buffer) => {
body += chunk.toString("latin1");
});
request.on("end", () => {
const model = /name="model"\r\n\r\n([^\r]*)/.exec(body)?.[1];
models.push(model ?? "<none>");
keys.push(request.headers.authorization);
paths.push(request.url);
response.writeHead(200, { "content-type": "application/json" });
response.end(JSON.stringify({ text: "hello" }));
});
});
await new Promise<void>((resolve) => server.listen(0, "127.0.0.1", resolve));
const { port } = server.address() as AddressInfo;
return { server, baseUrl: `http://127.0.0.1:${port}/v1`, models, keys, paths };
}
async function transcribeOnce(stt: SpeechToTextProvider | null): Promise<void> {
if (!stt) throw new Error("STT service was not created");
const session = stt.createSession({ logger: pino({ level: "silent" }), language: "en" });
const transcript = new Promise<void>((resolve, reject) => {
session.on("transcript", (event) => {
if (event.isFinal) resolve();
});
session.on("error", (error) =>
reject(error instanceof Error ? error : new Error(String(error))),
);
});
await session.connect();
session.appendPcm16(Buffer.alloc(3200));
session.commit();
await transcript;
session.close();
}
describe("OpenAI speech-to-text models", () => {
let server: Server | null = null;
afterEach(async () => {
await new Promise<void>((resolve) => (server ? server.close(() => resolve()) : resolve()));
server = null;
});
test("dictation and voice mode each transcribe with the model chosen for them", async () => {
const endpoint = await startTranscriptionServer();
server = endpoint.server;
let persisted = PersistedConfigSchema.parse({
features: {
dictation: {
stt: {
provider: "openai",
apiKey: "dictation-key",
baseUrl: `${endpoint.baseUrl}/dictation`,
model: "gpt-4o-mini-transcribe",
},
},
voiceMode: {
enabled: true,
stt: {
provider: "openai",
apiKey: "listening-key",
baseUrl: `${endpoint.baseUrl}/listening`,
model: "whisper-1",
},
},
},
});
persisted = mergeSpeechSettingsPatch(persisted, {
voiceMode: { stt: { provider: "openai", model: "whisper-1" } },
});
const providers = {
dictationStt: { provider: "openai", explicit: true },
voiceTurnDetection: { provider: "local", explicit: false, enabled: false },
voiceStt: { provider: "openai", explicit: true },
voiceTts: { provider: "local", explicit: false },
} as const;
const services = initializeOpenAiSpeechServices({
providers,
openaiConfig: resolveOpenAiSpeechConfig({ env: {}, persisted, providers }),
existing: {
turnDetectionService: null,
sttService: null,
ttsService: null,
dictationSttService: null,
},
logger: pino({ level: "silent" }),
});
await transcribeOnce(services.dictationSttService);
await transcribeOnce(services.sttService);
expect(endpoint.models).toEqual(["gpt-4o-mini-transcribe", "whisper-1"]);
expect(endpoint.keys).toEqual(["Bearer dictation-key", "Bearer listening-key"]);
expect(endpoint.paths).toEqual([
"/v1/dictation/audio/transcriptions",
"/v1/listening/audio/transcriptions",
]);
});
});
packages/server/src/server/speech/providers/openai/runtime.ts
+26 -13
View File
@@ -3,7 +3,11 @@ import type { Logger } from "pino";
import type { SpeechToTextProvider, TextToSpeechProvider } from "../../speech-provider.js";
import type { RequestedSpeechProviders } from "../../speech-types.js";
import type { TurnDetectionProvider } from "../../turn-detection-provider.js";
import { DEFAULT_OPENAI_TTS_MODEL, type OpenAiSpeechProviderConfig } from "./config.js";
import {
DEFAULT_OPENAI_TTS_MODEL,
type OpenAiSpeechProviderConfig,
type OpenAiSttFeature,
} from "./config.js";
import { OpenAISTT } from "./stt.js";
import { OpenAITTS } from "./tts.js";
@@ -33,7 +37,7 @@ function resolveOpenAiCredentials(
return {
openaiSttApiKey: sttApiKey,
openaiTtsApiKey: openaiConfig?.tts?.apiKey,
openaiDictationApiKey: sttApiKey,
openaiDictationApiKey: openaiConfig?.dictationStt?.apiKey,
};
}
@@ -94,13 +98,16 @@ export function validateOpenAiCredentialRequirements(params: {
}
}
function createOpenAiStt(
apiKey: string,
openaiConfig: OpenAiSpeechProviderConfig | undefined,
logger: Logger,
): SpeechToTextProvider {
const { apiKey: _sttApiKey, ...sttConfig } = openaiConfig?.stt ?? {};
return new OpenAISTT({ apiKey, ...sttConfig }, logger);
function createOpenAiStt(params: {
apiKey: string;
openaiConfig: OpenAiSpeechProviderConfig | undefined;
feature: OpenAiSttFeature;
logger: Logger;
}): SpeechToTextProvider {
const site =
params.feature === "dictation" ? params.openaiConfig?.dictationStt : params.openaiConfig?.stt;
const { apiKey: _sttApiKey, ...sttConfig } = site ?? {};
return new OpenAISTT({ apiKey: params.apiKey, ...sttConfig }, params.logger);
}
function createOpenAiTts(
@@ -154,7 +161,12 @@ export function initializeOpenAiSpeechServices(params: {
logger.info("OpenAI speech provider initialized");
if (needsOpenAiStt && openAiCredentials.openaiSttApiKey) {
sttService = createOpenAiStt(openAiCredentials.openaiSttApiKey, openaiConfig, logger);
sttService = createOpenAiStt({
apiKey: openAiCredentials.openaiSttApiKey,
openaiConfig,
feature: "voiceMode",
logger,
});
}
if (needsOpenAiTts && openAiCredentials.openaiTtsApiKey) {
@@ -162,11 +174,12 @@ export function initializeOpenAiSpeechServices(params: {
}
if (needsOpenAiDictation && openAiCredentials.openaiDictationApiKey) {
dictationSttService = createOpenAiStt(
openAiCredentials.openaiDictationApiKey,
dictationSttService = createOpenAiStt({
apiKey: openAiCredentials.openaiDictationApiKey,
openaiConfig,
feature: "dictation",
logger,
);
});
}
} else if (needsAnyOpenAi) {
// validateOpenAiCredentialRequirements already warned about missing credentials
packages/server/src/server/speech/speech-config-resolver.test.ts
+38 -35
View File
@@ -6,7 +6,7 @@ import { PersistedConfigSchema } from "../persisted-config.js";
import { resolveSpeechConfig } from "./speech-config-resolver.js";
describe("resolveSpeechConfig", () => {
test("resolves local-first defaults without env overrides", () => {
test("resolves local-first defaults with voice mode off until turned on", () => {
const paseoHome = "/tmp/paseo-home";
const persisted = PersistedConfigSchema.parse({});
const env = {} as NodeJS.ProcessEnv;
@@ -26,17 +26,17 @@ describe("resolveSpeechConfig", () => {
expect(result.speech.providers.voiceTurnDetection).toEqual({
provider: "local",
explicit: false,
enabled: true,
enabled: false,
});
expect(result.speech.providers.voiceStt).toEqual({
provider: "local",
explicit: false,
enabled: true,
enabled: false,
});
expect(result.speech.providers.voiceTts).toEqual({
provider: "local",
explicit: false,
enabled: true,
enabled: false,
});
expect(result.speech.local).toEqual({
modelsDir: path.join(paseoHome, "models", "local-speech"),
@@ -61,13 +61,11 @@ describe("resolveSpeechConfig", () => {
const persisted = PersistedConfigSchema.parse({
features: {
voiceMode: {
enabled: true,
turnDetection: { provider: "local" },
stt: { provider: "openai", model: "gpt-4o-transcribe" },
stt: { provider: "openai", apiKey: "persisted-key", model: "gpt-4o-transcribe" },
},
},
providers: {
openai: { apiKey: "persisted-key" },
},
});
const env = {
PASEO_DICTATION_LOCAL_STT_MODEL: "parakeet-tdt-0.6b-v2-int8",
@@ -130,40 +128,45 @@ describe("resolveSpeechConfig", () => {
voice: "pt",
});
expect(result.openai?.stt?.apiKey).toBe("persisted-key");
expect(result.openai?.tts?.apiKey).toBe("persisted-key");
expect(result.openai?.tts?.apiKey).toBe("env-key");
expect(result.openai?.stt?.model).toBe("gpt-4o-transcribe");
});
test("resolves STT language from env, settings, and voice-to-dictation fallback", () => {
const persisted = PersistedConfigSchema.parse({
features: {
dictation: {
stt: {
language: "fr",
test.each(["local", "openai"])(
"resolves STT language from env, settings, and voice-to-dictation fallback (%s)",
(provider) => {
const persisted = PersistedConfigSchema.parse({
features: {
dictation: {
stt: {
provider,
language: "fr",
},
},
voiceMode: {
stt: {
provider,
language: "de",
},
},
},
voiceMode: {
stt: {
language: "de",
},
},
},
});
});
const result = resolveSpeechConfig({
paseoHome: "/tmp/paseo-home",
env: {
PASEO_DICTATION_LANGUAGE: "es",
PASEO_VOICE_LANGUAGE: " ",
} as NodeJS.ProcessEnv,
persisted,
});
const result = resolveSpeechConfig({
paseoHome: "/tmp/paseo-home",
env: {
PASEO_DICTATION_LANGUAGE: "es",
PASEO_VOICE_LANGUAGE: " ",
} as NodeJS.ProcessEnv,
persisted,
});
expect(result.speech.sttLanguages).toEqual({
dictation: "es",
voice: "es",
});
});
expect(result.speech.sttLanguages).toEqual({
dictation: "es",
voice: "es",
});
},
);
test("respects disabled dictation and voice mode feature flags", () => {
const persisted = PersistedConfigSchema.parse({
packages/server/src/server/speech/speech-config-resolver.ts
+11 -3
View File
@@ -43,8 +43,13 @@ const RequestedSpeechProvidersSchema = z.object({
voiceTts: OptionalSpeechProviderSchema.default("local"),
});
function resolveOptionalBooleanFlag(value: unknown): boolean {
return OptionalBooleanFlagSchema.parse(value) ?? true;
// Dictation is on unless turned off. Voice mode is opt-in: it downloads its own models and
// runs a hidden agent, so a host only pays for it once someone turns it on.
export const DICTATION_ENABLED_BY_DEFAULT = true;
export const VOICE_MODE_ENABLED_BY_DEFAULT = false;
function resolveOptionalBooleanFlag(value: unknown, fallback: boolean): boolean {
return OptionalBooleanFlagSchema.parse(value) ?? fallback;
}
interface FeatureProviderInputs {
@@ -70,6 +75,7 @@ function buildFeatureProviderInputs(params: {
params.env.PASEO_VOICE_MODE_ENABLED,
params.persisted.features?.voiceMode?.enabled,
]),
VOICE_MODE_ENABLED_BY_DEFAULT,
);
return {
dictationStt: {
@@ -82,6 +88,7 @@ function buildFeatureProviderInputs(params: {
params.env.PASEO_DICTATION_ENABLED,
params.persisted.features?.dictation?.enabled,
]),
DICTATION_ENABLED_BY_DEFAULT,
),
},
voiceTurnDetection: {
@@ -119,7 +126,7 @@ function buildRequestedFeatureProvider(
};
}
function resolveRequestedSpeechProviders(params: {
export function resolveRequestedSpeechProviders(params: {
env: NodeJS.ProcessEnv;
persisted: PersistedConfig;
}): RequestedSpeechProviders {
@@ -173,6 +180,7 @@ export function resolveSpeechConfig(params: {
openai,
speech: {
providers,
localModelsDir: local.modelsDir,
sttLanguages: local.sttLanguages,
...(local.local ? { local: local.local } : {}),
},
packages/server/src/server/speech/speech-runtime.test.ts
+107 -10
View File
@@ -2,13 +2,14 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
import pino from "pino";
import type { PaseoSpeechConfig } from "../bootstrap.js";
import type { LocalSpeechModelId } from "./providers/local/models.js";
import type { InitializedLocalSpeech } from "./providers/local/runtime.js";
import type { SpeechToTextProvider, TextToSpeechProvider } from "./speech-provider.js";
import type { TurnDetectionProvider } from "./turn-detection-provider.js";
import { createSpeechService } from "./speech-runtime.js";
import { createSpeechService, LocalSpeechModelInUseError } from "./speech-runtime.js";
const { ensureLocalSpeechModelsMock, initializeLocalSpeechServicesMock } = vi.hoisted(() => ({
ensureLocalSpeechModelsMock: vi.fn(async () => ({})),
const { ensureSherpaOnnxModelMock, initializeLocalSpeechServicesMock } = vi.hoisted(() => ({
ensureSherpaOnnxModelMock: vi.fn(async (_options: { signal?: AbortSignal }) => ""),
initializeLocalSpeechServicesMock: vi.fn<(args: unknown) => Promise<InitializedLocalSpeech>>(),
}));
@@ -34,9 +35,10 @@ vi.mock("./providers/openai/runtime.js", () => ({
validateOpenAiCredentialRequirements: () => {},
}));
vi.mock("./providers/local/models.js", () => ({
ensureLocalSpeechModels: ensureLocalSpeechModelsMock,
listMissingLocalSpeechModels: vi.fn(async ({ modelIds }: { modelIds: string[] }) => modelIds),
vi.mock("./providers/local/sherpa/model-downloader.js", () => ({
ensureSherpaOnnxModel: ensureSherpaOnnxModelMock,
isSherpaOnnxModelInstalled: vi.fn(async () => false),
deleteSherpaOnnxModel: vi.fn(async () => {}),
}));
function createStubStt(id: string): SpeechToTextProvider {
@@ -75,6 +77,7 @@ function rejectWhenAborted(signal?: AbortSignal): Promise<never> {
function createSpeechConfig(providers: PaseoSpeechConfig["providers"]): PaseoSpeechConfig {
return {
providers,
localModelsDir: "/tmp/missing-local-speech-models",
sttLanguages: {
dictation: "en",
voice: "en",
@@ -84,8 +87,8 @@ function createSpeechConfig(providers: PaseoSpeechConfig["providers"]): PaseoSpe
describe("createSpeechService readiness", () => {
beforeEach(() => {
ensureLocalSpeechModelsMock.mockReset();
ensureLocalSpeechModelsMock.mockResolvedValue({});
ensureSherpaOnnxModelMock.mockReset();
ensureSherpaOnnxModelMock.mockResolvedValue("");
initializeLocalSpeechServicesMock.mockReset();
});
@@ -169,7 +172,7 @@ describe("createSpeechService readiness", () => {
it("aborts and joins an in-flight model download when stopped", async () => {
let downloadSignal: AbortSignal | undefined;
ensureLocalSpeechModelsMock.mockImplementation(({ signal }: { signal?: AbortSignal }) => {
ensureSherpaOnnxModelMock.mockImplementation(({ signal }: { signal?: AbortSignal }) => {
downloadSignal = signal;
return rejectWhenAborted(signal);
});
@@ -201,10 +204,104 @@ describe("createSpeechService readiness", () => {
});
runtime.start();
await runtime.ready;
expect(ensureLocalSpeechModelsMock).toHaveBeenCalledOnce();
expect(ensureSherpaOnnxModelMock).toHaveBeenCalledOnce();
await runtime.stop();
expect(downloadSignal?.aborted).toBe(true);
});
});
describe("createSpeechService settings changes", () => {
const dictationOnly = createSpeechConfig({
dictationStt: { provider: "local", enabled: true, explicit: true },
voiceTurnDetection: { provider: "local", enabled: false, explicit: true },
voiceStt: { provider: "local", enabled: false, explicit: true },
voiceTts: { provider: "local", enabled: false, explicit: true },
});
const nothingEnabled = createSpeechConfig({
dictationStt: { provider: "local", enabled: false, explicit: true },
voiceTurnDetection: { provider: "local", enabled: false, explicit: true },
voiceStt: { provider: "local", enabled: false, explicit: true },
voiceTts: { provider: "local", enabled: false, explicit: true },
});
function localSpeech(input: {
dictationStt: SpeechToTextProvider | null;
defaultModelIds: LocalSpeechModelId[];
}): InitializedLocalSpeech {
return {
turnDetectionService: null,
sttService: null,
ttsService: null,
dictationSttService: input.dictationStt,
localVoiceTtsProvider: null,
localModelConfig: {
modelsDir: "/tmp/missing-local-speech-models",
defaultModelIds: input.defaultModelIds,
},
availability: { configured: true, modelsDir: "/tmp/missing-local-speech-models" },
cleanup: () => {},
};
}
beforeEach(() => {
ensureSherpaOnnxModelMock.mockReset();
ensureSherpaOnnxModelMock.mockResolvedValue("");
initializeLocalSpeechServicesMock.mockReset();
});
it("applies new providers without a restart", async () => {
initializeLocalSpeechServicesMock.mockResolvedValue(
localSpeech({ dictationStt: createStubStt("dictation-local"), defaultModelIds: [] }),
);
const runtime = createSpeechService({
logger: pino({ level: "silent" }),
speechConfig: dictationOnly,
});
runtime.start();
await runtime.ready;
expect(runtime.getReadiness().dictation.available).toBe(true);
const disabled = new Promise<void>((resolve) => {
runtime.onReadinessChange((snapshot) => {
if (snapshot.dictation.reasonCode === "disabled") resolve();
});
});
initializeLocalSpeechServicesMock.mockResolvedValue(
localSpeech({ dictationStt: null, defaultModelIds: [] }),
);
runtime.reconfigure({ speechConfig: nothingEnabled });
await disabled;
expect(runtime.resolveDictationStt()).toBeNull();
await runtime.stop();
});
it("refuses to delete a model an enabled feature uses", async () => {
initializeLocalSpeechServicesMock.mockResolvedValue(
localSpeech({
dictationStt: createStubStt("dictation-local"),
defaultModelIds: ["parakeet-tdt-0.6b-v2-int8"],
}),
);
ensureSherpaOnnxModelMock.mockImplementation(({ signal }) => rejectWhenAborted(signal));
const runtime = createSpeechService({
logger: pino({ level: "silent" }),
speechConfig: dictationOnly,
});
runtime.start();
await runtime.ready;
const models = await runtime.listLocalModels();
expect(models.filter((model) => model.inUse).map((model) => model.spec.id)).toEqual([
"parakeet-tdt-0.6b-v2-int8",
]);
await expect(runtime.deleteLocalModel("parakeet-tdt-0.6b-v2-int8")).rejects.toBeInstanceOf(
LocalSpeechModelInUseError,
);
await expect(runtime.deleteLocalModel("kokoro-int8-en-v0_19")).resolves.toBeUndefined();
await runtime.stop();
});
});
packages/server/src/server/speech/speech-runtime.ts
+134 -64
View File
@@ -2,7 +2,10 @@ import type { Logger } from "pino";
import type { PaseoOpenAIConfig, PaseoSpeechConfig } from "../bootstrap.js";
import type { LocalSpeechModelId } from "./providers/local/config.js";
import { ensureLocalSpeechModels, listMissingLocalSpeechModels } from "./providers/local/models.js";
import {
LocalSpeechModelLibrary,
type LocalSpeechModelEntry,
} from "./providers/local/model-library.js";
import { initializeLocalSpeechServices } from "./providers/local/runtime.js";
import {
getOpenAiSpeechAvailability,
@@ -92,7 +95,7 @@ function buildRealtimeVoiceReadiness(params: {
enabled: false,
available: false,
reasonCode: "disabled",
message: "Realtime voice is disabled in daemon config.",
message: "Voice mode is off. Turn it on in Settings > Voice.",
retryable: false,
missingModelIds: [],
};
@@ -147,7 +150,7 @@ function buildDictationReadiness(params: {
enabled: false,
available: false,
reasonCode: "disabled",
message: "Dictation is disabled in daemon config.",
message: "Dictation is off. Turn it on in Settings > Voice.",
retryable: false,
missingModelIds: [],
};
@@ -292,6 +295,23 @@ function resolveEffectiveProviderIds(params: {
};
}
export interface LocalSpeechModelStatus extends LocalSpeechModelEntry {
// Selected by an enabled feature, so the runtime keeps it installed.
inUse: boolean;
}
export class LocalSpeechModelInUseError extends Error {
constructor(public readonly modelId: LocalSpeechModelId) {
super(`${modelId} is selected for dictation or voice mode. Choose another model first.`);
this.name = "LocalSpeechModelInUseError";
}
}
export interface SpeechServiceConfig {
openaiConfig?: PaseoOpenAIConfig;
speechConfig?: PaseoSpeechConfig;
}
export interface SpeechService {
resolveStt: () => SpeechToTextProvider | null;
resolveSttLanguage: () => string;
@@ -301,37 +321,49 @@ export interface SpeechService {
resolveDictationSttLanguage: () => string;
getReadiness: () => SpeechReadinessSnapshot;
onReadinessChange: (listener: (snapshot: SpeechReadinessSnapshot) => void) => () => void;
// Applies new provider/model settings without restarting the daemon.
reconfigure: (config: SpeechServiceConfig) => void;
listLocalModels: () => Promise<LocalSpeechModelStatus[]>;
downloadLocalModel: (modelId: LocalSpeechModelId) => void;
deleteLocalModel: (modelId: LocalSpeechModelId) => Promise<void>;
start: () => void;
stop: () => Promise<void>;
ready: Promise<void>;
}
export function createSpeechService(params: {
logger: Logger;
openaiConfig?: PaseoOpenAIConfig;
speechConfig?: PaseoSpeechConfig;
}): SpeechService {
export function createSpeechService(
params: SpeechServiceConfig & {
logger: Logger;
},
): SpeechService {
const logger = params.logger.child({ module: "speech-runtime" });
const speechConfig = params.speechConfig ?? null;
const openaiConfig = params.openaiConfig;
const providers = resolveRequestedSpeechProviders(speechConfig);
const requestedProviders = describeRequestedProviders(providers);
let speechConfig = params.speechConfig ?? null;
let openaiConfig = params.openaiConfig;
let providers = resolveRequestedSpeechProviders(speechConfig);
let requestedProviders = describeRequestedProviders(providers);
// The models directory is fixed for the daemon's lifetime; settings changes only pick
// which of its models are used.
const modelLibrary = speechConfig
? new LocalSpeechModelLibrary(speechConfig.localModelsDir, logger)
: null;
validateOpenAiCredentialRequirements({
providers,
openaiConfig,
logger,
});
logger.info(
{
requestedProviders,
availability: {
openai: getOpenAiSpeechAvailability(openaiConfig),
const logReconciliationStart = (): void => {
validateOpenAiCredentialRequirements({
providers,
openaiConfig,
logger,
});
logger.info(
{
requestedProviders,
availability: {
openai: getOpenAiSpeechAvailability(openaiConfig),
},
},
},
"Speech provider reconciliation started",
);
"Speech provider reconciliation started",
);
};
logReconciliationStart();
let sttService: SpeechToTextProvider | null = null;
let ttsService: TextToSpeechProvider | null = null;
@@ -346,12 +378,11 @@ export function createSpeechService(params: {
let missingLocalModelIds: LocalSpeechModelId[] = [];
let backgroundDownloadInProgress = false;
let backgroundDownloadAbortController: AbortController | null = null;
let backgroundDownloadPromise: Promise<void> | null = null;
let backgroundDownloadError: string | null = null;
let stopped = false;
let monitorTimeout: ReturnType<typeof setTimeout> | null = null;
let reconcileInFlight: Promise<void> | null = null;
let reconcileQueue: Promise<void> = Promise.resolve();
const readinessListeners = new Set<(snapshot: SpeechReadinessSnapshot) => void>();
let lastReadinessFingerprint: string | null = null;
let lastPublishedReadinessSnapshot: SpeechReadinessSnapshot | null = null;
@@ -445,12 +476,10 @@ export function createSpeechService(params: {
};
const refreshMissingLocalModels = async (): Promise<void> => {
missingLocalModelIds = localModelConfig
? await listMissingLocalSpeechModels({
modelsDir: localModelConfig.modelsDir,
modelIds: localModelConfig.defaultModelIds,
})
: [];
missingLocalModelIds =
localModelConfig && modelLibrary
? await modelLibrary.listMissing(localModelConfig.defaultModelIds)
: [];
};
const reconcileServices = async (): Promise<void> => {
@@ -520,16 +549,12 @@ export function createSpeechService(params: {
}
};
// Reconciles run one after another, and each starts after it was requested, so a
// reconfigure that lands mid-reconcile is applied instead of joining the stale run.
const runReconcile = async (): Promise<void> => {
if (reconcileInFlight) {
await reconcileInFlight;
publishReadinessIfChanged();
return;
}
reconcileInFlight = reconcileServices().finally(() => {
reconcileInFlight = null;
});
await reconcileInFlight;
const next = reconcileQueue.catch(() => undefined).then(reconcileServices);
reconcileQueue = next;
await next;
publishReadinessIfChanged();
};
@@ -544,12 +569,12 @@ export function createSpeechService(params: {
};
const startBackgroundDownload = (): void => {
if (stopped || backgroundDownloadInProgress) {
if (stopped || backgroundDownloadInProgress || !modelLibrary) {
return;
}
const modelsDir = localModelConfig?.modelsDir ?? null;
const modelsDir = modelLibrary.getModelsDir();
const modelIds = [...missingLocalModelIds];
if (!modelsDir || modelIds.length === 0) {
if (modelIds.length === 0) {
return;
}
@@ -565,16 +590,9 @@ export function createSpeechService(params: {
"Starting background download for missing local speech models",
);
const abortController = new AbortController();
backgroundDownloadAbortController = abortController;
backgroundDownloadPromise = (async () => {
try {
await ensureLocalSpeechModels({
modelsDir,
modelIds,
logger,
signal: abortController.signal,
});
await Promise.all(modelIds.map((modelId) => modelLibrary.download(modelId)));
await runReconcile();
backgroundDownloadError = null;
} catch (error) {
@@ -588,7 +606,6 @@ export function createSpeechService(params: {
"Background local speech model download failed",
);
} finally {
backgroundDownloadAbortController = null;
backgroundDownloadInProgress = false;
await refreshMissingLocalModels().catch((error) => {
logger.warn({ err: error }, "Failed to refresh local speech model status after download");
@@ -630,6 +647,62 @@ export function createSpeechService(params: {
}
};
const reconcileAndFetchMissingModels = async (): Promise<void> => {
await runReconcile();
const snapshot = computeReadinessSnapshot();
if (snapshot.voiceFeature.enabled && !snapshot.voiceFeature.available) {
if (missingLocalModelIds.length > 0) {
startBackgroundDownload();
}
scheduleMonitor();
}
};
const reconfigure = (next: SpeechServiceConfig): void => {
speechConfig = next.speechConfig ?? null;
openaiConfig = next.openaiConfig;
providers = resolveRequestedSpeechProviders(speechConfig);
requestedProviders = describeRequestedProviders(providers);
// A failure for the previous selection must not block downloading the new one.
backgroundDownloadError = null;
logReconciliationStart();
if (!started || stopped) {
return;
}
void reconcileAndFetchMissingModels().catch((error) => {
logger.error({ err: error }, "Speech runtime failed to apply new configuration");
});
};
const requireModelLibrary = (): LocalSpeechModelLibrary => {
if (!modelLibrary) {
throw new Error("Local speech models are unavailable on this daemon");
}
return modelLibrary;
};
const listLocalModels = async (): Promise<LocalSpeechModelStatus[]> => {
const inUse = new Set(localModelConfig?.defaultModelIds ?? []);
const entries = await requireModelLibrary().list();
return entries.map(({ spec, state }) => ({ spec, state, inUse: inUse.has(spec.id) }));
};
const downloadLocalModel = (modelId: LocalSpeechModelId): void => {
void requireModelLibrary()
.download(modelId)
.then(() => runMonitorTick())
.catch((error) => {
logger.warn({ err: error, modelId }, "Local speech model download failed");
});
};
const deleteLocalModel = async (modelId: LocalSpeechModelId): Promise<void> => {
if (localModelConfig?.defaultModelIds.includes(modelId)) {
throw new LocalSpeechModelInUseError(modelId);
}
await requireModelLibrary().delete(modelId);
};
const start = (): void => {
if (started || stopped) {
return;
@@ -637,14 +710,7 @@ export function createSpeechService(params: {
started = true;
void (async () => {
try {
await runReconcile();
const snapshot = computeReadinessSnapshot();
if (snapshot.voiceFeature.enabled && !snapshot.voiceFeature.available) {
if (missingLocalModelIds.length > 0) {
startBackgroundDownload();
}
scheduleMonitor();
}
await reconcileAndFetchMissingModels();
if (!readySettled) {
readySettled = true;
resolveReady();
@@ -666,7 +732,7 @@ export function createSpeechService(params: {
monitorTimeout = null;
}
localCleanup();
backgroundDownloadAbortController?.abort();
await modelLibrary?.stop();
await backgroundDownloadPromise?.catch(() => undefined);
backgroundDownloadPromise = null;
};
@@ -680,6 +746,10 @@ export function createSpeechService(params: {
resolveDictationSttLanguage: () => speechConfig?.sttLanguages?.dictation ?? "en",
getReadiness: () => lastPublishedReadinessSnapshot ?? computeReadinessSnapshot(),
onReadinessChange: subscribeSpeechReadiness,
reconfigure,
listLocalModels,
downloadLocalModel,
deleteLocalModel,
start,
stop,
ready,
packages/server/src/server/speech/speech-settings.ts
+202
View File
@@ -0,0 +1,202 @@
import type { SpeechSettings, SpeechSettingsPatch } from "@getpaseo/protocol/speech/rpc-schemas";
import {
PersistedConfigSchema,
SpeechSttConfigSchema,
SpeechTtsConfigSchema,
type PersistedConfig,
} from "../persisted-config.js";
import { resolveRequestedSpeechProviders, resolveSpeechConfig } from "./speech-config-resolver.js";
import type { SpeechService } from "./speech-runtime.js";
import { LocalSttModelIdSchema, LocalTtsModelIdSchema } from "./providers/local/models.js";
interface SpeechRead {
features: SpeechSettings;
envOverrides: Record<string, string>;
}
// Owns the per-site wire boundary: credential redaction, env provenance and patch
// semantics. The store never assembles or merges provider objects itself.
export interface SpeechSettingsOwner {
read(persisted: PersistedConfig): SpeechRead;
apply(persisted: PersistedConfig): void;
}
export function createSpeechSettingsOwner(params: {
paseoHome: string;
env: NodeJS.ProcessEnv;
speechService: Pick<SpeechService, "reconfigure">;
}): SpeechSettingsOwner {
const { paseoHome, env, speechService } = params;
return {
read: (persisted) => readSpeechSettings(persisted, env),
apply: (persisted) => {
const resolved = resolveSpeechConfig({ paseoHome, env, persisted });
speechService.reconfigure({ openaiConfig: resolved.openai, speechConfig: resolved.speech });
},
};
}
export function readSpeechSettings(persisted: PersistedConfig, env: NodeJS.ProcessEnv): SpeechRead {
const dictation = persisted.features?.dictation;
const voiceMode = persisted.features?.voiceMode;
const providers = resolveRequestedSpeechProviders({ persisted, env });
const envOverrides: Record<string, string> = {};
const recordOverride = (path: string, names: string[], fallback?: unknown) => {
if (fallback !== undefined) return;
const name = names.find((candidate) => env[candidate]?.trim());
if (name) envOverrides[path] = name;
};
recordOverride("features.dictation.enabled", ["PASEO_DICTATION_ENABLED"]);
recordOverride("features.voiceMode.enabled", ["PASEO_VOICE_MODE_ENABLED"]);
recordOverride("features.voiceMode.turnDetection.provider", [
"PASEO_VOICE_TURN_DETECTION_PROVIDER",
]);
function readSite(
site: Record<string, unknown> | undefined,
provider: string,
path: string,
kind: "stt" | "tts",
providerEnv: string,
localModelEnv: string,
) {
recordOverride(`${path}.provider`, [providerEnv]);
const configured = site?.provider === provider ? site : undefined;
const { apiKey, ...fields } = configured ?? {};
const openaiModelEnv = kind === "stt" ? "STT_MODEL" : "TTS_MODEL";
recordOverride(`${path}.model`, [provider === "openai" ? openaiModelEnv : localModelEnv]);
if (kind === "stt") {
const languageVars =
path === "features.dictation.stt"
? ["PASEO_DICTATION_LANGUAGE"]
: ["PASEO_VOICE_LANGUAGE", "PASEO_DICTATION_LANGUAGE"];
recordOverride(`${path}.language`, languageVars);
}
if (provider === "openai") {
for (const [field, suffix] of [
["apiKey", "API_KEY"],
["baseUrl", "BASE_URL"],
] as const) {
recordOverride(
`${path}.${field}`,
[`OPENAI_${kind.toUpperCase()}_${suffix}`, `OPENAI_${suffix}`],
configured?.[field],
);
}
if (kind === "stt") {
recordOverride(`${path}.confidenceThreshold`, ["STT_CONFIDENCE_THRESHOLD"]);
} else recordOverride(`${path}.voice`, ["TTS_VOICE"]);
} else if (kind === "tts") {
recordOverride(`${path}.speakerId`, ["PASEO_VOICE_LOCAL_TTS_SPEAKER_ID"]);
recordOverride(`${path}.speed`, ["PASEO_VOICE_LOCAL_TTS_SPEED"]);
}
return {
...fields,
provider,
...(provider === "openai"
? { hasApiKey: typeof apiKey === "string" && apiKey.trim().length > 0 }
: {}),
};
}
return {
features: {
dictation: {
enabled: providers.dictationStt.enabled !== false,
stt: readSite(
dictation?.stt,
providers.dictationStt.provider,
"features.dictation.stt",
"stt",
"PASEO_DICTATION_STT_PROVIDER",
"PASEO_DICTATION_LOCAL_STT_MODEL",
),
},
voiceMode: {
enabled: providers.voiceStt.enabled !== false,
...(voiceMode?.llm ? { llm: voiceMode.llm } : {}),
...(voiceMode?.turnDetection
? {
turnDetection: {
...voiceMode.turnDetection,
provider: providers.voiceTurnDetection.provider,
},
}
: {}),
stt: readSite(
voiceMode?.stt,
providers.voiceStt.provider,
"features.voiceMode.stt",
"stt",
"PASEO_VOICE_STT_PROVIDER",
"PASEO_VOICE_LOCAL_STT_MODEL",
),
tts: readSite(
voiceMode?.tts,
providers.voiceTts.provider,
"features.voiceMode.tts",
"tts",
"PASEO_VOICE_TTS_PROVIDER",
"PASEO_VOICE_LOCAL_TTS_MODEL",
),
},
},
envOverrides,
};
}
function mergeFields(current: Record<string, unknown> | undefined, patch: Record<string, unknown>) {
const next = { ...current };
for (const [key, value] of Object.entries(patch)) {
if (value === undefined) continue;
if (value === null || (typeof value === "string" && !value.trim())) delete next[key];
else next[key] = typeof value === "string" ? value.trim() : value;
}
return next;
}
function mergeSite(
current: Record<string, unknown> | undefined,
patch: Record<string, unknown>,
kind: "stt" | "tts",
) {
const providerChanged =
patch.provider !== undefined && patch.provider !== (current?.provider ?? "local");
const next = mergeFields(providerChanged ? undefined : current, patch);
next.provider ??= "local";
const site = (kind === "stt" ? SpeechSttConfigSchema : SpeechTtsConfigSchema).parse(next);
if (site.provider === "local" && site.model !== undefined) {
const schema = kind === "stt" ? LocalSttModelIdSchema : LocalTtsModelIdSchema;
if (!schema.safeParse(site.model).success) {
throw new Error(
`Unknown local ${kind === "stt" ? "speech-to-text" : "voice"} model: ${site.model}`,
);
}
}
return site;
}
export function mergeSpeechSettingsPatch(
persisted: PersistedConfig,
patch: SpeechSettingsPatch,
): PersistedConfig {
const features = { ...persisted.features };
if (patch.dictation) {
const { stt, ...rest } = patch.dictation;
const next = mergeFields(features.dictation, rest);
if (stt) next.stt = mergeSite(features.dictation?.stt, stt, "stt");
features.dictation = PersistedConfigSchema.shape.features.unwrap().shape.dictation.parse(next);
}
if (patch.voiceMode) {
const { stt, tts, llm, turnDetection, ...rest } = patch.voiceMode;
const next = mergeFields(features.voiceMode, rest);
if (stt) next.stt = mergeSite(features.voiceMode?.stt, stt, "stt");
if (tts) next.tts = mergeSite(features.voiceMode?.tts, tts, "tts");
if (llm) next.llm = mergeFields(features.voiceMode?.llm, llm);
if (turnDetection)
next.turnDetection = mergeFields(features.voiceMode?.turnDetection, turnDetection);
features.voiceMode = PersistedConfigSchema.shape.features.unwrap().shape.voiceMode.parse(next);
}
return { ...persisted, features };
}
packages/server/src/server/test-utils/paseo-daemon.ts
+11 -2
View File
@@ -39,7 +39,8 @@ interface TestPaseoDaemonOptions {
staticDir?: string;
cleanup?: boolean;
openai?: PaseoOpenAIConfig;
speech?: PaseoSpeechConfig;
// localModelsDir defaults to the local provider's modelsDir, then to a dir under paseoHome.
speech?: Omit<PaseoSpeechConfig, "localModelsDir"> & { localModelsDir?: string };
voiceLlmProvider?: PaseoDaemonConfig["voiceLlmProvider"];
voiceLlmProviderExplicit?: boolean;
voiceLlmModel?: string | null;
@@ -209,7 +210,15 @@ async function prepareTestDaemonConfig(
webUi: options.webUi,
trustedProxies: options.trustedProxies,
openai: options.openai,
speech: options.speech,
speech: options.speech
? {
...options.speech,
localModelsDir:
options.speech.localModelsDir ??
options.speech.local?.modelsDir ??
path.join(paseoHome, "models", "local-speech"),
}
: undefined,
voiceLlmProvider: options.voiceLlmProvider ?? null,
voiceLlmProviderExplicit: options.voiceLlmProviderExplicit ?? false,
voiceLlmModel: options.voiceLlmModel ?? null,
packages/server/src/server/websocket-server.ts
+3
View File
@@ -1620,6 +1620,7 @@ export class VoiceAssistantWebSocketServer {
stt: () => this.speech?.resolveStt() ?? null,
sttLanguage: this.speech?.resolveSttLanguage() ?? "en",
tts: () => this.speech?.resolveTts() ?? null,
speechModels: this.speech,
terminalManager: this.terminalManager,
providerSnapshotManager: this.providerSnapshotManager,
hubExecutionAgents: options.hubExecutionAgents,
@@ -1991,6 +1992,8 @@ export class VoiceAssistantWebSocketServer {
...(this.advertiseDaemonStatusRpc ? { daemonStatusRpc: true } : {}),
// COMPAT(daemonConfigReload): added in v0.4.0, remove gate after 2027-02-14.
daemonConfigReload: true,
// COMPAT(speechSettings): added in v0.12.0, remove gate after 2027-04-18.
speechSettings: true,
// COMPAT(relayConfig): added in v0.2.6, remove gate after 2027-01-31.
...(this.advertiseRelayConfig ? { relayConfig: true } : {}),
// COMPAT(pushTokenRevocation): added in v0.3.2, remove gate after 2027-02-10.