puter/paseo
1
0
Fork 0
mirror of https://github.com/getpaseo/paseo.git synced 2026-07-29 12:01:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ad98a3f265b4497134d778ff719c34a4e67bb5f5
paseo/packages/web/REALTIME_API_CONFIGURATION.md
Mohamed Boudra fee0f4fd28 Initial monorepo setup with mcp-server and web packages
2025-10-13 14:03:54 +02:00

7.7 KiB
Raw Blame History

OpenAI Realtime API Configuration Options

This document lists all available configuration parameters for the OpenAI Realtime API session.

Core Parameters

model

  • Type: string
  • Default: 'gpt-4o-realtime-preview-2024-12-17'
  • Options: 'gpt-realtime', 'gpt-4o-realtime-preview-2024-12-17'
  • Description: The Realtime model to use. gpt-realtime is the latest production model.

voice

  • Type: string
  • Default: 'alloy'
  • Options: 'alloy', 'echo', 'fable', 'onyx', 'nova', 'shimmer'
  • Description: Voice to use for AI speech generation

modalities

  • Type: array of strings
  • Default: ['text', 'audio']
  • Options: ['text'], ['audio'], ['text', 'audio']
  • Description: Response modalities to use. Set to ['text'] for text-only mode with separate TTS.

temperature

  • Type: float
  • Default: 0.8
  • Range: 0.6 to 1.2
  • Description: Controls response randomness. Lower values = more deterministic, higher values = more creative.

max_response_output_tokens

  • Type: integer
  • Default: Not specified
  • Example: 2048
  • Description: Maximum number of tokens in the model's response.

instructions

  • Type: string
  • Default: None
  • Description: System instructions for the model (like system messages in Chat API)

Audio Configuration

input_audio_format

  • Type: string
  • Default: 'pcm16'
  • Options: 'pcm16', 'g711_ulaw', 'g711_alaw'
  • Description: Format for input audio

output_audio_format

  • Type: string
  • Default: 'pcm16'
  • Options: 'pcm16', 'g711_ulaw', 'g711_alaw'
  • Description: Format for output audio

input_audio_transcription

  • Type: object
  • Default: None
  • Properties:
    • model: Transcription model (e.g., 'gpt-4o-transcribe')
  • Description: Configuration for transcribing input audio
  • Example: 
    {
  "model": "gpt-4o-transcribe"
}

input_audio_noise_reduction (New in 2025)

  • Type: object | null
  • Default: null
  • Description: Noise reduction configuration for input audio
  • Note: Using noise reduction can impact latency, especially with semantic VAD turn detection
  • Status: Recently added parameter, check latest API docs for configuration details

Turn Detection Configuration

Controls when the AI should start/stop responding based on voice activity.

turn_detection

  • Type: object | null
  • Default: Server VAD mode
  • Description: Configuration for turn detection

Server VAD Mode (Default)

{
  "type": "server_vad",
  "threshold": 0.5,
  "prefix_padding_ms": 300,
  "silence_duration_ms": 500,
  "create_response": true,
  "interrupt_response": true
}

Parameters:

  • threshold (float, 0.0-1.0): Voice activity detection threshold. Higher values (e.g., 0.7) work better in noisy environments.
  • prefix_padding_ms (integer): Audio to include before speech starts (ms)
  • silence_duration_ms (integer): Silence duration to wait before considering speech ended (ms)
  • create_response (boolean): Automatically create response when turn detected
  • interrupt_response (boolean): Allow user to interrupt AI responses

Semantic VAD Mode

{
  "type": "semantic_vad",
  "eagerness": "auto",
  "create_response": true,
  "interrupt_response": true
}

Parameters:

  • eagerness (string): 'low', 'medium', 'high', or 'auto'. Controls how quickly the model responds.
  • create_response (boolean): Automatically create response
  • interrupt_response (boolean): Allow interruptions

Note: Semantic VAD can have higher latency, especially with noise reduction enabled.

Disable Turn Detection

{
  "type": "none"
}

Use this if you want manual control over when responses are generated.

Tool Configuration

tools

  • Type: array of objects
  • Default: []
  • Description: Tools available for the model to use

Remote MCP Server Tool

{
  "type": "mcp",
  "server_label": "my-mcp-server",
  "server_url": "https://example.com/mcp",
  "authorization": "Bearer token",  // Optional
  "require_approval": "never"       // or "always"
}

Parameters:

  • type: Must be 'mcp' for MCP servers
  • server_label: Friendly name for the server
  • server_url: HTTPS URL of the MCP server
  • authorization: Optional auth header value
  • require_approval: 'never' for auto-execution, 'always' for manual approval

Function Tool (Client-Side)

{
  "type": "function",
  "name": "function_name",
  "description": "What the function does",
  "parameters": {
    "type": "object",
    "properties": {
      "param1": {
        "type": "string",
        "description": "Parameter description"
      }
    },
    "required": ["param1"]
  }
}

tool_choice

  • Type: string
  • Default: 'auto'
  • Options: 'auto', 'required', 'none'
  • Description: Controls when tools are used
    • 'auto': AI decides when to use tools
    • 'required': AI must use a tool
    • 'none': Tools disabled for this response

Current Configuration (app/api/session/route.ts)

{
  model: 'gpt-realtime',
  voice: 'shimmer',
  input_audio_transcription: {
    model: 'gpt-4o-transcribe',
  },
  tools: [
    {
      type: 'mcp',
      server_label: 'local-mcp',
      server_url: 'https://mohameds-macbook-pro.tail8fe838.ts.net/mcp?password=dev-password',
      require_approval: 'never',
    },
  ],
}

Recommended Settings for Different Use Cases

High Quality Conversation (Default)

{
  "model": "gpt-realtime",
  "temperature": 0.8,
  "turn_detection": {
    "type": "server_vad",
    "threshold": 0.5
  }
}

Noisy Environment

{
  "model": "gpt-realtime",
  "input_audio_noise_reduction": {},
  "turn_detection": {
    "type": "server_vad",
    "threshold": 0.7,  // Higher threshold for noisy environments
    "silence_duration_ms": 700  // Wait longer for silence
  }
}

More Deterministic Responses

{
  "model": "gpt-realtime",
  "temperature": 0.6,
  "max_response_output_tokens": 1024
}

Fast, Responsive Conversation

{
  "model": "gpt-realtime",
  "turn_detection": {
    "type": "semantic_vad",
    "eagerness": "high"
  }
}

Manual Control (No Auto-Response)

{
  "model": "gpt-realtime",
  "turn_detection": {
    "type": "none"
  }
}

Notes on Noise Reduction

Availability: input_audio_noise_reduction is a newly added parameter (2025).

Pros:

  • Improves recognition in noisy environments
  • Better handling of background noise

Cons:

  • Can increase latency, especially with semantic VAD
  • May affect audio quality in some cases

Recommendation: Test with and without noise reduction to see if the latency trade-off is worth it for your use case.

Client-Side Audio Processing

In addition to server-side configuration, consider client-side audio processing:

  1. Browser WebRTC: Modern browsers include:

    • Echo cancellation
    • Noise suppression
    • Automatic gain control

  2. getUserMedia constraints:

    navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,
    noiseSuppression: true,
    autoGainControl: true
  }
});

These are currently not configured in the app but can be added in use-webrtc-voice.ts for additional audio quality improvements.

References