This PR introducesreal-time voice pipeline support for OpenAI’sgpt-4o-realtime-preview model, enabling seamless, low-latency speech-to-speech interactions in the Speect framework. The update brings a modern, streaming audio interface, integrated tool execution, and robust event handling—while maintaining full compatibility with the existing STT/TTS pipeline.

Key Features & Changes

• RealtimeVoicePipeline:

  • New pipeline for direct, continuous audio-to-audio conversations with OpenAI’s real-time models.
  • Handles streaming microphone input and speaker output at 24kHz, as required by the API.
  • Supports push-to-talk and half-duplex operation to prevent echo/feedback.

• Integrated Tool Calls:

  • Tools are registered with the pipeline and executed automatically when the model requests a function call.
  • Tool results are sent back to the model using the correct OpenAI Realtime API protocol.

• Event Handling & Debugging:

  • Full support for all major OpenAI Realtime API events, including:
    • Audio and text deltas
    • Tool call arguments (streamed and completed)
    • Transcription events (conversation.item.input_audio_transcription.delta and.completed)
    • Session and rate limit updates
  • Example logs all transcription events for easy debugging of what the model “hears.”

• Echo & Feedback Mitigation:

  • Implements a buffer window after assistant audio playback to prevent microphone echo from triggering new turns.
  • Optionally enables server-side noise/echo reduction viainput_audio_noise_reduction in the session config.

• Sample Rate Fixes:

  • Ensures both input and output audio are always 24kHz PCM, as required by the OpenAI API (fixes “slow motion” audio bug).

• Backwards Compatibility:

  • All changes are fully compatible with the existing STT/TTS pipeline and configuration.
  • Legacy examples and workflows continue to work without modification.

• Documentation & Examples:

  • Updateddocs/voice/pipeline.md with new real-time usage, configuration, and troubleshooting sections.
  • New example:continuous_realtime_assistant.py demonstrates push-to-talk, tool calls, and event handling.

🛠️ How to Use

• Realtime Pipeline:
  See the new example and documentation for how to useRealtimeVoicePipeline with your OpenAI API key and tools.
• Classic Pipeline:
  No changes required—existing STT/TTS flows are unaffected.