Repository files navigation

this is a svelte component for using the openai realtime api (and mostly a direct port to svelte of theopenai-realtime-console repository).

it allows you to easily use the openai realtime api in your svelte(kit) project.

work in progress, but it should be functional.

demos here (enter your own api key, as always be careful with stuff like that and e.g. use a 2$ limited api key):

basic chat:https://flo-bit.dev/svelte-openai-realtime-api/

chat with visualizations:https://flo-bit.dev/svelte-openai-realtime-api/visualizations-chat

1. copy thesrc/lib/realtime/ folder from this repository into your sveltekit projectssrc/lib/ folder

2. install the dependency:

$ npm i openai/openai-realtime-api-beta

3. import theRealtime component into your svelte file and use it e.g. like so:

<scriptlang="ts">importRealtimefrom'$lib/realtime/Realtime.svelte';importtype {ItemType }from'@openai/realtime-api-beta/dist/lib/client';let startConversation: ()=>Promise<void>;let endConversation: ()=>Promise<void>;let turnDetection:'server_vad'|'none'='server_vad';let items:ItemType[]= [];let isConnected=false;let isRecording=false;let startRecording: ()=>Promise<void>;let stopRecording: ()=>Promise<void>;// set your api key herelet apiKey='';</script>{#ifapiKey}<Realtimebind:startConversationbind:endConversationbind:isConnectedbind:isRecordingbind:startRecordingbind:stopRecordingbind:items{turnDetection}{apiKey}/>{/if}

seesrc/routes/+page.svelte for a full example.

seesrc/routes/visulizations-chat/+page.svelte for an example of how to visualize the audio input and output. andsrc/routes/visulizations-input/+page.svelte for all currently available visualizations.

demo here:https://flo-bit.dev/svelte-openai-realtime-api/visualizations-input

for production use, you will need to use a relay server to use the realtime api.

1. addOPENAI_API_KEY to your.env file

2. tell the Realtime component to use the relay server:

<RealtimeuseRelayServer />

then you have two options:

change yourvite.config.ts to this:

import{realtimeWebSocketServer}from'./src/lib/realtime/realtime_server';import{sveltekit}from'@sveltejs/kit/vite';import{defineConfig}from'vite';exportdefaultdefineConfig({plugins:[sveltekit(),realtimeWebSocketServer]});

• copy therelay-server/ folder (identical to the code in theopenai-realtime-console repository)from this repository into your project

• install the dependencies:

npm i dotenv openai/openai-realtime-api-beta

• run the relay server:

$ node relay-server/index.js

• add the relay server url to your Realtime component:

<Realtime useRelayServer relayServer="http://localhost:8081" />

• show waveforms
• add tests
• update main demo ui
• add more documentation
  • simple setup client side, server side
  • all props
• tool calling
• events
  • onConnect
  • onDisconnect
  • onError
  • onInterrupt
  • onRealtimeEvent
  • onConversationUpdate
  • speechStart
  • speechStop
• error handling
• pause/resume recording
• send text message
• update props
  • instructions
  • voice
  • vad settings

MIT