Streaming

AI providers can be slow when producing responses, but many make their responses available in chunks as they're processed. Streaming enables you to show users those chunks of data as they arrive rather than waiting for the full response, improving the perceived speed of AI-powered apps.

Vercel recommends usingVercel's AI SDK to stream responses from LLMs and AI APIs. It reduces the boilerplate necessary for streaming responses from AI providers and allows you to change AI providers with a few lines of code, rather than rewriting your entire application.

The following example shows how to send a message to one of OpenAI's models and streams:

1. You should understand how to setup a Vercel Function. See theFunctions quickstart for more information.
2. You should also have a fundamental understanding of how streaming works on Vercel. To learn more seeWhat is streaming?.
3. You should be using Node.js 18 or later and thelatest version of the Vercel CLI.
4. You should copy your OpenAI API key in the.env.local file with nameOPENAI_API_KEY. See theAI SDK docs for more information on how to do this.
5. Install theai and@ai-sdk/openai packages:
   pnpmyarnnpmbun

   pnpm i ai openai

import { streamText }from'ai';import { openai }from'@ai-sdk/openai';// This method must be named GETexportasyncfunctionGET() {// Make a request to OpenAI's API based on// a placeholder promptconstresponse=streamText({    model:openai('gpt-4o-mini'),    messages: [{ role:'user', content:'What is the capital of Australia?' }],  });// Respond with the streamreturnresponse.toTextStreamResponse({    headers: {'Content-Type':'text/event-stream',    },  });}

If your workload requires longer durations, you should consider enablingfluid compute, which hashigher default max durations and limits across plans.

Maximum durations can be configured for Node.js functions to enable streaming responses for longer periods. Seemax durations for more information.

You can stream responses from Vercel Functions that use the Python runtime.

When your function is streaming, it will be able to take advantage of the extendedruntime logs, which will show you the real-time output of your function, in addition to larger and more frequent log entries. Because of this potential increase in frequency and format, yourLog Drains may be affected. We recommend ensuring that your ingestion can handle both the new format and frequency.

• What is streaming?
• AI SDK
• Vercel Functions
• Fluid compute
• Streaming and SEO: Does streaming affect SEO?
• Processing data chunks: Learn how to process data chunks
• Handling backpressure: Learn how to handle backpressure