Insertable Streams for MediaStreamTrack API

Limited availability

This feature is not Baseline because it does not work in some of the most widely-used browsers.

Experimental:This is anexperimental technology
Check theBrowser compatibility table carefully before using this in production.

Note: This feature is only available inDedicated Web Workers.

TheInsertable Streams for MediaStreamTrack API provides a way to process the video frames of aMediaStreamTrack as they are consumed.

Concepts and Usage

When processing real-time video, you sometimes want to insert visual elements or otherwise process the stream of video frames. For example, an application might include two tracks that need to be combined, such as a weather map and video of a presenter explaining the map. Or, you may want to do processing on a track to blur backgrounds, or introduce other elements (such as adding funny hats to people, and so on). The APIs described here provide direct access to the video stream, allowing you to manipulate it in real time.

To ensure optimal performance, the APIs are only available indedicated workers (unless otherwise stated).

Interfaces

MediaStreamTrackProcessorExperimental: Consumes a MediaStreamTrack object's source and produces a stream of video frames.
VideoTrackGeneratorExperimental: Creates a WritableStream that acts as aMediaStreamTrack video source.
MediaStreamTrackGeneratorExperimental Non-standard: Creates a WritableStream that acts as aMediaStreamTrack source for either video or audio. Only available on themain thread.

Examples

The following example is from the articleUnbundling MediaStreamTrackProcessor and VideoTrackGenerator. Ittransfers a cameraMediaStreamTrack to a worker for processing. The worker creates a pipeline that applies a sepia tone filter to the video frames and mirrors them. The pipeline culminates in aVideoTrackGenerator whoseMediaStreamTrack is transferred back and played. The media now flows in real time through the transform off themain thread.

const stream = await navigator.mediaDevices.getUserMedia({ video: true });const [track] = stream.getVideoTracks();const worker = new Worker("worker.js");worker.postMessage({ track }, [track]);const { data } = await new Promise((r) => (worker.onmessage = r));video.srcObject = new MediaStream([data.track]);

worker.js:

onmessage = async ({ data: { track } }) => {  const vtg = new VideoTrackGenerator();  self.postMessage({ track: vtg.track }, [vtg.track]);  const { readable } = new MediaStreamTrackProcessor({ track });  await readable    .pipeThrough(new TransformStream({ transform }))    .pipeTo(vtg.writable);};