Movatterモバイル変換

MediaStream Processing API

W3C Working Group Note 31 May 2012

This version:

http://www.w3.org/TR/2012/NOTE-streamproc-20120531/

Latest published version:

http://www.w3.org/TR/streamproc/

Latest editor's draft:

https://dvcs.w3.org/hg/audio/raw-file/tip/streams/StreamProcessing.html

Previous version:

http://www.w3.org/TR/2011/WD-streamproc-20111215/

Public Comments:

public-audio@w3.org

Working Group:

Audio WG

Editor:

Robert O'Callahan, Mozilla Corporation <robert@ocallahan.org>

Copyright © 2012W3C^® (MIT,ERCIM,Keio), All Rights Reserved. W3Cliability,trademark anddocument use rules apply.

---

Table of Contents

1. 1. Introduction
   1. 1.1. Scenarios
2. 2. MediaStreams
   1. 2.1. The Semantics Of MediaStreams
   2. 2.2. Media Formats
   3. 2.3. MediaStream Extensions
3. 3. Media Elements
   1. 3.1. Media Element Extensions
   2. 3.2. Audio Element Extensions
4. 4. Stream Mixing And Processing
   1. 4.1. Time-varying Attributes
   2. 4.2. ProcessedMediaStream
   3. 4.3. MediaInput
   4. 4.4. Worker Processing
5. 5. Built-In Processing Engines
   1. 5.1. Default Processing Engine
   2. 5.2. "LastInput" Processing Engine
6. 6. Media Graph Considerations
7. 7. Canvas Recording
8. 8. Implementation Considerations
9. 9. Examples

1. Introduction

The ideas here build onIan Hickson's proposal for HTML Streams, adding features partly inspired by the Mozilla audio API andthe Chrome audio API. Unlike previous audio API proposals, the API presented here integrates with proposed API for media capture from local devices, integrates with proposed API for peer-to-peer media streaming, handles audio and video in a unified framework, incorporates Worker-based Javascript audio processing, and specifies synchronization across multiple media sources and effects. The API presented here does not include a library of "native" effects; those should be added as new "named effects" (described below), perhaps as a "level 2" spec.

The work here is nascent. Until a prototype implementation exists, this proposal is likely to be incomplete and possibly not even implementable.

1.1. Scenarios

These are concrete usage scenarios that have helped guide the design of the API. They are higher-level than use-cases.

1. Play video with processing effect applied to the audio track (e.g. high-pass filter)
2. Play video with processing effects mixing in out-of-band audio tracks (in sync) (e.g. mixing in an audio commentary with audio ducking)
3. Capture microphone input and stream it out to a peer with a processing effect applied to the audio (e.g. XBox 360 chat with voice distortion)
4. Capture microphone input and visualize it as it is being streamed out to a peer and recorded (e.g. Internet radio broadcast)
5. Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record (e.g. Internet radio broadcast)
6. Receive audio streams from peers, mix them with spatialization effects, and play (e.g. live chat with spatial feature)
7. Seamlessly chain from the end of one input stream to another (e.g. playlists, audio/video editing)
8. Seamlessly switch from one input stream to another (e.g. adaptive streaming)
9. Synthesize samples from JS data (e.g. game emulators or MIDI synthesizer)
10. Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking (e.g. game sound effects)
11. Trigger a sound sample to be played through the effects graph at a given time (e.g. game sound effects)
12. Capture video from a camera and analyze it (e.g. face recognition)
13. Capture video and audio, record it to a file and upload the file (e.g. Youtube upload)
14. Capture video from a canvas element, record it and upload (e.g. Screencast/"Webcast", or composite multiple video sources with effects into a single canvas then record)

2. MediaStreams

2.1. The Semantics Of MediaStreams

The description of MediaStreams here extends and must remain compatible with

Each MediaStream DOM object has an underlying media stream. The underlying media streams form a graph;

To avoid interruptions due to script execution, script execution can overlap with media stream processing;

• Changes to the MediaStream and MediaInput DOM objects are batched together betweenstable states (as defined by the HTML spec, while script is not running), and propagate as an atomic change to the media stream graph to take effect after some delay. This ensures that incomplete changes to the media stream graph during script execution do not result in transient glitches in the output, and that scripted graph changes do not interfere with concurrent media processing.
  Thus the following script would never cause an interruption in audio output, since no stable state occurs between the two volume changes:

    stream.inputs[0].volume = 0;  if (needToPlay()) {    stream.inputs[0].volume = 1.0;  }

  Specify exactly which attributes (and methods) are subject to this regime, possibly extending to attributes and methods already defined in HTML for media elements etc.

• State changes from the media stream graph are only propagated back to the MediaStream DOM objects during astable state. This ensures that between stable states (usually, during the execution of a script event handler), MediaStream DOM APIs always reflect a consistent snapshot of the state of the media stream graph.

In this spec, references toMediaStreams andMediaInputs refer to the DOM-visible state, and references tomedia streams andinput ports refer to the underlying real-time media stream graph.

A stream is an abstraction of a time-varying video and/or audio signal. At a given point in time, a media stream can beblocked, that is, not playing for some reason. All non-blocked streams play at the same constant rate --- real time. Streams cannot be seeked or played at a rate other than real time. For convenience we define a stream's "current time" as the duration it has played since it was created, but (unless blocking is involved) streams do not have independent timelines; they are synchronized.

At the implementation level, and when processing media data with Workers, we assume that each stream has a buffered window of media data available, containing the sample it is currently playing (or that it will play the next time it unblocks). This buffer defines the stream's contents into the future.

A stream can be in thefinished state. A finished stream is always blocked and can

A stream that has no consumers must block. Stream consumers defined in this specification are media elements andProcessedMediaStreams (see below). This avoids situations where streams that aren't connected

A media stream contains video and audio tracks. Tracks can start and end at any time. Each track

2.2 Media Formats

This spec mostly treats the formats used for stream audio and video data

Built-in audio processing filters guarantee that if all the audio inputs constantly have the same uncompressed format

When samples are exposed to a Worker for processing, the user-agent chooses a fixed uncompressed audio format (sample rate and channel configuration) for its inputs and outputs; see section 4.4.

However, suggested resampling algorithms will be provided in an appendix.

2.3 MediaStream Extensions

partial interface MediaStream {  readonly attribute double currentTime;  ProcessedMediaStream createProcessor(optional in DOMString namedEffect);  ProcessedMediaStream createWorkerProcessor(in Worker worker);};

ThecurrentTime attribute returns the amount of time that thisMediaStream has played since it was created.

ThecreateProcessor(namedEffect) method returns a newProcessedMediaStream with thisMediaStream as its sole input.

ThecreateWorkerProcessor(worker) method returns a newProcessedMediaStream with thisMediaStream as its sole input.

Add event handlers or callback functions for all finished and blocking state changes?

3. Media Elements

3.1 Media Element Extensions

We extend HTML media elements to produce and consume streams. When an HTML media element

partial interface HTMLMediaElement {  readonly attribute MediaStream stream;   MediaStream captureStream();  MediaStream captureStreamUntilEnded();  readonly attribute boolean audioCaptured;  attribute any src;};

Thestream attribute returns a stream which always plays whatever the element is playing. The stream is blocked while the media element is not playing. It is never finished, even when playback ends

Currently the HTML media element spec says thatplaying would fire on an element that is able to play except that a downstreamMediaController is blocked. This is incompatible with the above. I think that part of the HTML media spec should be changed so that only elements that are actually going to play fireplaying.

ThecaptureStream() method returns a newMediaStream that plays the same audio and video asstream.

ThecaptureStreamUntilEnded() method returns a newMediaStream that plays the same audio and video asstream, until the element next reaches

While the media element is playing a resource whose origin is not the same as the media element itself,MediaStreams for the element do not play the content; they block.

This prevents leaking of media contents to scripts in the page. In the future we could relax this and allow the streams to play as long as there's no scripted processing of the data downstream, but that's trickier.

While theaudioCaptured attribute is true, the element does not produce direct audio output.

Thesrc attribute is extended to allow it to be set to aMediaStream. The element

TheURL.createObjectURL(stream) method defined for HTML MediaStreams can create a URL to be

3.2 Audio Element Extensions

We add anAudio constructor taking aMediaStream as a parameter.

[NamedConstructor=Audio(MediaStream src)]partial interface HTMLAudioElement {}

4. Stream Mixing And Processing

4.1 Time-varying Attributes

To enable precise control over the timing of attribute changes, many attributes can be set using a

Using a timed setter method never changes the observed attribute value immediately. Setter method changes always take effect after the nextstable state, as described in section 2.1. Setting the attribute value changes the observed attribute value immediately, but the change to the underlying media stream will still not take effect until after

Multiple pending changes to an attribute are allowed. Calling the setter method with

4.2 ProcessedMediaStream

AProcessedMediaStream combines zero or more input streams and applies some processing to

[Constructor(), Constructor(in Worker worker, in optional long audioSampleRate, in optional short audioChannels)]interface ProcessedMediaStream : MediaStream {  readonly attribute MediaInput[] inputs;  MediaInput addInput(in MediaStream inputStream, in optional double outputStartTime, in optional double inputStartTime);  attribute any params;  void setParams(in any params, in optional double startTime);  readonly attribute boolean autofinish;};

The constructors create a newProcessedMediaStream with no inputs.

Specify valid values foraudioChannels andaudioSampleRate.

Theinputs attribute returns an array ofMediaInputs, one for

AMediaInput represents an input port. An input port isactive while it is enabled (see below) and the input stream is not finished.

TheaddInput(inputStream, outputStartTime, inputStartTime) method adds a newMediaInput to the end of the

More precisely, when theaddInput call takes effect (see section 2.1),

1. If theoutputStartTime was omitted, set it to the output stream's current time.
2. If theinputStartTime was omitted, set it to the input stream's current time.
3. Compute thedeadline miss delay: the maximum of
   • The input stream's curent time minusinputStartTime
   • The output stream's current time minusoutputStartTime
4. If the deadline miss delay is greater than zero, add it to theinputStartTime andoutputStartTime. (This would be a good place for user-agents to emit a developer-accessible warning.)
5. While the input stream's current time is less thaninputStartTime, or the output stream's current time is less thanoutputStartTime:
   • Whenever the input stream's current time is equal toinputStartTime, block the input stream.
   • Whenever the output stream's current time is equal tooutputStartTime, block the output stream.
6. Enable the input port.

  var p = inputStream.createProcessor();  p.addInput(inputStream, 5);

Theparams attribute and thesetParams(params, startTime) timed setter method set the paramters for this stream. On setting, astructured clone of this object is made. The clone is sent to

When an input stream finishes, at the next stable state anyMediaInputs for that

When theautofinish attribute is true, then when all stream inputs are finished (including if there are no inputs), the stream will automatically enter the finished state and never produce any more output (even if new inputs are attached).

4.3 MediaInput

AMediaInput object controls how an input stream contributes to the combined stream.

interface MediaInput {  readonly attribute MediaStream stream;  MediaInput addFollowing(MediaStream inputStream);  attribute double volume;  void setVolume(in double volume, in optional double startTime, in optional double fadeTime);  attribute any params;  void setParams(in any params, in optional double startTime);  attribute boolean blockInput;  attribute boolean blockOutput;  void remove(in optional double time);};

Thestream attribute returns theMediaStream connected to this input.

TheaddFollowing(inputStream) method adds a newMediaInput to

Sequencing media resources using media element durations is possible, e.g.processor.addInput(video1.captureStreamUntilEnded()); processor.addInput(video2.captureStreamUntilEnded(),video1.duration);. However, media element durations may not be perfectly reliable due to limitations of resource formats or commonly-used encoders. For example, WebM resources often have a slight mismatch between the number of audio samples and the duration metadata. For other resources, such as live streaming sources, the duration may not be knowable in advance. Thus, using the duration is often unreliable and additional API is warranted.

Add additional API to select particular tracks.

Thevolume volume attribute and thesetVolume timed setter method

Specify the exact transition function. Tim says "w=cos((pi/2)*t)^2 for t=0...1".

Theparams attribute and thesetParams(params, startTime) timed setter method set the paramters for this input. On setting, astructured clone of this object is made. The clone is sent to

For the timed setter methods ofMediaInput, the subject stream is the output stream, so changes take effect when the output stream's current time is equal tostartTime.

TheblockInput andblockOutput attributes control

Need to look again at these. It's not clear we have use cases for both attributes, and I haven't implemented them yet and they could be hard to implement.

Theremove(time) method removes thisMediaInput from the inputs array of its owning

4.4 Worker Processing

AProcessedMediaStream with a worker computes its output by dispatching a sequence ofonprocessmedia callbacks to the worker, passing each aProcessMediaEvent parameter. AProcessMediaEvent provides audio sample buffers for each input stream. Each sample buffer for a givenProcessMediaEvent has the same duration, so the inputs presented to the worker are always in sync. (Inputs may be added or removed betweenProcessMediaEvents, however.) The sequence of buffers provided for an input stream is the audio data to be played by that input stream. The user-agent will precompute data for the input streams as necessary.

For example, if a Worker computes the output sample for time T as a function of the [T - 1s, T + 1s] interval of an input stream, then initially the Worker would simply refuse to output anything until it has received at least 1s of input stream data, forcing the user-agent to precompute the input stream at least 1s ahead of the current time. (Note that large Worker latencies will increase the latency of changes to the media graph.)

Note thatWorkers do not have access to most DOM API objects. In particular,Workers have no direct access toMediaStreams.

Note that aProcessedMediaStream's worker cannot be aSharedWorker. This ensures that the worker can run in the same process as the page in multiprocess browsers, so media streams can be confined to a single process.

CurrentlyProcessMediaEvent does not offer access to video data. This should be added later.

partial interface DedicatedWorkerGlobalScope {  attribute Function onprocessmedia;};

Theonprocessmedia attribute is the function to be called whenever stream data needs to be processed.

interface ProcessMediaEvent : Event {  readonly attribute double inputTime;  readonly attribute any params;  readonly attribute double paramsStartTime;  readonly attribute MediaInputBuffer inputs[];  readonly attribute long audioSampleRate;  readonly attribute short audioChannels;  reaodnly attribute long audioLength;  void writeAudio(in Float32Array data);  void finish();};

TheinputTime attribute returns the duration of the input that has been consumed by the

Theparams attribute provides a structured clone of the parameters object set by

TheparamsStartTime attribute returns the first time (measured in duration of input consumed for this stream) that thisparams object was set.

Note that the parameters objects are constant over the duration of the inputs presented in the event. Frequent changes to parameters will reduce the length of the input buffers that can be presented to the worker.

inputs provides access toMediaStreamBuffers for each active input stream

audioSampleRate andaudioChannels represent the format of the input and

audioLength is the duration of the input(s) multiplied by the sample rate. If there are no inputs,

writeAudio(data) writes audio data to the stream output.

It is permitted to write less audio than the duration of the inputs (including none). This indicates latency in the filter. Normally the user-agent will dispatch another event to provide

A synthesizer with no inputs can output as much data as it wants; the UA will buffer data and fire events as necessary. Filters that misbehave, e.g. by always writing zero-length buffers, will cause the stream to block due to an underrun.

IfwriteAudio is not called during the event handler, then the input audio buffers are added together and written to the output.

IfwriteAudio is called outside the event handler, the call is ignored.

Callingfinish() puts the stream into the finished state (once any previously buffered output has been consumed). The event callback will never be called again.finish() can be called at any time, inside or outside the event handler.

The output video track is computed as if there was no worker (see above).

This will change when we add video processing.

interface MediaInputBuffer {  readonly attribute any params;  readonly attribute double paramsStartTime;  readonly attribute Float32Array audioSamples;};

Theparams attribute provides a structured clone of the parameters object set by

TheparamsStartTime attribute returns the first time (measured in duration of input consumed for this stream) that thisparams object was set.

audioSamples gives access to the audio samples for each input stream. The array length will beevent.audioLength multiplied byevent.audioChannels. The samples are floats ranging from -1 to 1, laid out non-interleaved, i.e. consecutive segments ofaudioLength samples each. The durations of the input buffers for the input streams will be equal. TheaudioSamples object will be a fresh object in each event. For inputs with no audio track,audioSamples will be all zeroes.

5. Built-In Processing Engines

5.1. Default Processing Engine

AProcessedMediaStream with the default processing engine produces output as follows:

• If no active input has an audio track, the output has no audio track. Otherwise, the output has a single audio track whose metadata (id,kind,label, andlanguage) is equal to that of the audio track for the last active input that has an audio track. The output audio track is produced by adding the samples of the audio tracks of the active inputs together.
• If no active input has a video track, the output has no video track. Otherwise, the output has a single video track whose metadata (id,kind,label, andlanguage) is equal to that of the video track for the last active input that has a video track. The output video track is produced by compositing together all the video frames from the video tracks of the active inputs, with the video frames from higher-numbered inputs on top of the video frames from lower-numbered inputs; each video frame is letterboxed to the size of the video frame for the last active input that has a video track.

  This means if the last input's video track is opaque, the video output is simply the video track of the last input.

5.2. "LastInput" Processing Engine

AProcessedMediaStream with the "LastInput" processing engine simply produces the last active input stream as output. If there are no active input streams, it produces the same output as the default processing engine.

6. Media Graph Considerations

6.1. Cycles

While aProcessedMediaStream has itself as a direct or indirect input stream (considering only active inputs), it is blocked.

6.2. Blocking

At each moment, every stream should not be blocked except as explicitly required by this specification.

7. Canvas Recording

To enable video synthesis and some easy kinds of video effects we can record the contents of a canvas:

partial interface HTMLCanvasElement {  readonly attribute MediaStream stream;};

Thestream attribute is a stream containing a video track with the "live" contents of the canvas as video frames whose size is the size of the canvas, and no audio track. It always returns the same stream for a given element.

8. Implementation Considerations

Here will be some non-normative implementation suggestions.

9. Examples

Add Worker scripts for these examples.

1. Play video with processing effect applied to the audio track

   <video src="foo.webm" controls></video><audio autoplay></audio><script>document.getElementById("out").src =   document.getElementById("v").captureStream().createWorkerProcessor(new Worker("effect.js"));</script>

2. Play video with processing effects mixing in out-of-band audio tracks (in sync)

   <video src="foo.webm"></video><audio src="back.webm"></audio><audio autoplay></audio><script>  var mixer = document.getElementById("v").captureStream().createWorkerProcessor(new Worker("audio-ducking.js"));  mixer.addInput(document.getElementById("back").captureStream());  document.getElementById("out").src = mixer;  function startPlaying() {    document.getElementById("v").play();    document.getElementById("back").play();  }  // MediaController is a more convenient API because it ties together control of the elements,  // but using streams is more flexible (e.g. they can be seeked to different offsets).</script>

3. Capture microphone input and stream it out to a peer with a processing effect applied to the audio

   <script>  navigator.getUserMedia('audio', gotAudio);  function gotAudio(stream) {    peerConnection.addStream(stream.createWorkerProcessor(new Worker("effect.js")));  }</script>

4. Capture microphone input and visualize it as it is being streamed out to a peer and recorded

   <canvas></canvas><script>  navigator.getUserMedia('audio', gotAudio);  var streamRecorder;  function gotAudio(stream) {    var worker = new Worker("visualizer.js");    var processed = stream.createWorkerProcessor(worker);    worker.onmessage = function(event) {      drawSpectrumToCanvas(event.data, document.getElementById("c"));    }    streamRecorder = processed.record();    peerConnection.addStream(processed);  }</script>

5. Capture microphone input, visualize it, mix in another audio track and stream the result to a peer and record

   <canvas></canvas><audio src="back.webm"></audio><script>  navigator.getUserMedia('audio', gotAudio);  var streamRecorder;  function gotAudio(stream) {    var worker = new Worker("visualizer.js");    var processed = stream.createWorkerProcessor(worker);    worker.onmessage = function(event) {      drawSpectrumToCanvas(event.data, document.getElementById("c"));    }    var mixer = processed.createProcessor();    mixer.addInput(document.getElementById("back").captureStream());    streamRecorder = mixer.record();    peerConnection.addStream(mixer);  }</script>

6. Receive audio streams from peers, mix them with spatialization effects, and play

   <script>  var worker = new Worker("spatializer.js");  var spatialized = stream.createWorkerProcessor(worker);  peerConnection.onaddstream = function (event) {    spatialized.addInput(event.stream).params = {x:..., y:..., z:...};  };  (new Audio(spatialized)).play();</script>

7. Seamlessly chain from the end of one input stream to another

   This method requires that you know each stream's duration, which is a bit suboptimal. To get around that we'd need new API, perhaps a new kind of ProcessedMediaStream that plays streams in serial.

   <audio src="in1.webm" preload></audio><audio src="in2.webm"></audio><script>  var in1 = document.getElementById("in1");  in1.onloadedmetadata = function() {    var mixer = in1.captureStreamUntilEnded().createProcessor();    var in2 = document.getElementById("in2");    mixer.addInput(in2.captureStreamUntilEnded(), in1.duration);    (new Audio(mixer)).play();    in1.play();  }</script>

8. Seamlessly switch from one input stream to another, e.g. to implement adaptive streaming

   There are two ways to implement seamless switching: seek the second resource to before the current time and then run the decoder faster than real-time to catch up to the first resource's play point, or seek the second resource to after the current time and enable it when the first resource catches up to the seek point. The first is more robust if the seek takes unexpectedly long, but the second is less demanding on the decoder. Only the second method is currently implementable with this API (since by design there is no way to drive MediaStreams faster than real-time). If we want to support the first method as well, the right way would be to add API to media elements to let them seek to synchronize with a given MediaStream.

   <audio src="in1.webm" preload></audio><audio src="in2.webm"></audio><audio autoplay></audio><script>  var stream1 = document.getElementById("in1").captureStream();  var mixer = stream1.createProcessor("LastInput");  document.getElementById("out").src = mixer;  function switchStreams() {    var in2 = document.getElementById("in2");    in2.currentTime = in1.currentTime + 10; // arbitrary, but we should be able to complete the seek within this time    mixer.addInput(in2.captureStream(), mixer.currentTime + 10);    in2.play();    // in2 will be blocked until the input port is enabled    in2.onplaying = function() { mixer.inputs[0].remove(); };  }</script>

9. Synthesize samples from JS data

   <audio autoplay></audio><script>  document.getElementById("out").src =    new ProcessedMediaStream(new Worker("synthesizer.js"));</script>

10. Trigger a sound sample to be played through the effects graph ASAP but without causing any blocking

    <script>  var effectsMixer = ...;  function playSound(src) {    var audio = new Audio(src);    audio.oncanplaythrough = function() {      var stream = audio.captureStreamUntilEnded();      var port = effectsMixer.addInput(stream);      port.blockOutput = false;      audio.play();    }  }</script>

11. Trigger a sound sample to be played through the effects graph in five seconds

    <script>  var effectsMixer = ...;  var audio = new Audio(...);  function triggerSound() {    var sound = audio.clone();    var stream = sound.captureStreamUntilEnded();    sound.play();    effectsMixer.addInput(stream, effectsMixer.currentTime + 5);  }</script>

12. Capture video from a camera and analyze it (e.g. face recognition)

    <script>  navigator.getUserMedia('video', gotVideo);  function gotVideo(stream) {    stream.createWorkerProcessor(new Worker("face-recognizer.js"));  }</script>

13. Capture video, record it to a file and upload the file (e.g. Youtube)

    <script>  navigator.getUserMedia('video', gotVideo);  var streamRecorder;  function gotVideo(stream) {    streamRecorder = stream.record();  }  function stopRecording() {    streamRecorder.getRecordedData(gotData);  }  function gotData(blob) {    var x = new XMLHttpRequest();    x.open('POST', 'uploadMessage');    x.send(blob);  }</script>

14. Capture video from a canvas, record it to a file then upload

    <canvas width="640" height="480"></canvas><script>  var canvas = document.getElementById("c");    var streamRecorder = canvas.stream.record();  function stopRecording() {    streamRecorder.getRecordedData(gotData);  }  function gotData(blob) {    var x = new XMLHttpRequest();    x.open('POST', 'uploadMessage');    x.send(blob);  }  var frame = 0;  function updateCanvas() {    var ctx = canvas.getContext("2d");    ctx.clearRect(0, 0, 640, 480);    ctx.fillText("Frame " + frame, 0, 200);    ++frame;  }  setInterval(updateCanvas, 30);</script>

Related Work

• W3C-RTC charter (Harald et. al.)
• WhatWG proposal (Ian Hickson et. al.)
• Chrome audio API
• Mozilla audio API
• WhatWG MediaController API