@kalradivyanshu@fippo PTAL.

@aboba Looks good. So we just set L3T3 in encoder, and each frame will tell us which spatial and temporal layer it belongs in, and which frames are its dependencies, then in the decoder nothing changes, we just make sure all dependencies are fed in before feeding in the frame, and it just works, right?

For a frame to be decodable, all its dependencies need to have been received and decoded without an error callback. From the conference server perspective, this means not only tracking what frames were sent to each participant, but also the transport status (whether the frame was completely received) and whether it was successfully decoded. Currently the underlying encoder API limits avenues available for repair to keyframe generation, retransmission and forward error correction. Alternate Long Term Reference (LTR) frames or layer refresh (LRR) are not yet supported.

@tonyherre PTAL

Thankyou so much for this@aboba , what all is left in this PR to get it accepted in the spec?

@kalradivyanshu It has been noted that spatial scalability is not widely used today because it is not hardware accelerated and therefore create power and thermal issues on mobile devices. As a result, applications are using spatial simulcast instead. Also, the current WebCodecs API does not support layer refresh, which means that if a spatial frame is lost, a base layer keyframe is required, rather than just creating a new spatial frame referencing a received base-layer frame (e.g. moving to a new Long-Term Reference).

@Djuffin has argued that these problems need to be fixed before spatial scalability could become popular in WebCodecs, and therefore that it would make sense to focus on a new encoder API that can address the problems rather than just shipping a (potentially unusable) feature.

Oh ok. Couple of things:

1. Spatial scalability is critical for low network restraint according to me, whenever thinking about video streaming/communication, its always a trade off b/w cpu and network, yes CPU used will be more, but simulcast needs keyframes whenever switching layers (specially since for AV1 webcodecs doesnt support switch framesSupport for AV1 switch frames #747). And even with hardware acceleration, keyframes are notoriouslya lot bigger, so not only will they clog the network, but will cause issues with CPU as well.

2. If I am using simulcast, if I lose a frame, I still have to rely on a keyframe, so its still a problem as I have mentioned, switching layers mean another keyframe, so either application have to regularly generate keyframe, or create feedback machanisms like PLI, which cause scale issues when broadcasting to larger crowd.

While I agree with the issues@Djuffin raised, I honestly feel that since the new API is atleast a year away, spatial scalability should be added or at the very least stuff like switch frames should be added to make simulcast more usable. Without any of these, the only solution is to do simulcast with keyframes requests for every switch, which in turn will add a huge load on encoder and also the decoder and network.

Thank you both for all your work!