📦 Bundle size

Full ESM build, minified and gzipped.

🌳 Bundle size after tree-shaking

Minimal build including a renderer, camera, empty scene, and dependencies.

It took a bit to design how to go about it. The only issue is you have to toggle the video mesh layers when going into XR and ending the session. Maybe there is other ways to do it.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L130
https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L151

live examplehttps://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

I have tried to improve the JSDoc a bit. For a functional review I hope@cabanier can have a look.

BTW: Please create a E2E screenshot for the new examplewebgpu_xr_media_layer. You can do that withnpm run make-screenshot webgpu_xr_media_layer.

The requirement to toggle off the video texture with the meshes. I'm not sure how to make better. It causes a crash if the texture is also rendered.

I've refactored it quite a bit. And followed the structure of modifying the geometry externally to keep that external. I could add that in. It returns a group of meshes which works in 2D, Rift, Quest.

There is still the problem of needing to turn off meshes for native layers. Can be done internally or keep it external. I'm not across everything to figure out a better way to render from 2D to native.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

mono examplehttps://electroteque.org/dev/threejs/examples/webgpu_xr_media_mono_layer.html

Is this looking good setting the geometry externally.

I have it working implementing and tested. Unfortunately media layer needs to be added in the manager as doing it external caused a context loss fault. It may be a problem adding other layers externally as the media layers list needs to be prepended in the list as backgrounds as I have done.

The only issue still I can't figure out when going from the 2D mesh to native layer in XR. It will be rendered including the video texture in the projection layer as its added to the scene so those layers need to be turned off. Or maybe simply removing those mesh groups from the scene ?. I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

... doing it external caused a context loss fault.

Do you have an example for me to look at? You shouldn't be getting context losses.

I can look at doing it in the manager perhaps. Turn off mesh layers for all the media layers in the list on session start. Unless there is a better way to not render those meshes in the scene into the projection layer.

Yes, can't you do the same as what I did for quad/cylinder layer?

If the external geometry setter is ok. Or I make them default SphereGeometry requiring to update it's settings. I just set them to PlaneGeometry.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L105

I add the mesh group to the layers params.

https://github.com/danrossi/three.js/blob/equirect-layers/src/renderers/common/XRManager.js#L667

So this requirement to turn off texture rendering in the projection layer can be done internally perhaps and make it transparent.

https://github.com/danrossi/three.js/blob/equirect-layers/examples/webgpu_xr_media_layer.html#L114

There is an issue I can't figure out if it's the browser or three. Native media layer rendering is blank if it's an immersive-vr session. Only works in immersive-ar session. The controllers don't connect all the time either and can't figure out where to report. There is still the makeXRCompatible error in the console I can't find where to report.

It might need a blank render target like the other layer examples. When using immersive-vr mode instead of immersive-ar mode like the XRButton uses. The projection layer render target blocks out the background native media layer. So it seems it's a blank render.

https://github.com/mrdoob/three.js/blob/dev/src/renderers/common/XRManager.js#L1480
https://github.com/mrdoob/three.js/blob/dev/src/renderers/webgl-fallback/WebGLBackend.js#L354

https://electroteque.org/dev/threejs/examples/webgpu_vr_media_layer.html

Doing something like this in the renderLayers method seems to work but not sure if its worth it and just keep it an immersive-ar session. If it peforms the same. Is just to be able to not have the projection layer render target cover it up. I noticed all the demos show up weird because they are AR sessions though. So stuff around the camera.

for(constlayer2ofthis._mediaLayers){layer2.renderTarget.isXRRenderTarget=this._session!==null;layer2.renderTarget.hasExternalTextures=layer2.renderTarget.isXRRenderTarget;layer2.renderTarget.autoAllocateDepthBuffer=!layer2.renderTarget.isXRRenderTarget;if(layer2.renderTarget.isXRRenderTarget&&this._supportsLayers){constglSubImage=this._glBinding.getSubImage(layer2.xrlayer,this._xrFrame);this._renderer.backend.setXRRenderTargetTextures(layer2.renderTarget,glSubImage.colorTexture,glSubImage.depthStencilTexture);}}

I was able to figure out fixing the conflict in the editor but have been testing on a merged branch. I'm still unable to figure out how to work around three projection layer render target hiding the native layer in immersive-vr mode. Only works in immersive-ar mode. Which is what all the XR examples use.

Hopefully disabling the 2D mesh layer internally to not render in the projection layer is the right solution.

This might need some time before integrating if it all looks good. The media layer feature is incredibly unstable causing crashes. It can't be used in video playlists, still has crashes replacing video elements but emptying the video source first before replacement is less unstable. Exiting XR its as if the layers don't clear, and causes freezing, low memory and high cpu. It's not the integration or three however.

I'm still maintaining this and syncing fork. However the project I did this for is on hold due to the native layers crashing the browser very badly and not stable. Its a potential memory leak and causes 100% cpu spikes then crashes, reporting the crashes has done nothing. I also lost a week trying to recover from an update power cycle bug that crashed the filesystem it seems like in v72, wouldnt power down and kept rebooting and trying to run the same update. I think the OS is not production ready at all.

It seems unfit for purpose and not ready for production sadly. It performs better than with multiview in the projection layer however. But changing videos and swapping native layers to change the layout or angle between 360 and 180, and even exiting XR will make it crash badly eventually. Its cost me 30+ hours I cant recover making tests and reporting. And it's still a problem for a few months now.

I may have to convince them to use the non native layers configuration for as long as it takes after spending time integrating it all. To keep moving forward. But the project is also on hold also because of the regression 5.7k h265 unable to be played in the browser when i spent a month beforehand testing it then broke after updates, disrupting things. I can't test any of their content at all now in Quest 2 v78

update example. Trigger/select will replicate a video playlist change. 3-6 changes is enough to crash it.https://electroteque.org/dev/webxr-samples/layers/media-layer-sample.html

I've just finally caught another problem with the browser. If the video is added to the DOM and its a large size it will crash also regardless of layers. But the layers crash tests aren't in the DOM so it's a seperate issue.

So I've made hacks in my feature to remove the element from the DOM while in XR and add it back when exiting.

To get them going and still use native media layers for better framerate performance. I had to build a work around prototype of launching completely new players and XR sessions in a video grid. It thankfully works around user gestures as XR launch has to happen after the video is ready but the gesture happened to load the whole player and then the video first. When exiting it will dispose the player. Its vr for videojs features and it requires the element in the DOM.

They would still like a video item grid in XR sessions to switch videos or the controls to move to the next item which I have a controlbar for. So in the playlist option I've had to completely disable native layers features for now to avoid the crashes and it's fine.

I fixed up the demo to improve Quest performance hit issues. Remove video element from DOM and remove the whole container and canvas from the DOM while in XR and return after.

The large video still causes the OS to be sticky with movements and fan ramps up when exiting XR. So the layers are not clearing when exiting XR and causing cpu spikes it seems. Definitely a memory leak is happening I watch the memory drop when re-entering and exiting XR.

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

I fixed up the demo to improve Quest performance hit issues. Remove video element from DOM and remove the whole container and canvas from the DOM while in XR and return after.

The large video still causes the OS to be sticky with movements and fan ramps up when exiting XR. So the layers are not clearing when exiting XR and causing cpu spikes it seems. Definitely a memory leak is happening I watch the memory drop when re-entering and exiting XR.

https://electroteque.org/dev/threejs/examples/webgpu_xr_media_layer.html

Quest browser 39.3 will release soon with a potential fix for this.