electron/shell/common/api/shared_texture

Importing Shared Texture

This document describes the design of the sharedTexture API that imports external shared textures into Electron as a VideoFrame. Written based on Electron 37 and Chromium 137. Note that Chromium's SharedImage interfaces may iterate quickly, especially regarding the lifecycle management of SharedImage-related resources, including GpuMemoryBuffer and VideoFrame.

Design

The main goal of the current implementation is to import external shared texture descriptions while reusing as much as possible of the existing Chromium infrastructure. I chose SharedImage as the underlying holder of the external texture because it provides SharedImageInterface::CreateSharedImage, which accepts a GpuMemoryBufferHandle containing native shared handle types: NT HANDLE for Windows, IOSurfaceRef for macOS, and NativePixmapHandle with file descriptors for each plane for Linux.

However, I encountered multiple challenges during implementation. The current implementation was designed by overcoming these obstacles:

1. Shared handle is local to process

   For Windows, a shared D3D11 texture can be created by GetSharedHandle (deprecated) or CreateSharedHandle. The deprecated method generates a non-NT HANDLE that can be globally accessed, while the newer one generates an NT HANDLE that is local to the current process. To share it with other processes, you need to call DuplicateHandle to create this handle in the remote process.

   For macOS, IOSurface can also be global by setting kIOSurfaceIsGlobal, which is also a deprecated option. If you want to share an IOSurface with other processes, you need to create a mach_port from it and pass the mach_port through a previously created IPC (which also uses mach_port as transport), making it significantly more complex than Windows.

   Given these obstacles, you must ensure that when calling sharedTexture.importSharedTexture, the handle is already available to the current process. You might be able to use global IOSurface, but a non-NT HANDLE is not an option as described in problem 2 below. In fact, Chromium's IPC internally handles all the concerns about this (duplicate handles for remote processes, passing mach_port through mach_port), which is why the OSR paint event can use the handle directly - because IPC has transparently handled these issues.

2. Shared handle ownership management

   Chromium takes ownership of the GpuMemoryBuffer and its native representation. Once the resource is destroyed, the handle will be closed.

   For Windows, calling CloseHandle on a non-NT HANDLE is an invalid operation and will cause Chromium to crash. Therefore, you cannot use a non-NT HANDLE when importing. In the future, we may provide a helper for this. To work with this design, when calling importSharedTexture, an NT HANDLE will be duplicated.

   For macOS, IOSurface is a reference-counted resource. When calling importSharedTexture, instead of taking ownership, we can simply let Chromium retain this resource and increment the reference count.

3. Transfer the shared texture between processes

   I initially considered using WebGPU Dawn Native API to import the external texture as a WGPUSharedTextureMemory, but encountered more problems. For example, it was unable to export, difficult to manage the lifetime of a frame, and working with WebGPU was non-ideal.

   SharedImage has advantages when it comes to sharing across processes because it holds a reference to a Mailbox, which points to the corresponding SharedImageBacking in the GPU process. Therefore, I use SharedImageInterface->ImportSharedImage and ClientSharedImage->Export to serialize sufficient information to retrieve the SharedImage reference in another process, and it can also reuse the mojo serializer to serialize as a string.

4. Interprocess resource management

   The final obstacle is managing the lifetime of a frame. Currently, I use OSR to get an exported shared texture generated by Chromium itself. As the documentation states, the texture needs to be manually released. By importing this texture into a SharedTextureImported in Electron, we must ensure the imported one is released before the source texture is released.

   What's more challenging is that the paint event occurs in the main process, while we have to render in renderer processes, so we must use startTransferSharedTexture to pass the underlying SharedImage to another process.

   Most GPU calls are asynchronous, sending to the command buffer of the GPU process through the GpuChannel of each client process. When we have two SharedImage instances referencing the same Mailbox in two different processes, we don't know when the GPU has finished using the resources. Typically, this is guaranteed by SyncToken. For example, we can simply schedule the destruction of a SharedImage with an empty SyncToken in the main process (where the texture was first imported), but when we use the same SharedImage in the renderer process and use it in WebGPU (or WebGL) pipelines, the destruction token will be generated by WebGPU to prevent destruction before the GPU uses it. The destruction won't occur until WebGPU rendering finishes.

   Ideally, if we are in Chromium, we can use mojo to create a PendingRemote callback and update the main process destruction SyncToken to prevent the main process from releasing the frame before the GPU starts working on it. In the future, we may implement this in Electron code to wrap the mojo functionality. Eventually, I found a way to use gpu::ContextSupport and register a callback when a specific SyncToken is released (signaled). When you call release() on the imported shared texture object, if you've used VideoFrame and imported it into a WebGPU pipeline, it will wait for WebGPU to finish rendering, then run a callback to notify you to release dependent resources, such as the original imported object in the main process, the source texture, etc.

Example

I use built-in OSR offscreen: { useSharedTexture: true } to obtain a exported shared texture, but you can use whatever you want with same requirements, you can even import at renderer process, in case you choose to load your native code in renderer process and correctly put the handle under that process. However, you need to make sure the shared handle is visible to that process.

To read the example, visit the test spec of this feature, start from here. There's a detailed step by step comment in it, you'll also need to read preload and renderer code to navigate throught all steps.