WebRTC encoded transform - Modify Metadata functions

Category: Realtime / Communication
Type: New or changed feature
Status: Enabled by default (Chrome 130)
Intent stage: Start prototyping

Summary

Allow WebRTC Encoded Transform API to manipulate audio and video frame metadata. Some WebRTC Encoded Transform use cases involve manipulation of not only the payload of encoded video / audio frames but also its metadata. Some examples: * Altering the timestamp of a frame to introduce a delay, * Changing the mime type of the frame if the transform changes the type of the payload. * Forwarding of media to a new peer connection set up to use different metadata values Use cases: https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#auction Issue link: https://github.com/w3c/webrtc-nv-use-cases/issues/77

Motivation

This feature is needed to support use cases that involve manipulation of encoded video / audio frames beyond inline processing of its payload. We want to support the creation of new frames with custom metadata. Some example use cases: * Altering the timestamp of a frame to introduce a delay, * Forwarding of media to a new peer connection set up to use different metadata values * Changing the mime type of the frame if the transform changes the type of the payload. Use cases: https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#auction

Standards & signals

Specification: https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor
Firefox: Positive — During WebRTC WG meetings, Mozilla has shown positive signals and agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223
Safari: Closed Without a Position — Marked as "Invalid" in the position request because this is a small change and the position was addressed in the PR review. The comments from WebKit in the PR review and during WebRTC WG meetings are positive and they have agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223
Web developers: Positive
Tracking bug: https://issues.chromium.org/issues/40248396

Explainers: https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

View on chromestatus.com