An Introduction to the Supplemental Enhancement Information Message for Coding Video with Constituent Rectangles

The Constituent Rectangles Supplemental Enhancement Information (SEI) message is under development by the Joint Video Experts Team (JVET) for coding videos formed from compositing multiple videos as constituent rectangles. Many applications require multiple synchronized videos for multiview content or different types of content, such as alpha channels, depth maps, object masks, or image features. Use cases for the SEI message include VR/AR, 3D, free-viewpoint video, game streaming, and cloud gaming. Pictures containing frame-packed rectangular regions have previously been coded with existing single-layer video codecs/profiles in an ad hoc non-standardized manner, without clear description within the video bitstream. The Constituent Rectangles SEI message provides a standardized way to describe a video containing pictures formed by compositing multiple rectangular videos of one or more types using a single-layer video codec profile such as High-Efficiency Video Encoding (HEVC) Main or Versatile Video Coding (VVC) Main profile. The multiple videos otherwise could have been coded as separate pictures in layers of a multi-layer bitstream, requiring the use of a multi-layer video codec profile. The Constituent Rectangles SEI message is targeted for inclusion in a new version of the Versatile Supplemental Enhancement Information (VSEI) standard, a partner standard to the VVC standard, which defines SEI messages providing in-band metadata targeting a variety of use cases. SEI messages defined in VSEI may also be used with other video coding standards, including Advanced Video Coding (H.264/AVC) and HEVC.

Print ISSN

1545-0279

Electronic ISSN

2160-2492

Published

2025-01

Content type

Original Research

Keywords

supplemental enhancement information (sei), versatile supplemental enhancement information (vsei)

DOI

10.5594/JMI.2025/LZES6606