Advanced Volumetric Video Format for Enhancing Photo-realistic Lighting Reproduction

This study proposes a volumetric capturing studio, called NHK Meta Studio, that can acquire various material attributes and light-field parameters to enable photo-realistic relighting. Volumetric videos captured from Meta Studio comprise a multidimensional extended point cloud for each frame, and hence, have large amounts of data. Although it is possible to export all parameters into existing formats, the large amounts of data make this approach less optimal. Therefore, we propose a hierarchical volumetric video format that comprises base and enhancement layers. The base layer is designed for compatibility with the existing format and rendering for a simple relighting method. The uncompressed base-layer format is represented in the Stanford triangle format (.ply) and the compressed base layer is used for an extension of the MPEG volumetric video format called visual volumetric video-based coding (V3C). As a conventional file format is adopted, integration with existing 3D applications becomes easy. The enhancement layer includes the other components and the advanced rendering shader program, which enables us to fully utilize our advanced photo-realistic rendering method; however, full real-time rendering is still under development. Prior to full real-time rendering, we developed an efficient rendering system for the base layer. This system was verified for integration with game engines. Even real-time rendering using full information is not accomplished, real-time rendering will make it possible to make wide use of Meta Studio's volumetric data.

Published

2023-10

Content type

Original Research

Keywords

olumetric videos, rendering, shading, virtual video production, immersive, point cloud, free- viewpoint, XR

DOI

10.5594/M002011

ISBN

978-1-61482-964-5