Enhancements to Media Transport in ICVFX using SMPTE 2110

In this paper, we propose an enhancement to clustered rendering for In-Camera Visual Effects (ICVFX) that leverages the quality, low latency, and timing guarantees in SMPTE 2110, a standard for professional media over managed IP networks. — We show how SMPTE 2110 can be used to multicast multiple camera views of variable or overscanned resolutions, each rendered by dedicated camera render nodes, and received by the collection of render nodes in the cluster that warp and compose it on top of the rendered out-of-camera pixels, resulting in a more efficient, scalable, and higher performing rendering cluster. Since the pixels are now transported between render and composer nodes, we describe techniques to mitigate, and in some cases completely hide, the time required to multicast the pixels over the network, resulting in no added frame latency. — We also demonstrate that the final pixels can then be output via SMPTE 2110 to synchronously drive each section of the LED wall, which results in low latency, simplified and more flexible hardware setup that also unifies the media transport strategy in an ICVFX stage. We show how SMPTE ST 2110 and IEEE-1588 Precision Time Protocol (PTP) with SMPTE ST 2059 enable the ability to synchronize the video output of the multiple render nodes that drive an LED wall, ensuring temporal coherence and spatial alignment of the virtual scene. We explain the clustered media output genlock and framelock algorithm and demonstrate the benefits of our approach with a proof-of-concept implementation and some experimental results. — Find our implementation in the Unreal Engine source code: https://www.unrealengine.com/en-US/ue-on-github

Published

2023-10

Content type

Original Research

Keywords

Virtual Production, ICVFX, Visual Effects, LED Wall, Multicast, Inner Frustum, Framelock, PTP, IEEE-1588, ST 2110, ST 2059

DOI

10.5594/M001996

ISBN

978-1-61482-964-5