Twenty years of Frame Interpolation for Retiming in the Movies

Frame interpolation is the process of synthesising a new frame in-between existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects since its use at a large scale in the making of the movie “The Matrix”. This paper presents a review and a new unified view of the classical algorithms used to create inbetween frames, representing most of the last 20 years of their evolution. This is used to benchmark the recent Deep Learning algorithms against two of the best industrial retimers available. A significantly expanded dataset of 140,000 frames is used for testing. In the context of high resolution material we find that techniques relying principally on DNNs do not clearly outperform the classical ideas. It is only with the emergence of hybrid approaches since 2019 that we see DNNs adding significantly to the performance in this space. Despite the hype surrounding DNNs, we find that there is still something left to do.

Published

2020-11

Content type

Original Research

Keywords

Frame interpolation, Retiming, Inbetweening, Motion Estimation, Optical Flow

DOI

10.5594/M001927