Video Production with Generative AI

The number of applications for Generative AI (GenAI) in video production is rapidly increasing. Many kinds of GenAI models are used, from audio generation to text generation models. However, video generation models (VGMs) are foundational for video production and are gaining importance as their power and expressiveness increases. In regard to expressiveness, VGMs are different from prior kinds of GenAI models in that VGM users must have domain-specific knowledge in order to optimally leverage VGMs to express their artistic vision. More specifically, it can be demonstrated that VGM users must know at least the basics of the visual language of cinematography in order to properly translate their artistic vision into quality VMG output. The tool by which users apply cinematographic language to VGMs is prompt engineering, which is used to craft VGM output into something useful and aesthetically viable. Using prompt engineering, it is possible to ask a VGM to instantiate the basic elements of cinematography such as: camera placement, camera movement, shot composition, shot size, focus/depth of field, and lighting. Additionally, GenAI can be used to create storyboards and animatics to help plan out these cinematographic elements. VGMs and other GenAI models also are useful in post-production workflows. This includes not only editing, but also other post-production tasks such as visual effects.

Published

2024-10-21

Content type

Original Research

Keywords

generative ai, genai, video generation, video production

DOI

10.5594/MOO/3041

ISBN

978-1-61482-965-2