Volumetric video is an emerging media format that captures a subject in three dimensions but allows playback from any conceivable angle. Volumetric video is much different than 3D movies or 360-degree video in that a user can experience a recording with six degrees of freedom (6DoF), including X, Y, and Z axes, in addition to pitch, yaw, and roll. This freedom allows users to experience, rather than watch, a recorded event through VR (virtual reality) or AR (augmented reality) headsets.
Most people are familiar with holograms, which accurately reflect the output of a volumetric video. Volumetric video captures actors or subjects in 3D to create holograms that viewers can watch from any conceivable angle. Users would consume volumetric videos through the use of VR or AR devices.
The technological and cost barrier to entry for VR has been lowered drastically with the release of Facebook's Oculus Quest 2, and the interest in the media format will likely ignite again with Apple's impending entry into VR/AR, most likely around the corner. The decline in hardware cost and increased competition in the field will soon create an insatiable hunger for VR & AR content — but the cost of producing VR or AR content remains high.
Game studios or teams of professional creatives who have mastered the art of computer graphics produce the vast majority of VR experiences today. Volumetric video offers a cheaper alternative to make an immersive experience that could unlock VR & AR content production to a new set of creative teams and individuals with prior experience in video production.
How do you capture Volumetric Video?
Depth-sensing cameras (like the Microsoft Azure Kinect) paired with a traditional camera to capture color can capture volumetric video. The depth camera captures depth by analyzing infrared light and generating a 3D mesh of the subject.
Companies like Depthkit offer software solutions to combine depth and color from these two recordings into a single volumetric video. Creators often use multiple cameras from different angles to create a fully immersive experience.
Microsoft has even created several full Mixed Reality Capture Studios to capture subjects in high fidelity. These studios employ the use of a multitude of cameras on a green screen sound stage to produce a professional-grade volumetric video or what they call holograms.
How do you view volumetric videos?
Creators must consider both the delivery mechanism and the device used for volumetric video playback.
Volumetric video creates a ton of data, and the current playback method requires preloading bespoke apps or experiences. Users would download the app on an AR-enabled smartphone or for a more immersive experience on an AR or VR headset.
The delivery mechanism and device requirements severely inhibit the growth of volumetric video. Users have become accustomed to instantaneous playback of video thanks to YouTube, Netflix, and the advancement of streaming video. Choosing content beforehand and waiting for a large file download is counter to the instantaneous experiences expected by users today.
Many users have an augmented reality device in their pocket today, thanks to the addition of AR technology to the latest iPhones. However, looking through a tiny smartphone window does not provide the level of immersion that makes volumetric video so compelling.
The popularity of volumetric video correlates directly with the household penetration of VR and AR headsets. Facebook is leading the way with the Oculus Quest 2, the most consumer-ready product out of all VR headsets, since it is wireless and has a (relatively) friendly price point.
Current Volumetric Video Applications
If you have watched the highlights of an NFL game in the past few years, you might have already experienced volumetric video. The NFL and other major sports leagues partnered with Intel to capture volumetric videos of sports matches.
Powered by an array of 5k cameras, this capture process allows Intel to create replays from any angle or views that would not even be possible, like watching a play unfold from the perspective of a quarterback. Producers can then weave these unique perspectives into one seamless uncut highlight.
While providing an interesting 3D take on a traditional 2D experience, this application of volumetric video suffers the same drawbacks of watching volumetric video on a smartphone, a lack of immersion. Viewers are shown different angles from a single moment in time but do not experience it.
Microsoft used its Mixed Reality Capture Studio to create a more immersive experience to help promote the movie Blade Runner 2049. The Blade Runner 2049: Memory Lab experience immerses a user in the world of Blade Runner through volumetric video capture and a VR headset. In this experience, creators blend volumetric video captures of actors with 3D worlds to create a fully realized sense of presence.
The Case for Volumetric Video
Volumetric videos allow real humans, rather than 3D characters, to perform in virtual reality experiences. Actors captured through volumetric videos enable users to perceive them from different angles. Creating an artificial three-dimensional character could cheapen the experience and degrade the level of immersion.
Using actors and specialized software can rapidly speed up and reduce the cost of virtual or augmented reality production. Rather than relying on game designers and engineers, traditional TV and movie producers could jump to creating immersive experiences with a small amount of training.
The popularity and proliferation of volumetric video depend on consumer interest in VR and AR. Without devices to realize the full immersive potential of volumetric video, the need to produce and invest money in this emerging format wanes.
With Facebook investing billions and Apple working on competing applications, VR and AR are most likely here to stay. The proliferation of consumer hardware will necessitate the need for content, and volumetric video provides a lower barrier of entry for creators to deliver that content.