Volumetric video is an emerging media format that captures a subject in three dimensions but allows playback from any conceivable angle. Volumetric video is much different than 3D movies or 360-degree video in that a user can experience a recording with six degrees of freedom (6DoF), including X, Y, and Z axes, in addition to pitch, yaw, and roll. This freedom allows users to experience, rather than watch, a recorded event through the use of VR (virtual reality) or AR (augmented reality) headsets.
Most people are familiar with the concept of holograms, which accurately reflects the output of a volumetric video. Volumetric video captures actors or subjects in 3D, allowing the creation of holograms that a viewer could behold from any conceivable angle. Users would consume volumetric videos through the use of VR or AR devices.
The technological and cost barrier to entry for VR has been lowered drastically with the release of Facebook's Oculus Quest 2, and with an impending entry into AR by Apple all but confirmed. The decline of hardware cost and the increase of competition in the field will soon create an insatiable hunger for VR & AR content. But the cost to produce VR or AR content remains high.
Game studios or teams of professional creatives who have mastered the art of computer graphics produce the vast majority of VR experiences today. Volumetric video offers a cheaper alternative to make an immersive experience that could unlock VR & AR content production to an entirely new set of creative teams and individuals with prior experience in video production.
How do you capture Volumetric Video?
Depth-sensing cameras like the Microsoft Azure Kinect in tandem with a traditional camera to capture color can capture volumetric video. The depth camera captures depth by analyzing infrared light and generating a 3D mesh of the subject.
Companies like Depthkit offer software solutions to combine depth and color from these two recordings into a single volumetric video. Creators often use multiple cameras from different angles to create a fully immersive experience.
Microsoft has even created several full Mixed Reality Capture Studios around the world to capture subjects in high fidelity. These studios employ the use of a multitude of cameras on a green screen sound stage to produce a professional-grade volumetric video or what they call holograms.
How do you view volumetric video?
Creators must consider both the delivery mechanism and the device used for volumetric video playback.
Volumetric video demands a vast amount of data and the current method of playback requires pre-downloading bespoke apps or experiences. Users would download the app on an AR-enabled smartphone or for a more immersive experience on an AR or VR headset.
The delivery mechanism and device requirements severely inhibit the growth of volumetric video. Users have become accustomed to instantaneous playback of video thanks to YouTube, Netflix, and the advancement of streaming video. Choosing content beforehand and waiting for a large file download is counter to the instantaneous experiences expected by users today.
Many users have an augmented reality device in their pocket today thanks to the addition of AR technology of the latest iPhones, but looking through a tiny window of a smartphone does not provide the level of immersion that makes volumetric video so compelling.
The popularity of volumetric video correlates directly with the household penetration of VR and AR headsets. Facebook is leading the way with the Oculus Quest, the most consumer-ready product out of all VR headsets since it is wireless and has a friendly $299 price point.
Current Volumetric Video Applications
If you have watched highlights of an NFL game in the past few years, you might have already experienced Volumetric video in action. The NFL and other major sports leagues partnered with Intel to capture volumetric videos of sports matches.
Powered by an array of dozens of 5k cameras, this capture process allows Intel to create replays from any angle or create views that would not even be possible, like watching a play unfold from the perspective of a quarterback. Producers can then weave all of these unique perspectives into one seamless uncut highlight.
While providing an interesting 3D take on a traditionally 2D experience, this application of volumetric video suffers the same drawbacks of consuming volumetric video on a smartphone, lack of immersion. Viewers are shown different angles from a single moment in time but do not experience it.
Microsoft used their Mixed Reality Capture Studio to create a more immersive experience to help promote the movie, Blade Runner 2049. The Blade Runner 2049: Memory Lab experience immerses a user in the world Blade Runner through the use of volumetric video capture and a VR headset. In this experience, creators blend volumetric video captures of actors with 3D worlds to create a fully realized sense of presence.
The Case for Volumetric Video
Actors captured through volumetric videos allow users to perceive them from different angles. Creating an artificial three-dimensional character could cheapen the experience and degrade the level of immersion. Volumetric videos allow real humans, rather than 3D characters, to perform in virtual reality experiences.
The use of actors and specialized software can rapidly speed up and bring down the cost of a virtual or augmented reality production. Rather than relying on game designers and engineers, traditional TV and movie producers could make the jump to creating immersive experiences with a small amount of training.
The popularity and proliferation of volumetric video depend on consumer interest in VR and AR. Without devices to realize the full immersive potential of volumetric video, the need to produce and invest money in this emerging format wanes.
Facebook recently reported that Oculus surpassed $100 million in content and game sales. Recent estimates peg the total number of Oculus Quest sales are close to 1 million and rising. Facebook has had trouble keeping the Quest in stock, and now with the Oculus Quest 2 coming in at $100 cheaper than its predecessor, that trend may continue.
Facebook is all in on VR, evidenced by this recent interview of Mark Zuckerberg by MKBHD. The Facebook CEO displays a real sense of excitement about the future of VR and its potential as a new platform with the radical potential to change the world as smartphones did with the introduction of the iPhone.
With Facebook investing millions and Microsoft and Apple working on competing applications, VR and AR are most likely here to stay. The proliferation of consumer hardware will necessitate the need for content, and volumetric video provides a lower barrier of entry for creators to deliver that content.