Unity demos real-time volumetric 3D for UFC

Interview with Peter Moore, SVP and GM of sports and live entertainment, Unity

What is Metacast?

Metacast empowers content creators to build volumetric experiences. Once content is captured and translated into volumetric data, Metacast provides the tools and technologies that allows the data to be ingested into the Unity editor (encoded, compressed and prepped), rendered, and streamed agnostic of network or device.

How is content distributed?

Metacast enables a suite of products from consumer to B2B. These include interactive streamed 3D mobile applications like the UFC app, to more feature-rich apps for broadcast hosts doing post-game analysis presentations. Metacast will also support the rendering of 2D videos for cinematic replay type in-stadium jumbotron + broadcast video ‘impossible camera’ replay applications.

Metacast is capture/format agostic, powerful, and easy to use; it has the potential to be used in industries ranging from industrial to medical and music. Anywhere one wants to interact and have agency over time and space with volumetrically captured data, Metacast can be used.

What are the challenges with distribution?

Delivering volumetric content requires very high bandwidth and low latency, as it needs to handle anywhere from tens to hundreds of megabits per second in sustained bitrate. To do this at scale requires purpose-built, data delivery, and data reduction capabilities. As a result, Unity is partnering with communications service provider networks and Cloud Delivery Networks to get the data as close to viewers as possible. This ensures the most scalable bandwidth and the lowest latency available.

What cameras are used and how many?

Metacast is built to be as volumetric-format agnostic as possible. We currently support polygonal and point-cloud solutions from capture partners such as Canon, 8i, Microsoft and its mixed reality capture partners such as Metastage and Dimension, TetaVi and others.

For the UFC demo, we used 100+ cameras. It’s early days for the volumetric medium and with advances in camera technology, 3D reconstruction methods, and ML, we expect to see the required number of cameras drop significantly. There are techniques that work with only one camera, which while not yet at an acceptable level of quality may rapidly change. We expect within a few years the required number of cameras will be half of what is currently required, perhaps even less.

What is the normal camera set-up?

Every capture provider does it a little differently, depending on the quality and physical requirements of the application. For example, some providers use a symmetrical spherical camera configuration setup, while others place cameras strategically with different field of views to capture the entire field of action.

What are the challenges in positioning?

In order to produce a good volumetric reconstruction, cameras must be placed such that they maximize coverage of the subject (i.e., can see the subject without obstruction from all angles) with a good amount of overlap between cameras. The subjects must also be able to be separated from the background. Furthermore, they must not interfere with the subjects and the physical setup that they perform in. While all of these requirements are easy to meet in a studio setup, it is much trickier to do for real sports. For instance, in the case of UFC, the cameras must not interfere with the referee and athletes.