HP Case Study: How Planet X mixed realities in real-time

Advances in production technology and computing power are inspiring creators to transcend long-standing barriers between film, television, games, and live events—and between audiences and creators.

Nowhere was this more evident than at the 2023 Cannes Film Festival when audiences were treated to a live presentation at which several of the celebrity panelists appeared in person and others joined in as ‘Xavatars’, real-time animated virtual doppelgängers of themselves.

The show is a collaboration between creative content agency Planet X delivering virtual production services, and startup company Xavatar, with HP supplying high-performance Z4 G5 workstations equipped with Intel Xeon W series processors that powered the innovative workflow from creation through live production.

With participants including filmmaker David Lynch, musicians Elvis Costello and George Clinton, rapper Redman, and 60’s icon Donovan, the Xavatar panel production drew a lot of attention, and more importantly, showed audiences what’s possible in terms of mixed-reality entertainment. But the live event was just a taste of what’s to come, proof of an innovative multi-platform concept that will be an ongoing series.

In addition to its collaboration on Xavatar, Planet X produces the Netflix original rom com doc, Longest Third Date, and is delivering animation for The Proof is Out There, airing on The History Channel, among many other commercial, live event, and episodic projects.

Powering real-time visual adventures

In its downtown Brooklyn, NY studio, Planet X has an LED volume (a 26-foot curved LED wall, LED ceiling, and rolling LED panel) where they do Unreal Engine-based virtual productions and build virtual environments for clients who rent out their space.

But the horsepower they needed to create and operate Xavatar’s complex real-time workflow was more than their older computers could handle.

Planet X managing partner Darin Friedmann, said: “We create all of the dynamic 3D environments for the show in Unreal Engine. The Z4 G5 machines were so much faster in terms of the rendering times we were seeing in Maya, 3ds Max and Blender, and with the frame rates we could get out of Unreal.

“Our team was blown away. We didn’t realise how much we had been suffering in the past until we saw the performance of the HP machines.”

Environments heavy with 3D particles in particular didn’t perform well in real-time on their older machines, but when processed on the Z4 G5 workstations with the new Intel Xeon W7-2495X CPUs, production was seamless. 

Friedmann said: “Once we saw what the machines could handle, we just kept adding elements to make the virtual scenes more interesting. With real-time raytracing in space, we added a lot more metallic elements, things that would have been impossible on our older workstations because they would not have been able to process the visuals in real-time.”

Xavatar co-founder Kevin Sharpley added: “The HP Z series computers are just incredible. With this live project in front of the Cannes audience, we didn’t have any room for snags, and we were able to just not worry about whether the computers were going to perform. It went off without a hitch. It was seamless; The audience was mesmerised.”

Flying through the challenges of a live production

For the show’s opening sequence, a robot host flies through space, through all of the different virtual environments Planet X created, and past billboards introducing each guest, before landing at the space station. “This was all designed to be done in real-time so we had to make sure we had the performance to pull that off with no lag. That was one piece of the challenge,” said Friedmann.

The other challenge was getting the Xavatars into the environments. At the live Cannes show, there were five people on stage, each with an HP ZBook Studio laptop, using headphones and the laptop camera.

Planet X streamed audio and video data from each laptop and used it to drive each of the Xavatars using an AI-based animation system they developed specifically for this project. The data came into Unreal Engine, where they had set up virtual cameras within the environment, each with separate outputs. This enabled Sharpley to direct the action just like it was a regular TV show. Planet X was also able to manipulate 3D objects in the Unreal Engine environments in real-time. The live composited output of the Xavatars performing within the virtual environments is what appeared on the LED screens above the hosts on stage at the Cannes event and in the produced version of the show’s first episode.

Friedmann said, “We had to minimise the latency. If any of that fairly complex path broke down you’d have frozen, stuttering things on the screens onstage. It would have been a disaster. Also, given the compressed schedule of the project and the complexity of managing live and remote guests, there was no rehearsal. The Z4s were so fast and so robust that we were able to make last-minute changes and tweak just about everything, including the fly-in. 

Live litmus test leads to greenlight

The Cannes event was intended to showcase the show’s concept and demonstrate how the real-time technology worked. Friedmann explained: “People were just amazed to see Donovan sitting there and then see the animated avatar version of him on the screen above him. And as he’s talking, the Xavatar is mirroring his expressions. And then David Lynch came on and he was having a conversation with Donovan, and just seeing the character on the screen in real-time conversing with a person on the stage was really a trip for everybody. It got people excited. That’s when the TV networks that we were already talking to finally greenlit the series.”

The show takes place on the Xavatar space station where each celebrity guest joins via their Xavatar. Audiences are transported to locales around the world following the featured guest’s personal journey – from childhood to content created focusing on their career highlights. Xavatar also provides a portal for audiences to actively join the sessions via the show’s cross-platform Xataverse.

Each environment will also include gaming, social networking, 3D retail experiences, and more. The Xavatar Show (TXS) will premiere on Ovation TV in Spring 2024.

Sharpley said: “We had such a tremendous time at The Xavatar Show Live event at the Cannes Film Festival 2023 and we’re looking to do it again at Cannes 2024, this time even bigger. We plan to debut The Xavatar Show, which will be premiering on the Ovation network and UP Entertainment with a reach of over 64 million households, and produce a live event featuring the hosts and guests of the show brought in as real-time avatars to Unreal Engine, and of course we wouldn’t dream of doing it without HP and leveraging the Z series to create an even greater presentation.”