Virtual production in the spotlight

MARCUS BRODERSEN (chief executive, Pixotope) It is becoming a more solid option for a range of productions across all genres. We’ve left the ‘innovators’ stage and are squarely in the ‘early adopters’ phase. From big-budget Hollywood blockbusters to live news and sports broadcasts and advertising campaigns, virtual production is making waves.

Time and cost savings, increased operational efficiency and the ability to create content that immerses, engages and retains audiences are just a handful of the compelling advantages it brings to the table.

Virtual production (VP) can be used to create immersive, real-time data visualisation graphics during live sport, enhancing the viewing experience for audiences at home.

Moreover, augmented reality (AR) has introduced a new dimension to in-game advertising. With AR, advertisements can now be seamlessly integrated onto the playing field during live broadcasts, removing the need for traditional commercial breaks, as seen in examples from Gillette and Chipotle.

This not only enhances viewer engagement but also creates a more fluid and uninterrupted viewing experience.

Similarly, in the advertising industry, VP offers the potential to create visually stunning and highly personalised ads that can be tailored to specific demographics.

“VP is maturing fast and there are now a lot of stages globally, but productions need to figure out how to use them e ectively”
James Uren

It’s still early days. While we’re seeing more productions do more creative, boundary-pushing things, we need to think bigger. Virtual production enables a future where traditional viewing formats for things like news broadcasts are completely upended to create an entirely new experience for the audience.

PAUL FRANKLIN (co-founder, DNEG VFX) Virtual production is maturing in all areas, but the cost of the LED volumetric approach and the scarcity of talented, skilled crew who understand how to mount a VP workflow is still a limiting factor. We need a breakthrough in technology and training to make it commonplace. VP is still finding its place in the wider world of production.

LUIS GARRIDO (executive director, Alfalite) The challenge of virtual production and extended reality (XR) is to change the workflow and production concept stage from post-production to pre-production. The sector is in the process of training and assessing the advantages that a VP XR environment offers.

Many news and other programmes already use AR, XR and VR, and countless advertising production companies, esports producers, corporate production houses, education providers and the AV event rental market are considering investments to implement a complete XR set.

JAMES UREN (technical director, Mo Sys) Virtual production technology is getting smarter and faster, and crews are becoming more familiar with the techniques. This is closing the knowledge gap.

The market is still a bit unsure, but as more productions dip their toe in the water, VP gets easier and the cost, convenience and carbon benefits begin to shine through.

LOUISA BREMNER (virtual production supervisor, NantStudios) Virtual production has had a huge amount of press and interest, with fi lms like The Batman and high-profile TV shows like The Mandalorian using the technology. Between that and the increased interest due to the restrictions during the pandemic, knowledge of the technology has skyrocketed.

More productions than ever are aware of the technology and regard it as an essential tool. While some remain cautious due to the instabilities during the technology’s early days, this caution can be beneficial for the industry.

Producers are now making more informed decisions about adopting and implementing the technology, and there is a significantly greater wealth of knowledge and training compared with 2020.

Over the past few years, there has been a surge in innovation, leading to solutions for many of the limitations that the technology initially faced. This has resulted in a remarkable improvement in photorealism and affordability.

Compared with 2020, there is a notable increase in the number of available stages, particularly ones well-suited and appropriately sized in tax-incentive regions. Governments are becoming increasingly aware of the expansion in this industry and are customising their incentive policies to encompass virtual production rentals and the generation of real-time 3D assets as integral components of their programmes.

The UK came in early with a cash rebate of up to 25% of UK qualifying expenditure, which attracted a lot of fi lm industry business to the country. Many more countries are now increasingly competitive. Australia offers up to a 50% cash rebate, and many countries in Europe off er around the 30-45% mark.

MB The biggest challenge is about helping creators understand the value of implementing virtual production. It comes down to making the technology easier to use, more affordable, and educating the talent in technical and creative problem-solving.

Camera tracking remains one of the biggest obstacles preventing widespread use of VP. It’s technically complex and can require a lot of resources to get it set up and keep it running. But there have been gradual enhancements in workflows and pricing, resulting in quicker set-up times and improved ease of operation, ultimately making VP more accessible.

Our innovative Pixotope Fly offering, launched last year, simplifies creating and tracking AR elements with drones and cable cameras. Its unique markerless, through-the-lens technology eliminates the complex set-up and creative constraints imposed by tracking markers, enabling productions and live events to engage audiences with dynamic real-time aerial graphics without additional hardware.

Customers have been able to more easily incorporate AR graphics indoors, outdoors or in studio environments with precision and flexibility to create some truly unique content.

VP education and the lack of trained talent remains a problem limiting the potential growth of this industry. As demand for this type of technology grows, it is important that we equip the next generation of talent with the necessary skills, tools and expertise to be able to meet it.

However, we have seen virtual production gain attention this year within media production training and education, as well as indications that 2023 has marked a turning point for schools, which are beginning to seriously consider VP as a core component of their curricula. We have seen this echoed within the ongoing growth and general interest in our own Pixotope Education Program.

All that said, we are seeing greater adoption within broadcast, and expect to see this continue to grow in 2024. This surge in demand for broadcast graphics is largely due to the immersive nature of the media landscape, with modern viewers having heightened expectations – seeking high-quality and visually captivating graphics that elevate their viewing experience.

Broadcasters are acutely aware of the necessity to upgrade their infrastructure to support advanced graphics capabilities and have expressed their readiness. Solutions like Pixotope Live Controller are important for this shift, as they empower broadcasters with a­ ordable and easy-to-use methods to seamlessly integrate cutting-edge graphics.

PF The biggest challenge is that a lot of crews and facilities have been disbanded or mothballed due to the various Hollywood strikes in 2023. It will take a few more months to get things back to where they were, plus a lot of people will have left the industry during the strikes.

It’s a challenging time, but those outfits that have managed to keep their core crew together will be well placed to take advantage of the return of demand for their services.

LG After evangelising for three to four years about the benefits of VP XR for the production market, we are used to clarifying doubts about limitations that already don’t exist in products such as LED screens.

The screens have been designed and manufactured specifically to minimise problems such as flickering. They have colour ranges close to 90% of Rec. 2020 and achieve super-pure blacks for a magnificent contrast.

The evolution of the design of materials with new compounds also allows us to minimise reflections. Companies evaluating VP XR technology will see their creative capacity is not limited – they just have to adapt their workflow.

JU The technology is maturing fast, and there are now a lot of stages globally, but productions need to figure out how to use them effectively. This means planning, testing and thinking about VP earlier on.

Technically, colour management is still a challenge, and in-camera VFX needs to be able to work in parallel with traditional finishing and VFX. From our side, this means improving colour workflows and recording high-quality tracking data that can be used to automate VFX pipelines.

LB There is a big push for better quality lighting from LED panels, along with anti-reflective coatings to solve some of the challenges around magenta casts from screens and contrast lifting from overlighting the screens. These are things that can be managed, but when the new generation of panels is distributed en masse, there will be a lot less to worry about when working with DoPs and ga­ffers.

Innovation is ahead of distribution, so only a select number of stages have transitioned to these kinds of panels. For example, in our stages in Melbourne, Australia, we have a new ceiling panel that has removed much of the noticeable o­ -axis red-to-green shifts that were common in earlier volumes.

The biggest change at the moment is the start of the switchover to SMPTE 2110, sending digital media over IP. This is going to completely disrupt the current limitations we have around bandwidth, latency and flexibility.

MB Since VP is used in so many di­fferent types of production, kit requirements will vary. The core components are a VP graphics solution, such as Pixotope Graphics, which provides the rendering, video processing and compositing features needed. Most virtual productions will require camera tracking, either as a standalone system, such as our Pixotope Vision or Fly, or through PTZ or grip equipment with built-in mechanical tracking.

In the case of Pixotope, these applications are software that requires a high-end workstation, with an Nvidia GPU. If SDI is required, you need a Video I/O device from either AJA or Blackmagic Design.

For multi camera and large-scale LED-based workflows, you also need solid ethernet infrastructure, and reference signal and timecode generators to synchronise all the equipment. For these workflows, the cameras also need to be able to genlock to a reference signal, which is typically available in more high-end cinema and broadcast cameras.

For green screen-based productions, you will need a chroma keyer, which comes built in with Pixotope. We also highly recommend using a talent tracking solution, such as Pixotope TalenTrack or the built-in solution, to facilitate more advanced integration of people and virtual graphics.

The key to a successful virtual production is planning – specifically, around how the di­fferent components are connected and integrated. A poorly configured ethernet network or reference signal distribution can severely hold back a production.

PF The answer to this question continually changes. Rather than kit, the essential element is crew who understand the production’s creative needs and are agile enough to respond to this while also making sure they’re taking care of business and delivering what the show needs to keep moving forward.

To keep the VP shoot running smoothly, it’s all about communication and making sure every department understands how they will work in a VP workflow.

LG To carry out VP XR production, it is essential to have an LED wall designed specifically for such a mission. For a film or series set, an LED ceiling and LED totems should be considered to provide support and coverage to lighting the scene.

The main things to keep in mind with LED technology for a VP XR production include brightness – it must be greater than 1,500 nits for the LED wall and 6,000 nits for the LED ceiling and LED totems.

It is also necessary for the colour temperature of the LED wall to be maintained regardless of whether the brightness varies from scene to scene, and for the composite that covers the face of the LEDs to o­ er a high level of contrast and pure blacks. The LED screens must allow high shooting angles (of at least 170 degrees) to be reached while maintaining quality of brightness and colorimetry.

JU The great news here is that for 3D in-camera VFX, you don’t need anything more than Unreal Engine with nDisplay to render your real-time backgrounds. You can use Unreal for 2D and 2.5D plates too, but I’d recommend a playout system for this, such as Assimilate Live FX. Of course, you need camera tracking, and inside-out optical systems like StarTracker are the most accurate and convenient for in-studio applications.

You’ll need a camera you can shutter-sync with Genlock and if you are doing additional AR/XR graphics on top – set extensions, for example – then you’ll need another render engine downstream of the camera running Unreal-based software like VP Pro.

If you intend to do further VFX in post, you’ll want to record the tracking and lens data too. VP Pro provides the tools for this if you are using a StarTracker.

For a smooth-running VP shoot, I’d recommend involving your VFX team early and reaching out to manufacturers like Mo-Sys for support and guidance. Working in a game engine means you can pre-viz ‘for free’, so planning can be very detailed – down to the precise shots and moves.

While everyone is still finding their feet with VP, it’s critical to schedule enough time to test – to prep the tech and give the crew time to get used to it. Try to keep your options open for post-production. Don’t be afraid to put some green in behind the subject.

Multi-camera needs careful planning too. Unlike with green/ traditional production, you can’t just throw another camera in for more coverage.

LB A kit list is hard to lock in since there is so much choice. A number of recommended cameras sync to the walls and global shutter cameras are becoming more popular as they avoid rolling shutter artefacts.

“The biggest question people should be asking is whether a 2D, 3D or 2.5D workflow is best suited to their needs”
Louisa Bremner

You’ll want a good tracking system suited to your space and requirements, and check what additional hardware you might need to attach to your camera for tracking – like ‘crowns’, depth cameras and anything that you’ll need to receive sync or transmit focus data.

The biggest question people should be asking is whether a 2D plates workflow, a full 3D environment or the middle ground 2.5D workflow is best suited to their needs as there’s a big di erence in both prep time and cost.

A lot of the decisions will revolve around budget and the story, so getting a good script breakdown with experts in VP will help you make smart decisions. At Nant, we’re very familiar with taking companies that are either fresh to virtual production or have been burned in the early days, and getting their confidence in the technology back to where it should be.

MB Unreal Engine is still widely recognised as the most cutting-edge game engine, able to create the highest quality photo-realistic images while maintaining performance, so will continue to remain the preferred choice for real-time VP.

At Pixotope, we have developed our own core VP technology that integrates seamlessly with the Unreal Engine, but allows for complete control and optimisation of the user experience. Our customers are safeguarded and future-proofed from any changes in the VP workfl ow.

PF Epic is a very important player in this area, but there are other technology partners that have staked a lot on the success of VP, so I don’t think it’s game over by any means.

LG The different manufacturers of real-time 3D graphics server solutions have their own creation tools and rendering engines, so Epic is one more tool, but not the only one.

Currently they coexist, but if Epic stops betting on this sector, there are alternatives that will continue to be developed that o er a complete range of 3D creation and rendering tools in real-time with all the necessary functionalities.

JU It’s not Epic’s style to remove functionality, but they may decide to not grow it further or commercialise certain parts. VP is a general trend, and while Unreal is the most popular engine currently, it is not the only option, and if Epic step back I expect others will step forward.

LB The support that VP has received from Epic over the past few years has been over and above what any of us in the fi eld could have dreamed of and the innovations it has made have catapulted us forward by huge leaps.

The world economic climate is different now than in 2020 so we can’t necessarily expect that things will be the same. The engine is still developing at a pace, and we benefit from a lot of the graphics improvements that are made to support other complementary industries, like the gaming sector.

The vast majority of software supporting the film industry is on a seat-based model, so if anything this is a move that fits in with the industry status quo and speaks to the increased stability of the engine for commercial use. I’m still confident that the support we continue to receive from Epic in VP is higher than most others would provide.