My Top Tips for using AI-based VFX tools

The rapid rollout of AI-based technology is causing understandable concern in the creative industries. With jobs and revenue streams under threat, it’s no surprise to see writers and actors in the US attempting to establish ground-rules that will protect talent’s livelihoods.

At StudioB, a digital-first content creation studio, we fully support these efforts. But at the same time, it’s clear to us that AI also represents a significant opportunity for our segment of the business. In a sector characterised by tight budgets and tough deadlines, AI is enabling us to unlock our creative potential by reducing costs and accelerating workflows.

We’re not alone. In a recent survey commissioned by Descript, a US-based editing kit company, almost two-thirds of content creators said they had tried using generative AI tools like ChatGPT and Dall-E. More than 70% of the sample said they expected AI-powered tools would lead to “better quality content, greater accuracy and time savings”.

The buzz around AI can be traced back to last November’s launch of ChatGPT by OpenAI – a move that has triggered a new wave of investment from the likes of Google, Apple, Meta and Microsoft. But studios like ours have been gradually integrating AI into our workflows for years, it’s only recently that it has been brought into the spotlight. To date, these are the kind of areas where we have found it makes a difference:

Brainstorming

Not surprisingly, we’ve been playing around with OpenAI tools like GPT-3 and ChatGPT. As an experiment, a member of our team ran a test at the studio where we trained a GPT-3 model how to understand a StudioB script. We fed 300 scripts and videos into the test and asked GPT-3 to generate more scripts. The results were incoherent and unusable – but they did throw up interesting ideas and words. So while we couldn’t use the texts provided, it did give us a new way of brainstorming. It’s not a replacement for writers, but it can aid development.

The Pitch

One of my favourite AI tools is Midjourney, which creates images based on text prompts. Available at a low subscription price, it allows creators to generate realistic visuals quickly and easily. For us, the best use case is the pitch decks we send to clients. We always want to paint a picture of what the eventual production will look like to help inspire them.

Greenscreen efficiency

Another tool we’re using a lot comes from Runway. The firm has developed a great greenscreen feature that saves time on rotoscoping. Rotoscoping is a key part of production but it is not very creative and it’s time-consuming. Runaway’s solution trims off about 80% of the fat, so content producers can spend more time being creative.

Simplified CG/live action composition

A firm called Wonder Dynamics has created an AI tool that automatically animates, lights and composes CG characters into a live-action scene. As a VFX-led studio, we spend a lot of time on processes like this – but now instead of weeks the same effect can be achieved in hours. The more time that can be saved on technical processes, the more energy and resource we can put into creating more ambitious content.

Democratised VFX

Usually when people talk about virtual production, the discussion centres on LED volumes and the ability to shoot actors in real-time against digital backdrops. But there’s also a powerful AI dimension to VP that is making high-end VFX accessible to a much broader range of producers. Recently, we created an awesome Spider-Man video which shows the main character swinging from building to building. It got 100 million views in five days. 10 years ago, that video would have taken a team of 100 artists three months to produce. But we were able to do it with five artists in about three weeks using a brand new piece of tech from HTC called Vive Mars.

This democratisation of VFX-based production is accelerating. One innovation I’m watching is PixarGPT, created by a 19-year old TikToker. He’s building an AI model that allows users to create Pixar-style movies by inputting books. It’s early days, but the fact a TikToker in his bedroom using an old computer can attempt such a huge technological leap is a gamechanger for content creators.

Reversioning

Google has been working on a tool called LipSync3D, which makes it possible to map over a person’s lips in a realistic way – so that you can attach a new audio track to a piece of video. There are dozens of ways this kind of tech can be used (and abused), but where I see an opportunity for digital-first creators is reversioning for different language markets. Another use would be to strip out bad language, so content can be used for different demos.

Final Thought

As I said at the outset, StudioB’s sympathies are with creative talent as they seek to get a fair deal from media owners. But at this end of the business, AI looks set to enhance production and make the work more satisfying. We have no plans to replace artists with AI – we see AI as a tool that can help small studios create great work and hire more talent.

Brandon Baum is founder and CEO of StudioB

He is speaking at the forthcoming AI Creative Summit - tickets are available now at https://www.aicreativesummit.co.uk/