An explanation of Lidar and photogrammetry to build realistic environments, such as the Ritzy in London
In the fifth part of the Virtual Production Innovation series, Dimension Studios provides an overview of two scanning techniques (Lidar and photogrammetry) to build realistic environments – in this case, the Ritzy in London. The benefits of this approach are it saves you having to film on-set, which is especially convenient in busy locations. The scan also provides you with a realistic, accurately detailed, and organic environment to work with.
“The huge benefit is you get all this tiny detail. Even as an artist you might not think to add these little stories in the scene. Using traditional techniques you often see repeated patterns, with scanning you get natural, organic environments.”
VIRTUAL PRODUCTION INNOVATION SERIES – SCANNING THE RITZY
Broadcast teamed up with virtual production and volumetric capture specialist Dimension to present this six-part series of short films providing insight into shooting in LED Volume stages.
The series is focused on the ‘Virtual Production Innovation Project’, which was designed to explore the state-of-the-art in virtual production, testing different workflows and scenarios.
The project was conceived and produced by Dimension, DNEG, and Sky Studios, and filmed at Arri’s Uxbridge studio with support from partners Creative Technology and NEP Group. The aim is to share insights and expertise to the creative community, while helping upskill the next generation of creative talent.
The fifth episode in the series is below:
Featured in the video:
Craig Stiff, lead real-time artist, Dimension
The series of six films explaining different learnings from the project will be released weekly throughout March and April, exclusively on Broadcastnow.
Each episode discusses the tests the team has undertaken and demonstrates some of the advantages of virtual production in capturing in-camera VFX and final pixel shots using an LED volume.
Alongside the film, we present an exclusive interview with one of the key members of the production team, talking about the insights offered in that week’s episode and the lessons they learned during the making of the Virtual Production Innovation Project films.
The series will cover lighting, tracking, simulcam, in-camera effects, scanning, and 360 photospheres. Broadcast’s interview with Dimension lead real-time artist Craig Stiff about scanning the Ritzy.
Broadcast interview with Dimension lead real-time artist, Craig Stiff
Why did you scan the Ritzy and how long did it take to complete?
A huge advantage of virtual production is the ability to instantly travel to any location and have complete control over the environment. One of the challenges is making the digital locations look realistic with detailed signs of life. Traditionally, this would involve teams of talented artists to create detailed props and environments and light it all realistically. We tested 3D scanning and photogrammetry of a location to see if we could overcome this challenge.
We chose to scan The Ritzy building in London so that we can use it whenever we want in all different lighting conditions at any time. This venue is on a busy street and would have been a difficult and costly task to secure filming here. For instance, the owners would have to close the establishment, involving compensation; parts of the street would have to be closed, which would require lengthy paperwork; and the whole crew would have to travel to it, taking up the team’s time to travel and shoot on location; there would also be a lot of potential for public intrusion with the noise from the busy London street, which can’t be controlled.
To bypass all of that, we decided to create it digitally for a virtual production LED volume. This gave us complete control over the environment and the ability to create any atmosphere the director and DoP desired. It also allowed us to film at any fixed time of day for as long as we needed, with no unwanted sounds or interruptions.
We could have gone down the traditional route of using reference images to manually recreate the location, but this is a lengthy and technical art process. Instead, we utilised 3D scanning and photogrammetry to capture the location.
This involved a small team of three going to the location armed with a DSLR camera, a Lidar scanner, and a photogrammetry rig (three or more cameras on a staff). It took just an hour to completely scan the outside of the location and surrounding streets.
It then only took me and two other artists just a couple of weeks to turn that data into a photo-realistic digital version of the scene which was lit and ready to go straight up on the LED wall.
What process was involved in doing the scan and making it look so photo-real?
We used a Lidar scanner to capture the geometry of the buildings. This uses a laser to accurately find the distance to surfaces, giving a very high detailed model. However, a Lidar scanner alone doesn’t give colour or texture information and it often has large holes that the scanner doesn’t pick up.
This is where photogrammetry comes in. Photogrammetry is a method of taking lots of photos of a location and using special software that matches points in the photos to build the 3D model. Combining the two techniques gives us a very high-quality model with very high-quality textures.
It is worth noting that on the morning we went to scan the Ritzy, we were very lucky with the weather. A successful scan requires very flat lighting with no hard shadows or highlights. A bright grey overcast day is the best lighting and that’s what we had that morning. If it was a bright sunny day we would have to put a lot of effort into painting out hard shadows so that we can re-light the scene how we want later.
What difference does the scan make to the quality and authenticity of the final image, compared to either just filming there or using a standard 3D model of the building?
Of course, there is nothing like being able to actually film in a real location. However, with the aforementioned difficulties and even impossible shoot locations that can not be accessed or do not exist, 3D scanning can help to achieve a realistic-looking scene in-camera.
When filming on location, there are additional elements that can make filming more difficult, such as glass. The Ritzy has large glass front doors, which were in the shot and in reality, we would have seen the reflections of the crew, the lights, and the traffic on the street. In a digital scene, we do not have to worry about reflections.
The main benefit of 3D scanning a scene is being given all the intricate details, models, textures, and the layout of the location for free. At the Ritzy, there are all sorts of signs of life with patches of graffiti, leaves, and cigarette butts on the ground, as well as ageing with scrapes, scratches, and grime on the building surfaces with weeds growing between cracks, as well as detailed sculptures and decorations on the buildings.
All this kind of detailing would take a lot of time, care, and consideration for prop and environment artists to recreate using traditional modelling. If many artists are working on different objects, often getting them all to work in a scene together can be a tricky balancing act of materials and lighting. With a 3D scan, all the artists are working with consistent data, meaning the assets will all look correct together in a finished scene without much effort.
Another benefit of digital locations and props is that when you’re done with them, you don’t have to destroy them or put them into expensive storage. A whole city street or an entire mountain can fit in your pocket on a small hard disk, to be used again in any lighting condition for reshoots or other projects whenever needed.
The project was directed by Paul Franklin at DNEG. He said: “It would have been great to capture the Ritzy environment under a range of different lighting conditions, ideally something that was more dramatic than what we got. The more of the original lighting that you can retain in an environment like the Ritzy, the better. However, we only had a small window within which to capture the building and the lighting in the shot is what the clouds and sun gave us on the day. In the event, we had enough flexibility within Unreal Engine to be able to dial back in the drama that was missing from the finished asset.”
You can find out more about Dimension Studio and its work with virtual production, by clicking here.
No comments yet