Faster than a ray of light

Despite swapping celluloid for silicon, the fundamentals of recording images haven’t changed since the invention of photography.

New advances in light-field technology, first conceived in the 1840s, are about to change all that. Light-field is a way to capture or project rays of light in a scene.

Technically, a light field is 5D and includes three spatial dimensions (X, Y, Z) plus two angular dimensions, describing the direction and intensity of the light ray.

Record all of this with enough fidelity and in theory you have a holograph. While there is currently no commercially available display capable of showing a holographic video (see box), there are potential applications in using light-field data flattened into 2D to create visual effects and other image manipulations in post.

Moving rapidly from science-fiction to experimentation, light-field has made it onto the agenda at the SMPTE Cinema Conference at NAB and at the MPEG JPEG committees, where a new working group on the topic, plus sound fields, has just been established.

What’s more, the company that brought light-field to the mainstream in 2012, with a consumer stills camera that enables users to refocus after capture, claims to have developed the world’s first light-field cinema camera.

“Light-field is the future of imaging,” declares Jon Karafin, head of light-field video product management at Lytro.

“It is not a matter of ‘if ’ but ‘when’.” Light fields can be captured in two ways. One is to sychronise an array of cameras, each recording a different point within the same space; the other is to place a microlens array (MLA), comprising hundreds of thousands of tiny lenses, in front of a single sensor. Lytro has chosen the latter approach, building on a decade of software R&D, 18 months of intensive hardware design and $50m (£35m) in funding.

Last November, the company released stills camera Lytro Illum and announced Lytro Immerge, a video camera designed for virtual reality, which it has not yet released. These products are stepping stones to the company’s new system, the specifications of which are astounding.

Currently in alpha test and due to launch during the third quarter of this year, the Lytro Cinema Camera carries the highest resolution video sensor ever made at 755 megapixels (MP), capable of recording 300 frames per second (fps) through an MLA comprising more than 2 million lenslets.

The next highest resolution sensor publicly announced is a 250MP model being development by Canon.

By contrast, HD equates to 2MP and 4K to 9MP. The resolution needs to be vast for the system to process the unprecedented volume of information. According to Lytro, the effective output resolution will be 4K. “We are leapfrogging the leapfrog, if you will,” says Karafin. “We are massively oversampling the 2D to be able to change the optical para meters in post.

Everything that makes the image unique is retained but you can re-photograph every single point in the field of view.”

For example, the shutter angle and frame rate can be computationally altered in post. As studios move towards higher frame rate cinema (Ang Lee’s Billy Lynn’s Long Halftime Walk is shot at 120fps), Lytro thinks there’s a market for being able to render the same project at 24fps or 120fps for theatrical release and 60fps for broadcast, at the touch of a button.

With plug-ins for The Foundry’s Nuke available on launch, the system has a route into facilities where calibrations including depth of field refocus, tracking, stereo 3D and Matrix-style VFX can be created from the same raw information. Lytro admits the technology is “very challenging” and that work needs to be done on the software algorithms, compression system and hardware.

Nor will it be cheap. Too large to be handheld, the camera needs to be physically mounted to a dolly or crane and be cabled to a server up to 100 metres away.

Video village

The server itself is powerful enough to crunch data up to 300GB/s. Even then, the server will sit in a ‘video village’ supervised by technicians. The camera requires “a non-standard optical format” and Lytro will offer the unit with custom focal lengths.

The whole system, including operators, is being offered for rental. Cinematographers, already wary of virtual production technologies eating into their core crafts of camera position and light exposure, are likely to feel threatened.

Anticipating this, Lytro has incorporated a physical focus control and aperture check on the camera to give DoPs reassurance that the decisions they make on set are preserved down the line.

“There are those who swear by film as a capture medium, but for most cinematographers there is no right or wrong, just a tool that best meets the creative requirements,” says Karafin.

“Ultimately, this is a disruptive technology and we need the feedback of the studio, cinematographer and VFX community to make sure what we are creating meets their needs, as well as helping them understand the creative control this unleashes.”

Light-field-captured video may appear little different to the viewer until there are holographic screens capable of projecting a three-dimensional image.

Unsurprisingly, there are major developments here too, with companies including Zebra, Holografika and Leia3D among those with designs.

“We are not there today but we will cross that threshold,” says Karafin. “Holographic is the next generation of display technology and will truly change the way we all think about capturing native imagery.”