CGSociety

{digg}

John Gaeta channels his unique perspective on
experimental photography and CG into filmmaking
like no one has seen before.

Start with a Long Island high school student who has friends in theater arts. Send him to classes taught by an inspirational film criticism teacher named Bernard Shearer. Stir in a little civil disobedience by residents in his beautiful and tiny town, who fought a proposed nuclear power station. Send him to film school at NYU’s Tisch School of the Arts and encourage his experimental photography. Give him an apartment in Greenwich Village and a job in television as a PA for the comedy series ‘Saturday Night Live’. Let him hang out with Robin Williams, Tom Hanks, Steve Martin, and Al Franken.

Next, move him on to a second job as an assistant for Peter Wallach, a stop-motion animator, and Mike Sullivan, a cinematographer who created surreal sculptures of impossible looking cameras. Perch him in a bird’s nest high above a river between New Jersey and Staten Island, with plumes of fire from smoke stacks in the background, and hire him to film baby birds all day for National Geographic Explorer documentaries.

And then, give him his first big break: a job as camera assistant for a motion control shoot of the Enterprise for the film Star Trek V. And suddenly, you have a starting line for vfx supervisor John Gaeta’s career. Woven through that career are people and technologies that have forever altered visual effects and filmmaking.“Peter Wallach got the contract to shoot the Enterprise through Bran Ferren,” Gaeta says of that first job in visual effects. “There wasn’t much money, so I got to do everything.”

Twenty-some years later, Gaeta has become one of the most innovative visual effects supervisors on the planet, winning an Oscar and a BAFTA award for ‘The Matrix,’ supervising effects for the second two films in the ‘Matrix’ trilogy for which he won VES awards, and leading the game-changing effects in ‘Speed Racer.’

He attributes his visual effects open mindedness to his start 3,000 miles from Hollywood. “Being far from the California scene allowed me to move through theory uncontaminated by whispers of works within the schools of thought there,” he says. “If anything, I identified with whispered happenings in Europe as much as California.”

After Star Trek, Gaeta found himself far from New York City, as well, working with a master: the visionary Doug Trumbull in Massachusetts, who had been photographic effects supervisor for “2001: A Space Odyssey,” “The Andromeda Strain,” “Close Encounters of the Third Kind,” Star Trek: The Motion Picture,” and “Blade Runner.” “When Doug Trumbull escaped from LA and went to the Berkshires, he needed as many technicians as he could find and there weren’t many on the East Coast,” Gaeta says.

Among those technicians were friends who opened the gates to Gaeta.Trumbull built his dream studio, the Berkshires Motion Picture Corporation, in 1987 inside a former textile mill. There, a subsidiary, Berkshire Ridefilm created Universal Studios’ the 70mm, seven-story “Back to the Future Ride” film for the theme park’s first high-tech thrill ride. Later, the Berkshire companies would evolve into the Trumbull Company, which merged with IMAX.

“Every aspect of the [Back to the Future] project from beginning to end was state of the art to jump start this new format,” Gaeta says. “Carbon fiber motion control systems, new screens, new cameras. It was a pivotal experience. I learned that if you can see the image in your head, you can get where you want to be by making your tools.” By the end of the ride film project, Gaeta was operating the cameras and shooting scenes himself.

Along the way, with that project and others, he also learned from visual effects cinematographers Dave Stewart who had worked on “Blade Runner,” and Jim Dickson, who had been on “2001.” Dickson eventually went on to develop a nine-camera rig that, like CircleVision, points the cameras at mirrors to take 360-degree moving images.

“It’s essentially the same idea we used for ‘Speed Racer,’ but in motion and in film [rather than HD],” Gaeta says.
The crew also experimented with what we now call previsualization. Gaeta remembers that it came about because the miniature designers used CAD data to build models for Trumbull’s motion control systems. “Because some of the guys good at CAD design also knew Softimage, they began exporting 3D data into the motion control system to see what the miniatures would look like through different lenses,” he says. “Doug pushed every button on new technology.”

When Trumbull moved into new digs, a spin-off visual effects company Mass.Illusion grabbed the space. Joel Hynek, who had worked with Trumbull on the 1993 ride film “In Search of the Obelisk,” founded the studio in 1994 by to create effects for ‘Judge Dredd’ and ‘Die Hard with a Vengeance.’ But, Mass.Illusion would gain visibility with the romantic fantasy film ‘What Dreams May Come,’ for which Hynek won a visual effects Oscar.

“I learned about layout, mapping distortion, and other esoteric things from Joel when we were both working for Doug Trumbull,” says Gaeta. “He was one of the best optical compositors in the industry.” And, Gaeta joined Hynek at Mass.Illusion.

There, Colin Green (now at PLF), Michael Schmitt (now at Giant Killer Robots) and Gaeta pushed the previsualization experiments further. “We tried to lay out the dimensions of all the things they would shoot, and design camera moves in previs,” he says. “It’s taken for granted to some degree now, but it was not done at the time. That led to the idea that there is a direct relationship between the physical world, the optics of the camera, and the CG visualization of it.”

You can probably see where this is going – straight toward the legendary “Bullet-Time” shot in “The Matrix. But, not so fast. The immediate goal was accuracy. “We realized we could have a direct coupling of a CG construct to a physical object if we understood its form and precise relationship to the nodal point of a lens,” Gaeta says. “And that led to mad investigations of measurement technologies of all sorts. We’d get into whisper rooms at measurement conferences.

That’s how we got a prototype of a Lidar scanner.” For ‘Judge Dredd,’ the crew had composited a shot using a matte based on measurements not bluescreens. “We did it just to see if we could,” Gaeta says. They had also embedded measurement markers with miniature planes for “Eraser.”“We weren’t sure what we were doing, but we wanted to make it possible to design everything in the computer to visualize shots ahead of time and then export the data,” he says. Although the point was to have extra time to conceive shots – to previsualize the visual effects shots – the process led them in radical new directions.

“It allowed us to understand that you could uncover complex camera paths,” Gaeta says. “And then we started going beyond that. If you could create any camera pass you wanted, if you understood the spatial relationships, could you do something with time? With stop motion? With motion control shot one frame at a time? And that’s what made it possible to make the conceptual leap to Bullet Time.”
“It was a good time for me,” he adds. “I was starting to experiment with laser radar and weird forms of previs. When ‘What Dreams May Come’ came up, I helped Nick Brooks with pitches by doing experimental explorations for moving paint.”

Brooks became the visual effects supervisor for the painted world in “What Dreams,” and he won an Oscar for that work. Film critic Roger Ebert described the film as among the most visually exciting he’d seen, pointing, in particular, to a sequence in which Chris (Robin Williams) “occupies a landscape that is a painting, and as he plucks a flower it turns to oil paint in his hand.”

Gaeta was the ‘reality capture supervisor.’ “We needed to track fluttering leaves on a mountain and convert them into dobs of paint,” he says. “So we studied painting and experimented with optical flow and reality capture.” Brooks introduced Gaeta to Kim Libreri from the UK who was doing pioneering work in optical flow with Dan Piponi. (Optical flow is the name given to the math for tracking every pixel in a moving image.) To scan a mountainside in Montana, Gaeta hired a French company that had some of the first Lidar scanners.

“It was an unbelievable job,” Gaeta says of the reality capture. “It was wildflower season in Glacier National Park. I’d hang out on the set for part of the day with Robin Williams, and then when the crew left, we determined which part of the mountainside we had to shoot. We surveyed the mountains with Lidar radar from dusk to dawn every night and listened our Indian guide tell stories.”

About the time “What Dreams May Come” moved into post-production, Larry and Andy Wachowski showed up at Mass.Illusion. “They were talking to studios about their concepts, about how to accomplish what they had storyboarded,” Gaeta says, referring to what we now know as Bullet Time. “I thought about it and basically pitched the idea of trying to use still cameras.”

In addition, he convinced his friend Rudy Poot to work with him on experimental creature studies. “I bombed Larry and Andy with renegade studies,” Gaeta says. “We did test renders of creatures and bugs in Geof Darrow’s designs. Larry and Andy have a sort of spiritual radar for figuring out who they want to work with. It worked out well.”

Mass.Illusion, by then under new management and renamed Manex Visual Effects, got the job and opened offices in Alameda, California. Gaeta became visual effects supervisor. In 2000, he received an Oscar for Best Visual Effects; in 2001, George Borshukov (who joined the ‘Matrix’ team in Alameda from the University of California at Berkeley), Kim Libreri and Dan Piponi won an Academy Award for Technical Achievement for the development of a system for image-based rendering allowing choreographed camera movements through computer graphic reconstructed sets.

“The real concept in Bullet Time was to move the distance,” Gaeta says. “It was a simulation of Neo [Keanu Reeves] falling over with the bullet flying past. We had to go from in front to behind instantly. We wound up compositing to display the concept of a virtual camera; the technology couldn’t produce the virtual image, but the concept was there.”

Gaeta knew what he wanted to do – not just create virtual backgrounds as had Paul Debevec for his film “The Campanile Movie,” which had inspired him and which Borshukov had worked on – but to extend that concept to people. Together, Borshukov, Libreri and Piponi began working on the problem. “Larry and Andy [Wachowski] had the concept of Neo self-actualized at the end of ‘The Matrix,’” Gaeta says. “So, we needed to explore how to create a super human, a virtual human.”

Borshukov jumped on the idea quickly. “He showed me a research study from Carnegie Mellon of some boys playing basketball who were captured with multiple cameras and their forms reconstructed,” Gaeta says. “Something had crossed his mind there, so we decided to set up a test.”

They set up three digital cameras and shot tests of a friend saying lines of dialog. From those tests, Borshukov extrapolated the form of her face frame by frame and projected her face onto a CG model.

“That led to universal capture,” Gaeta says. “My dream at the time, I don’t know if it’s so much that now, was to create dynamic virtual components of everything; to push this into sampled cinema. I was still trained on the idea of real-time cinema, on virtual cinematography because I think it’s absolutely a horizon that will be a gateway for many things. But I went into this honeymoon dream mode with esoteric ideas that included things having nothing to do with movies. I’d be like, we can go to Saudi Arabia and get a king to pay us 10 million dollars to make a virtual version of himself. It could be a side business.”

For “The Matrix Revolutions” the crew used universal capture most notably for the “Super Burly Brawl” and Neo’s (Keanu Reeves’) remarkable punch that distorts Agent Smith’s (Hugo Weaving’s) photorealistic face in slow motion. But, after “Revolutions,” Gaeta put virtual cinematography aside. I wanted to deconstruct and Larry and Andy were in the same mood,” he says. “‘Matrix was a strange ride. The sequels were such a challenge in every possible way for all involved. Exhausting physically, emotionally and intellectually.”

“The over sell created a vacuum that caused disappointment from people who wanted a different meaning from the film,” he continues, “and people lost interest in seeing deeper into the fabric of its making. I thought our teams did revolutionary work, so it was a bit of a strange aura after. We all took a break to rethink. Look at the world. Consume life.”

When they reconnected, it was for “Speed Racer.” The Wachowski brothers brought back production designer Owen Paterson and art director Hugh Bateup from ‘The Matrix’ as well as Gaeta and Dan Glass, who had co-supervised the last two films in the trilogy. Gaeta and Glass pulled together a team that included several ‘Matrix’ trilogy alumni including Kim Libreri, Mohen Leo, and Haarm-Pieter Duiker now at Digital Domain, Lubo Hristov at Christov Design, Matt McDonald at Evil Eye, and many others. In addition to those studios, the visual effects supervisors enlisted help from Sony Pictures Imageworks, Industrial Light & Magic, BUF, and several other artists and studios all of whom would find themselves tossing most of what they’d learned out the window.

“’Speed Racer’ is the antithesis of ‘The Matrix,’” Gaeta says. “It’s bright, colorful, poptomistic. The whole frame of mind is different. We went a level away from photorealism and the perfect integration of all things.”

To create the X-game action of ‘Speed Racer,” CG cars zip around roller-coaster tracks and the drivers practice “car fu.” In these sequences, the highly reflective racecars speed inside CG stadiums with cheering digital crowds. The drivers are often the only live action elements.

For the narrative sequences, and some of the races, visual effects artists led by Hristov created “bubbles,” 360-degree virtual backgrounds from photographs stitched together by Dennis Martin, another Matrix alum, photographed elements, matte paintings, and 3D models. They layered these bubbles one inside another, cut holes between them using transparency, and produced animated backgrounds, much as if they were creating 2D animation – an homage to the anime television series of the same name that inspired “Speed Racer.” Gaeta has named the technique “photo anime.”

On the “Speed Racer” set, the director of photography David Tattersall, who was the cinematographer for “Star Wars Episodes I, II and III,” referenced these virtual locations to light the actors. Also on set, Gaeta and McDonald used a game-engine based, real-time compositing system developed at Digital Domain to composite the live action footage from the HD cameras into the bubbles – that is, into the virtual locations - for the directors, DP and actors to see.

The filmmaking process was fluid: The visual effects team changed the cinematography. The editors moved layers in the backgrounds. The filmmakers used the virtual backgrounds and other effects to make emotional story points using heart-shaped defocus blurs, for example, in a romantic scene – much as in a 2D cartoon or anime.

“‘Speed Racer is the first movie I’ve worked on where it seemed we had the potential to carve out a new format for a movie,” Gaeta says. “It’s a work in progress, of course. But we could let our hair down and break some conventions of cinematography.”

“You can see a thread through all the Wachowski projects and my collaborations in visual effects design through all these years,” he adds. “Visual effects always serves stories; the glue of this work is in changing the perspective and perception of events in stories.”