•  Each year, seven films compete for an Oscar nomination at the Academy of Motion Pictures Arts & Sciences’ visual effects branch annual show and tell, the visual effects “bakeoff.” Once the accountants tally the votes, 12 people, four representing each of the three winning films, become Oscar nominees.

    This year’s bakeoff mix included 'Casino Royale', 'Eragon', 'Night at the Museum', 'Pirates of the Caribbean: Dead Man’s Chest', 'Poseidon', 'Superman Returns', and 'X-Men: The Last Stand'.  During the bakeoff, representatives for each film showed a 15 minute reel with shots from the final film; no making-of’s, no breakdowns, no “how to’s” – the rules are very strict.

    On January 23, the Academy announced the three winning films and the Oscar nominees

     chosen for their work on these films:
    'Pirates of the Caribbean: Dead Man’s Chest' (Buena Vista): John Knoll, Hal Hickel, Charles Gibson and Allen Hall;
    'Poseidon' (Warner Bros.): Boyd Shermis, Kim Libreri, Chaz Jarrett and John Frazier;
    and 'Superman Returns' (Warner Bros.): Mark Stetson, Neil Corbould, Richard R. Hoover and Jon Thum.

    These three films arguably showcase the most ambitious and technically innovative visual effects work this year. For ‘Pirates,’ Industrial Light & Magic invented iMoCap, which helped the crew create the remarkable, all-digital Davy Jones and his crew. For ‘Poseidon,’ ILM developed state of the art systems for simulating enormous volumes of water that sank the ship, and the Moving Picture Company and Scanline raised the bar by
     mixing fire, dust, water and other elements within one simulation. For ‘Superman Returns,’ Sony Pictures Imageworks turned to technology from Paul Debevec to push the state of digital doubles even closer to the camera. But those innovations are only part of each film’s visual effects story. We talked with Oscar nominees from each film about the bakeoff and about this year’s batch of visual effects films. For ‘Superman Returns,’ VFX Supervisor Mark Stetson. For ‘Poseidon,’ VFX Supervisor Boyd Shermis. And for ‘Pirates,’ VFX Supervisor John Knoll and animation director Hal Hickel, both of whom generously gave their time to answer questions in the Davy Jones Appreciation forum last summer. 


    Mark Stetson’s first international recognition was in 1982 for his miniature effects work on Ridley Scott's ‘Blade Runner,’ for which he was the chief model maker. Since then, he has received three visual effects nominations (‘2010,’ ‘Lord of the Rings: The Fellowship of the Rings,’ and ‘Superman Returns’) and won an Oscar for ‘Fellowship.’ In addition, he’s won two BAFTA awards (‘Fellowship’ and ‘The Fifth Element’).

    For ‘Superman Returns,’ he supervised the work of 11 studios: Sony Pictures Imageworks, Framestore CFC, Rhythm & Hues, Rising Sun Pictures, The Orphanage, Photon VFX, Frantic Films, Lola Visual Effects, Pixel Liberation Front, Eden FX, and New Deal Studios, all of which had a hand in bringing home the nomination.

    “In my introductory comments at the bakeoff,” Stetson says, “I talked about all the scenes I had to cut from the demo reel to end up with 15 minutes. On the first cut, I chopped off all the fat and left only the meaty visual effects shots and I still had a half-hour reel. I had to cut a lot of great stuff.”


    Even so, he believes the bakeoff voters recognized the scope of the work, the integration of the effects into the story, and the quality of the work.

    “We had a bunch of environments,” he says, “from a Kansas cornfield to the crystal environments. We had the sky over the eastern seaboard of Manhattan. We had ocean work for a sea rescue that Rhythm & Hues did. Ocean work and crystal island that Framestore CFC’s did. It was a huge body of work. Every place the movie took you was another visual effects environment.”

    The crew called the environments without visual effects “safe zones.” And even one of those few environments, the interior of the Daily Planet, became a visual effects shot when a shockwave hit Metropolis and the crew had to composite a damaged city outside the Perry’s office window.

    The shots left on the Stetson’s reel still showed a wide range of work in ‘Superman Returns.’ The reel begins with a visit to the “fortress of solitude.” Stetson cut from there to young Superman running through the cornfields, and then showed the shuttle disaster sequence created at Imageworks. “I carried it from the main engines firing on the shuttle until Superman guides the Boeing 777 to a touchdown in the ball field,” he says, describing the sequence created at Imageworks. “Then we had a few shots from the ‘listening post’ sequence.” That’s when Superman hovers in the air outside Lois Lane’s house.

    From there, he cut to the bank job, the sea rescue, and Superman hoisting Lex’s island out of the ocean. “We had to show the bullet in Superman’s eye,” he says, referring to a shot created at The Orphanage during the bank heist. He skipped Superman’s return to the fortress of solitude, though, as well as the entire Metropolis ‘disaster’ sequence.

     “I cut Superman’s fight with Lex Luther,” he says, “and instead went from Lex stepping out onto the ledge all the way to the cliff side where Superman falls off. We have a close-up of his face when he starts to fall. I’m sure no one knows that’s a digital shot.”


    The plan had been to use a greenscreen shot of Brandon Routh (Superman) falling into a bag, but the in-camera stunt didn’t work. Instead, Imageworks created a digital Superman. “Because Imageworks did a terrific job with digital Superman, we were able to make the best choices for the film,” Stetson says. “We always tried to use Brandon, but when we couldn’t, we could use CG to improve the shot.”

    To create Routh’s face for digital double close-ups, Imageworks used the Light Stage 2 system that Paul Debevec and his team at the Institute for Creative Technology developed. For facial expressions, the studio used motion capture data based on the FACS (Facial Action Coding System). Motion capture data also helped animators create his physical performance.

    One such performance takes place during one of Shermis’s favorite shots. Even though it was a difficult shot, he didn’t include it in the bakeoff reel, which suggests the depth of the effects in this film. “At the end of the movie, Superman flies past the camera over the entire city of Metropolis,” he says. “It was the hardest shot and one of the prettiest. We shot [the city beneath] backwards. Imageworks had to paint out all the traffic in the island of Manhattan.”


    As for his competitors in the race for the Oscar?
    “Davy Jones was really well rendered and animated and it was cool. It really helped tell the story,” he says. “And the crews did a great job on ‘Poseidon’ – the water effects were really different.  But, actually, I didn’t see any clinker shots in any of the seven films in the bakeoff. All those films deserved to be there and probably others.”

    Stetson singles out ‘Charlotte’s Web’ as a film that deserved to be in the bakeoff, but didn’t make it. “The effects in ‘Charlotte’s Web’ looked really good, but I don’t know which film I’d take off the list to make room for it,” he says. “We’re all getting so good at integrating the work into the film and pushing these barriers all the time, creating characters, duplicating humans, creating environments. As long as we can keep our exploration going and keep creating these different looks the audience will stay interested and perhaps enthralled.”



    Boyd Shermis received a BAFTA nomination for visual effects in the first film he supervised – ‘Speed’ – in 1994. In addition to the Oscar nomination this year for ‘Poseidon,’ he, Rhonda C. Gunner, Kim Libreri, and Philippe Rebours were nominated for a VES Award for Best Single Visual Effect of the Year for the opening sequence in ‘Poseidon.’ In that sequence, the camera closely follows Josh Lucas as he jogs around a photorealistic digital model of the Poseidon.

     “The film didn’t do well,” Shermis says, “So, I was surprised by the Oscar nomination. To be nominated by members of the branch means a lot to me.”

    Shermis believes ‘Poseidon’ pushed the state of the art in two, perhaps three respects. “We took water to another level in volumetric space,” he says. “We had a combination of water, fire, smoke and oil – four different properties all in the same simulation, all aware of and interactive with one another. And, the ship is the largest piece of geometry to be modeled and rendered. ILM made it look perfectly real.”



    Although Industrial Light & Magic and The Moving Picture Company handled most of the shots, CIS Hollywood, Hydraulx, Gentle Giant Studios, Lola Visual Effects, Giant Killer Robots, and Pixel Playground also contributed to the film.

    When Shermis takes on a visual effects project, he looks for studios with expertise in particular areas. “Almost any studio can do any number of things,” he says, “but most have strengths. If I have a specific task within a show, I take it to a studio doing that specific thing.”

    After working in the industry for around 20 years – Shermis began his career at Apogee in 1985 –  he has a good idea about which studios do what best. “I have friends in almost every facility,” he says, “and I keep my eye on what all the studios are doing. They don’t give away their secrets, but they show what they’re up to.”

    So, before settling on ILM to create the 200-foot wave that rolls the ship, he checked around. “[Digital] water is one of the hardest things to do, but more and more studios are doing it because it always comes up,” he says. “Trying to do miniatures in water is a royal pain in the ass and it never works very well because of scale problems. There were three or four companies that did different types of water, but I didn’t get the confidence that the other studios could handle the scope and scale that we needed.”

    To create the water, ILM used a fluid solver developed in conjunction with researchers at Stanford University. To manage the huge number of calculations needed to simulate the ocean of water around the ship, the ocean surface, splashes, spray, foam, bubbles as well as the debris in the water, the studio developed ways to run multiple pieces of the simulation for one frame in parallel on separate processors. Four ILM’ers are up for a second VES award – for Outstanding Created Environment – for their ocean shots.


    Similarly, Shermis needed a studio that could handle the unique requirements for interior shots when the water capsizes the ship. “The inside volume of water was a fraction of what we needed for the exterior,” he says. “And running simulations on smaller scales in internal environments can be accomplished by a broader range of companies. But what complicated it for us was the material inside the lobby and the ballroom. And, we needed water, oil, and smoke. The Moving Picture Company had been working with Scanline to develop fire and water sims, so that was who we chose to do that next most difficult water.”

    And other studios contributed to water shots, as well. “Giant Killer Robots, CIS, and Hydraulx all did different types of dripping liquids in their miniature work,” Shermis says.

    Guessing that many people attending the bakeoff might not have seen the film, Shermis created a reel without regard to the linear narrative. “Everyone knows that the boat rolls over,” he says. “So, I explained the work, not the movie; I showed all the set extensions together, for example.”

    In addition, Shermis wanted to show branch members the range of effects in Poseidon. “There is a large contingent of practical effects people who recognize and vote on their effects,” he says. For example, when the wave rolled the boat, a set piece tipped a ballroom filled with furniture. “We had a gimbal that was about 110 feet long, one of the biggest stage-bound gimbals ever built.”

    But, it’s the water – and the digital ship – that won the day. Shermis believes the quality of work will impact future effects. “People have to move beyond the status quo,” he says. “Anytime we raise the bar, that becomes the standard.”

    As for the other nominees?
    “If you look at Davy Jones and the other characters in ‘Pirates’ – applying motion from their onset iMoCap system to fully rendered 3D characters is state of the art. The onset motion capture that ILM developed for ‘Pirates’ is so brilliant. And when you look at ‘Superman’ – the use of water, the close up CG characters – there are all kinds of things in that film. I think we all felt the pinch of condensed schedules, but we didn’t have the budgetary constraints of many films.”

    “I’m thrilled to be considered in their company,” he adds. “The process has its flaws. I’m surprised that ‘Charlotte’s Web’ didn’t make the bakeoff. But, for all its flaws, being as humble as I can, I think the cream rises to the top. It’s an honor to be nominated by the branch, by our peers.”




    Knoll joined Industrial Light & Magic in 1985 as a motion control camera operator for ‘Captain Eo.’ He moved to the CG side for ‘The Abyss’ and about the same time helped his brother create Photoshop. He’s been nominated for a visual effects Oscar four times – for ‘Star Wars: Episodes I’ and ‘II’ and for ‘Pirates of the Caribbean - The Black Pearl’ and ‘Dead Man’s Chest.’ In addition, he’s been nominated for six VES awards including two this year, for Best Single Visual Effect of the Year (for the Flying Dutchman sequence) and for Outstanding Visual Effects in a Visual Effects Driven Motion Picture.

    Industrial Light & Magic handled most of the visual effects for ‘Pirates’ under Knoll’s supervision. In addition, several other studios helped out: Asylum, The Orphanage, CIS Hollywood, Evil Eye Pictures, Gentle Giant Studios, Pacific Title & Art Studio, Method, Proof, and Tippett Studio.


    “Looking at the whole show, there was a mixture of things in ‘Pirates,’” Knoll says. “We had physical effects, miniatures, CG matte paintings, CG water. But, Davy Jones was the significant aspect; I think everyone really liked Davy Jones. It took a lot of effort to put a system in place that preserved the nuances of Bill Nighy’s performance and was actor and director friendly and a lot of the things we did to make that character happen succeeded.”

    That ILM used a new “production-friendly” motion capture system dubbed iMoCap to help create Davy Jones and his crew is no secret (although many of the technical details of the invention still are). Indeed, we’ve covered ‘Pirates’ in several articles on the site (see links below).

    With iMoCap, ILM captured the actors playing the pirates on location and on set in any environment and lighting condition without using any special equipment; that is, without interfering with the filming in any way. The ability to do this affected everything from how the director framed the shots, to how the actors interacted with each other, to how the film was edited, to how the animators were able to apply the actors’ performances to digital characters.

    But Davy Jones also relied on the skills of animators who painstakingly studied and applied Nighy’s facial performance to the many tentacled character by hand. And, Davy Jones’ success depended on advances in lighting and rendering. “I think we got a high level of realism partly from using more sophisticated shading models that were not computationally practical on previous shows,” says Knoll.

    That high level of realism was evident in every detail. Take Davy Jones’ remarkable digital eyes, for example. For the first ‘Pirates’ film, ILM used actor Geoffrey Rush’s eyes as he transitioned into the pirate Barbarossa. “We hung onto his real life action eyes for about two seconds after the transition,” says Knoll. “It was a sneaky trick that helped us sell the transition.”

    So, Gore [Verbinski] wanted us to do the same thing for Davy,” Knoll continues. “He felt he hadn’t seen anyone create a good character with emotion in close-ups.” (That’s why you sometimes see photographs of Bill Nighy with dots around his eyes. The dots were the backup plan, there to make it easier to extract the real eyes from the live action plates if need be.)

    Knoll knew from his experience on ‘Star Wars,’ that even though the performer is on set, the director sometimes wants to change things later for continuity. “As soon as you know you can replace a performance in CG, the director feels, and he should feel, that he can depart from what was shot on set,” says Knoll. “But if he had to use the eyes, he couldn’t.”

    Also, Knoll wanted to be free to light the character to make it look the best. “We generally started with the lighting from Dariusz [Wolski, director of photography] on set, then we might do things like bring the fill light down a bit. Move the rim light a little to the right. The same things he’d do if he could see the image I was seeing. But if we’d been stuck with live action eyes, we would have been constrained. So one of the first things we did was a close up of Davy’s eyes. Happily, Gore bought it.”

    So, how did they do it?
    “Attention to detail and having really good reference,” says Knoll. “When you don’t have reality, it’s all subjective. But, we had Bill Nighy on set in that lighting reading those lines. And we had really fantastic TDs on the show. They flipped between the real photography on set and our character all the time. This specular highlight needs to be brighter here, darker there…”

    Knoll also singles out the compositing work on ‘Pirates,’ Davy’s tentacle controls that James Tooley and Karen Durlich masterminded, the Kraken controls, the ships at sea, the breaching Dutchman, the Black Pearl miniature on the beach at Cannibal Island. . .
    “And Cannibal Island itself,” he says. “In 75 percent of the shots in that sequence, we replaced the skylines with matte paintings, so there were really difficult composites. When you look at the shots and think about everything in front, the natives with straw headdresses and spears, the cooking fires, the shacks – that was the hardest roto we’ve ever done.”
     As for the other Oscar contenders?
    “I was impressed with the ‘Poseidon’ reel at the bakeoff,” He says. “That’s really, really hard stuff and it was well executed. We’ve never before seen water like it. And ‘Superman’ has a broad scope of work. Digital doubles, environments, vehicles and the crystal island. I think that always appeals to the membership, that it’s not a one trick film.”

    Ironically, one of John Knoll’s first jobs out of college was as a freelance model maker for Greg Jein’s project “Tour of the Universe,” a ride film. Sharing the shop was his fellow Oscar nominee this year, Mark Stetson, who at that time was working on models for ‘Bladerunner.

    Like Stetson and Shermis, Knoll is surprised that Charlotte’s Web didn’t make the bakeoff. “In previous years, it seemed that talking animals equaled a nomination,” he jokes, “So, for a while, Hal [Hickel] and I were talking about adding a talking animal to ‘Pirates,’” he laughs. “Hal suggested we give Kraken some dialog and have Ben Kingsley deliver it.” What would the Kraken say? Knoll deepens his voice and says, “So, Jack, we meet again.”

    In May, we’ll all get a chance to meet Jack again, in ‘Pirates of the Caribbean: At World’s End.’ How will they top last year’s effects for the third ‘Pirates’ film? “The big storm sequence,” Knoll says. We can’t wait.


     Hal Hickel joined ILM in 1996 to work as an animator on ‘Lost World: Jurassic Park.’ In 2001, he became animation supervisor for ‘Artificial Intelligence: AI’ and continued in that role for ‘Star Wars: Episode II – Attack of the Clones,’ ‘Dreamcatcher,’ and both ‘Pirates’ films. He, too, points to Davy Jones as the most visible aspect of ‘Pirates’.”

    “We had great and difficult work throughout the film,” he says. “The Kraken attacks, the work we did augmenting the ships at sea, the CG Flying Dutchman breaching and emerging. But, the biggest single challenge was the crew of the Flying Dutchman and Captain Davy Jones.”

    To create Davy Jones’ body performance, animators applied data captured with ILM’s new iMoCap system of Bill Nighy’s performance on location. “In the past, you might have one person perform the body motion, a famous actor do the voice in a sound studio, and use a third person as reference for the facial performance,” Hickel says. “It’s all sort of fractured. With this way, Bill [Nighy] was on set like everyone else.”


    Video reference of Bill Nighy also helped animators create Davy Jones’ facial performance. “ILM is R&Ding the heck out of facial motion capture, but for this film, where we were taking Bill’s performance and applying it to Davy, we were interpreting one language into another,” he says. “If I’m looking at a shot in dailies with Bill smiling and Davy smiling, but Bill looks sarcastic and Davy looks jolly, I didn’t want to ask what the ‘box’ was doing. I wanted to turn to an animator and say, “Let’s adjust the way the brows are shaped.’ I wanted to say those things to a human brain.”

    Ironicially, using the director-friendly iMoCap system meant retraining the visual effects-savvy director, in a way. “We didn’t want Gore [Verbinski] to shoot a couple of takes and think, ‘We’ll make it cool in post,’” Hickel says. “If he went into it thinking that we’d replace everything anyway, that Bill [Nighy] was a placeholder, then he wouldn’t get the benefit of a terrific actor doing that great thing that happens in the moment.”

     But in addition to the technical challenges inherent in creating Davy Jones, Hickel points to a particular artistic challenge as well, one that’s rarely articulated.

    “It’s tricky,” he explains. “You don’t want to create a character that people think is a guy in makeup. But, on the other hand, you don’t want audiences to know instantly that it’s a CG character. Today, if you show people something that couldn’t possibly be real, they don’t say, ‘Wow.’ And then ask ‘How did you do that?’ They say, ‘Oh, it’s CG.’ Saying ‘Oh, it’s CG,’ is only slightly better than saying it’s terrible CG. It’s a trick they’re not buying.”

    “So because Davy is human sized,” he continues, “and because of his design, and because we didn’t do any press before the release, people looked at him and were confused. They wondered if that was a guy in makeup or maybe an animatronic. They didn’t know which file folder to put him in. It set up a tension between what they think is real and what they saw on screen, and I think that’s one of the things that caught people’s eyes. That tension is like magic.”

    Now, the question becomes whether Davy Jones’ magic will hold for the next film. “I think people responded so well to the character, they won’t care,” Hickel says. “And he gets to do more things.” We can’t wait. Oh, we’ve already said that.

    Related links:
    Pirates of the Caribbean site
    CGSociety ‘Pirates’ feature ‘A Treasure Chest of Techniques’
    CGSociety ‘Davy Jones’ feature ‘Shades of Davy Jones’
    CGSociety ‘Pirates’ concept feature ‘Art Booty’
    ILM’s work on Pirates:
    John Knoll
    Hal Hickel
    Superman Returns’ site
    CGSociety ‘Superman Returns’ feature
    Mark Stetson
    Poseidon site
    Boyd Shermis

    Discuss this article on CGTalk

blog comments powered by Disqus