CGSociety

{digg}

{stumble}

The Beginning of the end for the Uncanny Valley.
Digital Domain reveals its part in Holy Grail of VFX.

CGSociety :: Production Focus
6 January 2009, by Renee Dunlop

The Curious Case of Benjamin Button has one of the strangest character arcs, and involved a most unusual production pipeline and timeline. The Benjamin Button project first came to Digital Domain back in the mid 1990’s when it belonged to Ron Howard, but the technology to bring the story to life was simply not available. Then in 2002, Director David Fincher got involved. Digital Domain (DD) had worked with Fincher for fifteen years, and considers him, according to Button’s VFX Executive Producer Ed Ulbrich, “the patron saint of Digital Domain, one of our most loyal clients. He is a rare director in that he is absolutely fearless, perfectly comfortable pioneering into areas where no one has gone before.”

THE HISTORY OF THE AGE
With a new screenplay by Eric Roth in hand and new technology begging for its moment of truth, the task of how to proceed was back on the table. Fincher didn’t want to use different sized actors wearing makeup, concerned that each time the role was handed off to a new actor the audience would lose the emotional thread and sympathy for the character. The primary goal to Fincher was that the character Benjamin carry the likeness and performance of actor Brad Pitt throughout the film. Various approaches were considered and eliminated. Though Benjamin begins his life as a diminutive old man, he wouldn’t be convincing by simply shrinking Pitt to the appropriate size. Fincher had the idea of casting petite actors to portray the childhood bodies at various stages of Benjamin’s life, but there was no way to photograph Pitt in makeup and stitch his head onto an actor’s body while avoiding lighting, continuity, and tracking issues. Gesticulating while talking, a smile pulling tendons in the neck and chest, the timing of the breathing, were all interconnected. It became clear that DD had to do a photorealistic and perfectly animated CG head, the holy grail of CG, and attach it seamlessly to various sized actors.

Ed Ulbrich. VFX Executive Producer.

By 2004 the film was getting traction from the studios and DD was chewing on the details. “We studied the latest advancements in facial animation,” explained Ulbrich. “One large budget CG movie in production at the time was using state of the art marker based capture. It was stylized, but edging towards photoreal. We looked at some of the best work in that and other films and were very clear that, if our worst shot looked as good as the best shot in that film, we would fail. It’s not a critique of that film, but a comment of the state of the art of using marker based capture. You can put 500 markers on the face, but you still don’t get what happens between those markers, and that was what was critically important to us.” Keyframing was not an option either, since Brad Pitt’s behavioral and gestural performance would be filtered through an animator’s interpretation, not to mention continuity issues working across a team.

Warner Brothers and Paramount commissioned DD to do a one shot test. In five weeks DD created Benjamin, worked out the tracking issues and put a CG head on a body. Everyone was happy, the film was greenlit. But now, DD had to make a character speak, hold up in a close-up, and handle several hundred shots of a character that would make the audience laugh and cry and carry the first 52 minutes of the film. Though the test had worked, it was not sufficient for the entire project. That method was limited to how Pitt looks today, but DD had to model and animate Pitt over a series of decades.

It must be said that several other VFX houses did a lot of other work for the Benjamin Button feature. Extensive Matte Painting work was tackled by the team at Matte World Digital; Greg Strause and his crew at Hydraulx did the Russian snow and the matte painting of scenes in Paris, New York and Russia, as well as head replacements for Cate Blanchett's dance performances and the CG elements for the baby Button. Asylum VFX handled the many scenes in the tugboat, and Lola VFX also worked on the 'youthening' effects, with supervision by Edson Williams.

PIECING IT TOGETHER
Enter Rick Baker. Starting with life casts of Pitt, Baker sculpted three different lifelike maquettes, Benjamin at 60, 70, and 80 years of age. Baker also did life casts from the shoulders up of the various actors playing Benjamin’s body, then grafted on the various heads, resulting in three different busts of Pitt at different ages. Those sculptures were then scanned into a mesh to be retargeted with Pitts acting.

DD’s Character Supervisor Steve Preeg was aware of the work of psychologist Dr. Paul Ekman who researched human emotion and reaction to stimuli. Ekman believed that the face had a series of basic poses he could catalogue, thus creating FACS (Facial Action Coding System), a library of everything the human face can do. Ekman contended our facial muscles are more or less hard wired with root expressions that are universal. It’s a theory used by animators for years, but is usually used by referencing the shapes for keyframing emotion.

By utilizing this theory along with Mova Contour, DD volumetrically captured Pitts’ face doing roughly 120 expressions and applied the information to various 3D models, resulting in literally thousands of models of Pitt. At the same time, they scanned the three maquettes of Benjamin’s head. This type of volumetric capture, where a head is scanned in real time in three dimensions at 24 FPS, results in millions of polygons of surface capture without the dead zones that result between markers. The result was the old Benjamin lined up perfectly to the Mova Contour scan of Pitt’s facial poses. By retargeting the expressions onto the scanned Benjamin heads, DD effectively had the three old Benjamin CG characters performing the FACS with Brad Pitt’s full range of emotions and expressions.

DD then cut up the mesh heads into sub shapes of different poses, creating a relational database of thousands of micro expressions or components of expressions entirely based on the volumetric capture of Pitt. By using this process, DD avoided the issue of human interpretation of how the face would change according to an animator, instead staying completely true to Pitt’s acting. This became the basis of what DD referred to as “emotion capture”.

The next task was bringing this model to life, creating software tools that allowed DD, on a frame by frame basis, to compare the live action performance of Pitt playing Benjamin to the database of micro expressions, then reconstruct a CG model that matched, and seamlessly attach it to the various live action actors in the film. A piece of cake, right? Maybe not. And there is more to go.

ACTING THE PERFECT PART
VFX Supervisor Eric Barba put Tracking Supervisor Marco Maldonado and Jessie James Chisholm, the Integration Lead, on set for all the VFX work over a seven month period. “DD would parse out whatever details the other venders might need. We put together a plan for the elaborate system of tracking the heads in. Traditional tracking is pretty great for most things in a scene because, if the track is not quite right, normally you do 2D reconciliation to make that little bit seamless. That is what this industry is used to. But trying to track the spine and neck together with the CG head is incredibly challenging. When the body and camera is moving around, the lens distortion, that traditional tracking doesn’t give the view depth information to get the accuracy necessary.”

Eric Barba, VFX Supervisor.

Barba proposed that first unit run a set of witness cameras during live action photography that would be synced to the main system. Doug Roble, winner of an Academy Scientific Achievement Award for tracking software, rewrote DD’s tracking code to utilize the data from the witness cams. “It was still an incredible amount of labor for the tracking team, a huge hurdle that scared everybody because it had never been done before.”

The lighting needed particular attention. “The human head,” said Barba, “is extremely sensitive to color and light, and Claudio Miranda’s photography is so beautiful. Our digital heads had to work with the way he lit the set, the way the eye highlights hit and the subtleties they strove for on set. We had to devise a system that would give us more fidelity than we’ve had previously, so wrote a whole new system of rendering to take that data and make it work with the skin. We spent months writing Mental ray shaders and a slew of tools to take all the on-set data from traditional techniques like HDR’s and set surveys, and apply that to shaders and geometry to rebuild 3D sets that were physically accurate, had the right bounce and falloff and color temperature, so that our head always existed as if it were in that set at the same time it was being shot. That was a huge process, day to day during principle photography, to feed all that information back to a system we were building here at the same time.”

On a sound stage in Burbank, while watching select scenes, Pitt stood motionless in the center of the stage surrounded by four Viper cameras in a semicircle while he performed the part of Benjamin. There were no markers on his face, no need for tracking data. Image analysis technology was used to match those shots of Pitt performing to the appropriate background footage, deriving timings of, for example, when an eyebrow or lip moved. DD then mapped out the timings on Pitts face. Character Supervisor Steve Preeg’s team developed code that took the rough data and searched the relational database for shapes that best match the various facial components on a frame by frame basis to reconstruct a 3D GC head. The head motion was taken from the on-location actor.

ANIMATION
Animation Lead Matthias Wittmann, whom Ulbrich rightly referred to as the “genius animator”, was one of only 6 animators on a team of 150 artists. “I was in charge of testing the blend shapes from the Mova scan, of talking to the modelers to see what they could keep and lose, what was the neutral or zero position of the head. For example, a smile. When our rig is smiling, the whole face is somehow deforming. We took into account every millimeter on the face of the real Brad, determining if a deformation was due to the muscles driving the smile, or was it due to, for example, him raising his eyebrows? If you have too much deformation on every part of the face and you mix those with other shapes you get errors of strange deformations happening on other sections of the head. We did this to every single shape.”

Matthias Wittman, Animation Lead.

DD put together a plan using optical flow and machine vision cameras and doing what the Mova Contour system offered. This led them to a facial system based off of Pitts’ face and not off of skin and muscle systems that are traditionally used, and the riggers best guess, a method that allowed them to build a sophisticated rig that only moved like Pitt. As blend shapes were defined, a sophisticated system gradually emerged. The blend shapes used for Pitts face totaled 180, though more were built in.

“That is not a lot, but if they are well developed you don’t need more,” Wittmann explained. “Our eye system was tricky. The animators just needed to move the eyes around and the blend shapes drove the movement around the eyes. Some techniques were added to the head to detect shape animation rotations, and would dial in certain neck deformations and wrinkles under the jaw. There were systems built to animate Benjamin’s tongue and his adams apple when he swallows.

“We thought we would need way more animators. We started with three, and built a pipeline for them that made it really easy to get all the reference images of Brad from the witness cameras on set. All this was in sync and the sound was there, and all they had to press one button and the scene was set up and they could start animating, and click one button to send it on to lighting. The animation was really handy and convenient.” By setting up such a pipeline, they team could pick and choose their best talent without worry of their technical prowess.

“We figured we would be famous after this, or they would chase us out of the country! It’s thrilling working on this not knowing. I’m really happy all the departments were so high class on this, everything from tracking to lighting to compositing, everything was so good and you didn’t feel you were wasting your time because somebody was messing it up.”

THE END OR THE BEGINNING
“There were so many parts to make it seamless,” Eric Barba exclaimed. “So much had to happen during the live action shoot to get the lighting to work. The team writing tools for lighting had to get it right so that head felt like it belonged there. And even if it felt like it belonged there, if it didn’t feel alive and didn’t emote, and it didn’t make you laugh, then it wouldn’t work. That is a huge credit to the animation team, and Steve Preeg and his team of riggers and animators who built that system so that this character becomes believable, the tracking, to get that head perfectly on that spine so that you buy that it exists. And to get the paint and roto work just right, and the loving last bit of artistry which is the composite.

On the tug boat heading for Russia, the first time we see Pitt as Button, only with makeup.

“There is a sensitivity in compositing, and Janelle Croshaw, Paul Lambert and Sonja Burchard, our compositing sups, did an amazing job keeping that consistent through our 325 shots. Once someone works on human skin you realize you have to have such a fine sensitivity to color. The skin falls apart very easily if it gets mooshed around. Our color temperature process of taking in the correct balance of on set lighting, and adjusting the HDR imagery and reapplying it, that whole pipe was built by some really talented people so when the renders came out all the color temperature was correct and the compositors didn’t have to moosh around too much.

“All those pieces have to be there. The eyes, the eyeglasses, the teeth, the tongue, the lips, the moisture, the hair, the right type of light, it all has to work or it all falls apart. There are so many people involved with each one of those shots. That is what I am most proud of, I am most proud of the team.”

WHERE'S BRAD PITT?
The baby is an animatronic puppet. The first time the audience sees Benjamin is when he is a toddler eating at the table, playing with his food. For the next 52 minutes in the movie it’s all the CG head spliced on to other bodies.

The CG Brad Pitt is handed off to the real Pitt in makeup, who remains in the film until he returns as a child again, a feat handled by Lola in a process Fincher called “youthenizing”. Lola also made the Daisy character, played by Cate Blanchett, appear younger.

Pitt, in makeup, is first seen when he is on the tug boat heading to Russia, when Benjamin is sitting on the side of the tugboat reading postcards from Daisy. Brad Pitt was on that site and on that ship. When he saw the movie, he said to Fincher ‘I don’t remember shooting that’. It was because it wasn’t Pitt, it was the CG head on another actor. Said VFX Executive Producer Ed Ulbrich, “That was very satisfying, because if we can fool him…”

This film could have the first all CG haircut in a film. In an extreme close-up on the back of Benjamin’s head. His character spans 35 years of life in this movie and so needed some 60 hair styles.

“We had a screening up at ILM and up until that time I was still a bit nervous, but there is a group of people up there working on something similar. One of them asked, ‘what was my favorite shot?’ and when I told them, they looked at each other and said, ‘that was CG too?’ and I knew that we were in good shape.” -- Eric Barba, VFX Supervisor, Digital Domain.

Ironically, one of the hurdles was finding animators. “We started hiring, thinking everyone would want to work on this movie. It’s Brad Pitt and David Fincher and facial performance that had never been done before! But people find themselves cleaning up motion capture curves or that they don’t have time to go into detail, or they are pushing around deformers that are so slow you are 20 years older after the project. It was hard to get any animator interested!” - Matthias Wittmann, Animation Lead.

Society of Digital Artists

CGWorkshops

CGTODs

Forums

WIKI

Members

CGPortfolio

CGChallenge

CGForum

BSW

CGGallery

Features

CGJobs

CGS Members

Ballistic Publishing

The Society

Learn

Discover

Create & Share

Shop

Social