"Pixar Points"
PIXAR POINTS - Technology Focus, 15 April, 2010
CGSociety revisits a popular article published so Pixar's leading technologist could reveal the details inside the RenderMan engine.
CGSociety :: Technology Focus
15 April 2010, by Nils O Sandys
Over the last few years, a brand new technology has emerged for creating CGI effects that has already made a big impact on feature film production. It is called point-based rendering, and this powerful technique makes creating global illumination effects for feature film far more practical and efficient than before. In fact, this year the Academy of Motion Picture Arts and Sciences awarded the creators of this innovation, Per Christensen, Michael Bunnell and Christophe Hery, with a Scientific and Engineering Academy award.
In this article, we will look into the development of this important new technology, how point-based rendering works, and what this all means to the future of feature film production as we know it.
Let's start with color bleeding, which is a common problem in computer graphics. (For the uninitiated, color bleeding is an effect that simulates the bouncing of diffuse light between objects, such as a red wall bouncing colored red light onto an adjacent white floor, as seen in the image below.) In fact, color bleeding has been one of the Holy Grails in CGI, because it can be used to render images of extreme photorealism.
Over the years there have been many methods of creating this type of effect, however there has never been a suitable solution for feature film production.
Historically, there have been three main techniques used to create color bleeding. The first method is to simply 'fake it,' by placing lights in a scene to mimic diffuse light bouncing between objects, artistically cheating the effect. This method is not very accurate, requires a lot of time for a set up, and is impractical for scenes with complicated topologies and multiple objects. However for simple scenes where color bleeding can be easily faked, it's the ideal method to render the effect.
The other two methods involve ray tracing and radiosity solutions, both processes which require casting thousands upon thousands of rays (in ray-tracing) or calculating thousands upon thousands of form factors (in traditional radiosity) to calculate the transport of light in a scene.
Both of these methods are easy to set up and can create physically accurate results, but neither technique has seen wide use in major feature film production. While ray tracing and radiosity can create stunning results, the memory consumption and computational overhead when used in actual production has been extremely high. For production-sized datasets these methods were simply impractical to deploy, if not impossible.
In 2004, from the seemingly unlikely world of real-time rendering, came a completely new idea. Enter Michael Bunnell from NVIDIA, with a chapter published in GPU Gems 2 in which he documented his method for generating real-time ambient occlusion (a diffuse shadowing effect) without the use of ray tracing, instead approximating the calculations from an asset called a point cloud.
This was a pioneering idea, and soon Pixar's Per Christensen, a senior RenderMan developer, began to investigate how to implement these same ideas in Pixar's RenderMan, by building on point-based workflows originally developed for calculating subsurface scattering. During early stages, Christensen worked with Sony's Rene Limberger who integrated point-based rendering on pre-production tests for Surf's Up.
Christensen further refined these techniques with ILM's Christophe Hery who applied these methods to Pirates of the Caribbean: Dead Man's Chest. It quickly became apparent that these techniques had major implications for feature film rendering. Bunnell's single chapter had kicked off a minor revolution in CGI for feature film, point-based rendering.
On Pirates, Hery had great success using point-based rendering. Initial tests using the new point-based ambient occlusion cut the rendering times for occlusion from ten hours down to two. Even more incredibly, adding color bleeding was essentially free because illumination could be baked directly into the point clouds. It was a significant breakthrough. Previously, color bleeding would have proved impossible on such demanding shots. With just a few custom tools, ILM's shading and lighting team quickly migrated all the characters to using the new point-based rendering methods to calculate ambient occlusion, color bleeding, and image based lighting. These new shading and lighting tools helped allow Davy Jones, and crew, to be realized so successfully on screen.
Per Christensen was quite impressed with Hery's efforts, "It's frightening, almost, how quickly Christophe was able to integrate it into their pipeline!" Soon Christensen ported his implementation from prototype DSOs to code directly in Pixar's RenderMan, improving performance, accuracy, and expanding the feature set based on the production demands of ILM, Sony, and other studios.
Until this point, Pixar had not used point-based color bleeding in production, but that was about to change in a big way with Pete Doctor's feature film Up. It turned out to be quite easy to integrate the point-based color bleeding methods in Pixar's sophisticated pipeline. In fact, over 90% of the shots in the movie used the new technique. Up's final look was enhanced with the entrance of this new production-ready feature.
Point-based rendering techniques are significant, because they provide an extremely efficient alternative when compared to ray tracing and radiosity for rendering color bleeding and other effects, and the subtle lighting effects on Up would not have been possible without point-based rendering. It is a technique that can efficiently handle a classic CGI problem.
Great, but how does it work?
First, let's talk about point clouds. Basically a point cloud is just what it sounds like, a cloud of points in 3d space that contains one or more channels of data (lighting, occlusion, area, etc) at each point.
Point clouds are important for many purposes beyond point-based rendering, as they are simply a useful structure for caching 3d textures, and Pixar's RenderMan is very efficient at generating point clouds because of the way the REYES algorithm dices geometry.
RenderMan can simply write a point to the point cloud, that contains the point's location in space, its normal, and any data that the user specifies. The image at left shows a typical view of a point cloud.
As mentioned earlier, Michael Bunnell's key idea was to avoid ray tracing and radiosity altogether, relying instead on a point cloud format. So how do we get occlusion and color bleeding with this method? From the point of RenderMan, it is a two step process:
First, a pre-pass generates a point cloud of the scene geometry and illumination in a process informally called "baking a point cloud." Each point includes data about the area of the micropolygon and its surface normal. Once that data is generated it can be cached and reused.
An added benefit of point-based rendering is that color bleeding calculations can be computed nearly as quickly as occlusion due to the fact that shaders only need to be run when the colors are baked into a point cloud, which is a big time saving over ray tracing. The only thing the algorithm needs is for the point baking pass to include the radiance at each point (surface color and illumination). Notice the points are being used to "approximate" the geometric primitives in the scene, which is a more efficient calculation than attempting to ray trace against the actual geometry in the scene.
The second phase occurs at render time, when the baked point cloud is referred to by shaders attached to the scene geometry to calculate the point-based effect. Each of these points is then approximated with an oriented disk in space.
It is simple to compute the analytical color and occlusion contribution of a disk without having to use ray tracing. For efficiency, distant disks, which individually have a weak contribution to the overall result, are clustered and treated as one entity. The inter-disk occlusion calculation also computes the average non-occluded normal which is used for an environment lookup for image based lighting.
This is analogous to the same functionality in ray-traced occlusion. An added benefit of point-based rendering is that color bleeding calculations can be computed nearly as quickly as occlusion due to the fact that shaders only need to be run when the colors are baked into a point
The images below compare the quality of ray-traced occlusion versus point-based approximate occlusion. The visual results are very close in this instance, but a slight over-darkening can be seen in heavily occluded areas of the point-based image. However, for roughly the same visual quality, the point-based approach rendered nearly eight times faster and with a fraction of the memory overhead.
The benefits of point-based approximate GI include:
Noise free images - high frequency "spotty" artifacts associated with ray tracing are avoided.
Faster computation times - 4x to 10x speedup versus ray tracing large scenes.
Color bleeding and image-based lighting (including support for HDRI) - nearly as fast as ambient occlusion.
Lowered memory requirements - geometry does not need to be visible for ray tracing.
Displaced surfaces - no penalty is incurred for rendering displacements, a huge advantage over ray tracing.
Lights and objects can be easily included or excluded from computation.
There are some drawbacks:
The effect can sometimes compute too much occlusion and color bleeding, because the points are being used to approximate the actual scene geometry. This can lead to an over darkening from occlusion or stray color bleeding from hidden surfaces.
Point-based rendering techniques are a welcome addition to the shading and lighting pipeline at Pixar, and, because of their success on Up, are currently being used extensively on all productions.
The use of point-based rendering techniques is reserved for complicated scenes, scenes that have lots of geometry and displacement shaders. Keep in mind point-based rendering is not a silver bullet, but is rather another tool for the technical director's bag of tricks.
At Pixar it is the supervising lighting TDs who must make the decisions as to which rendering features are most appropriate for a particular shot. Color bleeding can still be faked when possible, but when a scene is complex enough, point-based color bleeding is used. Take the case of ambient occlusion, both point-based rendering and ray tracing are used at Pixar to create this effect, and it just depends on the specific details of a shot.
What point-based rendering brings to the table is the ability to render global illumination effects on scenes that aren't possible any other way. Some of the most complicated shots currently being rendered at Pixar are taking over 20 hours and generating point clouds over 300 gigabytes. These shots simply would have not looked the same without point-based rendering.
Point-based color bleeding was used extensively on Up and Toy Story 3.
One of the key benefits of point-based rendering techniques is that they are directable due to the fact that they are not constrained to the behavior of physical lights, as is the case with ray tracing and radiosity solutions. At Pixar, point clouds are generated using a limited set of lights or specific geometry, balancing the efficient use of memory and resources with the best looking result.
Additionally, the effect can be precisely positioned; for example, in some shots it is only the area directly in front of the camera that requires point-based color bleeding. Further away from the camera, the effect is not noticeable, and with this kind of intelligent manipulation the point cloud only needs to include the objects closest to the camera, ignoring the rest of the geometry in the scene. Because of the extreme complexity of certain shots these point clouds can still be over a whopping 300 gigabytes, and generating the same effect on these types of datasets is just not possible with ray tracing or radiosity methods, because of the higher memory cost.
These point-based techniques are now being adopted by the entire industry, as Per Christensen notes, "Point-based color bleeding has now been used in more than 30 movies. It seems that color bleeding has finally made it into the standard production pipeline in most of the large studios. This means that the movies are able to achieve new levels of realism."
Point-based rendering is becoming a useful technique for creating color bleeding and ambient occlusion in feature film productions, as it avoids the inherent limitations of ray tracing or radiosity solutions. Some scenes are simply too complex to be ray-traced, and for scenes of such extreme complexity, point-based rendering is the only viable option. With Pixar's implementation of the technique directly in RenderMan, it is already allowing directors to create movies of unprecedented realism and complexity.
It's certainly something you'll be hearing more of in the future.
At Pixar, the point-based color bleeding is often baked out and read back in using a light. Adjusting a light's intensity makes it easy to locally control the 'global' illumination effect, and, because it's an ordinary light, penumbra, gobos, and other light attributes just work for free. Consequently, incorporating the point-based rendering into Pixar's own pipeline was relatively straightforward, as it involved familiar workflows. To a TD, it behaves just like any other light.
