Every summer, the Vivid festival briefly transforms Sydney into an enchanting wonderland of light, color and music. At the center of the event is Lighting the Sails, the Sydney Opera House’s tradition of turning its unique architecture, which resembles sails or shells, into a breathtaking canvas.



This year’s Lighting the Sails production, Audio Creatures, was directed by Sydney’s Ash Bolland with soundscapes by renowned Brazilian sound designer, Amon Tobin. The elaborate projection-mapped show, which required 16 projectors, reflects Bolland’s interest in the relationship between nature and humans and offers viewers a glimpse of abstract creatures, otherworldly plants and a futuristic chrome world.

In the ice pods scene, a tentacle emerges from a hole in one pod. After exploring its environment, the tentacle appears to feed the other pods from which other tentacles emerge.

Tim Clapham, creative director of Luxx, a Sydney-based motion graphics and 3D animation studio, says Luxx was very privileged to be invited by Spinifex Group to create visuals for Audio Creatures. Working with Mike Tosetto, creative director of Sydney motion design studio, Never Sit Still, and using Cinema 4D, After Effects, V-Ray and X-Particles, as well as custom code, Luxx collaborated with Spinifex to create about 8 minutes of the 15-minute production in seven weeks.

I asked Tim Clapham and Mike Tosetto about their experience working on Lighting the Sails, Audio Creatures, and here’s what they said.

How did you get invited to contribute to this prestigious event?

Clapham: It was a huge honor to be asked to do this. It gets millions and millions of views, and is probably the most viewed projection mapping in the world. It was a once- in-a-lifetime opportunity to do this. We had a couple of other great artists working with us, and everybody put their all into it. You could tell people really knew it was a privilege.

Luxx used Cloth to create the delicate butterfly that emerges from a chrysalis that they also made in C4D.

Luxx used a combination of pose morphs and Cloth simulations for full control of the butterfly reveal.

Tosetto: It’s really cool because when you go and see it, every hand is up holding a camera in the air and there are tripods everywhere. So many people want to share what they’re watching.

Clapham: The Spinifex Group and Ash Bolland approached Luxx independently to collaborate on the project. It was a wonderful coincidence and great to know that both the production company and the director had plenty of faith in our abilities. The project was pretty vast, and Luxx produced around eight minutes of the 3D content, all rendered at 4K. That’s a lot of material to create in seven weeks, which required many long days and nights to craft.

What kind of direction did you get from Ash Bolland?

Clapham: Ash gave great direction and was a pleasure to work with. He had some very specific ideas, but he was open to our interpretations too. If there was a cool way to execute his concept, he was open to that. Ash also had some wonderful references for how he thought the creatures should look, and it was our job to interpret them. He created style frames for each creature, some montaged in Photoshop using 3D renders, and others were hand-drawn sketches. We worked together to rebuild those fantastic creatures in Cinema 4D.

Huge organic shapes grow and morph across the Sydney Opera House’s sails, glowing and pulsing in vibrant colors.

Thousands of tiny parasites were placed across the surface, along with larger specimens that were placed and animated by hand.

Tosetto: He wanted the creatures to feel big and heavy, like Godzilla stomping through a city. The opera house is huge, and the creatures needed to convey weight through slow and substantial movement. That’s a challenge because when you look on a computer screen, it’s so small in comparison. We animated much more slowly than usual to compensate for the large-scale of the projection.

What kinds of creatures did you make?

Clapham: We built many of the creatures and components, including a butterfly, a chrysalis, a jellyfish, ice pods, an octopus, parasites, shells and a magnetic core. Each creature presented its own unique challenge. One shot in particular starts with a chrysalis tearing open to reveal a butterfly. Ash wanted to reveal the butterfly’s wings as individual segments from the wing patters. We ran Cloth simulations on each segment, creating flags flapping in the wind as they unfurl to reveal the butterfly wings.

Parasites resembling jellybeans grow and smother the butterfly wings.

Tosetto: We initially used Cloth for the chrysalis tearing apart, and controlling it was really difficult. We needed full control over where each torn section would go, as well as how long it would take to tear open. We tried several other approaches before eventually rigging each tearing section with joints. After trying a procedural approach, we ended up hand-animating each tear and adding wiggle deformers for a natural look.

Talk about the challenges of getting everything to fit on the opera house’s sails.

Clapham: The opera house’s sails are an incredible canvas, but it is quite an unusual shape, kind of like pieces of an orange. We needed the animation to be constrained within that shape so it didn’t break out and kill the illusion. We had a 3D model of the opera house and the hero projection in position, but we needed to make sure all of the elements fit perfectly. We did a lot with deformers in C4D, and also used reshaping tools in After Effects to ensure everything was pixel perfect to the shape.


The shards surrounding the magnetic core were solved using MoGraph. And a Python Effector simulated the Fibonacci sequence.

We explored multiple avenues in Cinema, which gave us the flexibility to experiment with the ideas we had. For example, when we were working on the magnetic core, which is more like geometry and swirling shapes than a creature, we thought it was going to be a simple shot, but it turned out to be really challenging. We used a custom Python Effector that allowed us to arrange objects using the Fibonnaci sequence. [A series of numbers wherein the next number is found by adding up the two numbers before it.] This was very similar to the natural distribution of plant growth, such as the center of a sunflower. We see that kind of arrangement in everyday life, and it is related to the golden angle, a feature that is often used when laying out design. It worked beautifully, but it was kind of a happy accident that we ended up using that technique.

Tosetto: This was one of those projects where we reached for a little bit of everything, Hair, Cloth, Dynamics, X-Particles, character rigging and lots of simulations. We rendered with V-Ray and completed all of our compositing in After Effects. Spinifex worked on some shots of their own and they took our renders and completed the final composite that was ready for projection. One of the cool things about projection is that the camera never moves, which gives you a lot of opportunities to bend the rules and fix things in post much more easily.

Describe the projection setup for this?

Tosetto: The projections were split into four separate parts, and projected all the way across the harbor. Four projectors were used for the two sails on the left, six projectors for the center sail, three projectors for the right sail and three projectors for the two smaller sales, for a total of 16.

Making the shells wrap around the sails to create a natural curve was difficult, but it was doable using a combination of deformers in C4D and After Effects’ Reshape.

The shells crack open to reveal a squid-like creature that lays thousands of shiny, chrome eggs.

What did you think when you saw it actually being projected?

Clapham: Ash really had a crazy vision. He walks though life seeing things most of us take for granted, but he stops and looks and sees so much beauty in nature and he sees how some creatures are almost like alien beings. Watching this, I thought the creatures felt really accessible, abstract and beautiful. We went out on a limb with this because the whole subject matter is abstract and bizarre, but we kept it fun and light-hearted with creatures that aren’t spiky and aggressive, but kind of fluffy and bouncy.

Tosetto: To be completely honest, I questioned how accessible this would be to the general public, but it has been incredibly well received. People genuinely loved it, and we’ve had very positive feedback. I saw a video of my friend’s children watching Audio Creatures and it was amazing to see their reactions because it was clear they really loved it.

Credits:

Clients: Destination NSW & Sydney Opera House
Sydney Opera House: Brooke Webb, Ben Marshall
Destination NSW: Adam Lowe, Julie Turpie
Interrogate Director: Ash Bolland
Interrogate Executive Producer: Tara Riddell
Audio: Amon Tobin
Spinifex Managing Director: Cyril de Baecque
Spinifex Creative Director: Jason French
Luxx Creative Director: Tim Clapham
Never Sit Still Creative Director: Mike Tosetto
2D / 3D Artists: Tim Clapham, Mike Tosetto, Dan Braga, Alex Barnet, Nejc Polovsak, Rich Nosworthy, Will Skinner, Duncan Dix, Roy Christian, Pepin Portingale, Nicholas Hunter, Marcus Coblyn.
Sound FX: Marcus Longfoot – Full Circle Audio
Projection Company: The Electric Canvas.

Meleah Maynard is a writer and editor in Minneapolis, Minnesota