Pajama Club 3D : Kinect + WebGL

We’ve created a pair of interactive music videos for Pajama Club, Neil Finn’s (of Crowded House fame) latest band. The project was a collaboration with brilliant director Sam Peacocke.

Visit the site here.

All the 3D footage in the video was captured with a Kinect camera and saved with custom software written in openFrameworks. The depth information is saved as a color-coded video. To decode it in the browser, I’ve written a shader for WebGL that transforms the color-coded video into a moving point cloud. The online experience also makes heavy use of ThreeJS, a great 3d engine for WebGL.

There are two songs to play with. In “TNT for Two,” you control the camera as the video cuts between shots of the band performing and a dreamy narrative journey from the sea to forest.

In “These Are Conditions,” things are a bit more unconventional. You, the viewer, have complete control of the scene. You can grab the drummer’s head and swap it for the bass player’s as they’re both still playing and singing. You can duplicate the keyboard players limbs a dozen times making a sea of arms writhing to the beat or create a penguin and re-scale it so it’s 100 metres tall.

I’m hoping to clean up the Kinect recording software and release it for others to use. I’m looking into James George’s RGBDToolkit at the moment, and am hoping I can add to the great work his team have already done.

What I liked about the approach we used on this project is that we always worked with straight video, never OBJ files or PCD files or any other strange format. The depth is saved as video, and the texture data is saved as video. This meant that we could edit it in Final Cut and do clean-up in After Effects.

Neither of us are very experienced with 3D editing software like Maya or Max and luckily we never touched any during the entire project. I think many filmmakers are in the same boat, so if we can come up with a workflow that fits in with the tools they already know, then we’ll start to see some really interesting work come out of this technology. I feel like we’re just scratching the surface of something big here. Depth is a new dimension to filmmaking… at the moment it feels like a gimmick at times, but so did sound and color when they were first introduced to cinema.

[PS]
I’ve got a few more ideas on how to improve the interaction for “These are Conditions.” I think the most helpful one would be a simple way to save your creations. I haven’t built it yet because I’m not sure if anyone would use it. But if you think it’s a good idea, let me know and I’ll get to work!