Augmented reality and sound

Flartoolkit

Yes I know, it’s the hype of 2009 and we’re in 2010 now, but I haven’t seen anything yet where augmented reality is combined with sound. If you do know examples of augmented reality and audio, please leave a comment. I’d love to see what other people do with it.

This is a small experiment I started half a year ago but never finished. I looked it up again last week because I needed a Flash AR example for a project at work. Here I improved it a bit to make it more presentable.

What it is

It’s basically a sort of theremin. Hold the marker in front of your computer’s webcam and a sound starts to play. Move the marker and the sound changes.

The Flash file

All source files can be downloaded here. They’re a bit messy however. This was just a quick update to an abandoned older experiment, I didn’t take care to write clean code.

To try this you’ll need the marker to move in front of the webcam. Download it here as a PDF file. Print the file and – for the best results – glue it on a piece of cardboard.

The video

Because I know it's a lot of hassle to download and print the marker I recorded a short demonstration screencap video to show how it all works.

How to use it

Click "Allow" on the Flash player settings screen. This allows Flash to switch on the webcam and use the images it films. Then hold the printed marker in front of the webcam to generate some sounds:

• With the marker at the center of the video a sine wave plays a C note.
• Move the marker slowly up and the pitch rises in steps, move down and the pitch goes down. It plays a pentatonic scale in the key of C minor. Two octaves up and two down. With added flattened fifth to make it sound more bluesy.
• Move the marker to the right to add vibrato. The more to the right the faster and the more intense the vibrato gets.
• Move left from the center and a 'note repeat' effect starts. At the outer left edge of the webcam the repeat effect doubles in speed. When the marker is moved vertically at the left new notes start and combine with the previous notes that still play and produce a harp-like sound. Very nice.

This idea could be taken a lot further. There's more information to be had from the marker:

• The distance from the webcam can be measured. To lower the volume the farther the marker is away from the webcam for instance.
• The three rotations in 3D are available: Rotation around the X, Y and Z axis.

The only obstacle for direct musical use is the latency. The delay between a movement and hearing the resulting sound is too big to be able to play along with a record for example.

For the augmented reality code I used FLARToolkit. More information can be found at the English FLARToolKit page and the Start Up Guide.