The CLOUD MIRROR is an interactive augmented reality art installation by Eric Gradman of Monkeys & Robots. Live video captured by a camera and is re-projected on the wall behind the camera, functioning like a “magic mirror.” But the CLOUD MIRROR software alters the images on the way to the screen. It runs an algorithm that tracks faces from frame to frame and also examines each frame for 2D barcodes printed on attendee badges. By pairing each face with a badge, and each badge id with a database row, the CLOUD MIRROR can identify by name whoever is standing in front of the installation.
The CLOUD MIRROR then augments each frame, adding a thought bubble to each face in the image. The contents of that thought bubble are selected from a set of “tags” associated with that person. Tags come from various sources, including Facebook, Twitter, and SMS data.
When registering for the event, attendees were asked to optionally provide their Twitter name, Facebook profile ID, and to answer the question “Where is your favorite place in LA?” In the weeks leading up to the event, the CLOUD MIRROR software sent a friend request to any attendee that provided that information. The poor trusting souls who accepted this request had their personal profile gently data-mined. Specifically, the information captured was “Facebook updates,” “Twitter updates,” and “Facebook relationship status.”
CLOUD MIRROR also capitalized on peoples’ innate desire to embarrass their friends by allowing anyone to anonymously “graffiti” in a thought bubble by sending an SMS message to a special number containing the target’s unique badge ID.
Behind the scenes, CLOUD MIRROR is divided into two parts. The computer vision component is written in C++ using OpenCV’s Viola-Jones face detector; ARToolkitPlus for fiducial glyph identification; and Pango+Cairo for text and graphics rendering. The camera is a Point Grey Flea2 with a wide-angle fixed focus lens and a big LED ring light.
Once the vision component has identified a face and correlated it with a badge ID, it consults an SQLite database for the content of the thought bubble. The SQLite database is kept up to date by a background Python process which is using the Twitter API and Facebook API to periodically grab new statuses.
Both components ran on a Shuttle PC running Ubuntu Linux.
The SMS component was written by Chris Nelson and ran on a separate machine. It fires decoded SMS messages via UDP at yet another Python process, which also updated the database.
The CLOUD MIRROR was unveiled at the April 16 Mindshare event in Los Angeles. Each month, Mindshare features new interactive installations from the members of Mindshare Labs, a collective of artists, scientists, and engineers.
More pictures available here.