Being John Malkovich
ECCV 2010

One person’s expressions (top row) are mapped to another person’s face (bottom row) by real-time matching to an image database. In this case, the input is a video of Cameron Diaz, and the database is formed from a video (John Malkovich, bottom-left 4 images) or an unstructured set of photographs downloaded from the Internet (George W. Bush, bottom-right 4 images).

“Ever wanted to be someone else? Now you can.”
—tagline from the film Being John Malkovich

Abstract -- Given a photo of person A, we seek a photo of person B with similar pose and expression. Solving this problem enables a form of puppetry, in which one person appears to control the face of another. When deployed on a webcam-equipped computer, our approach enables a user to control another person’s face in real-time. This image-retrieval-inspired approach employs a fully-automated pipeline of face analysis techniques, and is extremely general—we can puppet anyone directly from their photo collection or videos in which they appear. We show several examples using images and videos of celebrities from the Internet.

Team

Ira Kemelmacher-Shlizerman - University of Washington
Aditya Sankar - University of Washington
Eli Shechtman - Adobe Systems
Steve Seitz - University of Washington & Google

Video

Being John Malkovich, Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman, and Steven M. Seitz. European Conference on Computer Vision. September 2010.

Files:

  Paper (7 MB PDF)
  Video (15 MB MP4)
  Poster (10 MB PDF)

Acknowledgments:

The authors gratefully acknowledge Jason Saragih for providing the face tracking software. This work was supported in part by Adobe and the University of Washington Animation Research Labs.

In this work we used the following video sequences of celebrities: Cameron Diaz, George Clooney, John Malkovich.

We also used the unstructured collection of photos of George W. Bush from the LFW database.

Questions? Please contact Ira Kemelmacher-Shlizerman

Also, check out our new SIGGRAPH 2011 paper: "Exploring Photobios", which is also the basis to Google's Picasa Face Movies feature!