Keyframe-Based Tracking for Rotoscoping and Animation

Aseem Agarwala1     Aaron Hertzmann2     David H. Salesin1,3     Steven M. Seitz1    

1University of Washington     2University of Toronto     3Microsoft Research




Abstract
We describe a new approach to rotoscoping --- the process of tracking contours in a video sequence --- that combines computer vision with user interaction. In order to track contours in video, the user specifies curves in two or more frames; these curves are used as keyframes by a computer-vision-based tracking algorithm. The user may interactively refine the curves and then restart the tracking algorithm. Combining computer vision with user interaction allows our system to track any sequence with significantly less effort than interpolation-based systems --- and with better reliability than "pure" computer vision systems. Our tracking algorithm is cast as a spacetime optimization problem that solves for time-varying curve shapes based on an input video sequence and user-specified constraints. We demonstrate our system with several rotoscoped examples. Additionally, we show how these rotoscoped contours can be used to help create cartoon animation by attaching user-drawn strokes to the tracked contours.

Citation
Aseem Agarwala, Aaron Hertzmann, David Salesin, Steven Seitz. Keyframe-Based Tracking for Rotoscoping and Animation. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2004), 2004.

Paper
SIGGRAPH 2004 pre-print (2.4MB PDF)

Video
Bigger video (720x480, 122 MB, MPEG4 AVI)
Smaller video (360x240, 40 MB, MPEG4 AVI)

Source code
The source code for this project is available here.