Video Matting of Complex Scenes

Yung-Yu Chuang1     Aseem Agarwala1     Brian Curless1     David Salesin1,2     Richard Szeliski2

1University of Washington     2Microsoft Research

This paper describes a new framework for video matting, the process of pulling a high-quality alpha matte and foreground from a video sequence. The framework builds upon techniques in natural image matting, optical flow computation, and background estimation. User interaction is comprised of garbage matte specification if background estimation is needed, and hand-drawn keyframe segmentations into ``foreground,'' ``background,'' and ``unknown''. The segmentations, called trimaps, are interpolated across the video volume using forward and backward optical flow. Competing flow estimates are combined based on information about where flow is likely to be accurate. A Bayesian matting technique uses the flowed trimaps to yield high-quality mattes of moving foreground elements with complex boundaries filmed by a moving camera. A novel technique for smoke matte extraction is also demonstrated.

Citation (bibTex)
Yung-Yu Chuang, Aseem Agarwala, Brian Curless, David H. Salesin, and Richard Szeliski. Video Matting of Complex Scenes. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2002), Vol. 21, No. 3, pages 243-248, July 2002


SIGGRAPH 2002 paper (3.0MB PDF)

Video in SIGGRAPH 2002 video proceedings
720x480 DivX Avi (52.9MB)
360x240 DivX Avi (39.2MB)
(Download DivX codec from


The results are best viewed in video form. Please watch the SIGGRAPH videos linked above.

Background replacement
Input Composite
Background editing
Input Composite
Input Composite
Additional results (Indeo 5.10 AVI videos)
Input Composite

cyy -a-t-