Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture

Yung-Yu Chuang1     Douglas Zongker1     Joel Hindorff1     Brian Curless1     David Salesin1,2     Richard Szeliski2

1University of Washington     2Microsoft Research

Environment matting is a generalization of traditional bluescreen matting. By photographing an object in front of a sequence of structured light backdrops, a set of approximate light-transport paths through the object can be computed. The original environment matting research chose a middle ground using a moderate number of photographs to produce results that were reasonably accurate for many objects. In this work, we extend the technique in two opposite directions: recovering a more accurate model at the expense of using additional structured light backdrops, and obtaining a simplified matte using just a single backdrop. The first extension allows for the capture of complex and subtle interactions of light with objects, while the second allows for video capture of colorless objects in motion.

Citation (bibTex)
Yung-Yu Chuang, Douglas E. Zongker, Joel Hindorff, Brian Curless, David H. Salesin, and Richard Szeliski. Environment Matting Extensions: Towards Higher Accuracy and Real-Time Capture. In Proceedings of ACM SIGGRAPH 2000, pages 121-130, July 2000


SIGGRAPH 2000 paper (1.5MB PDF)
Technical Report UW-CSE-2000-05-01
(This paper is identical to the SIGGRAPH 2000 paper, but also includes a proof in the appendix explaining why an object never amplifies light from a background.)

Video in SIGGRAPH 2000 video proceedings
720x480 DivX Avi (76.9MB)
360x240 DivX Avi (56.4MB)
(Download DivX codec from www.divx.com)


Towards Higher Accuracy

SIGGRAPH'99 SIGGRAPH'00 Photograph
Comparisons between the composite results of the previously published algorithm, the higher accuracy environment matting technique described here, and reference photographs of the matted objects in front of background images. Lighting in the room contributed a yellowish foreground color F that appears, e.g., around the rim of the pie tin in the bottom row. (a) A faceted crystal ball causes rainbowing due to prismatic dispersion, an effect successfully captured by the higher accuracy technique since shifted Gaussian weighting functions are determined for each color channel. (b) Light both reflects off and refracts through the sides of a glass. This bimodal contribution from the background causes catastrophic failure with the previous unimodal method, but is faithfully captured with the new multi-modal method. (c) The weighting functions due to reflections from a roughly-textured pie tin are smooth and fairly broad. The new technique with Gaussian illumination and weighting functions handles such smooth mappings successfully, while the previous technique based on square-wave illumination patterns and rectangular weighting functions yields blocky artifacts.

(a) (b) (c) (d)
Oriented weighting functions reflected from a pie tin. (a) The previous method yields blocky artifacts for smooth weighting functions. (b) Using the higher accuracy method with unoriented Gaussians (\theta = 0) produces a smoother result. (c) Results improve significantly when we orient the Gaussians and solve for \theta. In this case, \theta is about 25 degrees over most of the bottom surface (facing up) of the pie tin. (d) Reference photograph.

Towards Real-Time Capture

Sample frames from four environment matte video sequences. Rows (a) and (b) show bubbles being blown in an Ehrlenmeyer flask filled with glycerin, while rows (c) and (d) show a glass being filled with water. Sequences (a) and (c) were captured with no lighting other than the backdrop, so that the foreground color is zero. Sequences (b) and (d) were captured separately, shot with the lights on, and the foreground estimation technique is used to recover the highlights.

Additional results


cyy -a-t- cs.washington.edu