Panoramic Video Textures

Aseem Agarwala1     Colin Zheng1     Chris Pal3     Maneesh Agrawala2     Michael Cohen2     Brian Curless1     David Salesin1,2     Richard Szeliski2    

1University of Washington     2Microsoft Research     3University of Massachusetts, Amherst

Abstract
This paper describes a mostly automatic method for taking the output of a single panning video camera and creating a panoramic video texture (PVT): a video that has been stitched into a single, wide field of view and that appears to play continuously and indefinitely. The key problem in creating a PVT is that although only a portion of the scene has been imaged at any given time, the output must simultaneously portray motion throughout the scene. Like previous work in video textures, our method employs min-cut optimization to select fragments of video that can be stitched together both spatially and temporally. However, it differs from earlier work in that the optimization must take place over a much larger set of data. Thus, to create PVTs, we introduce a dynamic programming step, followed by a novel hierarchical min-cut optimization algorithm. We also use gradient-domain compositing to further smooth boundaries between video fragments. We demonstrate our results with an interactive viewer in which users can interactively pan and zoom on high-resolution PVTs.

Citation
Aseem Agarwala, Colin Zheng, Chris Pal, Maneesh Agrawala, Michael Cohen, Brian Curless, David Salesin, Richard Szeliski. Panoramic Video Textures. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2005), 2005.

Paper
SIGGRAPH 2005 pre-print (700K PDF)

Video
Video (5 min, 720x480, 112 MB, WMV format)

Using Windows Media Player 10? The video may appear a little dark.   Here's how to fix it.

Interactive viewer and data sets
By far, the best way to view our results at their full resolution is within our interactive viewer. Our viewer is a Windows XP application; however, it requires a strong computer with a good graphics card to play the results at their full frame-rate. In our experience, you need at least 1 GB of RAM, and a graphics card with at least 128 MB of texture memory. The waterfall data set is the most intensive, so try one of the others first. Also, the data sets are fairly large; the size of the download is listed in the table below. If you have success on a machine that doesn't meet the above requirements, please let us know!

Viewer (400K Zip file)

Data set
name & size
Input video
(WMV, 1/2 resolution)
Data Set
Waterfall   (104MB)
Yachts  (64MB)
Park  (105MB)
Waterfront  (112MB)



Figure 3, video form
Figure 3 of our paper shows one frame of a comparison between the original video, a result computing using a simple approach, and a result computed using our approach. The full videos of this comparison can be found here. Notice that our approach more faithfully mimics the input, and that it loops seamlessly.


Original video Simple Approach Our approach


Errata
We cited the incorrect paper when introducing the alpha-expansion algorithm and when describing the optimality properties of alpha-expansion. In the fourth paragraph of Section 3.5 and the third paragraph of Section 3.6 we should have cited the following paper rather than Kolmogorov and Zabih 2002.

Boykov, Y., Veksler, O., Zabih, R.. Fast Approximate Energy Minimization via Graph Cuts. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI). vol. 23, no.11, pp.1222-1239, 2001.

Contact Info
aseem@agarwala.org