A Light Stage on Every Desk

^{Soumyadip Sengupta}

^{Brian Curless}

^{Ira Kemelmacher-Shlizerman}

^{Steve Seitz}

^{University of Washington, Seattle}

(a) A light stage is a large spherical rig used to capture and relight subjects using multiple lights and cameras Wenger et al. 2005. (b) We show how any monitor can be used as a light stage by capturing a user's facial appearance as they watch a YouTube video to learn a person specific relighting model. Webcam is on top of the monitor. (c) Captured and (d) synthetically re-lit image with previously unseen lighting.

Every time you sit in front of a TV or monitor, your face is actively illuminated by time-varying patterns of light. This paper proposes to use this time-varying illumination for synthetic relighting of your face with any new illumination condition. In doing so, we take inspiration from the Light Stage work of Debevec et al., who first demonstrated the ability to relight people captured in a controlled lighting environment. Whereas existing light stages require expensive, room-scale spherical capture gantries and exist in only a few labs in the world, we demonstrate how to acquire useful data from a normal TV or desktop monitor. Instead of subjecting the user to uncomfortable rapidly flashing light patterns, we operate on images of the user watching a YouTube video or other standard content. We train a deep network on images plus monitor patterns of a given user and learn to predict images of that user under any target illumination (monitor pattern). Experimental evaluation shows that our method produces realistic relighting results.

Paper

Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, Steve Seitz

A Light Stage on Every Desk

ArXiv 2021.

[Pdf]

[Bibtex]

Results

(A) Post-Capture Performance Relighting: input image appears in the training data, while target lighting is unseen.

Input Ground-Truth Relit (Ours)		Input Ground-Truth Relit (Ours)

Input Relit (Ours)

(B) Personalized Relighting Model: input image and target lighting is not part of the training data.

Input Ground-Truth Relit (Ours)		Input Ground-Truth Relit (Ours)

Input Relit (Ours)

(C) Improving lighting for video calls. During a video call in a poorly-lit room, lighting on the face may change with the content onthe monitor. We can re-lit the video with a ring-light pattern producing temporally consistent well-lit video

Input Relit (Ours)

Acknowledgements

The authors thank the labmates from UW GRAIL especially Xuan Luo and Roy Or-El for helpful discussions and Ahana Mallick, Andrey Ryabstev, Daniel Lichy, Sohini Dutta for participating in data capture. This work was supported by the UW Reality Lab, Facebook, Google, Futurewei.This webpage template is taken from humans working on 3D who borrowed it from some colorful folks.