No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry
Edward Zhang1 Ricardo Martin-Brualla2 Janne Kontkanen2 Brian Curless1,2
1 University of Washington 2 Google
Abstract
Removing objects from images is a challenging problem that is important for many applications, including mixed reality. For believable results, the shadows that the object casts should also be removed. Current inpainting-based methods only remove the object itself, leaving shadows behind, or at best require specifying shadow regions to inpaint. We introduce a deep learning pipeline for removing a shadow along with its caster. We leverage rough scene models in order to remove a wide variety of shadows (hard or soft, dark or subtle, large or thin) from surfaces with a wide variety of textures. We train our pipeline on synthetically rendered data, and show qualitative and quantitative results on both synthetic and real scenes.Paper
Edward Zhang, Ricardo Martin-Brualla, Janne Kontkanen, and Brian Curless. 2021.
"No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry"
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
( Arxiv | PDF | Supplementary | BibTex )
"No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry"
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
( Arxiv | PDF | Supplementary | BibTex )
Additional Results
More object removal results from real scenes, including hard and soft shadows, very large shadows, and high-contrast textures.
Traditional inpainting approaches require the shadow to be indicated as part of the inpainting mask, whereas our system automatically determines where the shadows are. Our intrinsic decomposition allows us to inpaint shadows intelligently, while a naive inpainting approach hallucinates shadows.
Acknowledgements
This work was supported by Google and the University of Washington Reality Lab.