No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry

Edward Zhang1     Ricardo Martin-Brualla2     Janne Kontkanen2     Brian Curless1,2

1 University of Washington     2 Google





Abstract

Removing objects from images is a challenging problem that is important for many applications, including mixed reality. For believable results, the shadows that the object casts should also be removed. Current inpainting-based methods only remove the object itself, leaving shadows behind, or at best require specifying shadow regions to inpaint. We introduce a deep learning pipeline for removing a shadow along with its caster. We leverage rough scene models in order to remove a wide variety of shadows (hard or soft, dark or subtle, large or thin) from surfaces with a wide variety of textures. We train our pipeline on synthetically rendered data, and show qualitative and quantitative results on both synthetic and real scenes.

Paper

Edward Zhang, Ricardo Martin-Brualla, Janne Kontkanen, and Brian Curless. 2020.
"No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry"
Arxiv Preprint
Arxiv | PDF | Supplementary | BibTex )

Additional Results


More object removal results from real scenes, including hard and soft shadows, very large shadows, and high-contrast textures.

Traditional inpainting approaches require the shadow to be indicated as part of the inpainting mask, whereas our system automatically determines where the shadows are. Our intrinsic decomposition allows us to inpaint shadows intelligently, while a naive inpainting approach hallucinates shadows.

Acknowledgements

This work was supported by Google and the University of Washington Reality Lab.