No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry

Edward Zhang1     Ricardo Martin-Brualla2     Janne Kontkanen2     Brian Curless1,2

1 University of Washington     2 Google


Removing objects from images is a challenging problem that is important for many applications, including mixed reality. For believable results, the shadows that the object casts should also be removed. Current inpainting-based methods only remove the object itself, leaving shadows behind, or at best require specifying shadow regions to inpaint. We introduce a deep learning pipeline for removing a shadow along with its caster. We leverage rough scene models in order to remove a wide variety of shadows (hard or soft, dark or subtle, large or thin) from surfaces with a wide variety of textures. We train our pipeline on synthetically rendered data, and show qualitative and quantitative results on both synthetic and real scenes.


Edward Zhang, Ricardo Martin-Brualla, Janne Kontkanen, and Brian Curless. 2021.
"No Shadow Left Behind: Removing Objects and their Shadows using Approximate Lighting and Geometry"
The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Arxiv | PDF | Supplementary | BibTex )

Additional Results

More object removal results from real scenes, including hard and soft shadows, very large shadows, and high-contrast textures.

Traditional inpainting approaches require the shadow to be indicated as part of the inpainting mask, whereas our system automatically determines where the shadows are. Our intrinsic decomposition allows us to inpaint shadows intelligently, while a naive inpainting approach hallucinates shadows.


This work was supported by Google and the University of Washington Reality Lab.