PersonNeRF:
Personalized Reconstruction
from Photo Collections

Abstract

We present PersonNeRF, a method that takes a collection of photos of a subject (e.g., Roger Federer) captured across multiple years with arbitrary body poses and appearances, and enables rendering the subject with arbitrary novel combinations of viewpoint, body pose, and appearance. PersonNeRF builds a customized neural volumetric 3D model of the subject that is able to render an entire space spanned by camera viewpoint, body pose, and appearance. A central challenge in this task is dealing with sparse observations; a given body pose is likely only observed by a single viewpoint with a single appearance, and a given appearance is only observed under a handful of different body poses. We address this issue by recovering a canonical T-pose neural volumetric representation of the subject that allows for changing appearance across different observations, but uses a shared pose-dependent motion field across all observations. We demonstrate that this approach, along with regularization of the recovered volumetric geometry to encourage smoothness, is able to recover a model that renders compelling images from novel combinations of viewpoint, pose, and appearance from these challenging unstructured photo collections, outperforming prior work for free-viewpoint human rendering.

Video

Gallery

Roger Federer

Novak Djokovic

Rafael Nadal

Serena Williams

Exploring Federer's Space

Resources

@InProceedings{weng_personnerf,
    title     = {Person{N}e{RF}: Personalized Reconstruction From Photo Collections},
    author    = {Weng, Chung-Yi and Srinivasan, Pratul P. and Curless, Brian and Kemelmacher-Shlizerman, Ira},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {524-533}
}

Acknowledgement

We thank David Salesin and Jon Barron for their valuable feedback. This project is a tribute from the first author, a die-hard tennis fan, to Novak, Rafa, Roger, and Serena. He feels blessed to have lived in their era and wishes it would never come to an end.

This work was funded by the UW Reality Lab, Meta, Google, OPPO, and Amazon. Additional support was provided by a sponsored research award from Cisco Research.