FAU > Technische Fakultät > Informatik > Lehrstuhl 15 (Digital Reality)
Pramod Rao1, Mallikarjun B R1, Gereon Fox1, Tim Weyrich2, Bernd Bickel3, Hans-Peter Seidel1, Hanspeter Pfister4, Wojciech Matusik5, Ayush Tewari5, Christian Theobalt1, Mohamed Elgharib1
1 Max Planck Institute for Informatics, Saarbrücken, Germany
2 Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
3 Institute for Science and Technology Austria
4 Harvard University
5 Massachusetts Institute of Technology
Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handing both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take evena single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learnt in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illuminations.
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, Mohamed Elgharib. Proc. British Machine Vision Conference (BMVC), 14 pages, Nov 2022. Best Paper Honourable Mention Award.Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, November 2022. selected for oral presentation.Rao, P., B R, M., Fox, G., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Tewari, A., Theobalt, C., and Elgharib, M. 2022. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC), BMVA Press. selected for oral presentation.P. Rao, M. B R, G. Fox, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, A. Tewari, C. Theobalt, and M. Elgharib, “VoRF: Volumetric relightable faces,” in Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, Nov. 2022, selected for oral presentation. |
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, Mohamed Elgharib. Invited to International Journal of Computer Vision (IJCV), Springer, 2023. [Web Page] |
This work was supported by the ERC Consolidator Grant 4DReply (770784).