FAU  >  Technische Fakultät  >  Informatik  >  Lehrstuhl 15 (Digital Reality)

 
[A Deeper Analysis of Volumetric Relightiable Faces]

A Deeper Analysis of Volumetric Relightiable Faces

Pramod Rao1,  Mallikarjun B R1,  Gereon Fox1,  Tim Weyrich2,  Bernd Bickel3,  Hanspeter Pfister4,  Wojciech Matusik5,  Fangneng Zhan1,  Ayush Tewari5,  Christian Theobalt1,  Mohamed Elgharib1

1 Max Planck Institute for Informatics, Saarbrücken, Germany
2 Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
3 Institute for Science and Technology Austria
4 Harvard University
5 Massachusetts Institute of Technology

Abstract

Portrait viewpoint and illumination editing is an important problem with several applications in VR/AR, movies, and photography. Comprehensive knowledge of geometry and illumination is critical for obtaining photorealistic results. Current methods are unable to explicitly model in 3D while handling both viewpoint and illumination editing from a single image. In this paper, we propose VoRF, a novel approach that can take even a single portrait image as input and relight human heads under novel illuminations that can be viewed from arbitrary viewpoints. VoRF represents a human head as a continuous volumetric field and learns a prior model of human heads using a coordinate-based MLP with individual latent spaces for identity and illumination. The prior model is learned in an auto-decoder manner over a diverse class of head shapes and appearances, allowing VoRF to generalize to novel test identities from a single input image. Additionally, VoRF has a reflectance MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time (OLAT) images under novel views. We synthesize novel illuminations by combining these OLAT images with target environment maps. Qualitative and quantitative evaluations demonstrate the effectiveness of VoRF for relighting and novel view synthesis, even when applied to unseen subjects under uncontrolled illumination. This work is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide extensive evaluation and ablative studies of our model and also provide an application, where any face can be relighted using textual input.

Citation Style:    Publication

A Deeper Analysis of Volumetric Relightiable Faces.
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Fangneng Zhan, Ayush Tewari, Christian Theobalt, Mohamed Elgharib.
International Journal of Computer Vision (IJCV), Springer, 2023.
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Fangneng Zhan, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision (IJCV), October 2023.Rao, P., B R, M., Fox, G., Weyrich, T., Bickel, B., Pfister, H., Matusik, W., Zhan, F., Tewari, A., Theobalt, C., and Elgharib, M. 2023. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision (IJCV)(Oct.).P. Rao, M. B R, G. Fox, T. Weyrich, B. Bickel, H. Pfister, W. Matusik, F. Zhan, A. Tewari, C. Theobalt, and M. Elgharib, “A deeper analysis of volumetric relightiable faces,” International Journal of Computer Vision (IJCV), Oct. 2023.

Related Publications and Reports

[VoRF: Volumetric Relightable Faces]
VoRF: Volumetric Relightable Faces.
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, Mohamed Elgharib.
Proc. British Machine Vision Conference (BMVC), 14 pages, Nov 2022.
Best Paper Honourable Mention Award.
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, November 2022. selected for oral presentation.Rao, P., B R, M., Fox, G., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Tewari, A., Theobalt, C., and Elgharib, M. 2022. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC), BMVA Press. selected for oral presentation.P. Rao, M. B R, G. Fox, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, A. Tewari, C. Theobalt, and M. Elgharib, “VoRF: Volumetric relightable faces,” in Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, Nov. 2022, selected for oral presentation.
[Web Page][PDF (5.0 MB)][Suppl. Material (36 MB)][Video (39 MB)][Video (55 MB)][External Project Page][BibTeX]
[PhotoApp: Photorealistic Appearance Editing of Head Portraits]
PhotoApp: Photorealistic Appearance Editing of Head Portraits.
Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, Christian Theobalt.
Transactions on Computer Graphics (Proc. SIGGRAPH), 40(4), 44:1–44:16, July 2021.
Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, and Christian Theobalt. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH), volume 40, pages 44:1–44:16, July 2021.B R, M., Tewari, A., Dib, A., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Chevallier, L., Elgharib, M., and Theobalt, C. 2021. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH), vol. 40, 44:1–44:16.M. B R, A. Tewari, A. Dib, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, L. Chevallier, M. Elgharib, and C. Theobalt, “PhotoApp: Photorealistic appearance editing of head portraits,” in Transactions on Graphics (Proc. SIGGRAPH), vol. 40, no. 4, Jul. 2021, pp. 44:1–44:16. [Online]. Available: https://doi.org/10.1145/3450626.3459765
[Web Page][Preprint PDF (14 MB)][BibTeX][DOI]
[Monocular Reconstruction of Neural Face Reflectance Fields]
Monocular Reconstruction of Neural Face Reflectance Fields.
Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt.
Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.
Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, and Christian Theobalt. Monocular reconstruction of neural face reflectance fields. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.B R, M., Tewari, A., Oh, T.-H., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Elgharib, M., and Theobalt, C. 2021. Monocular reconstruction of neural face reflectance fields. InProc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).M. B R, A. Tewari, T.-H. Oh, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, M. Elgharib, and C. Theobalt, “Monocular reconstruction of neural face reflectance fields,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2021.
[Web Page][Preprint PDF (21 MB)][BibTeX]
[Analysis of Human Faces using a Measurement-Based Skin Reflectance Model]
Analysis of Human Faces using a Measurement-Based Skin Reflectance Model.
Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, Markus Gross.
ACM Transactions on Graphics (Proc. SIGGRAPH), pp. 1013–1024, Boston, MA, July 2006.
Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006), 25(3):1013–1024, 2006.Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006) 25, 3, 1013–1024.T. Weyrich, W. Matusik, H. Pfister, B. Bickel, C. Donner, C. Tu, J. McAndless, J. Lee, A. Ngan, H. W. Jensen, and M. Gross, “Analysis of human faces using a measurement-based skin reflectance model,” ACM Trans. on Graphics (Proc. SIGGRAPH 2006), vol. 25, no. 3, pp. 1013–1024, 2006.
[Web Page][PDF][Video][MERL/ETH Skin Reflectance Database][BibTeX][DOI]

Acknowledgments

Open Access funding enabled and organized by Projekt DEAL.


Privacy: This page is free of cookies or any means of data collection. Copyright disclaimer: The documents contained in these pages are included to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.