Monocular Reconstruction of Neural Face Reflectance Fields

Abstract

The reflectance field of a face describes the reflectance properties responsible for complex lighting effects including diffuse, specular, inter-reflection and self shadowing. Most existing methods for estimating the face reflectance from a monocular image assume faces to be diffuse with very few approaches adding a specular component. This still leaves out important perceptual aspects of reflectance as higher-order global illumination effects and self-shadowing are not modeled. We present a new neural representation for face reflectance where we can estimate all components of the reflectance responsible for the final appearance from a single monocular image. Instead of modeling each component of the reflectance separately using parametric models, our neural representation allows us to generate a basis set of faces in a geometric deformation-invariant space, parameterized by the input light direction, viewpoint and face geometry. We learn to reconstruct this reflectance field of a face just from a monocular image, which can be used to render the face from any viewpoint in any light condition. Our method is trained on a light-stage training dataset, which captures 300 people illuminated with 150 light conditions from 8 viewpoints. We show that our method outperforms existing monocular reflectance reconstruction methods, in terms of photorealism due to better capturing of physical primitives, such as sub-surface scattering, specularities, self-shadows and other higher-order effects.

Citation Style:    Display Alpha ACM SIGGRAPH IEEEtran Publication

Monocular Reconstruction of Neural Face Reflectance Fields. Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt. Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.Mallikarjun B R, Ayush Tewari, Tae-Hyun Oh, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, and Christian Theobalt. Monocular reconstruction of neural face reflectance fields. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021.B R, M., Tewari, A., Oh, T.-H., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Elgharib, M., and Theobalt, C. 2021. Monocular reconstruction of neural face reflectance fields. InProc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).M. B R, A. Tewari, T.-H. Oh, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, M. Elgharib, and C. Theobalt, “Monocular reconstruction of neural face reflectance fields,” in Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2021. Preprint PDF (21 MB) BibTeX

Related Publications and Reports

	A Deeper Analysis of Volumetric Relightiable Faces. Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Fangneng Zhan, Ayush Tewari, Christian Theobalt, Mohamed Elgharib. International Journal of Computer Vision (IJCV), Springer, 2023.Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Fangneng Zhan, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision (IJCV), October 2023.Rao, P., B R, M., Fox, G., Weyrich, T., Bickel, B., Pfister, H., Matusik, W., Zhan, F., Tewari, A., Theobalt, C., and Elgharib, M. 2023. A deeper analysis of volumetric relightiable faces. International Journal of Computer Vision (IJCV)(Oct.).P. Rao, M. B R, G. Fox, T. Weyrich, B. Bickel, H. Pfister, W. Matusik, F. Zhan, A. Tewari, C. Theobalt, and M. Elgharib, “A deeper analysis of volumetric relightiable faces,” International Journal of Computer Vision (IJCV), Oct. 2023. [Web Page][PDF (9.5 MB)][BibTeX][DOI]
	VoRF: Volumetric Relightable Faces. Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, Mohamed Elgharib. Proc. British Machine Vision Conference (BMVC), 14 pages, Nov 2022. Best Paper Honourable Mention Award.Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Ayush Tewari, Christian Theobalt, and Mohamed Elgharib. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, November 2022. selected for oral presentation.Rao, P., B R, M., Fox, G., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Tewari, A., Theobalt, C., and Elgharib, M. 2022. VoRF: Volumetric relightable faces. In Proceedings of the British Machine Vision Conference (BMVC), BMVA Press. selected for oral presentation.P. Rao, M. B R, G. Fox, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, A. Tewari, C. Theobalt, and M. Elgharib, “VoRF: Volumetric relightable faces,” in Proceedings of the British Machine Vision Conference (BMVC). BMVA Press, Nov. 2022, selected for oral presentation. [Web Page][PDF (5.0 MB)][Suppl. Material (36 MB)][Video (39 MB)][Video (55 MB)][External Project Page][BibTeX]
	PhotoApp: Photorealistic Appearance Editing of Head Portraits. Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, Christian Theobalt. Transactions on Computer Graphics (Proc. SIGGRAPH), 40(4), 44:1–44:16, July 2021.Mallikarjun B R, Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans-Peter Seidel, Hanspeter Pfister, Wojciech Matusik, Louis Chevallier, Mohamed Elgharib, and Christian Theobalt. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH), volume 40, pages 44:1–44:16, July 2021.B R, M., Tewari, A., Dib, A., Weyrich, T., Bickel, B., Seidel, H.-P., Pfister, H., Matusik, W., Chevallier, L., Elgharib, M., and Theobalt, C. 2021. PhotoApp: Photorealistic appearance editing of head portraits. In Transactions on Graphics (Proc. SIGGRAPH), vol. 40, 44:1–44:16.M. B R, A. Tewari, A. Dib, T. Weyrich, B. Bickel, H.-P. Seidel, H. Pfister, W. Matusik, L. Chevallier, M. Elgharib, and C. Theobalt, “PhotoApp: Photorealistic appearance editing of head portraits,” in Transactions on Graphics (Proc. SIGGRAPH), vol. 40, no. 4, Jul. 2021, pp. 44:1–44:16. [Online]. Available: https://doi.org/10.1145/3450626.3459765 [Web Page][Preprint PDF (14 MB)][BibTeX][DOI]
	Analysis of Human Faces using a Measurement-Based Skin Reflectance Model. Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, Markus Gross. ACM Transactions on Graphics (Proc. SIGGRAPH), pp. 1013–1024, Boston, MA, July 2006.Tim Weyrich, Wojciech Matusik, Hanspeter Pfister, Bernd Bickel, Craig Donner, Chien Tu, Janet McAndless, Jinho Lee, Addy Ngan, Henrik Wann Jensen, and Markus Gross. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006), 25(3):1013–1024, 2006.Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. on Graphics (Proc. SIGGRAPH 2006) 25, 3, 1013–1024.T. Weyrich, W. Matusik, H. Pfister, B. Bickel, C. Donner, C. Tu, J. McAndless, J. Lee, A. Ngan, H. W. Jensen, and M. Gross, “Analysis of human faces using a measurement-based skin reflectance model,” ACM Trans. on Graphics (Proc. SIGGRAPH 2006), vol. 25, no. 3, pp. 1013–1024, 2006. [Web Page][PDF][Video][MERL/ETH Skin Reflectance Database][BibTeX][DOI]

Acknowledgments

This work was supported by the ERC Consolidator Grant 4DReply (770784).