FAU  >  Technische Fakultät  >  Informatik  >  Lehrstuhl 15 (Digital Reality)

 
[Lite2Relight: 3D-aware Single Image Portrait Relighting]

Lite2Relight: 3D-aware Single Image Portrait Relighting

Pramod Rao1,  Gereon Fox1,  Abhimitra Meka2,  Mallikarjun B R1,  Fangneng Zhan1,  Tim Weyrich3,  Bernd Bickel4,  Hanspeter Pfister5,  Wojciech Matusik6,  Mohamed Elgharib1,  Christian Theobalt1

1 Max Planck Institute for Informatics, Saarbrücken, Germany
2 Google AR/VR
3 Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU)
4 Institute for Science and Technology Austria
5 Harvard University
6 Massachusetts Institute of Technology

Abstract

Achieving photorealistic 3D view synthesis and relighting of human portraits is pivotal for advancing AR/VR applications. Existing methodologies in portrait relighting encounter substantial limitations in terms of generalization and 3D consistency, coupled with inaccuracies in physically realistic lighting and identity preservation. Furthermore, personalization from a single view is difficult to achieve and often requires multiview images during the testing phase or involves slow optimization processes. This paper introduces Lite2Relight , a novel technique that can predict 3D consistent head poses of portraits while performing physically plausible light editing at interactive speed. Our method uniquely extends the generative capabilities and efficient volumetric representation of EG3D, leveraging a lightstage dataset to implicitly disentangle face reflectance and perform relighting under target HDRI environment maps. By utilizing a pre-trained geometry-aware encoder and a feature alignment module, we map input images into a relightable 3D space, enhancing them with a strong face geometry and reflectance prior. Through extensive quantitative and qualitative evaluations, our method outperforms the state-of-the-art methods in terms of efficacy, photorealism, and practical application. This includes producing 3D-consistent results of the full head, including hair, eyes, and expressions. Lite2Relight paves the way for large-scale adoption of photorealistic portrait editing in various domains, offering a robust, interactive solution to a previously constrained problem.

Citation Style:    Publication

Lite2Relight: 3D-aware Single Image Portrait Relighting.
Pramod Rao, Gereon Fox, Abhimitra Meka, Mallikarjun B R, Fangneng Zhan, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, Christian Theobalt.
SIGGRAPH 2024 Conference Papers, to appear, July 2024.
Pramod Rao, Gereon Fox, Abhimitra Meka, Mallikarjun B R, Fangneng Zhan, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Mohamed Elgharib, and Christian Theobalt. Lite2relight: 3d-aware single image portrait relighting. In SIGGRAPH Conference Papers, New York, NY, USA, July 2024. Association for Computing Machinery.Rao, P., Fox, G., Meka, A., B R, M., Zhan, F., Weyrich, T., Bickel, B., Pfister, H., Matusik, W., Elgharib, M., and Theobalt, C. 2024. Lite2relight: 3d-aware single image portrait relighting. InSIGGRAPH Conference Papers, Association for Computing Machinery, New York, NY, USA.P. Rao, G. Fox, A. Meka, M. B R, F. Zhan, T. Weyrich, B. Bickel, H. Pfister, W. Matusik, M. Elgharib, and C. Theobalt, “Lite2relight: 3d-aware single image portrait relighting,” in SIGGRAPH Conference Papers. New York, NY, USA: Association for Computing Machinery, Jul. 2024.

Acknowledgments

This work was supported by the ERC Consolidator Grant 4DReply (770784). We extend our gratitude to Shrisha Bharadwaj for providing feedback and constant support.


Privacy: This page is free of cookies or any means of data collection. Copyright disclaimer: The documents contained in these pages are included to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.