Scan&Paint: Image-based Projection Painting

Abstract

We present a pop-up projection painting system that projects onto an unknown three-dimensional surface, while the user creates the projection content on the fly. The digital paint is projected immediately and follows the object if it is moved. If unexplored surface areas are thereby exposed, an automated trigger system issues new depth recordings that expand and refine the surface estimate. By intertwining scanning and projection painting we scan the exposed surface at the appropriate time and only if needed. Like image-based rendering, multiple automatically recorded depth maps are fused in screen space to synthesize novel views of the object, making projection poses independent from the scan positions. Since the user's digital paint is also stored in images, we eliminate the need to reconstruct and parametrize a single full mesh, which makes geometry and color updates simple and fast.

Citation Style:    Display Alpha ACM SIGGRAPH IEEEtran Publication

Scan&Paint: Image-based Projection Painting. Vanessa Klein, Markus Leuschner, Tobias Langen, Philipp Kurth, Marc Stamminger, Frank Bauer. 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2021, pp. 517-525.Vanessa Klein, Markus Leuschner, Tobias Langen, Philipp Kurth, Marc Stamminger, and Frank Bauer. Scan&Paint: Image-based projection painting. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pages 517–525, 2021.Klein, V., Leuschner, M., Langen, T., Kurth, P., Stamminger, M., and Bauer, F. 2021. Scan&Paint: Image-based projection painting. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 517–525.V. Klein, M. Leuschner, T. Langen, P. Kurth, M. Stamminger, and F. Bauer, “Scan&Paint: Image-based projection painting,” in 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2021, pp. 517–525. PDF (10 MB) Low-res PDF (3.2 MB) Presentation (YouTube) Demo Video Video (German) BibTeX DOI