Progressive refinement imaging with depth-assisted disparity correction

Abstract

In recent years, the increasing on-board compute power of mobile camera devices gave rise to a class of digitization algorithms that dynamically fuse a stream of camera observations into a progressively updated scene representation. Previous algorithms either obtain general 3D surface representations, often exploiting range maps from a depth camera, such as, Kinect Fusion, etc.; or they reconstruct planar (or distant spherical, respectively) 2D images with respect to a single (perspective or orthographic) reference view, such as, panoramic stitching or aerial mapping. Our work sets out to combine aspects of both, reconstructing a 2.5-D representation (color and depth) as seen from a fixed viewpoint, at spatially variable resolution. Inspired by previous work on “progressive refinement imaging”, we propose a hierarchical representation that enables progressive refinement of both colors and depths by ingesting RGB-D images from a handheld depth camera that is carried through the scene. We evaluate our system by comparing it against state-of-the-art methods in 2D progressive refinement and 3D scene reconstruction, using high-detail indoor and outdoor data sets comprising medium to large disparities. As we will show, the restriction to 2.5-D from a fixed viewpoint affords added robustness (particularly against self-localization drift, as well as backprojection errors near silhouettes), increased geometric and photometric fidelity, as well as greatly improved storage efficiency, compared to more general 3D reconstructions. We envision that our representation will enable scene exploration with realistic parallax from within a constrained range of vantage points, including stereo pair generation, visual surface inspection, or scene presentation within a fixed VR viewing volume.

Citation Style:    Display Alpha ACM SIGGRAPH IEEEtran Publication

Progressive refinement imaging with depth-assisted disparity correction. Markus Kluge, Tim Weyrich, Andreas Kolb. Computers & Graphics, 115, pp. 446–460, Oct 2023.Markus Kluge, Tim Weyrich, and Andreas Kolb. Progressive refinement imaging with depth-assisted disparity correction. Computers & Graphics, 115:446–460, October 2023.Kluge, M., Weyrich, T., and Kolb, A. 2023. Progressive refinement imaging with depth-assisted disparity correction. Computers & Graphics 115 (Oct.), 446–460.M. Kluge, T. Weyrich, and A. Kolb, “Progressive refinement imaging with depth-assisted disparity correction,” Computers & Graphics, vol. 115, pp. 446–460, Oct. 2023. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0097849323001656 PDF (4.9 MB) Suppl. Material (208 MB) Low-res Suppl. Material (9.2 MB) External Project Page Publisher’s Page BibTeX DOI

Related Publication

Progressive Refinement Imaging. Markus Kluge, Tim Weyrich, Andreas Kolb. Computer Graphics Forum, 39(1), pp. 360–374, Feb 2020.Markus Kluge, Tim Weyrich, and Andreas Kolb. Progressive refinement imaging. Computer Graphics Forum, 39(1):360–374, February 2020.Kluge, M., Weyrich, T., and Kolb, A. 2020. Progressive refinement imaging. Computer Graphics Forum 39, 1 (Feb.), 360–374.M. Kluge, T. Weyrich, and A. Kolb, “Progressive refinement imaging,” Computer Graphics Forum, vol. 39, no. 1, pp. 360–374, Feb. 2020. [Web Page][PDF (51 MB)][Low-res PDF (2.7 MB)][External Project Page][BibTeX][DOI]