丝路文化虚拟体验中的多视角立体重建技术研究

丝路文化是联系“一带一路”倡议的重要纽带,其传承意义重大,但是由于历史地理原因,丝路文化中代表性的历史遗产分散或损坏,难以有效地呈现,因此,本文面向丝路文化的虚拟展示与数字化,提出并实现了基于虚拟现实技术的丝路文化传承平台,通过历史遗迹复原以及基于图像的三维重建,还原了丝路文化中重要节点宁夏固原有关的历史遗迹、文物和事件.特别地,本文提出一种面向高清图像的多视角立体三维重建算法,包括采用normal-aware PatchMatch stereo复原高质量的法线图,反映文物表面精细结构,以及提出一种基于GPU的增量式的深度融合方法,以较小的显存处理大规模的数据.在公共数据集和本文收集的室内外文物数据上的实验表明,本文提出的三维重建方法可恢复物体表面的精细结构,同时还对大规模数据具有良好的可扩展性.重建的模型可导入到虚拟互动系统中,对丝路文化的传播起到了积极的作用.

Research on Multi-View Stereo 3D Reconstruction in Virtual Reality System of Silk Road Cultural Inheritance

Silk Road culture is an important link in the Belt and Road initiative. Its heritage is of great significance. However, due to historical and geographical reasons, the representative historical heritage in the Silk Road culture is scattered or damaged, and it is difficult to present the historical heritage effectively.Therefore, in this work we propose and implement a virtual reality platform for the Silk Road Cultural Heritage. Through historical restoration and image-based 3D reconstruction, we effectively restored the historical sites, cultural relics and events of Guyuan of Ningxia in China, one of the important nodes in Silk Road Culture. For outdoor historical sites, we use a DJI Mavic Pro to capture 4K video clips of the giant Buddha of Xumi Mountain in a sunny day. For indoor cultural relics, we use a turntable with digital single lens reflex(DSLR) camera and multiple light sources to capture high-resolution images in 180degrees. Based on these image data, we propose a simple and efficient multi-view stereo 3D reconstruction method for high-resolution images, which consists of a normal-aware Patch Match stereo for the high-quality normal recovery to represent the detailed surface of the cultural relics, and a GPU-friendly incremental depth map fusion method which can fuse a large amount of depth maps by leveraging a small size of GPU memory. The high-resolution input images are essential for representing the geometric details in historical sites and cultural relics. However, the state-of-the-art depth map fusion method needs to import all depth maps and normal maps into the GPU memory, and then globally fuse the depth points into the 3D point clouds for each reference image. Nevertheless, the space complexity almost linearly increases when the amount of data and image resolution increase. For instance, doubling image size will result in a fourfold increase in GPU memory. Due to limitation of GPU memory, this kind of global fusion strategy cannot address high-resolution input image data. The proposed incremental depth map fusion method in this paper mainly consists of three steps:(1) We first set a reference view and a counter map for cross-view consistency check;(2) Then, we import α neighboring images of the reference view into GPU memory each time, and perform the cross-view consistency check for the depth points in reference view. And then, depth points are accumulated, and counter map is also updated. We then release the memory of these α images and import another α images into GPU and repeat the above operations;(3) When all neighboring views are processed, we can fuse the depth points whose values in counter map are larger than a threshold. The quantitative and qualitative experiment results on the public multi-view stereo benchmark as well as our captured datasets clearly highlight that the proposed method can recover the detailed surfaces while keeping a good scalability for the large-scale image data. The reconstructed high-quality 3D models of historical sites and cultural relics by our method can effectively support immersive virtual reality applications,playing a positive role in the dissemination of Silk Road culture.