Hierarchical deep semantic alignment for cross-domain 3D model retrieval

With the development of deep learning and the widespread application of 3D modeling technology, image-based cross-domain 3D model retrieval has attracted more and more researchers’ attention. Existing methods have achieved success by aligning the feature distributions from different domains. However, previous methods just statistically align the domain-level or class-level feature distributions, leaving sample discriminability a margin to be improved for retrieval. To address this issue, this paper proposes a Hierarchical Deep Semantic Alignment Network (HDSAN) for cross-domain 3D model retrieval, which combines the proposed sample-level semantic enhancement with global domain alignment and class semantic alignment. Concretely, we adopt adversarial domain adaptation at the domain level and dynamically align the class centers of two domains at the class level. To further improve sample discriminability, we design intra-domain and cross-domain triplet center alignment to enhance the semantic representation ability at the sample level. Experiments on two commonly-used cross-domain 3D model retrieval datasets MI3DOR-1 and MI3DOR-2 demonstrate the effectiveness of the proposed method.     

Wasserstein distance feature alignment learning for 2D image-based 3D model retrieval

2D image-based 3D model retrieval has become a hotspot topic in recent years. However, the current existing methods are limited by two aspects. Firstly, they are mostly based on the supervised learning, which limits their application because of the high time and cost consuming of manual annotation. Secondly, the mainstream methods narrow the discrepancy between 2D and 3D domains mainly by the image-level alignment, which may bring the additional noise during the image transformation and influence cross-domain effect. Consequently, we propose a Wasserstein distance feature alignment learning (WDFAL) for this retrieval task. First of all, we describe 3D models through a series of virtual views and use CNNs to extract features. Secondly, we design a domain critic network based on the Wasserstein distance to narrow the discrepancy between two domains. Compared to the image-level alignment, we reduce the domain gap by the feature-level distribution alignment to avoid introducing additional noise. Finally, we extract the visual features from 2D and 3D domains, and calculate their similarity by utilizing Euclidean distance. The extensive experiments can validate the superiority of the WDFAL method.     

A two-stage shape retrieval (TSR) method with global and local features

A robust two-stage shape retrieval (TSR) method is proposed to address the 2D shape retrieval problem. Most state-of-the-art shape retrieval methods are based on local features matching and ranking. Their retrieval performance is not robust since they may retrieve globally dissimilar shapes in high ranks. To overcome this challenge, we decompose the decision process into two stages. In the first irrelevant cluster filtering (ICF) stage, we consider both global and local features and use them to predict the relevance of gallery shapes with respect to the query. Irrelevant shapes are removed from the candidate shape set. After that, a local-features-based matching and ranking (LMR) method follows in the second stage. We apply the proposed TSR system to MPEG-7, Kimia99 and Tari1000 three datasets and show that it outperforms all other existing methods. The robust retrieval performance of the TSR system is demonstrated.     

Similarity retrieval of videos by using 3D C-string knowledge representation

We have proposed a new spatio-temporal knowledge structure called 3D C-string to represent symbolic videos accompanying with the string generation and video reconstruction algorithms. In this paper, we extend the idea behind the similarity retrieval of images in 2D C⁺-string to 3D C-string. Our extended approach consists of two phases. First, we infer the spatial relation sequence and temporal relations for each pair of objects in a video. Second, we use the inferred relations to define various types of similarity measures and propose the similarity retrieval algorithm. By providing various types of similarity between videos, our proposed similarity retrieval algorithm has discrimination power about different criteria. Finally, some experiments are performed to show the efficiency of the proposed approach.     

基于卷积神经网络的高效三维模型检索方法

目前基于视图的三维模型检索已经成为一个研究热点.该方法首先将三维模型表示为二维视图的集合,然后采用深度学习技术进行分类和检索.但是现有的方法在精度和效率方面都有待提升.本文提出了一种新的三维模型检索方法,该方法包括索引建立和模型检索.在索引建立阶段,选择代表性视图输入到训练好的卷积神经网络(Convolutional Neural Network,CNN)中以提取特征和分类,进而根据特征类别对特征进行组织以建立索引在检索阶段,使用CNN和投票算法将输入模型的代表性视图分类为一个类别,然后仅选择这个类别的特征而不是所有类别的特征进行相似度匹配,因此减少了搜索空间.此外,随着用于检索的视图数量的逐渐增加,一旦可以确定三维模型,检索过程将提前终止.实验的数据选用刚性三维模型数据集ModelNet10,ModelNet40和非刚性三维模型数据集McGill10.结果表明,该方法在提升检索效率的同时,确保检索准确率分别高达94％、92％和100％.     

3D object retrieval based on sparse coding in weak supervision

With the rapid development of computer vision and digital capture equipment, we can easily record the 3D information of objects. In the recent years, more and more 3D data are generated, which makes it desirable to develop effective 3D retrieval algorithms. In this paper, we apply the sparse coding method in a weakly supervision manner to address 3D model retrieval. First, each 3D object, which is represented by a set of 2D images, is used to learn dictionary. Then, sparse coding is used to compute the reconstruction residual for each query object. Finally, the residual between the query model and the candidate model is used for 3D model retrieval. In the experiment, ETH, NTU and ALOL dataset are used to evaluate the performance of the proposed method. The results demonstrate the superiority of the proposed method.     

SRI-Net: Similarity retrieval-based inference network for light field salient object detection

The cutting-edge RGB saliency models are prone to fail for some complex scenes, while RGB-D saliency models are often affected by inaccurate depth maps. Fortunately, light field images can provide a sufficient spatial layout depiction of 3D scenes. Therefore, this paper focuses on salient object detection of light field images, where a Similarity Retrieval-based Inference Network (SRI-Net) is proposed. Due to various focus points, not all focal slices extracted from light field images are beneficial for salient object detection, thus, the key point of our model lies in that we attempt to select the most valuable focal slice, which can contribute more complementary information for the RGB image. Specifically, firstly, we design a focal slice retrieval module (FSRM) to choose an appropriate focal slice by measuring the foreground similarity between the focal slice and RGB image. Secondly, in order to combine the original RGB image and the selected focal slice, we design a U-shaped saliency inference module (SIM), where the two-stream encoder is used to extract multi-level features, and the decoder is employed to aggregate multi-level deep features. Extensive experiments are conducted on two widely used light field datasets, and the results firmly demonstrate the superiority and effectiveness of the proposed SRI-Net.     

3D model retrieval by sample based alignment

In 3D model retrieval, preprocessing of 3D models is needed, in which alignment is a key factor that significantly affects retrieval performance. In particular, the anti-rotation image feature can obtain the alignment effect of 3D model views. In practice, the focus of many users of 3D models is not just on retrieval performance, but the use of aligned models for different purposes. In this paper, we propose a method, namely Sample Based Alignment (SBA) for better 3D model alignment and retrieval. In SBA, given a class, a sample model is used as the target for alignment, after which each 3D model in this class is then aligned one by one, i.e., the 3D model is actually rotated. Our experimental results, based on two 3D model datasets and performance comparisons with other methods, demonstrate the superiority of the SBA method over state-of-the-art methods in terms of 3D model retrieval and classification.     

基于视图感知的单视图三维重建算法

尽管由于丢弃维度将3维(3D)形状投影到2维(2D)视图看似是不可逆的,但是从可视化到计算机辅助几何设计,各个垂直行业对3维重建技术的兴趣正迅速增长。传统基于物体深度图或者RGB图的3维重建算法虽然可以在一些方面达到令人满意的效果,但是它们仍然面临若干问题:(1)粗鲁的学习2D视图与3D形状之间的映射;(2)无法解决物体不同视角下外观差异所带来的的影响;(3)要求物体多个观察视角下的图像。该文提出一个端到端的视图感知3维(VA3D)重建网络解决了上述问题。具体而言,VA3D包含多邻近视图合成子网络和3D重建子网络。多邻近视图合成子网络基于物体源视图生成多个邻近视角图像,且引入自适应融合模块解决了视角转换过程中出现的模糊或扭曲等问题。3D重建子网络使用循环神经网络从合成的多视图序列中恢复物体3D形状。通过在ShapeNet数据集上大量定性和定量的实验表明,VA3D有效提升了基于单视图的3维重建结果。     

一种SVM学习框架下的Web3D轻量级模型检索算法

随着Web3D技术的发展,对于互联网检索三维模型的需求越来越迫切,特别是基于草图的模型检索.本文对基于草图的三维模型检索相关技术进行了研究,提出了三维模型轻量化处理算法、基于支持向量机三维模型最佳视点选择算法.本文首先对模型进行简化处理,投影三维模型为多个视点图像.其次,使用支持向量机在草图数据集上学习规则,并根据规则进行相应的视点图像分类,获得最佳视点图像.再次,对视点图像提取梯度直方图特征并进行K-means聚类和索引,减少特征空间,获得三维模型的特征字典.最后,在开源数据集上进行相关的实验并对结果进行分析,相关结果表明方法具有很强鲁棒性、准确性.     

3D model retrieval using weighted bipartite graph matching

In this paper, we propose a view-based 3D model retrieval algorithm, where many-to-many matching method, weighted bipartite graph matching, is employed for comparison between two 3D models. In this work, each 3D model is represented by a set of 2D views. Representative views are first selected from the query model and the corresponding initial weights are provided. These initial weights are further updated based on the relationship among these representative views. The weighted bipartite graph is built with these selected 2D views, and the matching result is used to measure the similarity between two 3D models. Experimental results and comparison with existing methods show the effectiveness of the proposed algorithm.     

Colorful 3D reconstruction at high resolution using multi-view representation

High-quality 3D models should contain accurate shapes, as well as other correct attributes, such as realistic surface color. However, current researches were mostly focused on the reconstruction of shapes. We present a method to reconstruct high-resolution colorful 3D models from single images. Shapes and colors are learned separately, using a coarse-to-fine strategy in which the 3D color is expressed as 3-channel volumes. Colorful volumes share the same spatial dimension with generated shape volumes. We propose orthographic colorful maps to retain and recover projected coordinates and corresponding color for 3D surface points. To achieve a fine granularity increase in the quality of maps from low-resolution to high-resolution, we introduce 2D super resolution during reconstructing 3D shapes and color volumes. Models are carved by utilizing predicted high-resolution silhouette, depth and color details. Experimental results in a subset of the ShapeNet dataset and the Colorful Human dataset show the effectiveness of our method.     

Nonrigid 2D/3D registration of coronary artery models with live fluoroscopy for guidance of cardiac interventions

A 2D/3D nonrigid registration method is proposed that brings a 3D centerline model of the coronary arteries into correspondence with bi-plane fluoroscopic angiograms. The registered model is overlaid on top of interventional angiograms to provide surgical assistance during image-guided chronic total occlusion procedures, thereby reducing the uncertainty inherent in 2D interventional images. The proposed methodology is divided into two parts: global structural alignment and local nonrigid registration. In both cases, vessel centerlines are automatically extracted from the 2D fluoroscopic images, and serve as the basis for the alignment and registration algorithms. In the first part, an energy minimization method is used to estimate a global affine transformation that aligns the centerline with the angiograms. The performance of nine general purpose optimizers has been assessed for this problem, and detailed results are presented. In the second part, a fully nonrigid registration method is proposed and used to compensate for any local shape discrepancy. This method is based on a variational framework, and uses a simultaneous matching and reconstruction process to compute a nonrigid registration. With a typical run time of less than 3 s, the algorithms are fast enough for interactive applications. Experiments on five different subjects are presented and show promising results.     

基于多视图SIFT特征的三维模型检索

研究了一种基于多视图SIFT特征的三维模型检索算法。首先对三维模型进行多视图投影,得到其余方位的三维投影深度图,并在各投影深度图上提取SIFT特征。分别利用按模型特征数比例分次建立码本及建立模型库整体码本这两种方式建立了模型库的码本,继而将模型的SIFT特征聚类量化,并用一个多维向量将其表示出来。通过计算三维模型特征向...     

Multimodality Image Registration Using Spatial Procrustes Analysis and Modified Conditional Entropy

In this paper, we propose a new image registration technique using two kinds of information known as object shapes and voxel intensities. The proposed approach consists of two registration steps. First, an initial registration is carried out for two volume images by applying Procrustes analysis theory to the two sets of 3D feature points representing object shapes. During this first stage, a volume image is segmented by using a geometric deformable model. Then, 3D feature points are extracted from the boundary of a segmented object. We conduct an initial registration by applying Procrustes analysis theory with two sets of 3D feature points. Second, a fine registration is followed by using a new measure based on the entropy of conditional probabilities. Here, to achieve the final registration, we define a modified conditional entropy (MCE) computed from the joint histograms for voxel intensities of two given volume images. By using a two step registration method, we can improve the registration precision. To evaluate the performance of the proposed registration method, we conduct various experiments for our method as well as existing methods based on the mutual information (MI) and maximum likelihood (ML) criteria. We evaluate the precision of MI, ML and MCE-based measurements by comparing their registration traces obtained from magnetic resonance (MR) images and transformed computed tomography (CT) images with respect to x-translation and rotation. The experimental results show that our method has great potential for the registration of a variety of medical images.     

Multi-object image retrieval based on shape and topology

We aim at developing a geometry-based retrieval system for multi-object images. We model both shape and topology of image objects including holes using a structured representation called curvature tree (CT); the hierarchy of the CT reflects the inclusion relationships between the objects and holes. To facilitate shape-based matching, triangle-area representation (TAR) of each object and hole is stored at the corresponding node in the CT. The similarity between two multi-object images is measured based on the maximum similarity subtree isomorphism (MSSI) between their CTs. For this purpose, we adapt a continuous optimization approach to solve the MSSI problem and a very effective dynamic programming algorithm to measure the similarity between the attributed nodes. Our matching scheme agrees with many recent findings in psychology about the human perception of multi-object images. Experiments on a database of 1500 logos and the MPEG-7 CE-1 database of 1400 shape images have shown the significance of the proposed method.     

GPU加速三维面形测量

随着通用计算和图形显示需求的不断增加,图形处理器（Graphics Processing Unit,GPU）在医学、科学计算、图像处理等领域得到了广泛的应用。但它在三维测量领域的应用还只是一个开始。文中基于傅里叶变换轮廓术（Fourier Transform Profilometry,FTP）和三频外差法设计了两套三维测量系统,并利用计算统一设备架构（Compute Unified Device Architecture,CUDA）方法,加速了静态或动态物体的三维重建。在三频外差测量系统中,需要利用高速数字投影模块和相机,同步触发采集小视场表面的12个变形条纹图,然后对图像数据进行处理。实验结果表明:对12幅1 360 pixel1 024 pixel大小的图像进行相位展开运算,GPU方法比CPU方法的效率提高了2 089倍。在基于FTP方法的测量系统中,摄像机只需记录一幅变形条纹图,然后拷贝到显存中,并用CUDA编程的算法进行处理,进而重建出物体的三维面形。基于GPU的FTP方法对一幅1 024 pixel1 280 pixel大小的图像进行计算,其计算时间比CPU方法缩短了27倍。     

基于图像序列的YG算法二维相位恢复

迭代算法相位恢复计算中存在可能陷入局部极小值,导致算法收敛速度慢和不稳定的缺点,为了克服这一缺点,本文利用多个输出平面上光的强度信息,将YG算法推广到空域约束(光强度约束)二维相位恢复问题,提高算法收敛速度.模拟实验结果表明,基于图像序列的YG算法二维相位恢复收敛速度快,相位恢复精度高,同时对噪声具有较好的鲁棒性.