An inspection and classification method for chip solder joints using color grads and Boolean rules

In order to improve the comprehensive performance of solder joints inspection in three aspects, i.e. high recognition rate, detailed classification of defect types and fast inspection speed, a new detection and classification algorithm of the chip solder joints based on color grads and Boolean rules is developed in this paper. Firstly, the region features, evaluation features and color grads’ features are defined and extracted based on the special solder joint image, which is acquired by a particular image acquisition system composed of a 3-CCD color digital camera and a 3-color (red, green, and blue) hemispherical LED array illumination. Secondly, the models of solder joint types are built based on extracted features and statistical characteristics of solder joint types. Thirdly, the detection and classification method is designed and presented using Boolean rules, then eight common solder joint types, including the acceptable solder joint, pseudo, no solder, lacked solder, excess solder, shifted, tombstone, and miss component, can be classified and detected by the proposed algorithm. Fourthly, the proposed algorithm is optimized to improve the inspection speed based on a parallel computing method. Finally, to evaluate the performance of the proposed method, 79 pieces of PCBs with defects were inspected by the commercial AOI system developed by the authors which integrates the proposed algorithm. Experiment and result analysis illustrates that the proposed method is better than other methods in three aspects, it can detect and classify properly all the eight common types of solder joints, its detailed classification, and high correct rate, which is up to 97.7%, are more useful to the quality control in the manufacturing process, and its inspection speed is faster, thus helping us to improve the efficiency of the manufacturing process.     

CrackEmbed: Point feature embedding for crack segmentation from disaster site point clouds with anomaly detection

Laser-scanned point clouds can be used to represent the 3D as-damaged condition of building structures in a post-disaster scenario. Performing crack detection from the acquired point clouds is a critical component of disaster relief tasks such as structural damage assessment and risk assessment. Crack detection methods based on intensity or normals commonly result in noisy detections. On the other hand, deep learning methods can achieve higher accuracy but require a large dataset of annotated cracks. This research proposes an unsupervised learning framework based on anomaly detection to segment out cracked regions from disaster site point clouds. First, building components of interest are extracted from the point cloud scene using region growing segmentation. Next, a point-based deep neural network is used to extract discriminative point features using the geometry of the local point neighborhood. The neural network embedding, CrackEmbed, is trained using the triplet loss function on the S3DIS dataset. Then, an anomaly detection algorithm is used to separate out the points belonging to cracked regions based on the distribution of these point features. The proposed method was evaluated on laser-scanned point clouds from the 2015 Nepal earthquake as well as a disaster response training facility in the U.S. Evaluation results based on the point-level precision and recall metrics showed that CrackEmbed in conjunction with the isolation forest algorithm resulted in the best performance overall.     

一种基于RANSAC的点云特征线提取算法

点云中提取的特征线在点云处理中具有重要的应用价值,已被应用于对称性检测、表面重建及点云与图像之间的注册等。然而,已有的点云特征线提取算法无法有效地处理点云中不可避免的噪声、外点和数据缺失,而随机采样一致性RANSAC由于具有较高的鲁棒性,在图像和三维模型处理中具有广泛的应用。为此,针对由建筑物或机械部件等具有平面特征的物体扫描得到的点云,提出了一种基于RANSAC的特征线提取算法。本算法首先基于RANSAC在点云中检测出多个平面,然后将每个平面参数化域的边界点作为候选,在这些候选点上再应用基于全局约束的RANSAC得到最终的特征线。实验结果表明,该算法对点云中的噪声、外点和数据缺失具有很强的鲁棒性。     

基于改进YOLO的双网络桥梁表观病害快速检测算法

桥梁表观病害检测是确保桥梁安全的关键步骤. 然而, 桥梁表观病害类型多样, 不同病害间外观差异显著且病害之间可能发生重叠, 现有算法无法实现快速且准确的桥梁多病害检测. 针对这一问题, 对YOLO (You only look once) 进行了改进, 提出了YOLO-lump和YOLO-crack以提高网络检测多病害的能力, 进而形成基于双网络的桥梁表观病害快速检测算法. 一方面, YOLO-lump在较大的滑动窗口图像上实现块状病害的检测. 在YOLO-lump中, 提出了混合空洞金字塔模块, 其结合了混合空洞卷积与空间金字塔池化, 用于提取稀疏表达的多尺度特征, 同时可以避免空洞卷积造成的局部信息丢失; 另一方面, YOLO-crack在较小的滑动窗口图像上实现裂缝病害的检测. 在YOLO-crack中, 提出了下采样注意力模块, 利用1×1卷积和3×3分组卷积分别解耦特征的通道相关性和空间相关性, 可以增强裂缝在下采样阶段的前景响应, 减少空间信息的损失. 实验结果表明, 该算法能够提高桥梁表观病害检测的精度, 同时可实现病害的实时检测.     

基于图像处理的焊点缺陷识别方法的研究

针对生产线上的SMT（表面贴装技术）焊点图像的特点,研究基于图像处理的焊点缺陷识别算法,采用中值滤波、迭代阈值法、Sobel算子等一系列的图像预处理方法,有效抑制了噪声干扰,提高了图像的对比度,提取出较好的图像特征。采用径向基函数（RBF）神经网络对四种焊点缺陷进行识别。仿真结果表明,RBF神经网络很好地克服BP神经网络训练过程收敛依赖于初值和可能出现局部收敛的缺陷,具有较快的运算速度和较好的检测结果,基于图像处理的焊点识别方法是有效的。     

基于PCA和粒子群优化算法的焊点缺陷识别

针对生产线上的表面贴装技术(SMT）焊点图像的特点,提出了一种基于PCA和粒子群算法-误差反向传播（PSO-BP）神经网络的焊点缺陷识别方法。首先使用图像处理技术和CCD传感器对PCB焊点图像进行预处理,采用中值滤波、灰度图像增强、全局阈值法等方法,有效抑制噪声干扰并提高了图像对比度,提取出较好的图像特征。然后运用主成分分析法提取包含焊点86.6%特征信息的5个主成分,并输入到经粒子群算法改进后的BP神经网络。通过具体的实验分析,结果表明改进的BP神经网络具有较好的识别分类效果,能够对正常、多锡、少锡、漏焊四种不同类型的焊点进行识别,准确率达93.22%,算法可靠,在实际生产中能够有效的提高检测效率。     

R-VPCG: RGB image feature fusion-based virtual point cloud generation for 3D car detection

Although 3D object detection methods based on feature fusion have made great progress, the methods still have the problem of low precision due to sparse point clouds. In this paper, we propose a new feature fusion-based method, which can generate virtual point cloud and improve the precision of car detection. Considering that RGB images have rich semantic information, this method firstly segments the cars from the image, and then projected the raw point clouds onto the segmented car image to segment point clouds of the cars. Furthermore, the segmented point clouds are input to the virtual point cloud generation module. The module regresses the direction of car, then combines the foreground points to generate virtual point clouds and superimposed with the raw point cloud. Eventually, the processed point cloud is converted to voxel representation, which is then fed into 3D sparse convolutional network to extract features, and finally a region proposal network is used to detect cars in a bird’s-eye view. Experimental results on KITTI dataset show that our method is effective, and the precision have significant advantages compared to other similar feature fusion-based methods.     

基于球形视频机器人的管道内缺陷检测方法研究

传统的压力管道内表面缺陷检测管道运行安全十分重要,目前检测方法对待检测管道的形状结构要求较高,检测方法低效耗时费力。设计了一种球形视频管道内缺陷检测机器人,通过搭载的高分辨率立体相机拍摄视管道内部频流,编制基于深度卷积神经网络软件对视频流进行带表面缺陷图片分选,采用Fast-RCNN快速区域卷积神经网络对管道缺陷进行标记,通过该标记可实现管道内部缺陷的安全评估,实验结果表明了该检测方法的有效性。     

改进注意力机制的型钢表面微小缺陷检测方法

针对型钢表面缺陷种类多样、微小缺陷占比较大导致的检测效率低、检测精度差的问题,提出了一种基于双重多尺度注意力机制的表面缺陷检测方法DMSA-YOLOv3,实现型钢表面多尺度缺陷快速精确检测。构建了基于通道和空间的双重多尺度注意力模型DMSA,对不同尺度特征进行筛选融合,强化小尺度缺陷的特征权重;改进了YOLOv3模型,使用深度可分离卷积对DarkNet53特征提取主干网络实现轻量化处理,提高检测速度,并构建多尺度长距离上下文特征提取层,使用4种不同扩张率的并行空洞卷积替代全局池化,提高模型对小尺寸缺陷的特征提取能力;构建了融合DMSA模型和改进YOLOv3模型的DMSA-YOLOv3缺陷检测模型,并应用于型钢表面多尺度缺陷检测。实验结果表明：DMSA-YOLOv3模型具有97.6%的多类别平均检测精度和55.3?frame/s的检测速度,与YOLOv3模型相比分别提升了4.7个百分点和24.5?frame/s;最小可检出20×20像素（约10×10?mm2）缺陷,与YOLOv3模型相比提高了6.25倍,有效提升了型钢表面缺陷的检测精度与检测速度。     

基于置信域伪标签策略的半监督三维目标检测

当前基于点云的三维目标检测方法很大程度上依赖于大规模高质量的三维标注。为了减少所需标签量,基于SESS网络提出了一种新的三维目标检测方法：基于置信域伪标签策略的半监督三维目标检测。首先设计了一种置信域伪标签策略,将学生网络的输出分成有标签和无标签两部分,有标签部分利用ground truth进行全监督学习,无标签部分基于教师网络的类别和对象预测置信分数,利用一种有效的过滤机制,筛选出高质量教师预测,并转换成相应伪标签,用于监督学生网络无标签部分。其次,设计了一个TransVote模块,通过Transformer机制,增强每个点云与其邻域点之间的相互注意,聚合点云局部特征。在10%标记数据、mAP@0.25下,该算法在ScanNetV2和SUN RGB-D数据集上分别超越了基准线8.63%、6.75%,显著提高了半监督三维目标检测算法的检测精度。     

焊点质量检测新方法

提出一种基于小波神经网络的焊点质量检测算法。首先对焊点图像进行预处理,然后提出采用形态因子和曲率作为焊点图像特征,最后建立焊点质量检测的小波神经网络模型。实验结果表明,提出的焊点质量检测算法具有较快的处理速度以及较高的准确率。     

基于改进CenterNet的竹条表面缺陷检测方法

在竹条表面缺陷检测中,竹条表面缺陷形状各异,成像环境脏乱,现有基于卷积神经网络（CNN）的目标检测模型面对这样特定的数据时并不能很好地发挥神经网络的优势;而且竹条来源复杂且有其他条件限制,因此没办法采集所有类型的数据,导致竹条表面缺陷数据量少到CNN不能充分学习。针对这些问题,提出一种专门针对竹条表面缺陷的检测网络。该网络的基础框架为CenterNet,而且为提高CenterNet在较少的竹条表面缺陷数据中的检测性能,设计了一种基于从零开始训练的辅助检测模块：在网络开始训练时,冻结采用预训练模型的CenterNet部分,并针对竹条的缺陷特点从零开始训练辅助检测模块;待辅助检测模块损失趋于稳定时,通过一种注意力机制的连接方式将该模块与采用预训练的主干部分进行融合。将所提检测网络与CenterNet以及目前常用于工业检测的YOLO v3在相同训练测试集上进行训练和测试。实验结果表明,所提检测网络的平均精度均值（mAP）在竹条表面缺陷检测数据集上比YOLO v3和CenterNet的mAP分别提高了16.45和9.96个百分点。所提方法能够针对形状各异的竹条表面缺陷进行有效检测,且没有增加过多的时耗,在实际工业运用中具有很好的效果。     

基于轻量化重构网络的表面缺陷视觉检测

基于深度学习的方法在某些工业产品的表面缺陷识别和分类方面表现出优异的性能, 然而大多数工业产品缺陷样本稀缺, 而且特征差异大, 导致这类需要大量缺陷样本训练的检测方法难以适用. 提出一种基于重构网络的无监督缺陷检测算法, 仅使用容易大量获得的无缺陷样本数据实现对异常缺陷的检测. 提出的算法包括两个阶段: 图像重构网络训练阶段和表面缺陷区域检测阶段. 训练阶段通过一种轻量化结构的全卷积自编码器设计重构网络, 仅使用少量正常样本进行训练, 使得重构网络能够生成无缺陷重构图像, 进一步提出一种结合结构性损失和L1损失的函数作为重构网络的损失函数, 解决自编码器检测算法对不规则纹理表面缺陷检测效果较差的问题; 缺陷检测阶段以重构图像与待测图像的残差作为缺陷的可能区域, 通过常规图像操作即可实现缺陷的定位. 对所提出的重构网络的无监督缺陷检测算法的网络结构、训练像素块大小、损失函数系数等影响因素进行了详细的实验分析, 并在多个缺陷图像样本集上与其他同类算法做了对比, 结果表明重构网络的无监督缺陷检测算法有较强的鲁棒性和准确性. 由于重构网络的无监督缺陷检测算法的轻量化结构, 检测1024 × 1024像素图像仅仅耗时2.82 ms, 适合工业在线检测.     

基于联合注意力的复杂纹理瓷砖缺陷检测方法

复杂纹理瓷砖表面存在较多的低可视度小目标缺陷与严重的复杂纹理背景干扰,使应用目标检测方法时易出现较高的误检率和漏检率。为提升复杂纹理瓷砖表面缺陷检测效率,提出了基于通道与空间联合注意力的复杂纹理瓷砖表面缺陷检测方案。首先通过建模深浅层特征通道间关系设计了一种选择性特征融合方法,以提升模型对小目标缺陷的特征表达;其次,提出了通道与空间联合注意力模块,通过通道注意力和空间注意力来筛选关键特征通道和抑制纹理区域,使模型着重于学习缺陷特征以增强模型辨别缺陷与纹理的能力;最后,在复杂纹理瓷砖表面缺陷数据上进行了实验验证。实验结果表明,相较于AFF(attentional feature fusion)和CBAM(convolutional block attention module)方法,选择性特征融合方法和通道与空间联合注意力模块使模型检测性能分别提高了5.3 AP、6.32 AP。最终,实验证明了该方案分别优于现有的瓷砖检测方法YOLOv5和纹理织物缺陷检测AFAM方法1.32 AP、2.12 AP。     

PointSkelCNN: Deep Learning-Based 3D Human Skeleton Extraction from Point Clouds

A 3D human skeleton plays important roles in human shape reconstruction and human animation. Remarkable advances have been achieved recently in 3D human skeleton estimation from color and depth images via a powerful deep convolutional neural network. However, applying deep learning frameworks to 3D human skeleton extraction from point clouds remains challenging because of the sparsity of point clouds and the high nonlinearity of human skeleton regression. In this study, we develop a deep learning-based approach for 3D human skeleton extraction from point clouds. We convert 3D human skeleton extraction into offset vector regression and human body segmentation via deep learning-based point cloud contraction. Furthermore, a disambiguation strategy is adopted to improve the robustness of joint points regression. Experiments on the public human pose dataset UBC3V and the human point cloud skeleton dataset 3DHumanSkeleton compiled by the authors show that the proposed approach outperforms the state-of-the-art methods.     

Pattern classification of solder joint images using a correlation neural network

This paper presents a method of classifying solder joints on printed-circuit boards (PCB), using a neural-network approach. Inherently, the surface of the solder joints is curved, tiny and specularly reflective; it induces a difficulty of taking good images of the solder joints. The shapes of the solder joints tend to vary greatly with soldering conditions; solder joints, even when classified into the same soldering quality, have very different shapes. Furthermore, the position of the joints is not consistent within a registered solder pad on the PCB. Due to these aspects, it has been difficult to determine the visual features and classification criteria for automatic solder-joint inspection. In this research, the solder joints, imaged by using a circular, tiered illumination system of three colored lamps, are represented as red, green and blue colored patterns, showing their surface-slopes. Cross-correlation and auto-correlation of the colored patterns are used to classify the 3D shapes of the solder joints by their soldering qualities. To achieve this, a neural network is proposed, based on a functional link net, with two processing modules. The first preprocessing module is designed to implement the calculation of the correlations in functional terms. The subsequent, trainable module classifies the solder joints, based upon the capability learned from a human supervisor. The practical feasibility of the proposed method is demonstrated by testing numerous commercially manufactured PCBs.     

基于3维点云鸟瞰图的高精度实时目标检测

针对基于3维点云的目标检测问题,提出了一种高精度实时的单阶段深度神经网络,分别在网络特征提取、损失函数设计和训练数据增强等3个方面提出了新的解决方案.首先对点云直接进行体素化来构建鸟瞰图.在特征提取阶段,使用残差结构提取高层语义特征,并融合多层次特征输出稠密的特征图.在回归鸟瞰图上的目标框的同时,在损失函数中考虑二次偏移量以实现更高精度的收敛.在网络训练中,使用不同帧3维点云混合的方式进行数据增强,提高网络的泛化性能.基于KITTI鸟瞰图目标检测数据集的实验结果表明,本文提出的网络仅使用雷达点云的位置信息,在性能上不仅优于目前最先进的鸟瞰图目标检测网络,而且优于融合图像和点云的检测方案,且整个网络运行速度达到20帧/秒,满足实时性要求.     

局部几何与全局结构联合感知的三维形状分类方法

针对复杂结构的三维形状分析与识别问题,提出了新颖的图卷积分类方法,建立了局部几何与全局结构联合图卷积学习机制,有效提高了三维形状数据学习的鲁棒性与稳定性。首先,通过最远点采样与最近邻方法构造局部图,并建立动态卷积算子,有效提取局部几何特征;同时,基于特征域采样构造全局的特征谱图,通过卷积算子获得全局结构信息。进而,构建加权的联合图卷积学习网络模型,引入注意力机制,实现自适应的特征融合。最终,在联合优化目标函数约束下,有效提高特征学习的性能。实验结果表明,融合局部几何与全局结构的联合图卷积网络学习机制,有效提高了深度特征的表示能力及区分性,具有更为优秀的识别力和分类性能。提出的研究方法可应用于大规模三维场景识别、三维重建以及数据压缩,在机器人、产品数字化分析、智能导航、虚拟现实等领域具有着重要的工程意义与广泛的应用前景。     

基于原始点云的三维目标检测算法

针对当前三维目标检测中存在的数据降采样难、特征提取不充分、感受野有限、候选包围盒回归质量不高等问题,基于3DSSD三维目标检测算法,提出了一种基于原始点云、单阶段、无锚框的三维目标检测算法RPV-SSD(random point voxel single stage object detector),该算法由随机体素采样层、3D稀疏卷积层、特征聚合层、候选点生成层、区域建议网络层共五个部分组成,主要通过聚合随机体素采样的关键点逐点特征、体素稀疏卷积特征、鸟瞰图特征,进而实现对物体类别、3D包围盒以及物体朝向的预测。在KITTI数据集上的实验表明,该算法整体表现良好,不仅能够命中真值标签中的目标并且回归较好的包围盒,还能够从物体的不完整点云推测出物体的类别及其完整形状,提高目标检测性能。