基于深度学习的弱监督语义分割方法综述

语义分割是计算机视觉领域的基本任务,旨在为每个像素分配语义类别标签,实现对图像的像素级理解。得益于深度学习的发展,基于深度学习的全监督语义分割方法取得了巨大进展。然而,这些方法往往需要大量带有像素级标注的训练数据,标注成本巨大,限制了其在诸如自动驾驶、医学图像分析以及工业控制等实际场景中的应用。为了降低数据的标注成本并进一步拓宽语义分割的应用场景,研究者们越来越关注基于深度学习的弱监督语义分割方法,希望通过诸如图像级标注、最小包围盒标注、线标注和点标注等弱标注信息实现图像的像素级分割预测。首先对语义分割任务进行了简要介绍,并分析了全监督语义分割所面临的困境,从而引出弱监督语义分割。然后,介绍了相关数据集和评估指标。接着,根据弱标注的类型和受关注程度,从图像级标注、其他弱标注以及大模型辅助这3个方面回顾和讨论了弱监督语义分割的研究进展。其中,第2类弱监督语义分割方法包括基于最小包围盒、线和点标注的弱监督语义分割。最后,分析了弱监督语义分割领域存在的问题与挑战,并就其未来可能的研究方向提出建议,旨在进一步推动弱监督语义分割领域研究的发展。     

Weakly supervised semantic segmentation by iteratively refining optimal segmentation with deep cues guidance

Neural Computing and Applications - Weakly supervised semantic segmentation under image-level label supervision has undergone impressive improvements over the past years. These approaches can...     

A multi-view attention-based deep learning system for online deviant content detection

With the exponential growth of user-generated content, policies and guidelines are not always enforced in social media, resulting in the prevalence of deviant content violating policies and guidelines. The adverse effects of deviant content are devastating and far-reaching. However, the detection of deviant content from sparse and imbalanced textual data is challenging, as a large number of stakeholders are involved with different stands and the subtle linguistic cues are highly dependent on complex context. To address this problem, we propose a multi-view attention-based deep learning system, which combines random subspace and binary particle swarm optimization (RS-BPSO) to distill content of interest (candidates) from imbalanced data, and applies the context and view attention mechanisms in convolutional neural network (dubbed as SSCNN) for the extraction of structural and semantic features. We evaluate the proposed approach on a large-scale dataset collected from Facebook, and find that RS-BPSO is able to detect whether the content is associated with marijuana with an accuracy of 87.55%, and SSCNN outperforms baselines with an accuracy of 94.50%.

     

Weakly supervised approaches for quality estimation

Currently, quality estimation (QE) is mostly addressed using supervised learning approaches. In this paper we show that unsupervised and weakly supervised approaches (using a small training set) perform almost as well as supervised ones, for a significantly lower cost. More generally, we study the various possible definitions, parameters, evaluation methods and approaches for QE, in order to show that there are multiple possible configurations for this task.     

Weakly supervised temporal action localization with proxy metric modeling

Temporal localization is crucial for action video recognition. Since the manual annotations are expensive and time-consuming in videos, temporal localization with weak video-level labels is challenging but indispensable. In this paper, we propose a weakly-supervised temporal action localization approach in untrimmed videos. To settle this issue, we train the model based on the proxies of each action class. The proxies are used to measure the distances between action segments and different original action features. We use a proxy-based metric to cluster the same actions together and separate actions from backgrounds. Compared with state-of-the-art methods, our method achieved competitive results on the THUMOS14 and ActivityNet1.2 datasets.     

Weakly supervised 2D human pose transfer

We present a novel method for pose transfer between two 2D human skeletons.When the bone lengths and proportions between the two skeletons are significantly dif...     

Weakly supervised action anticipation without object annotations

Anticipating future actions without observing any partial videos of future actions plays an important role in action prediction and is also a challenging task. To obtain abundant information for action anticipation, some methods integrate multimodal contexts, including scene object labels. However, extensively labelling each frame in video datasets requires considerable effort. In this paper, we develop a weakly supervised method that integrates global motion and local fine-grained features from current action videos to predict next action label without the need for specific scene context labels. Specifically, we extract diverse types of local features with weakly supervised learning, including object appearance and human pose representations without ground truth. Moreover, we construct a graph convolutional network for exploiting the inherent relationships of humans and objects under present incidents. We evaluate the proposed model on two datasets, the MPII-Cooking dataset and the EPIC-Kitchens dataset, and we demonstrate the generalizability and effectiveness of our approach for action anticipation.     

基于弱监督的属性关系抽取方法

针对从中文百科中抽取属性关系时所面临的训练语料匮乏问题,提出一种利用极少人工参与的弱监督自动抽取方法。首先,利用中文百科条目信息模板中的半结构化属性关系回标条目文本自动获取训练语料;然后,根据朴素贝叶斯分类原理优化训练语料;最后,基于条件随机场(CRF)建立属性关系抽取模型。在互动百科中采集的数据集上进行实验,综合评价F值达到了80.9%。结果表明该方法能够获得质量较高的训练语料,并取得良好的抽取性能。     

弱监督深度语义分割网络的多源遥感影像水体检测

目的 深度语义分割网络的优良性能高度依赖于大规模和高质量的像素级标签数据。在现实任务中,收集大规模、高质量的像素级水体标签数据将耗费大量人力物力。为了减少标注工作量,本文提出使用已有的公开水体覆盖产品来创建遥感影像对应的水体标签,然而已有的公开水体覆盖产品的空间分辨率低且存在一定错误。对此,提出采用弱监督深度学习方法训练深度语义分割网络。方法 在训练阶段,将原始数据集划分为多个互不重叠的子数据集,分别训练深度语义分割网络,并将训练得到的多个深度语义分割网络协同更新标签,然后利用更新后的标签重复前述过程,重新训练深度语义分割网络,多次迭代后可以获得好的深度语义分割网络。在测试阶段,多源遥感影像经多个代表不同视角的深度语义分割网络分别预测,然后投票产生最后的水体检测结果。结果 为了验证本文方法的有效性,基于原始多源遥感影像数据创建了一个面向水体检测的多源遥感影像数据集,并与基于传统的水体指数阈值分割法和基于低质量水体标签直接学习的深度语义分割网络进行比较,交并比（intersection-over-union,IoU）分别提升了5.5%和7.2%。结论 实验结果表明,本文方法具有收敛性,并且光学影像和合成孔径雷达（synthetic aperture radar,SAR）影像的融合有助于提高水体检测性能。在使用分辨率低、噪声多的水体标签进行训练的情况下,训练所得多视角模型的水体检测精度明显优于基于传统的水体指数阈值分割法和基于低质量水体标签直接学习的深度语义分割网络。     

Weakly supervised object localization and segmentation in videos

We consider the problem of localizing and segmenting objects in weakly labeled video. A video is weakly labeled if it is associated with a tag (e.g. YouTube videos with tags) describing the main object present in the video. It is weakly labeled because the tag only indicates the presence/absence of the object, but does not give the detailed spatial/temporal location of the object in the video. Given a weakly labeled video, our method can automatically localize the object in each frame and segment it from the background. Our method is fully automatic and does not require any user-input. In principle, it can be applied to a video of any object class. We evaluate our proposed method on a dataset with more than 100 video shots. Our experimental results show that our method outperforms other baseline approaches.     

Visual attention-based deepfake video forgery detection

Pattern Analysis and Applications - The prime goal of creating synthetic digital data is to generate something very closer to real ones when the original data are scarce. However, the...     

动态生成掩膜弱监督语义分割

目的 传统图像语义分割需要的像素级标注数据难以大量获取,图像语义分割的弱监督学习是当前的重要研究方向。弱监督学习是指使用弱标注样本完成监督学习,弱标注比像素级标注的标注速度快、标注方式简单,包括散点、边界框、涂鸦等标注方式。方法 针对现有方法对多层特征利用不充分的问题,提出了一种基于动态掩膜生成的弱监督语义分割方法。该方法以边界框作为初始前景分割轮廓,使用迭代方式通过卷积神经网络（convolutional neural network,CNN） 多层特征获取前景目标的边缘信息,根据边缘信息生成掩膜。迭代的过程中首先使用高层特征对前景目标的大体形状和位置做出估计,得到粗略的物体分割掩膜。然后根据已获得的粗略掩膜,逐层使用CNN 特征对掩膜进行更新。结果 在Pascal VOC（visual object classes） 2012 数据集上取得了78.06% 的分割精度,相比于边界框监督、弱—半监督、掩膜排序和实例剪切方法,分别提高了14.71%、4.04%、3.10% 和0.92%。结论 该方法能够利用高层语义特征,减少分割掩膜中语义级别的错误,同时使用底层特征对掩膜进行更新,可以提高分割边缘的准确性。     

基于边界辅助的弱监督语义分割网络

由于弱监督语义分割任务中种子区域的随机生长机制,导致弱监督语义分割网络经常出现错分割和漏分割的问题。针对上述问题,提出一种基于边界辅助的弱监督语义分割网络。该网络利用边界信息和语义信息,为种子区域的生长提供参考,使种子区域可以自然生长至目标边界,并在目标被遮挡或重叠时正确区分目标类别,生成可以覆盖更完整目标的伪像素掩码。以此伪像素掩码作为监督信息训练分割网络,可以改善弱监督语义分割网络由于伪像素掩码无法准确覆盖目标区域导致的错分割和漏分割问题,提升弱监督语义分割网络精度。在通用数据集PASCAL VOC 2012验证集和测试集上对该网络进行评估,mIoU分别达到71.7%和73.2%。实验结果表明,其网络性能优于当前大多数图像级弱监督语义分割方法。     

基于注意力机制的弱监督动作定位方法

针对弱监督动作定位方法无法直接进行动作定位且定位准确性不高的问题,提出了一种基于注意力机制的弱监督动作定位方法,并设计和实现了一种基于动作前后帧信息和区分函数的动作定位模型.采用条件变分自编码器(CVAE)注意力值生成模型,将生成的帧级注意力值作为伪帧级标签;为了增强帧前后的关联性,改进CVAE注意力值生成模型,加入动...     

Weakly supervised learning of interactions between humans and objects

We introduce a weakly supervised approach for learning human actions modeled as interactions between humans and objects. Our approach is human-centric: We first localize a human in the image and then determine the object relevant for the action and its spatial relation with the human. The model is learned automatically from a set of still images annotated only with the action label. Our approach relies on a human detector to initialize the model learning. For robustness to various degrees of visibility, we build a detector that learns to combine a set of existing part detectors. Starting from humans detected in a set of images depicting the action, our approach determines the action object and its spatial relation to the human. Its final output is a probabilistic model of the human-object interaction, i.e., the spatial relation between the human and the object. We present an extensive experimental evaluation on the sports action data set from [1], the PASCAL Action 2010 data set [2], and a new human-object interaction data set.     

基于动作模板匹配的弱监督动作定位

为解决视频中的动作定位问题，提出一种基于模板匹配的弱监督动作定位方法。首先在视频的每一帧上给出若干个动作主体位置的候选框，按时间顺序连接这些候选框形成动作提名；然后利用训练集视频的部分帧得到动作模板；最后利用动作提名与动作模板训练模型，找到最优的模型参数。在UCF-sports数据集上进行实验，结果显示，与TLSVM方法相比，所提方法的动作分类准确率提升了0.3个百分点；当重叠度阈值取0.2时，与CRANE方法相比，所提方法的动作定位准确率提升了28.21个百分点。实验结果表明，所提方法不但能够减少数据集标注的工作量，而且动作分类和动作定位的准确率均得到提升。     

Online detection of bearing incipient fault with semi-supervised architecture and deep feature representation

Although researchers have made substantial progress in bearing fault detection and diagnosis recently, incipient fault detection, especially online detection, is still at an initial stage. Generally speaking, online detection of incipient faults is still subject to the following challenges: (1) improving discriminative ability of incipient fault features; (2) adaptive recognition of the distribution inconsistency that exists in online sequential data; (3) achieving automatic detections with avoiding manual adjustment of detection criterion; and (4) reducing false alarm rate. To address these challenges, this paper presents a new approach for bearing incipient fault online detection using semi-supervised architecture and deep feature representation. This approach is simple and effective. First, we extract deep features using stacked denoising auto-encoder from the target bearing's normal state data and an auxiliary bearing's fault state data. Second, we introduce safe semi-supervised support vector machine (S4VM), a kind of semi-supervised classifier, to identify the sequentially arrived data of the target bearing as normal or anomalous. To update the classifier effectively, we use the principal curve to generate synthetic fault data for keeping data classes balanced during online condition monitoring. Finally, we propose a new fault alarm criterion based on S4VM generalization error upper bound to adaptively recognize the occurrence of an incipient fault. The experimental results on three datasets (IEEE PHM Challenge 2012, IMS and XJTU-SY) demonstrate the effectiveness and high reliability of the proposed approach.     

结合局部标记序关系的弱监督标记分布学习

标记分布学习（label distribution learning,LDL）是一种用于解决标记多义性的新颖学习范式。现有的LDL方法大多基于完整数据信息进行设计,然而由于高昂的标注成本以及标注人员水平的局限性,很难获取到完整标注数据信息,且会导致传统LDL算法性能的下降。为此,本文提出了一种新型的结合局部序标记关系的弱监督标记分布学习算法,通过维持尚未缺失标记之间的相对关系,并利用标记相关性来恢复缺失的标记,在数据标注不完整的情况下提升算法性能。在14个数据集上进行了大量的实验来验证算法的有效性。