视觉语言预训练综述

近年来深度学习在计算机视觉(CV)和自然语言处理(NLP)等单模态领域都取得了十分优异的性能.随着技术的发展,多模态学习的重要性和必要性已经慢慢展现.视觉语言学习作为多模态学习的重要部分,得到国内外研究人员的广泛关注.得益于Transformer框架的发展,越来越多的预训练模型被运用到视觉语言多模态学习上,相关任务在性能上得到了质的飞跃.系统地梳理了当前视觉语言预训练模型相关的工作,首先介绍了预训练模型的相关知识,其次从两种不同的角度分析比较预训练模型结构,讨论了常用的视觉语言预训练技术,详细介绍了5类下游预训练任务,最后介绍了常用的图像和视频预训练任务的数据集,并比较和分析了常用预训练模型在不同任务下不同数据集上的性能.     

视觉-语言导航的研究进展与发展趋势EI北大核心CSCD

视觉-语言导航是近年来出现并蓬勃发展的新兴研究方向,是视觉-语言交互前沿领域中的代表性研究任务之一,其目标是根据人类给出的语言指令基于环境视觉感知实现自主导航.首先介绍该任务的研究内容,分析其面临的跨模态语义对齐、语义理解与推理和模型泛化能力增强3个方面的问题与挑战,然后列举了常用的数据集和评价指标;再从模仿学习、强化学习、自监督学习以及其他方法4个方面对该任务的研究进展进行归纳与总结,并对代表性方法的效果进行对比分析;从连续环境导航和高级复杂指令理解与常识推理2个方面论述该任务当前研究的热点趋势;最后对三维空间的视觉-语言导航、模糊导航、环境交互导航等未来发展方向进行讨论与展望.     

视觉提示学习综述

近年来,随着提示学习方法在自然语言处理领域被提出,其日益受到研究人员广泛关注.它通过将各类下游任务重构成预训练任务的形式,以参数高效和数据高效的方式将大规模预训练模型应用在各类自然语言相关下游任务中.其中以GPT系列为代表的模型通过提示学习在对话生成和多模态图文理解等任务上取得了巨大的成功.然而,这类模型及方法还不能解决视觉中的稠密任务.受此启发,一些研究人员逐渐将提示学习广泛应用到视觉相关的各类任务当中,如图像识别、目标检测、图像分割、领域适应、持续学习等.由于目前还没有提示学习应用在视觉相关领域中的综述,本文将对视觉单模态领域以及视觉语言多模态领域的提示学习方法展开全面论述和分析.作为回顾,我们首先简要介绍自然语言处理领域的预训练模型,并对提示学习的基本概念、下游应用形式以及提示模板类型进行阐述和分类.其次,我们分别介绍视觉单模态领域以及视觉语言多模态领域里提示学习方法适配的预训练模型和任务.再次,我们分别介绍视觉单模态领域以及视觉语言多模态领域的提示学习方法.在自然语言处理领域,提示学习方法以继承预训练形式实现多任务统一为主要目的;与此不同,在视觉相关领域,提示学习方法侧重于面向...     

视觉语言多模态预训练综述

在多模态机器学习领域,为特定任务而制作的人工标注数据昂贵,且不同任务难以进行迁移,从而需要大量重新训练,导致训练多个任务时效率低下、资源浪费。预训练模型通过以自监督为代表的方式进行大规模数据训练,对数据集中不同模态的信息进行提取和融合,以学习其中蕴涵的通用知识表征,从而服务于广泛的相关下游视觉语言多模态任务,这一方法逐渐成为人工智能各领域的主流方法。依靠互联网所获取的大规模图文对与视频数据,以及以自监督学习为代表的预训练方法的进步,视觉语言多模态预训练模型在很大程度上打破了不同视觉语言任务之间的壁垒,提升了多个任务训练的效率并促进了具体任务的性能表现。本文总结视觉语言多模态预训练领域的进展,首先对常见的预训练数据集和预训练方法进行汇总,然后对目前最新方法以及经典方法进行系统概述,按输入来源分为图像—文本预训练模型和视频—文本多模态模型两大类,阐述了各方法之间的共性和差异,并将各模型在具体下游任务上的实验情况进行汇总。最后,总结了视觉语言预训练面临的挑战和未来发展趋势。     

视觉问答研究综述

视觉问答是计算机视觉领域和自然语言处理领域的交叉方向,近年来受到了广泛关注.在视觉问答任务中,算法需要回答基于特定图片（或视频）的问题.自2014年第一个视觉问答数据集发布以来,若干大规模数据集在近5年内被陆续发布,并有大量算法在此基础上被提出.已有的综述性研究重点针对视觉问答任务的发展进行了总结,但近年来,有研究发现,视觉问答模型强烈依赖语言偏见和数据集的分布,特别是自VQA-CP数据集发布以来,许多模型的效果大幅度下降.主要详细介绍近年来提出的算法以及发布的数据集,特别是讨论了算法在加强鲁棒性方面的研究.对视觉问答任务的算法进行分类总结,介绍了其动机、细节以及局限性.最后讨论了视觉问答任务的挑战及展望.     

基于跨模态多维关系增强的多模态模型研究

针对当前多模态模型不能充分挖掘图像中非显著区域的空间关系和上下文间的语义关系,导致多模态关系推理效果不佳的问题,提出了一个基于跨模态多维关系增强的多模态模型（multi-dimensional relationship enhancement model,MRE）,用于提取潜层结构下图像各要素之间的空间关系信息,并推理出视觉—语言间的语义相关性。设计了特征多样性模块用于挖掘图像中与显著区域相关的次显著区域特征,从而增强图像空间关系特征表示;并设计了上下文引导注意模块来引导模型学习语言上下文在图像中的关系,实现跨模态关系对齐。在MSCOCO数据集上的实验表明所提模型获得了更好的性能,其中BLEU-4和CIDEr分数分别提升了0.5%和1.3%。将这种方法应用到视觉问答任务中,在VQA 2.0数据集上性能得到了0.62%的提升,证明该方法在多模态任务方面的广泛适用性。     

跨模态视觉问答与推理研究进展

随着社交媒体和人机交互技术的快速发展,视频、图像以及文本等多模态数据在互联网中呈爆炸式增长,因此多模态智能研究受到关注。其中,视觉问答与推理任务是跨模态智能研究的一个重要组成部分,也是人类实现人工智能的重要基础,已成功应用于人机交互、智能医疗以及无人驾驶等领域。本文对视觉问答与推理的相关算法进行了全面概括和归类分析。首先,介绍了视觉问答与推理的定义,并简述了当前该任务面临的挑战;其次,从基于注意力机制、基于图网络、基于预训练、基于外部知识库和基于可解释推理机制5个方面对现有方法进行总结和归纳;然后,全面介绍了视觉问答与推理常用公开数据集,并对相关数据集上的已有算法进行详细分析;最后,对视觉问答与推理任务的未来方向进行了展望。     

图像—文本多模态指代表达理解研究综述

指代表达理解（referring expression comprehension,REC）作为视觉—语言相结合的多模态任务,旨在理解输入指代表达式的内容并在图像中定位其所描述的目标对象,受到计算机视觉和自然语言处理两个领域的关注。REC任务建立了人类语言与物理世界的视觉内容之间的桥梁,可以广泛应用于视觉理解系统和对话系统等人工智能设备中。解决该任务的关键在于对复杂的指代表达式进行充分的语义理解;然后利用语义信息对包含多个对象的图像进行关系推理以及对象筛选,最终在图像中唯一地定位目标对象。本文从计算机视觉的视角出发对REC任务进行了综述,首先介绍该任务的通用处理流程。然后,重点对REC领域现有方法进行分类总结,根据视觉数据表征粒度的不同,划分为基于区域卷积粒度视觉表征、基于网格卷积粒度视觉表征以及基于图像块粒度视觉表征的方法;并进一步按照视觉—文本特征融合模块的建模方式进行了更细粒度的归类。此外,本文还介绍了该任务的主流数据集和评估指标。最后,从模型的推理速度、模型的可解释性以及模型对表达式的推理能力3个方面揭示了现有方法面临的挑战,并对REC的发展进行了全面展望。本文希望通过对REC...     

面向图像文本的多模态处理方法综述

在深度学习领域,解决实际应用问题往往需要结合多种模态信息进行推理和决策,其中视觉和语言信息是交互过程中重要的两种模态。在诸多应用场景中,处理多模态任务往往面临着模型架构组织方式庞杂、训练方法效率低下等问题。综合以上问题,梳理了在图像文本多模态领域的近五年的代表性成果。首先从主流的多模态任务出发,介绍了相关文本和图像多模态数据集以及预训练目标。其次,考虑以Transformer为基础结构的视觉语言模型,结合特征提取方法,从多模态组织架构、跨模态融合方法等角度进行分析,总结比较不同处理策略的共性和差异性。然后从数据输入、结构组件等多角度介绍模型的轻量化方法。最后,对基于图像文本的多模态方法未来的研究方向进行了展望。     

面向视觉语言理解与生成的多模态预训练方法

大多数现有的视觉语言预训练方法侧重于理解任务,并在训练时使用类似于BERT的损失函数(掩码语言建模和图像文本匹配).尽管它们在许多理解类型的下游任务中表现良好,例如视觉问答、图像文本检索和视觉蕴涵,但它们不具备生成信息的能力.为了解决这个问题,提出了视觉语言理解和生成的统一多模态预训练(unified multimodal pre-training for vision-language understanding and generation, UniVL). UniVL能够处理理解任务和生成任务,并扩展了现有的预训练范式,同时使用随机掩码和因果掩码,因果掩码即掩盖未来标记的三角形掩码,这样预训练的模型可以具有自回归生成的能力.将几种视觉语言理解任务规范为文本生成任务,并使用基于模版提示的方法对不同的下游任务进行微调.实验表明,在使用同一个模型时,理解任务和生成任务之间存在权衡,而提升这两个任务的可行方法是使用更多的数据. UniVL框架在理解任务和生成任务方面的性能与最近的视觉语言预训练方法相当.此外,实验还证明了基于模版提示的生成方法更有效,甚至在少数场景中它优于判别方法.     

基于Vega的舰载捷联惯导虚拟平台的航迹仿真

运用虚拟现实技术(Virtual Reality;VR)对捷联惯导系统(Strapdown Inertial Navigation System,SINS)进行模拟的方案,可以有效地提高研发效率,节约研发成本;现在利用Multigen Vega交互仿真环境与VC++6.0开发环境,建立虚拟方针环境;随之针对建立的舰载惯性导航系统航迹仿真的数学模型,分别利用毕卡逼近算法、双子样算法和三子样算法,进行导航解算;最后由真值和所得解算值,实时同步绘制了原始轨迹和解算轨迹,实现了在模拟平台下的航迹仿真;两组轨迹的结果显示的误差,能直观显示出算法的计算误差大小;如此,使得惯性系统的仿真数据可以更加直观的表现出来。     

Multi-source shared nearest neighbours for multi-modal image clustering

Shared Nearest Neighbours (SNN) techniques are well known to overcome several shortcomings of traditional clustering approaches, notably high dimensionality and metric limitations. However, previous methods were limited to a single information source whereas such methods appear to be very well suited for heterogeneous data, typically in multi-modal contexts. In this paper, we propose a new technique to accelerate the calculation of shared neighbours and we introduce a new multi-source shared neighbours scheme applied to multi-modal image clustering. We first extend existing SNN-based similarity measures to the case of multiple sources and we introduce an original automatic source selection step when building candidate clusters. The key point is that each resulting cluster is built with its own optimal subset of modalities which improves the robustness to noisy or outlier information sources. We experiment our method in the scope of multi-modal search result clustering, visual search mining and subspace clustering. Experimental results on both synthetic and real data involving different information sources and several datasets show the effectiveness of our method.     

IMMIView: a multi-user solution for design review in real-time

IMMIView is an interactive system that relies on multiple modalities and multi-user interaction to support collaborative design review. It was designed to offer natural interaction in visualization setups such as large-scale displays, head mounted displays or TabletPC computers. To support architectural design, our system provides content creation and manipulation, 3D scene navigation and annotations. Users can interact with the system using laser pointers, speech commands, body gestures and mobile devices. In this paper, we describe how we design a system to answer architectural user requirements. In particular, our system takes advantage of multiple modalities to provide a natural interaction for design review. We also propose a new graphical user interface adapted to architectural user tasks, such as navigation or annotations. The interface relies on a novel stroke-based interaction supported by simple laser pointers as input devices for large-scale displays. Furthermore, input devices such as speech and body tracking allow IMMIView to support multiple users. Moreover, they allow each user to select different modalities according to their preference and modality adequacy for the user task. We present a multi-modal fusion system developed to support multi-modal commands on a collaborative, co-located, environment, i.e. with two or more users interacting at the same time, on the same system. The multi-modal fusion system listens to inputs from all the IMMIView modules in order to model user actions and issue commands. The multiple modalities are fused based on a simple rule-based sub-module developed in IMMIView and presented in this paper. User evaluation performed over IMMIView is presented. The results show that users feel comfortable with the system and suggest that users prefer the multi-modal approach to more conventional interactions, such as mouse and menus, for the architectural tasks presented.     

Multi-modal emotive computing in a smart house environment

We determine hazards within a smart house environment using an emotive computing framework. Representing a hazardous situation as an abnormal activity, we model normality using the concept of anxiety, using an agent based probabilistic approach. Interactions between a user and the environment are determined using multi-modal sensor data. The anxiety framework is a scalable, real-time approach that is able to incorporate data from a number of sources, or agents, and able to accommodate interleaving event sequences. In addition to using simple sensors, we introduce a method for using audio as a pervasive sensor indicating the presence of an activity. The audio data enabled the detection of activity when interactions between a user and a monitored device didn’t occur, successfully preventing false hazardous situations from being detected. We present results for a number of activity sequences, both normal and abnormal.     

PDE-based robust robotic navigation

In robotic navigation, path planning is aimed at getting the optimum collision-free path between a starting and target locations. The optimality criterion depends on the surrounding environment and the running conditions. In this paper, we propose a general, robust, and fast path planning framework for robotic navigation using level set methods. A level set speed function is proposed such that the minimum cost path between the starting and target locations in the environment, is the optimum planned path. The speed function is controlled by one parameter, which takes one of three possible values to generate either the safest, the shortest, or the hybrid planned path. The hybrid path is much safer than the shortest path, but less shorter than the safest one. The main idea of the proposed technique is to propagate a monotonic wave front with a particular speed function from a starting location until the target is reached and then extracts the optimum planned path between them by solving an ordinary differential equation (ODE) using an efficient numerical scheme. The framework supports both local and global planning for both 2D and 3D environments. The robustness of the proposed framework is demonstrated by correctly extracting planned paths of complex maps.     

多模态张量数据挖掘算法及应用

近年来,多模态数据挖掘技术备受关注,如何高效地挖掘大量多模态数据成为一个研究热点。其中,基于张量表示的多模态数据挖掘,即多模态张量数据挖掘,是一个重要的研究问题。综述了多模态张量数据挖掘算法进展及其在计算机视觉中的应用。首先根据算法的样本标记、任务和核心技术的不同,对这些方法进行分类,并给出了相应的介绍和分析。其次,讨论了一些多模态张量数据挖掘算法在计算机视觉问题中的典型应用。最后,就多模态张量挖掘在计算机视觉领域的研究现状与研究前景进行了简要的分析。     

多模态控制在机器人路径规划中的应用

路径规划作为机器人基本动作实现的基础,其优劣将直接影响动作的实时性和准确性。经典PID控制虽然能准确跟踪目标,但缺乏轨迹优化能力,提出应用多模态控制方法,通过不同的感知驱动相应运动控制,以静态目标导航和动态目标导航为例详细介绍了多模态控制设计方法,并以上海交大的中型机器人Frontier-I为例,通过实验与PID进行比较,验证了方法的有效性。     

Going to town: Visualized perspectives and navigation through virtual environments

Two experiments examined how spatial learning perspectives support navigation through virtual urban environments. Participants briefly learned the overall layout of a virtual desktop environment, and then were taken on a simulated journey ending at a starting location within the environment. In Experiment 1, during the journey participants watched simulated video feeds either from the front of the vehicle (route perspective), above the vehicle (survey perspective), both feeds simultaneously, or no video at all. Participants then navigated between ten successive landmarks, and we measured indices of spatial and temporal efficiency, and heading error. Results indicated that the route perspective supported a restricted range of local navigation whereas the survey perspective better supported far-space navigation. Experiment 2 demonstrated that the survey perspective also better supports navigation around unexpected detours. Results are discussed with regard to theories of spatial memory and the design of computer-supported spatial visualization technologies.     

Multi-Modal Dialog Scene Detection Using Hidden Markov Models for Content-Based Multimedia Indexing

A class of audio-visual data (fiction entertainment: movies, TV series) is segmented into scenes, which contain dialogs, using a novel hidden Markov model-based (HMM) method. Each shot is classified using both audio track (via classification of speech, silence and music) and visual content (face and location information). The result of this shot-based classification is an audio-visual token to be used by the HMM state diagram to achieve scene analysis. After simulations with circular and left-to-right HMM topologies, it is observed that both are performing very good with multi-modal inputs. Moreover, for circular topology, the comparisons between different training and observation sets show that audio and face information together gives the most consistent results among different observation sets.     

车辆视频导航中道路检测方法研究进展

视频导航以其获取信息丰富、对环境影响小、成本低廉等诸多优点成为了辅助车辆自动驾驶和陆地自主车(ALV)常用的一种自动导航方式,而从视频导航图像中检测出道路区域则是实现视频导航的基础和关键。对近年来车辆视频导航过程中道路检测的实现算法进行了回顾和分析,并对道路检测算法以后的研究方向进行了探讨。