基于特征融合的无监督跨模态哈希

已有的无监督跨模态哈希（UCMH）方法主要关注构造相似矩阵和约束公共表征空间的结构,忽略了2个重要问题：一是它们为不同模态的数据提取独立的表征用以检索,没有考虑不同模态之间的信息互补;二是预提取特征的结构信息不完全适用于跨模态检索任务,可能会造成一些错误信息的迁移。针对第一个问题,提出一种多模态表征融合结构,通过对不同模态的嵌入特征进行融合,从而有效地综合来自不同模态的信息,提高哈希码的表达能力,同时引入跨模态生成机制,解决检索数据模态缺失的问题;针对第二个问题,提出一种相似矩阵动态调整策略,在训练过程中用学到的模态嵌入自适应地逐步优化相似矩阵,减轻预提取特征对原始数据集的偏见,使其更适应跨模态检索,并有效避免过拟合问题。基于常用数据集Flickr25k和NUS-WIDE进行实验,结果表明,通过该方法构建的模型在Flickr25k数据集上3种哈希位长检索的平均精度均值较DGCPN模型分别提高1.43%、1.82%和1.52%,在NUS-WIDE数据集上分别提高3.72%、3.77%和1.99%,验证了所提方法的有效性。     

基于语义融合和多重相似性学习的跨模态检索

针对现有跨模态检索方法不能充分挖掘模态之间的相似性信息的问题,提出一种基于语义融合和多重相似性学习（CFMSL）方法。首先,在特征提取过程中融合不同模态的语义信息,加强不同模态特征间的交互,使得模型能够充分挖掘模态间的关联信息。然后,利用生成器将单模态特征和融合模态特征映射到公共子空间中,通过最大化锚点与正例样本之间的相似性和最小化锚点与负例样本间的相似性得到具有判别性的特征进行模态对齐。最后,基于决策融合方式对相似性列表进行重排序,使得最终排序结果同时考虑单模态特征和融合模态特征,提高检索性能。通过在Pascal Sentences、Wikipedia、NUS-WIDE-10K这3个广泛使用的图文数据集上进行实验,实验结果表明CFMSL模型能够有效提高跨模态检索任务的性能。     

跨模态检索技术研究综述

跨模态检索可以通过一种模态检索出其他模态的信息,已经成为大数据时代的研究热点。研究者基于实值表示和二进制表示两种方法来减小不同模态信息的语义差距并进行有效的相似度对比,但仍会有检索效率低或信息丢失的问题。目前,如何进一步提高检索效率和信息利用率是跨模态检索研究面临的关键挑战。介绍了跨模态检索研究中基于实值表示和二进制表示两种方法的发展现状;分析对比了包含两种表示技术下以建模技术和相似性对比为主线的五种跨模态检索方法：子空间学习、主题统计模型学习、深度学习、传统哈希和深度哈希;对最新的多模态数据集进行总结,为相关的研究和工程人员提供有价值的参考资料;分析了跨模态检索面临的挑战并指出了该领域未来研究方向。     

跨模态检索研究进展综述

随着互联网上多媒体数据的爆炸式增长,单一模态的检索已经无法满足用户需求,跨模态检索应运而生.跨模态检索旨在以一种模态的数据去检索另一种模态的相关数据,其核心任务是数据特征提取和不同模态间数据的相关性度量.文中梳理了跨模态检索领域近期的研究进展,从传统方法、深度学习方法、手工特征的哈希编码方法以及深度学习的哈希编码方法等角度归纳论述了跨模态检索领域的研究成果.在此基础上,对比分析了各类算法在跨模态检索常用标准数据集上的性能.最后,分析了跨模态检索研究存在的问题,并对该领域未来发展趋势以及应用进行了展望.     

耦合保持投影哈希跨模态检索

目的 基于哈希的跨模态检索方法因其检索速度快、消耗存储空间小等优势受到了广泛关注。但是由于这类算法大都将不同模态数据直接映射至共同的汉明空间,因此难以克服不同模态数据的特征表示及特征维度的较大差异性,也很难在汉明空间中同时保持原有数据的结构信息。针对上述问题,本文提出了耦合保持投影哈希跨模态检索算法。方法 为了解决跨模态数据间的异构性,先将不同模态的数据投影至各自子空间来减少模态“鸿沟”,并在子空间学习中引入图模型来保持数据间的结构一致性;为了构建不同模态之间的语义关联,再将子空间特征映射至汉明空间以得到一致的哈希码;最后引入类标约束来提升哈希码的判别性。结果 实验在3个数据集上与主流的方法进行了比较,在Wikipedia数据集中,相比于性能第2的算法,在任务图像检索文本（I to T）和任务文本检索图像（T to I）上的平均检索精度（mean average precision,mAP）值分别提升了6%和3%左右;在MIRFlickr数据集中,相比于性能第2的算法,优势分别为2%和5%左右;在Pascal Sentence数据集中,优势分别为10%和7%左右。结论 本文方法可适用于两个模态数据之间的相互检索任务,由于引入了耦合投影和图模型模块,有效提升了跨模态检索的精度。     

基于对比学习和GIF标记的多模态对话回复检索

社交媒体网站上使用GIF(Graphics Interchange Format)作为消息的回复相当普遍。但目前大多方法针对问题“如何选择一个合适的GIF回复消息”,没有很好地利用社交媒体上的GIF附属标记信息。为此,提出基于对比学习和GIF标记的多模态对话回复检索（CoTa-MMD）方法,将标记信息整合到检索过程中。具体来说就是使用标记作为中间变量,文本→GIF的检索就被转换为文本→GIF标记→GIF的检索,采用对比学习算法学习模态表示,并利用全概率公式计算检索概率。与直接的文本图像检索相比,引入的过渡标记降低了不同模态的异质性导致的检索难度。实验结果表明,CoTa-MMD模型相较于深度监督的跨模态检索（DSCMR）模型,在PEPE-56多模态对话数据集和Taiwan多模态对话数据集上文本图像检索任务的召回率之和分别提升了0.33个百分点和4.21个百分点。     

视频片段检索研究综述

视频片段检索旨在利用用户给出的自然语言查询语句,在一个长视频中找到最符合语句描述的目标视频片段.视频中包含丰富的视觉、文本、语音信息,如何理解视频中提供的信息,以及查询语句提供的文本信息,并进行跨模态信息的对齐与交互,是视频片段检索任务的核心问题.系统梳理了当前视频片段检索领域中的相关工作,将它们分为两大类：基于排序的方法和基于定位的方法.其中,基于排序的方法又可细分为预设候选片段的方法和有指导地生成候选片段的方法,而基于定位的方法则可分为一次定位的方法和迭代定位的方法.同时对该领域的数据集和评价指标进行了介绍,并对一些模型在多个常用数据集上的性能进行了总结与整理.此外,介绍了该任务的延伸工作,如大规模视频片段检索工作等.最后,对视频片段检索未来的发展方向进行了展望.     

基于深度学习的跨模态检索综述

由于多模态数据的快速增长,跨模态检索受到了研究者的广泛关注,其将一种模态的数据作为查询条件检索其他模态的数据,如用户可以用文本检索图像或/和视频。由于查询及其检索结果模态表征的差异,如何度量不同模态之间的相似性是跨模态检索的主要挑战。随着深度学习技术的推广及其在计算机视觉、自然语言处理等领域的显著成果,研究者提出了一系列以深度学习为基础的跨模态检索方法,极大缓解了不同模态间相似性度量的挑战,本文称之为深度跨模态检索。本文从以下角度综述有代表性的深度跨模态检索论文,基于所提供的跨模态信息将这些方法分为3类：基于跨模态数据间一一对应的、基于跨模态数据间相似度的以及基于跨模态数据语义标注的深度跨模态检索。一般来说,上述3类方法提供的跨模态信息呈现递增趋势,且提供学习的信息越多,跨模态检索性能越优。在上述不同类别下,涵盖了7类主流技术,即典型相关分析、一一对应关系保持、度量学习、似然分析、学习排序、语义预测以及对抗学习。不同类别下包含部分关键技术,本文将具体阐述其中有代表性的方法。同时对比提供不同跨模态数据信息下不同技术的区别,以阐述在提供了不同层次的跨模态数据信息下相关技术的关注点与使用异同。为评估不同的跨模态检索方法,总结了部分代表性的跨模态检索数据库。最后讨论了当前深度跨模态检索待解决的问题以及未来的研究方向。     

联合线性判别和图正则的任务导向型跨模态检索

针对现有的基于公共子空间的跨模态检索方法对不同检索任务的差异性、检索模态的语义一致性考虑不足的问题,提出一种联合线性判别和图正则的任务导向型跨模态检索方法.该方法在一个联合学习框架中为不同的检索任务构建不同的映射机制,将不同模态的数据映射到公共子空间中以进行相似性度量;学习过程中结合相关性分析和单模态语义回归,保留成对数据间的相关性以及增强查询模态样本的语义准确性,同时利用线性判别分析保证检索模态样本的语义一致性;还为不同模态的数据构建局部近邻图以保留结构信息,从而提升跨模态检索的性能.在Wikipedia和Pascal Sentence这2个跨模态数据集上的实验结果表明,该方法在不同检索任务上的平均mAP值比12种现有方法分别提升了1.0％～16.0％和1.2％～14.0％.     

跨模态检索研究综述

近年来,各种类型的媒体数据,如音频、文本、图像和视频,在互联网上呈现爆发式增长,不同类型的数据通常用于描述同一事件或主题。跨模态检索提供了一些有效的方法,可以为任何模态的给定查询搜索不同模态的语义相关结果,使用户能够获得有关事件/主题的更多信息,从而达到以一种模态数据检索另外一种模态数据的效果。随着数据检索需求以及各种新技术的发展,单一模态检索难以满足用户需求,研究者提出许多跨模态检索的技术来解决这个问题。梳理近期跨模态检索领域研究者的研究成果,简要分析传统的跨模态检索方法,着重介绍近五年研究者提出跨模态检索方法,并对其性能表现进行对比;总结现阶段跨模态检索研究过程中面临的问题,并对后续发展做出展望。     

On the Complexity of Function Learning

The majority of results in computational learning theory are concerned with concept learning, i.e. with the special case of function learning for classes of functions with range {0, 1}. Much less is known about the theory of learning functions with a larger range such as or . In particular relatively few results exist about the general structure of common models for function learning, and there are only very few nontrivial function classes for which positive learning results have been exhibited in any of these models.We introduce in this paper the notion of a binary branching adversary tree for function learning, which allows us to give a somewhat surprising equivalent characterization of the optimal learning cost for learning a class of real-valued functions (in terms of a max-min definition which does not involve any learning model).Another general structural result of this paper relates the cost for learning a union of function classes to the learning costs for the individual function classes.Furthermore, we exhibit an efficient learning algorithm for learning convex piecewise linear functions from ^d into . Previously, the class of linear functions from ^d into was the only class of functions with multidimensional domain that was known to be learnable within the rigorous framework of a formal model for online learning.Finally we give a sufficient condition for an arbitrary class of functions from into that allows us to learn the class of all functions that can be written as the pointwise maximum ofk functions from . This allows us to exhibit a number of further nontrivial classes of functions from into for which there exist efficient learning algorithms.     

Transfer in variable-reward hierarchical reinforcement learning

Transfer learning seeks to leverage previously learned tasks to achieve faster learning in a new task. In this paper, we consider transfer learning in the context of related but distinct Reinforcement Learning (RL) problems. In particular, our RL problems are derived from Semi-Markov Decision Processes (SMDPs) that share the same transition dynamics but have different reward functions that are linear in a set of reward features. We formally define the transfer learning problem in the context of RL as learning an efficient algorithm to solve any SMDP drawn from a fixed distribution after experiencing a finite number of them. Furthermore, we introduce an online algorithm to solve this problem, Variable-Reward Reinforcement Learning (VRRL), that compactly stores the optimal value functions for several SMDPs, and uses them to optimally initialize the value function for a new SMDP. We generalize our method to a hierarchical RL setting where the different SMDPs share the same task hierarchy. Our experimental results in a simplified real-time strategy domain show that significant transfer learning occurs in both flat and hierarchical settings. Transfer is especially effective in the hierarchical setting where the overall value functions are decomposed into subtask value functions which are more widely amenable to transfer across different SMDPs.     

The Power of Self-Directed Learning

This article studies self-directed learning, a variant of the on-line (or incremental) learning model in which the learner selects the presentation order for the instances. Alternatively, one can view this model as a variation of learning with membership queries in which the learner is only charged for membership queries for which it could not predict the outcome. We give tight bounds on the complexity of self-directed learning for the concept classes of monomials, monotone DNF formulas, and axis-parallel rectangles in {0, 1, , n – 1} ^d . These results demonstrate that the number of mistakes under self-directed learning can be surprisingly small. We then show that learning complexity in the model of self-directed learning is less than that of all other commonly studied on-line and query learning models. Next we explore the relationship between the complexity of self-directed learning and the Vapnik-Chervonenkis (VC-)dimension. We show that, in general, the VC-dimension and the self-directed learning complexity are incomparable. However, for some special cases, we show that the VC-dimension gives a lower bound for the self-directed learning complexity. Finally, we explore a relationship between Mitchell's version space algorithm and the existence of self-directed learning algorithms that make few mistakes.     

Toward Efficient Agnostic Learning

In this paper we initiate an investigation of generalizations of the Probably Approximately Correct (PAC) learning model that attempt to significantly weaken the target function assumptions. The ultimate goal in this direction is informally termed agnostic learning, in which we make virtually no assumptions on the target function. The name derives from the fact that as designers of learning algorithms, we give up the belief that Nature (as represented by the target function) has a simple or succinct explanation. We give a number of positive and negative results that provide an initial outline of the possibilities for agnostic learning. Our results include hardness results for the most obvious generalization of the PAC model to an agnostic setting, an efficient and general agnostic learning method based on dynamic programming, relationships between loss functions for agnostic learning, and an algorithm for a learning problem that involves hidden variables.     

Learning from a Population of Hypotheses

We introduce a new formal model in which a learning algorithm must combine a collection of potentially poor but statistically independent hypothesis functions in order to approximate an unknown target function arbitrarily well. Our motivation includes the question of how to make optimal use of multiple independent runs of a mediocre learning algorithm, as well as settings in which the many hypotheses are obtained by a distributed population of identical learning agents.     

一种新的连续动作集学习自动机

学习自动机（Learning automation,LA）是一种自适应决策器。其通过与一个随机环境不断交互学习从一个允许的动作集里选择最优的动作。在大多数传统的LA模型中,动作集总是被取作有限的。因此,对于连续参数学习问题,需要将动作空间离散化,并且学习的精度取决于离散化的粒度。本文提出一种新的连续动作集学习自动机（Continuous action set learning automaton,CALA）,其动作集为一个可变区间,同时按照均匀分布方式选择输出动作。学习算法利用来自环境的二值反馈信号对动作区间的端点进行自适应更新。通过一个多模态学习问题的仿真实验,演示了新算法相对于3种现有CALA算法的优越性。     

Context counts: How learners' contexts influence learning in a MOOC

Massive Open Online Courses (MOOCs) require individual learners to self-regulate their own learning, determining when, how and with what content and activities they engage. However, MOOCs attract a diverse range of learners, from a variety of learning and professional contexts. This study examines how a learner's current role and context influences their ability to self-regulate their learning in a MOOC: Introduction to Data Science offered by Coursera. The study compared the self-reported self-regulated learning behaviour between learners from different contexts and with different roles. Significant differences were identified between learners who were working as data professionals or studying towards a higher education degree and other learners in the MOOC. The study provides an insight into how an individual's context and role may impact their learning behaviour in MOOCs.     

Indexing,Elaboration and Refinement: Incremental Learning of Explanatory Cases

This article describes how a reasoner can improve its understanding of an incompletely understood domain through the application of what it already knows to novel problems in that domain. Case-based reasoning is the process of using past experiences stored in the reasoner's memory to understand novel situations or solve novel problems. However, this process assumes that past experiences are well understood and provide good lessons to be used for future situations. This assumption is usually false when one is learning about a novel domain, since situations encountered previously in this domain might not have been understood completely. Furthermore, the reasoner may not even have a case that adequately deals with the new situation, or may not be able to access the case using existing indices. We present a theory of incremental learning based on the revision of previously existing case knowledge in response to experiences in such situations. The theory has been implemented in a case-based story understanding program that can (a) learn a new case in situations where no case already exists, (b) learn how to index the case in memory, and (c) incrementally refine its understanding of the case by using it to reason about new situations, thus evolving a better understanding of its domain through experience. This research complements work in case-based reasoning by providing mechanisms by which a case library can be automatically built for use by a case-based reasoning program.     

Exploring learnability between exact and PAC

We study a model of probably exactly correct (PExact) learning that can be viewed either as the Exact model (learning from equivalence queries only) relaxed so that counterexamples to equivalence queries are distributionally drawn rather than adversarially chosen or as the probably approximately correct (PAC) model strengthened to require a perfect hypothesis. We also introduce a model of probably almost exactly correct (PAExact) learning that requires a hypothesis with negligible error and thus lies between the PExact and PAC models. Unlike the Exact and PExact models, PAExact learning is applicable to classes of functions defined over infinite instance spaces. We obtain a number of separation results between these models. Of particular note are some positive results for efficient parallel learning in the PAExact model, which stand in stark contrast to earlier negative results for efficient parallel Exact learning.