基于运动特征的视频信息检索综述

随着各种视频压缩标准的制定、计算机软硬件与网络技术的发展，基于内容的视频信息检索从20世纪90年代开始，逐步成为研究热点，其中基于运动特征的视频信息检索则是一个重要的研究分支。本文从运动信息的提取、运动特征的表示和基于运动特征的检索算法这三个方面回顾基于运动特征的视频信息检索技术的研究进展。     

基于深度学习的图像检索系统

基于内容的图像检索系统关键的技术是有效图像特征的获取和相似度匹配策略.在过去,基于内容的图像检索系统主要使用低级的可视化特征,无法得到满意的检索结果,所以尽管在基于内容的图像检索上花费了很大的努力,但是基于内容的图像检索依旧是计算机视觉领域中的一个挑战.在基于内容的图像检索系统中,存在的最大的问题是“语义鸿沟”,即机器从低级的可视化特征得到的相似性和人从高级的语义特征得到的相似性之间的不同.传统的基于内容的图像检索系统,只是在低级的可视化特征上学习图像的特征,无法有效的解决“语义鸿沟”.近些年,深度学习技术的快速发展给我们提供了希望.深度学习源于人工神经网络的研究,深度学习通过组合低级的特征形成更加抽象的高层表示属性类别或者特征,以发现数据的分布规律,这是其他算法无法实现的.受深度学习在计算机视觉、语音识别、自然语言处理、图像与视频分析、多媒体等诸多领域取得巨大成功的启发,本文将深度学习技术用于基于内容的图像检索,以解决基于内容的图像检索系统中的“语义鸿沟”问题.     

Assessing the effects of different multimedia materials on emotions and learning performance for visual and verbal style learners

Multimedia materials are now increasingly used in curricula. However, individual preferences for multimedia materials based on visual and verbal cognitive styles may affect learners' emotions and performance. Therefore, in-depth studies that investigate how different multimedia materials affect learning performance and the emotions of learners with visual and verbal cognitive styles are needed. Additionally, many education scholars have argued that emotions directly affect learning performance. Therefore, a further study that confirms the relationships between learners' emotions and performance for learners with visual and verbal cognitive styles will provide useful knowledge in terms of designing an emotion-based adaptive multimedia learning system for supporting personalized learning. To investigate these issues, the study applies the Style of Processing (SOP) scale to identify verbalizers and visualizers. Moreover, the emotion assessment instrument emWave, which was developed by HeartMath, is applied to assess variations in emotional states for verbalizers and visualizers during learning processes. Three different multimedia materials, static text and image-based multimedia material, video-based multimedia material, and animated interactive multimedia material, were presented to verbalizers and visualizers to investigate how different multimedia materials affect individual learning performance and emotion, and to identify relationships between learning performance and emotion. Experimental results show that video-based multimedia material generates the best learning performance and most positive emotion for verbalizers. Moreover, dynamic multimedia materials containing video and animation are more appropriate for visualizers than static multimedia materials containing text and image. Finally, a partial correlation exists between negative emotion and learning performance; that is, negative emotion and pretest scores considered together and negative emotion alone can predict learning performance of visualizers who use video-based multimedia material for learning.     

Assessing the effects of different multimedia materials on emotions and learning performance for visual and verbal style learners

Multimedia materials are now increasingly used in curricula. However, individual preferences for multimedia materials based on visual and verbal cognitive styles may affect learners' emotions and performance. Therefore, in-depth studies that investigate how different multimedia materials affect learning performance and the emotions of learners with visual and verbal cognitive styles are needed. Additionally, many education scholars have argued that emotions directly affect learning performance. Therefore, a further study that confirms the relationships between learners' emotions and performance for learners with visual and verbal cognitive styles will provide useful knowledge in terms of designing an emotion-based adaptive multimedia learning system for supporting personalized learning. To investigate these issues, the study applies the Style of Processing (SOP) scale to identify verbalizers and visualizers. Moreover, the emotion assessment instrument emWave, which was developed by HeartMath, is applied to assess variations in emotional states for verbalizers and visualizers during learning processes. Three different multimedia materials, static text and image-based multimedia material, video-based multimedia material, and animated interactive multimedia material, were presented to verbalizers and visualizers to investigate how different multimedia materials affect individual learning performance and emotion, and to identify relationships between learning performance and emotion. Experimental results show that video-based multimedia material generates the best learning performance and most positive emotion for verbalizers. Moreover, dynamic multimedia materials containing video and animation are more appropriate for visualizers than static multimedia materials containing text and image. Finally, a partial correlation exists between negative emotion and learning performance; that is, negative emotion and pretest scores considered together and negative emotion alone can predict learning performance of visualizers who use video-based multimedia material for learning.     

Relevance feature mapping for content-based multimedia information retrieval

This paper presents a novel ranking framework for content-based multimedia information retrieval (CBMIR). The framework introduces relevance features and a new ranking scheme. Each relevance feature measures the relevance of an instance with respect to a profile of the targeted multimedia database. We show that the task of CBMIR can be done more effectively using the relevance features than the original features. Furthermore, additional performance gain is achieved by incorporating our new ranking scheme which modifies instance rankings based on the weighted average of relevance feature values. Experiments on image and music databases validate the efficacy and efficiency of the proposed framework.     

视频局部特征描述子的紧凑表示方法

目的 随着手持移动设备的迅猛发展和大数据时代的到来,以多媒体数据为核心的视觉搜索等研究和应用得到了广泛关注。其中局部特征描述子的压缩、存储和传输起到了举足轻重的作用。为此在传统图像/视频压缩框架中,提出一种高效的视觉局部特征的紧凑表示方法,使得传统内容编码可以适应广泛的检索分析等需求。方法 为了得到紧凑、有区分度、同时高效的局部特征表示,首先引入了多参考的预测机制,在消除了时空冗余的同时,通过充分利用视频纹理编码的信息,消除了来自纹理-特征之间的冗余。此外,还提出了一种新的率失真优化方法——码率-准确率最优化方法,使得基于匹配/检索应用的性能达到最优。结果 在不同数据集上进行验证实验,和最新的视频局部描述子压缩框架进行比较,本文方法能够在保证匹配和检索性能的基础上,显著地减少特征带来的比特消耗,达到大约150:1的压缩比。结论 本文方法适用于传统图像/视频编码框架,通过在码流中嵌入少量表示特征的信息,即可实现高效的检索性能,是一种面向检索等智能设备应用的新型多媒体内容编码框架。     

综合利用纹理和中心特征的图像检索

基于特征的图像检索在多媒体数据库管理和多媒体通信传输中得到越来越多的重视。本文介绍了基于纹理特征的特征提取方法和中心特征的提取方法，并进而提出了一种综合利用上述两个特征共同进行检索的方法。对真实图像库的检索实验表明，综合特征检索比单一特征检索更符合人的视觉特性，检索效果更好。     

A multi-access technique for broadband wireless local networks

This paper presents a new multi-access technique that allows multimedia information to be transported in indoor wireless networks. Growing emphasis on multimedia calls for the development of multi-access protocols capable of supporting demanding applications such as audio/video playback, image browsing, real-time voice/video transmission and interactive data exchange in a unified manner. To this end, a practical multi-access scheme called `reserved polling' is proposed. The technique handles a diverse mixture of delay classes and message priorities associated with multimedia traffic with an inherent ability to adapt to the traffic load profile. In order to gain a quantitative insight into its behavior, the protocol is analyzed and simulated in detail.     

A survey on content-based retrieval for multimedia databases

Conventional database systems are designed for managing textual and numerical data, and retrieving such data is often based on simple comparisons of text/numerical values. However, this simple method of retrieval is no longer adequate for multimedia data, since the digitized representation of images, video, or data itself does not convey the reality of these media items. In addition, composite data consisting of heterogeneous types of data also associates with the semantic content acquired by a user's recognition. Therefore, content-based retrieval for multimedia data is realized taking such intrinsic features of multimedia data into account. Implementation of the content-based retrieval facility is not based on a single fundamental, but is closely related to an underlying data model, a priori knowledge of the area of interest, and the scheme for representing queries. This paper surveys recent studies on content-based retrieval for multimedia databases from the point of view of three fundamental issues. Throughout the discussion, we assume databases that manage only nontextual/numerical data, such as image or video, are also in the category of multimedia databases     

Real-time EEG-based emotion monitoring using stable features

In human–computer interaction (HCI), electroencephalogram (EEG) signals can be added as an additional input to computer. An integration of real-time EEG-based human emotion recognition algorithms in human–computer interfaces can make the users experience more complete, more engaging, less emotionally stressful or more stressful depending on the target of the applications. Currently, the most accurate EEG-based emotion recognition algorithms are subject-dependent, and a training session is needed for the user each time right before running the application. In this paper, we propose a novel real-time subject-dependent algorithm with the most stable features that gives a better accuracy than other available algorithms when it is crucial to have only one training session for the user and no re-training is allowed subsequently. The proposed algorithm is tested on an affective EEG database that contains five subjects. For each subject, four emotions (pleasant, happy, frightened and angry) are induced, and the affective EEG is recorded for two sessions per day in eight consecutive days. Testing results show that the novel algorithm can be used in real-time emotion recognition applications without re-training with the adequate accuracy. The proposed algorithm is integrated with real-time applications “Emotional Avatar” and “Twin Girls” to monitor the users emotions in real time.     

一种基于内容相关性的跨媒体检索方法

针对传统基于内容的多媒体检索对单一模态的限制,提出一种新的跨媒体检索方法.分析了不同模态的内容特征之间在统计意义上的典型相关性,并通过子空间映射解决了特征向量的异构性问题,同时结合相关反馈中的先验知识,修正不同模态多媒体数据集在子空间中的拓扑结构,实现跨媒体相关性的准确度量.实验以图像和音频数据为例验证了基于相关性学习的跨媒体检索方法的有效性.     

论多媒体节目中影像交互的情感性与趣味性

随着社会的不断发展,人们审美趣味的提高,赋予多媒体影像交互的功能不断增多,对多媒体产品的精神化需求也不断增长。多媒体设计实际已经把人们的注意力更多地转移到影像交互的趣味性和情感性方面。如何更多的挖掘现代人的精神世界、情感世界,并将其转寄到多媒体影像的交互设计中,使它更符合人的心理、生理、情感的需要,是当代多媒体影像考虑的重点,也是充分体现人性化的时代特征。     

基于卷积神经网络的图文融合媒体情感预测

近年来,用户在社交媒体上越来越多地使用多媒体内容来分享经历和表达情绪。相比单独的文本和图像,融合文本和图像的多媒体内容能够更为充分地揭示用户的真实情感。针对单一文本或图像的情感不明显问题,提出了一种基于卷积神经网络(CNN)的图文融合媒体的情感分析方法。该方法融合图像特征与三个不同级别(词语级、短语级和句子级)的文本特征构建CNN模型,以分析比较不同层次的语义特征对情感预测的影响。在真实数据集上的实验结果表明,通过捕捉文本情感特征和图像情感特征之间的内部联系,可以更准确地实现对图文融合媒体情感的预测。     

Instantly indexed multimedia databases of real world events

We introduce a new paradigm for real-time conversion of a real world event into a rich multimedia database by processing data from multiple sensors observing the event. A real-time analysis of the sensor data, tightly coupled with domain knowledge, results in instant indexing of multimedia data at capture time. This yields semantic information to answer complex queries about the content and the ability to extract portions of data that correspond to complex actions performed in the real world. The power of such an instantly indexed multimedia database system, in content-based retrieval of multimedia data or in semantic analysis and visualization of the data, far exceeds that of systems which index multimedia data only after it is produced. We present LucentVision, an instantly indexed multimedia database system developed for the sport of tennis. This system analyzes video from multiple cameras in real time and captures the activity of the players and the ball in the form of motion trajectories. The system stores these trajectories in a database along with video, 3D models of the environment, scores, and other domain-specific information. LucentVision has been used to enhance live television and Internet broadcasts with game analyses and virtual replays in more than 250 international tennis matches.     

DEAR-MULSEMEDIA: Dataset for emotion analysis and recognition in response to multiple sensorial media

Traditionally, emotion recognition is performed in response to stimuli that engage either one (vision: image or hearing: audio) or two (vision and hearing: video) human senses. An immersive environment can be generated by engaging more than two human senses while interacting with multimedia content and is known as MULtiple SEnsorial media (mulsemedia). This study aims to create a new dataset of multimodal physiological signals to recognize emotions in response to such content. To this end, four multimedia clips are selected and synchronized with fan, heater, olfaction dispenser, and haptic vest to augment cold air, hot air, olfaction, and haptic effects respectively. Furthermore, physiological responses including electroencephalography (EEG), galvanic skin response (GSR), and photoplethysmography (PPG) are observed to analyze human emotional responses while experiencing mulsemedia content. A t-test applied using arousal and valence scores show that engaging more than two human senses evokes significantly different emotions. Statistical tests on EEG, GSR, and PPG responses also show a significant difference between multimedia and mulsemedia content. Classification accuracy of 85.18% and 76.54% is achieved for valence and arousal, respectively, using K-nearest neighbor classifier and feature-level fusion strategy.     

An approach of genetic programming for music emotion classification

In this paper, we suggest a new approach of genetic programming for music emotion classification. Our approach is based on Thayer’s arousal-valence plane which is one of representative human emotion models. Thayer’s plane which says human emotions is determined by the psychological arousal and valence. We map music pieces onto the arousal-valence plane, and classify the music emotion in that space. We extract 85 acoustic features from music signals, rank those by the information gain and choose the top k best features in the feature selection process. In order to map music pieces in the feature space onto the arousal-valence space, we apply genetic programming. The genetic programming is designed for finding an optimal formula which maps given music pieces to the arousal-valence space so that music emotions are effectively classified. k-NN and SVM methods which are widely used in classification are used for the classification of music emotions in the arousal-valence space. For verifying our method, we compare with other six existing methods on the same music data set. With this experiment, we confirm the proposed method is superior to others.