Intelligent multimedia data: data + indices + inference

In this paper, we present a novel approach for multimedia data indexing and retrieval that is machine independent and highly flexible for sharing multimedia data across applications. Traditional multimedia data indexing and retrieval problems have been attacked using the central data server as the main focus, and most of the indexing and query-processing for retrieval are highly application dependent. This precludes the use of created indices and query processing mechanisms for multimedia data which, in general, have a wide variety of uses across applications. The approach proposed in this paper addresses three issues: 1. multimedia data indexing; 2. inference or query processing; and 3. combining indices and inference or query mechanism with the data to facilitate machine independence in retrieval and query processing. We emphasize the third issue, as typically multimedia data are huge in size and requires intra-data indexing. We describe how the proposed approach addresses various problems faced by the application developers in indexing and retrieval of multimedia data. Finally, we present two applications developed based on the proposed approach: video indexing; and video content authorization for presentation.     

Content-based indexing of multimedia databases

Content-based retrieval of multimedia database calls for content-based indexing techniques. Different from conventional databases, where data items are represented by a set of attributes of elementary data types, multimedia objects in multimedia databases are represented by a collection of features; similarity of object contents depends on context and frame of reference; and features of objects are characterized by multimodal feature measures. These lead to great challenges for content-based indexing. On the other hand, there are special requirements on content-based indexing: to support visual browsing, similarity retrieval, and fuzzy retrieval, nodes of the index should represent certain meaningful categories. That is to say that certain semantics must be added when performing indexing. ContIndex, the context-based indexing technique presented in this paper, is proposed to meet these challenges and special requirements. The indexing tree is formally defined by adapting a classification-tree concept. Horizontal links among nodes in the same level enhance the flexibility of the index. A special neural-network model, called Learning based on Experiences and Perspectives (FEP), has been developed to create node categories by fusing multimodal feature measures. It brings into the index the capability of self-organizing nodes with respect to certain context and frames of reference. An icon image is generated for each intermediate node to facilitate visual browsing. Algorithms have been developed to support multimedia object archival and retrieval using Contlndex     

Inference and retrieval of facial images

Attempts have been made to extend SQL to work with multimedia databases. We are reserved on the representation ability of extended SQL to cope with the richness in content of multimedia data. In this paper we present an example of a multimedia database system, Computer Aided Facial Image Inference and Retrieval system (CAFIIR). The system stores and manages facial images and criminal records, providing necessary functions for crime identification. We would like to demonstrate some core techniques for multimedia database with CAFIIR system. Firstly, CAFIIR is a integrated system. Besides database management, there are image analysis, image composition, image aging, and report generation subsystems, providing means for problem solving. Secondly, the richness of multimedia data urges feature-based database for their management. CAFIIR is feature-based. A indexing mechanism,iconic index, has been proposed for indexing facial images using hierarchical self-organization neural network. The indexing method operates on complex feature measures and provides means for visual navigation. Thirdly, special retrieval methods for facial images have been developed, including visual browsing, similarity retrieval, free text retrieval and fuzzy retrieval.     

科学数据库基于内容的多媒体检索系统

科学数据库中存在大量的多媒体数据,为了实现对多媒体内容的有效存储、管理和检索．基于内容的多媒体综合检索技术将成为技术研究的重点。本文首先分析了科学数据库多媒体资源的特点和对多媒体内容管理的需求．然后探讨了基于内容的多媒体检索技术的原理、特点和检索方法。最后提出了科学数据库多媒体检索系统的一套设计方案,并说明了该体系的结构和功能。     

CORE: a content-based retrieval engine for multimedia information systems

Rapid advances in multimedia technology necessitate the development of a generic multimedia information system with a powerful retrieval engine for prototyping multimedia applications. We develop a content-based retrieval engine (CORE) that makes use of novel indexing techniques for multimedia object retrieval. We formalize the concepts related to multimedia information systems such as multimedia objects and content-based retrieval. We bring out the requirements and challenges of a multimedia information system. The architecture of CORE is described in detail along with the associated retrieval mechanisms and indexing techniques. Various modules developed for efficient retrieval are presented with some APIs. The efficacy of CORE is demonstrated in the development of two multimedia systems, a computer-aided facial image inference and retrieval (CAFIIR) system and a system for trademark archival and retrieval (STAR), which have been developed at the Institute of Systems Science (ISS). We expect that CORE will be useful for effective prototyping of other such multimedia applications.Mainly supported by National Science & Technology Board of SingaporePartly working in Real World Computing Partnership, Novel Function Institute of Systems Science Laboratory since April 1994.     

多媒体数据库索引技术的研究与实现

现有的数据库索引技术,普遍不能适应多维空间属性的搜索,特别是无法对大容量的多媒体数据进行基于内容的检索。文章分析研究了多媒体数据库的索引结构和索引算法,设计了一种用于大容量图像数据库的索引方法。实验在1万多幅的图像库上反复进行,结论证明该算法能够有效地支持大容量图像库的基于内容检索。     

Adapting metric indexes for searching in multi-metric spaces

An important research issue in multimedia databases is the retrieval of similar objects. For most applications in multimedia databases, an exact search is not meaningful. Thus, much effort has been devoted to develop efficient and effective similarity search techniques. A recent approach that has been shown to improve the effectiveness of similarity search in multimedia databases resorts to the usage of combinations of metrics (i.e., a search on a multi-metric space). In this approach, the desirable contribution (weight) of each metric is chosen at query time. It follows that standard metric indexes cannot be directly used to improve the efficiency of dynamically weighted queries, because they assume that there is only one fixed distance function at indexing and query time. This paper presents a methodology for adapting metric indexes to multi-metric indexes, that is, to support similarity queries with dynamic combinations of metric functions. The adapted indexes are built with a single distance function and store partial distances to estimate the dynamically weighed distances. We present two novel indexes for multimetric space indexing, which are the result of the application of the proposed methodology.     

RUSHES—an annotation and retrieval engine for multimedia semantic units

Multimedia analysis and reuse of raw un-edited audio visual content known as rushes is gaining acceptance by a large number of research labs and companies. A set of research projects are considering multimedia indexing, annotation, search and retrieval in the context of European funded research, but only the FP6 project RUSHES is focusing on automatic semantic annotation, indexing and retrieval of raw and un-edited audio-visual content. Even professional content creators and providers as well as home-users are dealing with this type of content and therefore novel technologies for semantic search and retrieval are required. In this paper, we present a summary of the most relevant achievements of the RUSHES project, focusing on specific approaches for automatic annotation as well as the main features of the final RUSHES search engine.     

Design issues of multimedia information indexing and retrieval systems

Multimedia information indexing and retrieval systems are required to manage and use ever increasing multimedia information effectively and efficiently. This paper first provides an over-view of general capabilities and architecture of multimedia information indexing and retrieval systems, and then discusses a number of main design issues including data modelling, user interface, feature extraction, indexing, feature similarity measures and quality of service guarantees.     

A probabilistic framework for semantic video indexing, filtering,and retrieval

Semantic filtering and retrieval of multimedia content is crucial for efficient use of the multimedia data repositories. Video query by semantic keywords is one of the most difficult problems in multimedia data retrieval. The difficulty lies in the mapping between low-level video representation and high-level semantics. We therefore formulate the multimedia content access problem as a multimedia pattern recognition problem. We propose a probabilistic framework for semantic video indexing, which call support filtering and retrieval and facilitate efficient content-based access. To map low-level features to high-level semantics we propose probabilistic multimedia objects (multijects). Examples of multijects in movies include explosion, mountain, beach, outdoor, music etc. Semantic concepts in videos interact and to model this interaction explicitly, we propose a network of multijects (multinet). Using probabilistic models for six site multijects, rocks, sky, snow, water-body forestry/greenery and outdoor and using a Bayesian belief network as the multinet we demonstrate the application of this framework to semantic indexing. We demonstrate how detection performance can be significantly improved using the multinet to take interconceptual relationships into account. We also show how the multinet can fuse heterogeneous features to support detection based on inference and reasoning     

基于JPEG的图像检索综述

基于内容的检索是近年来的研究热点之一，现在已有许多基于象素域的图像检索技术，目前数据压缩也已成为多媒体应用的标准模式，静态图像压缩主要采用JPEG技术，研究基于传统JPEG和JPEG2000的图像检索方法成为必然。本文综述近年来出现的基于JPEG核心算法离散余弦变换和JPEG2000核心算法离散小波变换的图像检索技术。     

多媒体数据库管理系统MMDBMS关键技术分析

在多媒体数据库管理系统体系结构的基础上,分析了多媒体数据模型、特征提取与索引、多媒体检索等关键技术。     

A real-time matching system for large fingerprint databases

With the current rapid growth in multimedia technology, there is an imminent need for efficient techniques to search and query large image databases. Because of their unique and peculiar needs, image databases cannot be treated in a similar fashion to other types of digital libraries. The contextual dependencies present in images, and the complex nature of two-dimensional image data make the representation issues more difficult for image databases. An invariant representation of an image is still an open research issue. For these reasons, it is difficult to find a universal content-based retrieval technique. Current approaches based on shape, texture, and color for indexing image databases have met with limited success. Further, these techniques have not been adequately tested in the presence of noise and distortions. A given application domain offers stronger constraints for improving the retrieval performance. Fingerprint databases are characterized by their large size as well as noisy and distorted query images. Distortions are very common in fingerprint images due to elasticity of the skin. In this paper, a method of indexing large fingerprint image databases is presented. The approach integrates a number of domain-specific high-level features such as pattern class and ridge density at higher levels of the search. At the lowest level, it incorporates elastic structural feature-based matching for indexing the database. With a multilevel indexing approach, we have been able to reduce the search space. The search engine has also been implemented on Splash 2-a field programmable gate array (FPGA)-based array processor to obtain near-ASIC level speed of matching. Our approach has been tested on a locally collected test data and on NIST-9, a large fingerprint database available in the public domain     

Semantic processing of multimedia data for e-government applications

Knowledge management has become a challenge for almost all e-government applications where the efficient processing of large amounts of data is still a critical issue. In the last years, semantic techniques have been introduced to improve the full automatic digitalization process of documents, in order to facilitate the access to the information embedded in very large document repositories. In this paper, we present a novel model for multimedia digital documents aiming at improve effectiveness of digitalization activities within an information system supporting e-government organizations. At the best of our knowledge, the proposed model is one of the first attempts to give a single and unified characterization of multimedia documents managed by e-government applications, whereas semantic procedures and multimedia facilities are used for the transformation of unstructured documents into structured information. Furthermore, we define an architecture for the management of multimedia documents “life cycle”, which provides advanced functionalities for information extraction, semantic retrieval, indexing, storage, presentation, together with long-term preservation. Preliminary experiments concerning an e-health scenario are finally presented and discussed.