SurvSurf: human retrieval on large surveillance video data

The volume of surveillance videos is increasing rapidly, where humans are the major objects of interest. Rapid human retrieval in surveillance videos is therefore desirable and applicable to a broad spectrum of applications. Existing big data processing tools that mainly target textual data cannot be applied directly for timely processing of large video data due to three main challenges: videos are more data-intensive than textual data; visual operations have higher computational complexity than textual operations; and traditional segmentation may damage video data’s continuous semantics. In this paper, we design SurvSurf, a human retrieval system on large surveillance video data that exploits characteristics of these data and big data processing tools. We propose using motion information contained in videos for video data segmentation. The basic data unit after segmentation is called M-clip. M-clips help remove redundant video contents and reduce data volumes. We use the MapReduce framework to process M-clips in parallel for human detection and appearance/motion feature extraction. We further accelerate vision algorithms by processing only sub-areas with significant motion vectors rather than entire frames. In addition, we design a distributed data store called V-BigTable to structuralize M-clips’ semantic information. V-BigTable enables efficient retrieval on a huge amount of M-clips. We implement the system on Hadoop and HBase. Experimental results show that our system outperforms basic solutions by one order of magnitude in computational time with satisfactory human retrieval accuracy.     

基于提升框架的实时视频降噪方法

针对基于离散小波变换的视频降噪方法难于实时处理的问题，提出了一种基于提升框架的可实时处理的视频降噪方法。首先，对每帧图像利用提升框架进行多级小波分解，得到尺度系数和小波系数；然后，对不同层次的小波系数采用软阈值收缩方法进行滤波；小波逆变换后，利用时间域滤波方法进一步提高降噪效果。实验结果表明，该方法具有较好的实时性和去噪效果。     

Real-time video watermarking system on the compressed domain for high-definition video contents: Practical issues

Everyday, we encounter high-quality multimedia contents from HDTV broadcasting, DVD, and high-speed Internet services. These contents are, unhappily, processed and distributed without protection. This paper proposes a practical video watermarking technique on the compressed domain that is real-time and robust against video processing attacks. In particular, we focus on video processing that is commonly used in practice such as downscaling resolution, framerate changing, and transcoding. Most previous watermarking algorithms are unable to survive when these processings are strong or composite. We extract low frequency coefficients of frames in fast by partly decoding videos and apply a quantization index modulation scheme to embed and detect the watermark. On an Intel architecture computer, we implement a prototype system and measure performance against video processing attacks frequently occur in the real world. Simulation results show that our video watermarking system satisfies real-time requirements and is robust to protect the copyright of HD video contents.     

Fast Gradient-Domain Video Processing Using Octree Data Structure

We present a fast gradient domain video processing using hierarchical data structure which subdivides the processing region into an octree data. It is hard to handle large video processing by solving a 3D Poisson equation, as the derived linear system is usually large. Solving the system requires large memory space and long computational time, which makes it intractable on a standard computer. To address the scalability problem, rather than processing the video in the gradient-domain pixel by pixel, we perform the video processing in a reduced space using octree data structure, which significantly reduces the variables. We show that the proposed octree approach is efficient in both seamless and mixing gradient-domain video processing. The method enables to perform video processing in greatly reduced computational time and memory space, while receiving visually identical results with that computed from the full solution.     

Modeling nuisance variabilities with factor analysis for GMM-based audio pattern classification

Audio pattern classification represents a particular statistical classification task and includes, for example, speaker recognition, language recognition, emotion recognition, speech recognition and, recently, video genre classification. The feature being used in all these tasks is generally based on a short-term cepstral representation. The cepstral vectors contain at the same time useful information and nuisance variability, which are difficult to separate in this domain. Recently, in the context of GMM-based recognizers, a novel approach using a Factor Analysis (FA) paradigm has been proposed for decomposing the target model into a useful information component and a session variability component. This approach is called Joint Factor Analysis (JFA), since it models jointly the nuisance variability and the useful information, using the FA statistical method. The JFA approach has even been combined with Support Vector Machines, known for their discriminative power. In this article, we successfully apply this paradigm to three automatic audio processing applications: speaker verification, language recognition and video genre classification. This is done by applying the same process and using the same free software toolkit. We will show that this approach allows for a relative error reduction of over 50% in all the aforementioned audio processing tasks.     

Super-resolution reconstruction of image sequences

In an earlier work (1999), we introduced the problem of reconstructing a super-resolution image sequence from a given low resolution sequence. We proposed two iterative algorithms, the R-SD and the R-LMS, to generate the desired image sequence. These algorithms assume the knowledge of the blur, the down-sampling, the sequences motion, and the measurements noise characteristics, and apply a sequential reconstruction process. It has been shown that the computational complexity of these two algorithms makes both of them practically applicable. In this paper, we rederive these algorithms as approximations of the Kalman filter and then carry out a thorough analysis of their performance. For each algorithm, we calculate a bound on its deviation from the Kalman filter performance. We also show that the propagated information matrix within the R-SD algorithm remains sparse in time, thus ensuring the applicability of this algorithm. To support these analytical results we present some computer simulations on synthetic sequences, which also show the computational feasibility of these algorithms     

Clustered blockwise PCA for representing visual data

Principal component analysis (PCA) is extensively used in computer vision and image processing. Since it provides the optimal linear subspace in a least-square sense, it has been used for dimensionality reduction and subspace analysis in various domains. However, its scalability is very limited because of its inherent computational complexity. We introduce a new framework for applying PCA to visual data which takes advantage of the spatio-temporal correlation and localized frequency variations that are typically found in such data. Instead of applying PCA to the whole volume of data (complete set of images), we partition the volume into a set of blocks and apply PCA to each block. Then, we group the subspaces corresponding to the blocks and merge them together. As a result, we not only achieve greater efficiency in the resulting representation of the visual data, but also successfully scale PCA to handle large data sets. We present a thorough analysis of the computational complexity and storage benefits of our approach. We apply our algorithm to several types of videos. We show that, in addition to its storage and speed benefits, the algorithm results in a useful representation of the visual data.     

A parallel LEGION algorithm and cell-based architecture for real time split and merge video segmentation

Split and merge segmentation is a popular region-based segmentation scheme for its robustness and computational efficiency. But it is hard to realize for larger size images or video frames in real time due to its iterative sequential data flow pattern. A quad-tree data structure is quite popular for software implementation of the algorithm, where a local parallelism is difficult to establish due to inherent data dependency between processes. In this paper, we have proposed a parallel algorithm of splitting and merging which depends only on local operations. The algorithm is mapped onto a hierarchical cell network, which is a parallel version of Locally Excitory Globally Inhibitory Oscillatory Network (LEGION). Simulation results show that the proposed design is faster than any of the standard split and merge algorithmic implementations, without compromising segmentation quality. The timing performance enhancement is manifested in its Finite State Machine based VLSI implementation in VIRTEX series FPGA platforms. We have also shown that, though segmentation qualitywise split-and-merge algorithm is little bit behind the state-of-the-art algorithms, computational speedwise it over performs those sophisticated and complex algorithms. Good segmentation performance with minimal computational cost enables the proposed design to tackle real time segmentation problem in live video streams. In this paper, we have demonstrated live PAL video segmentation using VIRTEX 5 series FPGA. Moreover, we have extended our design to HD resolution for which the time taken is less than 5 ms rendering a processing throughput of 200 frames per second.     

InsightVideo: toward hierarchical video content organization for efficient browsing, summarization and retrieval

Hierarchical video browsing and feature-based video retrieval are two standard methods for accessing video content. Very little research, however, has addressed the benefits of integrating these two methods for more effective and efficient video content access. In this paper, we introduce InsightVideo, a video analysis and retrieval system, which joins video content hierarchy, hierarchical browsing and retrieval for efficient video access. We propose several video processing techniques to organize the content hierarchy of the video. We first apply a camera motion classification and key-frame extraction strategy that operates in the compressed domain to extract video features. Then, shot grouping, scene detection and pairwise scene clustering strategies are applied to construct the video content hierarchy. We introduce a video similarity evaluation scheme at different levels (key-frame, shot, group, scene, and video.) By integrating the video content hierarchy and the video similarity evaluation scheme, hierarchical video browsing and retrieval are seamlessly integrated for efficient content access. We construct a progressive video retrieval scheme to refine user queries through the interactions of browsing and retrieval. Experimental results and comparisons of camera motion classification, key-frame extraction, scene detection, and video retrieval are presented to validate the effectiveness and efficiency of the proposed algorithms and the performance of the system.     

Improving nearest neighbor classification using Ensembles of Evolutionary Generated Prototype Subsets

One of the most accurate types of prototype selection algorithms, preprocessing techniques that select a subset of instances from the data before applying nearest neighbor classification to it, are evolutionary approaches. These algorithms result in very high accuracy and reduction rates, but unfortunately come at a substantial computational cost. In this paper, we introduce a framework that allows to efficiently use the intermediary results of the prototype selection algorithms to further increase their accuracy performance. Instead of only using the fittest prototype subset generated by the evolutionary algorithm, we use multiple prototype subsets in an ensemble setting. Secondly, in order to classify a test instance, we only use prototype subsets that accurately classify training instances in the neighborhood of that test instance. In an experimental evaluation, we apply our new framework to four state-of-the-art prototype selection algorithms and show that, by using our framework, more accurate results are obtained after less evaluations of the prototype selection method. We also present a case study with a prototype generation algorithm, showing that our framework is easily extended to other preprocessing paradigms as well.     

The priority curve algorithm for video summarization

In this paper, we introduce the concept of a priority curve associated with a video. We then provide an algorithm that can use the priority curve to create a summary (of a desired length) of any video. The summary thus created exhibits nice continuity properties and also avoids repetition. We have implemented the priority curve algorithm (PriCA) and compared it with other summarization algorithms in the literature with respect to both performance and the output quality. The quality of summaries was evaluated by a group of 200 students in Naples, Italy, who watched soccer videos. We show that PriCA is faster than existing algorithms and also produces better quality summaries. We also briefly describe a soccer video summarization system we have built on using the PriCA architecture and various (classical) image processing algorithms.     

Real-time medical video processing, enabled by hardware accelerated correlations

Image processing involving correlation based filter algorithms have proved extremely useful for image enhancement, feature extraction and recognition, in a wide range of medical applications, but is almost exclusively used with still images due to the amount of computations required by the correlations. In this paper, we present two different practical methods for applying correlation-based algorithms to real-time video images, using hardware accelerated correlation, as well as our results in applying the method to optical venography. The first method employs a GPU accelerated personal computer, while the second method employs an embedded FPGA. We will discuss major difference between the two approaches, and their suitability for clinical use. The system presented detects blood vessels in human forearms in images from NIR camera setup for the use in a clinical environment.     

On fast microscopic browsing of MPEG-compressed video

MPEG has been established as a compression standard for efficient storage and transmission of digital video. However, users are limited to VCR-like (and tedious) functionalities when viewing MPEG video. The usefulness of MPEG video is presently limited by the lack of tools available for fast browsing, manipulation and processing of MPEG video. In this paper, we first address the problem of rapid access to individual shots and frames in MPEG video. We build upon the compressed-video-processing framework proposed in [1, 8], and propose new and fast algorithms based on an adaptive mixture of approximation techniques for extracting spatially reduced image sequence of uniform quality from MPEG video across different frame types and also under different motion activities in the scenes. The algorithms execute faster than real time on a Pentium personal computer. We demonstrate how the reduced images facilitate fast and convenient shot- and frame-level video browsing and access, shot-level editing and annotation, without the need for frequent decompression of MPEG video. We further propose methods for reducing the auxiliary data size associated with the reduced images through exploitation of spatial and temporal redundancy. We also address how the reduced images lead to computationally efficient algorithms for video analysis based on intra- and inter-shot processing for video database and browsing applications. The algorithms, tools for browsing and techniques for video processing presented in this paper have been used by many in IBM Research on more than 30 h of MPEG-1 video for video browsing and analysis.     

Essential Matrix Estimation Using Gauss-Newton Iterations on a Manifold

A novel approach for essential matrix estimation is presented, this being a key task in stereo vision processing. We estimate the essential matrix from point correspondences between a stereo image pair, assuming that the internal camera parameters are known. The set of essential matrices forms a smooth manifold, and a suitable cost function can be defined on this manifold such that its minimum is the desired essential matrix. We seek a computationally efficient optimization scheme towards meeting the demands of on-line processing of video images. Our work extends and improves the earlier research by Ma et al., who proposed an intrinsic Riemannian Newton method for essential matrix computations. In contrast to Ma et al., we propose three Gauss-Newton type algorithms that have improved convergence properties and reduced computational cost. The first one is based on a novel intrinsic Newton method, using the normal Riemannian metric on the manifold consisting of all essential matrices. The other two methods are Newton-like methods, that are more efficient from a numerical point of view. Local quadratic convergence of the algorithms is shown, based on a careful analysis of the underlying geometry of the problem.     

High performance computing algorithms for land cover dynamics using remote sensing data

Global and regional land cover studies need to apply complex models on selected subsets of large volumes of multi-sensor and multi-temporal data sets that have been derived from raw instrument measurements using widely accepted pre-processing algorithms. The computational and storage requirements of most of these studies far exceed what is possible on a single workstation environment. We have been pursuing a new approach that couples scalable and open distributed heterogeneous hardware with the development of high performance software for processing, indexing and organizing remotely sensed data. Hierarchical data management tools are used to ingest raw data, create metadata and organize the archived data so as to automatically achieve computational load balancing among the available nodes and minimize input/output overheads. We illustrate our approach with four specific examples. The first is the development of the first fast operational scheme for the atmospheric correction of Landsat Thematic Mapper scenes, while the second example focuses on image segmentation using a novel hierarchical connected components algorithm. Retrieval of the global Bidirectional Reflectance Distribution Function in the red and near-infrared wavelengths using four years (1983 to 1986) of Pathfinder Advanced Very High Resolution Radiometer (AVHRR) Land data is the focus of our third example. The fourth example is the development of a hierarchical data organization scheme that allows on-demand processing and retrieval of regional and global AVHRR data sets. Our results show that substantial reductions in computational times can be achieved by the high performance computing technology.     

基于压缩域的关键帧快速提取方法

关键帧提取技术是基于内容检索和视频分析的基础。关键帧的使用减少了视频索引的数据量,同时也为视频摘要和检索提供了一个组织框架。首先介绍了目前的关键帧提取技术,然后提出了一种基于运动特征利用模糊推理算法从MPEG视频流中提取关键帧的方法。由于处理过程是直接从MPEG的压缩视频提取,不需对其解压,所以计算复杂度低,提高了提取速度。实验证明该方法效率高,可以比较好地代表视频内容。     

Video image processing with the Sonic architecture

Current industrial video-processing systems use a mixture of high-performance workstations and application-specific integrated circuits. However, video image processing in the professional broadcast environment requires more computational power and data throughput than most of today's general-purpose computers can provide. In addition, using ASICs for video image processing is both inflexible and expensive. Configurable computing offers an appropriate alternative for broadcast video image editing and manipulation by combining the flexibility, programmability, and economy of general-purpose processors with the performance of dedicated ASICs. Sonic is a configurable computing system that performs real-time video image processing. The authors describe how it implements algorithms for two-dimensional linear transforms, fractal image generation, filters, and other video effects. Sonic's flexible and scalable architecture contains configurable processing elements that accelerate software applications and support the use of plug-in software     

Moment Kernels for Regular Distributions

Many machine learning problems in natural language processing, transaction-log analysis, or computational biology, require the analysis of variable-length sequences, or, more generally, distributions of variable-length sequences.Kernel methods introduced for fixed-size vectors have proven very successful in a variety of machine learning tasks. We recently introduced a new and general kernel framework, rational kernels, to extend these methods to the analysis of variable-length sequences or more generally distributions given by weighted automata. These kernels are efficient to compute and have been successfully used in applications such as spoken-dialog classification with Support Vector Machines.However, the rational kernels previously introduced in these applications do not fully encompass distributions over alternate sequences. They are based only on the counts of co-occurring subsequences averaged over the alternate paths without taking into accounts information about the higher-order moments of the distributions of these counts.In this paper, we introduce a new family of rational kernels, moment kernels, that precisely exploits this additional information. These kernels are distribution kernels based on moments of counts of strings. We describe efficient algorithms to compute moment kernels and apply them to several difficult spoken-dialog classification tasks. Our experiments show that using the second moment of the counts of n-gram sequences consistently improves the classification accuracy in these tasks.Editors: Dan Roth and Pascale Fung     

MPEG Video Encryption Algorithms

Multimedia data security is important for multimedia commerce. Previous cryptography studies have focused on text data. The encryption algorithms developed to secure text data may not be suitable to multimedia applications because of the large data size and real time constraint. For multimedia applications, light weight encryption algorithms are attractive.We present four fast MPEG video encryption algorithms. These algorithms use a secret key to randomly change the sign bits of Discrete Cosine Transform (DCT) coefficients and/or the sign bits of motion vectors. The encryption is accomplished by the inverse DCT (IDCT) during the MPEG video decompression processing. These algorithms add a small overhead to MPEG codec. Software implementations are fast enough to meet the real time requirement of MPEG video applications. The experimental results show that these algorithms achieve satisfactory results. They can be used to secure video-on-demand, video conferencing, and video email applications.     

Fast parallel algorithms for discrete Gabor expansion and transform based on multirate filtering

The Gabor transform has long been recognized as a very useful tool for the joint time and frequency analysis in signal processing.Its real time applications,however,were limited due to the high computational complexity of the Gabor transform algorithms.In this paper,some novel and fast parallel algorithms for the finite discrete Gabor expansion and transform are presented based on multirate filtering.An analysis filter bank is designed for the finite discrete Gabor transform(DGT)and a synthesis filter bank is designed for the finite discrete Gabor expansion(DGE).Each of the parallel channels in the two filter banks has a unified structure and can apply the FFT and the IFFT to reduce its computational load.The computational complexity of each parallel channel does not change as the oversampling rate increases.In fact,it is very low and depends only on the length of the input discrete signal and the number of the Gabor frequency sampling points.The computational complexity of the proposed parallel algorithms is analyzed and compared with that of the major existing parallel algorithms for the finite DGT and DGE.The results indicate that the proposed parallel algorithms for the finite DGT and DGE based on multirate filtering are very attractive for real time signal processing.