Real-time camera motion tracking in planar view scenarios

We propose a novel method for real-time camera motion tracking in planar view scenarios. This method relies on the geometry of a tripod, an initial estimation of camera pose for the first video frame and a primitive tracking procedure. This process uses lines and circles as primitives, which are extracted applying classification and regression tree. We have applied the proposed method to high-definition videos of soccer matches. Experimental results prove that our proposal can be applied to processing high-definition video in real time. We validate the procedure by inserting virtual content in the video sequence.     

高速球形摄像机PTZ跟踪策略设计及实现

针对智能监控中基于高速球形摄像机的PTZ跟踪功能模块,设计了一种PTZ跟踪控制策略。该策略在球机机械参数未知的情况下,一方面能控制球机实时地跟踪目标使目标始终处于视野中央,另一方面可自动进行变倍动作来放大拍摄目标的局部细节。针对球机Zoom控制中跟踪窗口大小自适应调整的问题,利用SIFT算法设计了一种计算球机变倍率的方法。利用VS2005和OpenCV软件平台实现了PTZ跟踪的整体流程。实验表明,该策略能有效、稳定地进行PTZ跟踪。     

Augmented Panoramic Video

Many video sequences consist of a locally dynamic background containing moving foreground subjects. In this paper we propose a novel way of re‐displaying these sequences, by giving the user control over a virtual camera frame. Based on video mosaicing, we first compute a static high quality background panorama. After segmenting and removing the foreground subjects from the original video, the remaining elements are merged into a dynamic background panorama, which seamlessly extends the original video footage. We then re‐display this augmented video by warping and cropping the panorama. The virtual camera can have an enlarged field‐of‐view and a controlled camera motion. Our technique is able to process videos with complex camera motions, reconstructing high quality panoramas without parallax artefacts, visible seams or blurring, while retaining repetitive dynamic elements.     

Structure and event mining in sports video with efficient mosaic

Video is an information-intensive media with much redundancy. Therefore, it is desirable to be able to mine structure or semantics of video data for efficient browsing, summarization and highlight extraction. In this paper, we propose a mosaic based approach to key-event as well as structure mining, which is regarded as a complementary view for sports video analysis. Mosaic is generated for each shot by a novel efficient mosaicing scheme, which constructs a global motion path and selects a best subset of frames for mosaicing. These improved mosaics are then used as the representative image of shot content. Based on mosaic, the structure and event in sports video are mined by the methods with prior knowledge and without prior knowledge. Without prior knowledge, our system is able to locate global view shots taken by dominant camera. If prior knowledge is available, the events in these global view shots are detected using robust features extracted from mosaics. For global view mining, the experiments compared with key-frame-based scheme have demonstrated that this mosaic-based scheme presents better results in several kinds of sports videos; for events mining, the detection of key-plays and key-events in the specific-domain of soccer videos have proved its effectiveness.

Image mosaicing using sequential bundle adjustment

We describe the construction of accurate panoramic mosaics from multiple images taken with a rotating camera, or alternatively of a planar scene. The novelty of the approach lies in (i) the transfer of photogrammetric bundle adjustment techniques to mosaicing; (ii) a new representation of image line measurements enabling the use of lines in camera self-calibration, including computation of the radial and other non-linear distortion; and (iii) the application of the variable state dimension filter to obtain efficient sequential updates of the mosaic as each image is added.We demonstrate that our method achieves better results than the alternative approach of optimising over pairs of images.     

A fast and accurate moving object tracker in active camera model

Detecting and tracking moving objects within a scene is an essential step for high-level machine vision applications such as video content analysis. In this paper, we propose a fast and accurate method for tracking an object of interest in a dynamic environment (active camera model). First, we manually select the region of the object of interest and extract three statistical features, namely the mean, the variance and the range of intensity values of the feature points lying inside the selected region. Then, using the motion information of the background’s feature points and k-means clustering algorithm, we calculate camera motion transformation matrix. Based on this matrix, the previous frame is transformed to the current frame’s coordinate system to compensate the impact of camera motion. Afterwards, we detect the regions of moving objects within the scene using our introduced frame difference algorithm. Subsequently, utilizing DBSCAN clustering algorithm, we cluster the feature points of the extracted regions in order to find the distinct moving objects. Finally, we use the same statistical features (the mean, the variance and the range of intensity values) as a template to identify and track the moving object of interest among the detected moving objects. Our approach is simple and straightforward yet robust, accurate and time efficient. Experimental results on various videos show an acceptable performance of our tracker method compared to complex competitors.     

一种基于球场模型的广播足球视频摄像机自动定标方法

随着足球视频语义内容分析和理解、自动视频索引、虚拟增强现实表示等研究的不断深入,摄像机定标技术扮演了越来越重要的角色.基于此,提出了一种基于球场模型的广播足球视频摄像机自动定标算法,该算法建立在所定义的球场模型基础之上,首先通过改进的球场线探测算法得到球场平面的特征点,并求解摄像机同形 (Homography)矩阵,然后,利用所求解的同形矩阵来定位球门,得到球门平面的两个特征点,最后根据基于K均值聚类和Hough-like搜索的方法来得到最优的摄像机内外参数.初步的实验结果表明,依据本文算法所求解的摄像机参数最优值相比初始值精确度更高,并且其在连续帧中也更加稳定.     

足球视频球员感知跟踪算法

目的 足球比赛视频中的球员跟踪算法为足球赛事分析提供基础的数据支持。但足球比赛中球员跟踪存在极大的挑战：球员进攻、防守和争夺球权时,目标球员可能产生快速移动、严重遮挡和周围出现若干名干扰球员的情况,目前仍没有一种能够完美解决足球比赛中球员跟踪问题的算法。因此如何解决足球场景中的困难,提升球员跟踪的准确度,成为当前研究的热点问题。方法 本文在分析足球比赛视频中球员目标特点的基础上,通过融合干扰项感知的颜色模型和目标感知的深度模型,提出并设计了一种球员感知的跟踪算法。干扰项感知的颜色模型分别提取目标、背景和干扰项的颜色直方图,利用贝叶斯公式得到搜索区域中每个像素点属于目标的似然概率。目标感知的深度模型利用孪生网络计算搜索区域与目标的相似度。针对跟踪漂移问题,使用全局跟踪器和局部跟踪器分别跟踪目标整体和目标上半身,并且在两个跟踪器的跟踪结果出现较大差异的时候分析跟踪器有效性并进行定位修正。结果 在公共的足球数据集上将本文算法与10个其他跟踪算法进行对比实验,同时对于文本算法进行了局部跟踪器的消融实验。实验结果表明,球员感知跟踪算法的平均有效重叠率达到了0.560 3,在存在同队球员和异队球员干扰的情况下,本文算法比排名第2的算法的有效重叠率分别高出3.7%和6.6%,明显优于其他算法,但是由于引入了干扰项感知的颜色模型、目标感知的深度模型以及局部跟踪器等模块增加了算法的时间复杂度,导致本文算法跟踪速度较慢。结论 本文总结了跟踪算法的整体流程并分析了实验结果,认为干扰项感知、目标感知和局部跟踪这3个策略在足球场景中的球员跟踪问题中起到了重要的作用,为未来在足球球员跟踪领域研究的继续深入提供了参考依据。     

Extracting representative motion flows for effective video retrieval

In this paper, we propose a novel motion-based video retrieval approach to find desired videos from video databases through trajectory matching. The main component of our approach is to extract representative motion features from the video, which could be broken down to the following three steps. First, we extract the motion vectors from each frame of videos and utilize Harris corner points to compensate the effect of the camera motion. Second, we find interesting motion flows from frames using sliding window mechanism and a clustering algorithm. Third, we merge the generated motion flows and select representative ones to capture the motion features of videos. Furthermore, we design a symbolic based trajectory matching method for effective video retrieval. The experimental results show that our algorithm is capable to effectively extract motion flows with high accuracy and outperforms existing approaches for video retrieval.     

Seeing the Objects Behind the Dots: Recognition in Videos from a Moving Camera

Category-level object recognition, segmentation, and tracking in videos becomes highly challenging when applied to sequences from a hand-held camera that features extensive motion and zooming. An additional challenge is then to develop a fully automatic video analysis system that works without manual initialization of a tracker or other human intervention, both during training and during recognition, despite background clutter and other distracting objects. Moreover, our working hypothesis states that category-level recognition is possible based only on an erratic, flickering pattern of interest point locations without extracting additional features. Compositions of these points are then tracked individually by estimating a parametric motion model. Groups of compositions segment a video frame into the various objects that are present and into background clutter. Objects can then be recognized and tracked based on the motion of their compositions and on the shape they form. Finally, the combination of this flow-based representation with an appearance-based one is investigated. Besides evaluating the approach on a challenging video categorization database with significant camera motion and clutter, we also demonstrate that it generalizes to action recognition in a natural way. Electronic Supplementary Material  The online version of this article () contains supplementary material, which is available to authorized users. This work was supported in part by the Swiss national science foundation under contract no. 200021-107636.     

Graph-Based Multiplayer Detection and Tracking in Broadcast Soccer Videos

In this paper, we propose a graph-based approach for detecting and tracking multiple players in broadcast soccer videos. In the first stage, the position of the players in each frame is determined by removing the non player regions. The remaining pixels are then grouped using a region growing algorithm to identify probable player candidates. A directed weighted graph is constructed, where probable player candidates correspond to the nodes of the graph while each edge links candidates in a frame with the candidates in next two consecutive frames. Finally, dynamic programming is applied to find the trajectory of each player. Experiments with several sequences from broadcasted videos of international soccer matches indicate that the proposed approach is able to track the players reasonably well even under varied illumination and ground conditions.      

Rectified Surface Mosaics

We approach mosaicing as a camera tracking problem within a known parameterized surface. From a video of a camera moving within a surface, we compute a mosaic representing the texture of that surface, flattened onto a planar image. Our approach works by defining a warp between images as a function of surface geometry and camera pose. Globally optimizing this warp to maximize alignment across all frames determines the camera trajectory, and the corresponding flattened mosaic image. In contrast to previous mosaicing methods which assume planar or distant scenes, or controlled camera motion, our approach enables mosaicing in cases where the camera moves unpredictably through proximal surfaces, such as in medical endoscopy applications.     

Efficient multi-feature PSO for fast gray level object-tracking

Robust and real-time moving object tracking is a tricky job in computer vision systems. The development of an efficient yet robust object tracker faces several obstacles, namely: dynamic appearance of deformable or articulated targets, dynamic backgrounds, variation in image intensity, and camera (ego) motion. In this paper, a novel tracking algorithm based on particle swarm optimization (PSO) method is proposed. PSO is a population-based stochastic optimization algorithm modeled after the simulation of the social behavior of bird flocks and animal hordes. In this algorithm, a multi-feature model is proposed for object detection to enhance the tracking accuracy and efficiency. The object's model is based on the gray level intensity. This model combines the effects of different object cases including zooming, scaling, rotating, etc. into a single cost function. The proposed algorithm is independent of object type and shape and can be used for many object tracking applications. Over 30 video sequences and having over 20,000 frames are used to test the developed PSO-based object tracking algorithm and compare it to classical object tracking algorithms as well as previously published PSO-based tracking algorithms. Our results demonstrate the efficiency and robustness of our developed algorithm relative to all other tested algorithms.     

Summarization of Visual Content in Instructional Videos

In instructional videos of chalk board presentations, the visual content refers to the text and figures written on the boards. Existing methods on video summarization are not effective for this video domain because they are mainly based on low-level image features such as color and edges. In this work, we present a novel approach to summarizing the visual content in instructional videos using middle-level features. We first develop a robust algorithm to extract content text and figures from instructional videos by statistical modelling and clustering. This algorithm addresses the image noise, nonuniformity of the board regions, camera movements, occlusions, and other challenges in the instructional videos that are recorded in real classrooms. Using the extracted text and figures as the middle level features, we retrieve a set of key frames that contain most of the visual content. We further reduce content redundancy and build a mosaicked summary image by matching extracted content based on K-th Hausdorff distance and connected component decomposition. Performance evaluation on four full-length instructional videos shows that our algorithm is highly effective in summarizing instructional video content.     

Intrinsic and extrinsic active self-calibration of multi-camera systems

We present a method for active self-calibration of multi-camera systems consisting of pan-tilt zoom cameras. The main focus of this work is on extrinsic self-calibration using active camera control. Our novel probabilistic approach avoids multi-image point correspondences as far as possible. This allows an implicit treatment of ambiguities. The relative poses are optimized by actively rotating and zooming each camera pair in a way that significantly simplifies the problem of extracting correct point correspondences. In a final step we calibrate the entire system using a minimal number of relative poses. The selection of relative poses is based on their uncertainty. We exploit active camera control to estimate consistent translation scales for triplets of cameras. This allows us to estimate missing relative poses in the camera triplets. In addition to this active extrinsic self-calibration we present an extended method for the rotational intrinsic self-calibration of a camera that exploits the rotation knowledge provided by the camera’s pan-tilt unit to robustly estimate the intrinsic camera parameters for different zoom steps as well as the rotation between pan-tilt unit and camera. Quantitative experiments on real data demonstrate the robustness and high accuracy of our approach. We achieve a median reprojection error of $0.95$ pixel.     

足球视频中的目标检测与跟踪

提出了一种新颖的目标检测与跟踪算法来检测和跟踪足球运动中的球员。与普通的利用颜色分割的方法不同,考虑到足球视频中非目标的像素大体上都是单一的绿色这个特点,结合颜色的统计信息和像素的边缘特性来得到更完美的检测效果;接着利用灰度图像中的统计信息,轻松地完成球员队属的辨别;最后根据重叠面积提出一种简单的视频目标跟踪方案,结合图像匹配,解决运动中的遮挡问题。     

Video motion stitching using trajectory and position similarities

Stitching motions in multiple videos into a single video scene is a challenging task in current video fusion and mosaicing research and film production. In this paper, we present a novel method of video motion stitching based on the similarities of trajectory and position of foreground objects. First, multiple video sequences are registered in a common reference frame, whereby we estimate the static and dynamic backgrounds, with the former responsible for distinguishing the foreground from the background and the static region from the dynamic region, and the latter functioning in mosaicing the warped input video sequences into a panoramic video. Accordingly, the motion similarity is calculated by reference to trajectory and position similarity, whereby the corresponding motion parts are extracted from multiple video sequences. Finally, using the corresponding motion parts, the foregrounds of different videos and dynamic backgrounds are fused into a single video scene through Poisson editing, with the motions involved being stitched together. Our major contributions are a framework of multiple video mosaicing based on motion similarity and a method of calculating motion similarity from the trajectory similarity and the position similarity. Experiments on everyday videos show that the agreement of trajectory and position similarities with the real motion similarity plays a decisive role in determining whether two motions can be stitched. We acquire satisfactory results for motion stitching and video mosaicing.     

Common Visual Cues for Sports Highlights Modeling

Automatic annotation of semantic events allows effective retrieval of video content. In this work, we present solutions for highlights detection in sports videos. The proposed approach exploits the typical structure of a wide class of sports videos, namely those related to sports which are played in delimited venues with playfields of well known geometry, like soccer, basketball, swimming, track and field disciplines, and so on. For these sports, a modeling scheme based on a limited set of visual cues and on finite state machines that encode the temporal evolution of highlights is presented, that is of general applicability to this class of sports. Visual cues encode position and speed information coming from the camera and from the object/athletes that are present in the scene, and are estimated automatically from the video stream. Algorithms for model checking and for visual cues estimation are discussed, as well as applications of the representation to different sport domains.     

Autoconfiguration of a dynamic nonoverlapping camera network.

In order to monitor sufficiently large areas of interest for surveillance or any event detection, we need to look beyond stationary cameras and employ an automatically configurable network of nonoverlapping cameras. These cameras need not have an overlapping field of view and should be allowed to move freely in space. Moreover, features like zooming in/out, readily available in security cameras these days, should be exploited in order to focus on any particular area of interest if needed. In this paper, a practical framework is proposed to self-calibrate dynamically moving and zooming cameras and determine their absolute and relative orientations, assuming that their relative position is known. A global linear solution is presented for self-calibrating each zooming/focusing camera in the network. After self-calibration, it is shown that only one automatically computed vanishing point and a line lying on any plane orthogonal to the vertical direction is sufficient to infer the dynamic network configuration. Our method generalizes previous work which considers restricted camera motions. Using minimal assumptions, we are able to successfully demonstrate promising results on synthetic, as well as on real data.