An optimal time–space algorithm for dense stereo matching

The increasing demand for higher resolution images and higher frame rate videos will always pose a challenge to computational power when real-time performance is required to solve the stereo-matching problem in 3D reconstruction applications. Therefore, the use of asymptotic analysis is necessary to measure the time and space performance of stereo-matching algorithms regardless of the size of the input and of the computational power available. In this paper, we survey several classic stereo-matching algorithms with regard to time–space complexity. We also report running time experiments for several algorithms that are consistent with our complexity analysis. We present a new dense stereo-matching algorithm based on a greedy heuristic path computation in disparity space. A procedure which improves disparity maps in depth discontinuity regions is introduced. This procedure works as a post-processing step for any technique that solves the dense stereo-matching problem. We prove that our algorithm and post-processing procedure have optimal O(n) time–space complexity, where n is the size of a stereo image. Our algorithm performs only a constant number of computations per pixel since it avoids a brute force search over the disparity range. Hence, our algorithm is faster than “real-time” techniques while producing comparable results when evaluated with ground-truth benchmarks. The correctness of our algorithm is demonstrated with experiments in real and synthetic data.     

A local–global pattern matching method for subsurface stochastic inverse modeling

Inverse modeling is an essential step for reliable modeling of subsurface flow and transport, which is important for groundwater resource management and aquifer remediation. Multiple-point statistics (MPS) based reservoir modeling algorithms, beyond traditional two-point statistics-based methods, offer an alternative to simulate complex geological features and patterns, conditioning to observed conductivity data. Parameter estimation, within the framework of MPS, for the characterization of conductivity fields using measured dynamic data such as piezometric head data, remains one of the most challenging tasks in geologic modeling. We propose a new local–global pattern matching method to integrate dynamic data into geological models. The local pattern is composed of conductivity and head values that are sampled from joint training images comprising of geological models and the corresponding simulated piezometric heads. Subsequently, a global constraint is enforced on the simulated geologic models in order to match the measured head data. The method is sequential in time, and as new piezometric head become available, the training images are updated for the purpose of reducing the computational cost of pattern matching. As a result, the final suite of models preserve the geologic features as well as match the dynamic data. This local–global pattern matching method is demonstrated for simulating a two-dimensional, bimodally-distributed heterogeneous conductivity field. The results indicate that the characterization of conductivity as well as flow and transport predictions are improved when the piezometric head data are integrated into the geological modeling.     

Multi-scale spatial–temporal convolutional neural network for skeleton-based action recognition

Pattern Analysis and Applications - The skeleton data convey significant information for action recognition since they can robustly against cluttered backgrounds and illumination variation. In...     

Animated visualization of spatial–temporal trajectory data for air-traffic analysis

With increasing numbers of flights worldwide and a continuing rise in airport traffic, air-traffic management is faced with a number of challenges. These include monitoring, reporting, planning, and problem analysis of past and current air traffic, e.g., to identify hotspots, minimize delays, or to optimize sector assignments to air-traffic controllers. To cope with these challenges, cyber worlds can be used for interactive visual analysis and analytical reasoning based on aircraft trajectory data. However, with growing data size and complexity, visualization requires high computational efficiency to process that data within real-time constraints. This paper presents a technique for real-time animated visualization of massive trajectory data. It enables (1) interactive spatio-temporal filtering, (2) generic mapping of trajectory attributes to geometric representations and appearance, and (3) real-time rendering within 3D virtual environments such as virtual 3D airport or 3D city models. Different visualization metaphors can be efficiently built upon this technique such as temporal focus+context, density maps, or overview+detail methods. As a general-purpose visualization technique, it can be applied to general 3D and 3+1D trajectory data, e.g., traffic movement data, geo-referenced networks, or spatio-temporal data, and it supports related visual analytics and data mining tasks within cyber worlds.     

STTG-TTE: spatial–temporal gated multi-modality approach for travel time estimation based on temporal convolutional networks

Neural Computing and Applications - Travel time forecasting has become a core component of smart transportation systems, which assists both travelers and traffic organizers with route planning,...     

Mutual information-based selection of optimal spatial–temporal patterns for single-trial EEG-based BCIs

The common spatial pattern (CSP) algorithm is effective in decoding the spatial patterns of the corresponding neuronal activities from electroencephalogram (EEG) signal patterns in brain–computer interfaces (BCIs). However, its effectiveness depends on the subject-specific time segment relative to the visual cue and on the temporal frequency band that is often selected manually or heuristically. This paper presents a novel statistical method to automatically select the optimal subject-specific time segment and temporal frequency band based on the mutual information between the spatial–temporal patterns from the EEG signals and the corresponding neuronal activities. The proposed method comprises four progressive stages: multi-time segment and temporal frequency band-pass filtering, CSP spatial filtering, mutual information-based feature selection and naïve Bayesian classification. The proposed mutual information-based selection of optimal spatial–temporal patterns and its one-versus-rest multi-class extension were evaluated on single-trial EEG from the BCI Competition IV Datasets IIb and IIa respectively. The results showed that the proposed method yielded relatively better session-to-session classification results compared against the best submission.     

Robust spatial–temporal Bayesian view synthesis for video stitching with occlusion handling

Occlusion is visible in only one frame and cannot be seen in the other frame which is a vital challenge in video stitching. Occlusion always brings ghost artifacts in the blended area. Meanwhile, the traditional image stitching approaches ignore temporal consistency and cannot avoid flicking problem. To solve these challenges, we propose a unified framework in which the stitching quality and stabilization both perform well. Specifically, we explicitly detect the potential occlusion regions to indicate blending information. Then, based on the occlusion maps, we choose a proper strip in the overlapped region as the blending area. With spatial–temporal Bayesian view synthesis, spatial ghost-like artifacts can be significantly eliminated and the output videos can be kept stable. The experimental results show the out performance of the proposed approach compared to state-of-the-art approaches.     

Development of a novel system to quantify the spatial–temporal parameters for crutch-assisted quadrupedal gait

Abstract

Patients with gait disorders often use bilateral crutches along with their own two legs. It is a kind of quadrupedalism. Crutch-assisted gait is usually described and evaluated qualitatively. In this study, we developed a system to quantify the spatial and temporal parameters for crutch-assisted quadrupedalism. Our system consists of walkway hardware and our originally developed software. We specifically extended the measurable area to 1200 mm × 4800 mm, large enough to measure crutch gait. Using our system, we could describe crutch gait precisely. Our system has a capability to evaluate differences between patients and changes within a patient.     

A lightweight model with spatial–temporal correlation for cellular traffic prediction in Internet of Things

The Journal of Supercomputing - Accurate cellular traffic prediction becomes more and more critical for efficient network resource management in the Internet of Things (IoT). However, high-accuracy...     

Fast computation of 3D Meixner’s invariant moments using 3D image cuboid representation for 3D image classification

Multimedia Tools and Applications - In this paper, we propose a new fast computation method of 3D discrete orthogonal invariant moments of Meixner. This proposed method is based on two fundamental...     

Fast and exact unidimensional L2–L1 optimization as an accelerator for iterative reconstruction algorithms

This paper studies the use of fast and exact unidimensional L2–L1 minimization as a line search for accelerating iterative reconstruction algorithms. In L2–L1 minimization reconstruction problems, the squared Euclidean, or L2 norm, measures signal-data discrepancy and the L1 norm stands for a sparsity preserving regularization term. Functionals as these arise in important applications such as compressed sensing and deconvolution. Optimal unidimensional L2–L1 minimization has only recently been studied by Li and Osher for denoising problems and by Wen et al. for line search. A fast L2–L1 optimization procedure can be adapted for line search and used in iterative algorithms, improving convergence speed with little increase in computational cost. This paper proposes a new method for exact L2–L1 line search and compares it with the Li and Osher's, Wen et al.'s, as well as with a standard line search algorithm, the method of false position. The use of the proposed line search improves convergence speed of different iterative algorithms for L2–L1 reconstruction such as iterative shrinkage, iteratively reweighted least squares, and nonlinear conjugate gradient. This assertion is validated experimentally in applications to signal reconstruction in compressed sensing and sparse signal deblurring.     

Exploring spatial–temporal relations via deep convolutional neural networks for traffic flow prediction with incomplete data

Traffic flow prediction is a fundamental component in intelligent transportation systems. Various computational methods have been applied in this field, among which machine learning based methods are believed to be promising and scalable for big data. In general, most of machine learning based methods encounter three fundamental issues: feature representation of traffic patterns, learning from single location or network, and data quality. In order to address these three issues, in this work we present a deep architecture for traffic flow prediction that learns deep hierarchical feature representation with spatio-temporal relations over the traffic network. Furthermore, we design an ensemble learning strategy via random subspace learning to make the model be able to tolerate incomplete data. Accordingly the contributions of this work are summarized as the three points. First, we transform the time series analysis problem into the task of image-like analysis. Benefitting from the image-like data form, we can jointly explore spatio-temporal relations simultaneously by the two-dimension convolution operator. In addition, the proposed model can tolerate the incomplete data, which is very common in traffic application field. Finally, we propose an improved random search based on uniform design in order to optimize hyper-parameters for deep Convolutional Neural Networks (deep CNN). A large range of experiments with various traffic conditions have been performed on the traffic data originated from the California Freeway Performance Measurement System (PeMS). The experimental results corroborate the effectiveness of the proposed approach compared with the state of the art.     

3D reconstruction of specular surfaces using a calibrated projector–camera setup

Structured light is widely used for shape measurement of beamless surfaces using the triangulation principle. In the case of specular surfaces deflectometry is an appropriate method. Hereby the reflection of a light pattern is observed by a camera. The distortion of the reflected pattern is evaluated to obtain information about the reflecting surface. An important requirement for a 3d reconstruction of a specular surface by deflectometry is a calibrated measurement setup. We propose a method for the overall calibration of a setup consisting of a structured light source, a projection screen and a camera. We consider all extrinsic and intrinsic parameters for the optical mapping including a distortion model for the projector and for the camera, respectively. Given the deflectometric data obtained by the calibrated setup two methods are described which allow the 3d reconstruction of points on a specular surface. This is not trivial as a reconstruction using solely deflectometric data shows ambiguity. In the first method screen displacement between two deflectometric measurements is used to overcome this ambiguity. In the second method curvature-like features on the surface are determined to serve as starting points for a region growing approach. Results of the reconstruction of a specular surface are shown and the performance of the described reconstruction methods are compared to each other.     

Dynamic 3D surface reconstruction and motion modeling from a pan–tilt–zoom camera

3D surface reconstruction and motion modeling has been integrated in several industrial applications. Using a pan–tilt–zoom (PTZ) camera, we present an efficient method called dynamic 3D reconstruction (D3DR) for recovering the 3D motion and structure of a freely moving target. The proposed method estimates the PTZ measurements to keep the target in the center of the field of view (FoV) of the camera with the same size. Feature extraction and tracking approach are used in the imaging framework to estimate the target's translation, position, and distance. A selection strategy is used to select keyframes that show significant changes in target movement and directly update the recovered 3D information. The proposed D3DR method is designed to work in a real-time environment, not requiring all frames captured to be used to update the recovered 3D motion and structure of the target. Using fewer frames minimizes the time and space complexity required. Experimental results conducted on real-time video streams using different targets to prove the efficiency of the proposed method. The proposed D3DR has been compared to existing offline and online 3D reconstruction methods, showing that it uses less execution time than the offline method and uses an average of 49.6% of the total number of frames captured.     

Particle filters and occlusion handling for rigid 2D–3D pose tracking

In this paper, we address the problem of 2D–3D pose estimation. Specifically, we propose an approach to jointly track a rigid object in a 2D image sequence and to estimate its pose (position and orientation) in 3D space. We revisit a joint 2D segmentation/3D pose estimation technique, and then extend the framework by incorporating a particle filter to robustly track the object in a challenging environment, and by developing an occlusion detection and handling scheme to continuously track the object in the presence of occlusions. In particular, we focus on partial occlusions that prevent the tracker from extracting an exact region properties of the object, which plays a pivotal role for region-based tracking methods in maintaining the track. To this end, a dynamical choice of how to invoke the objective functional is performed online based on the degree of dependencies between predictions and measurements of the system in accordance with the degree of occlusion and the variation of the object’s pose. This scheme provides the robustness to deal with occlusions of an obstacle with different statistical properties from that of the object of interest. Experimental results demonstrate the practical applicability and robustness of the proposed method in several challenging scenarios.     

A multigrid scheme for 3D Monge–Ampère equations*

The elliptic Monge–Ampère equation is a fully nonlinear partial differential equation which has been the focus of increasing attention from the scientific computing community. Fast three-dimensional solvers are needed, for example in medical image registration but are not yet available. We build fast solvers for smooth solutions in three dimensions using a nonlinear full-approximation storage multigrid method. Starting from a second-order accurate centred finite difference approximation, we present a nonlinear Gauss–Seidel iterative method which has a mechanism for selecting the convex solution of the equation. The iterative method is used as an effective smoother, combined with the full-approximation storage multigrid method. Numerical experiments are provided to validate the accuracy of the finite difference scheme and illustrate the computational efficiency of the proposed multigrid solver.     

A 3D ellipsoidal volumetric foot–ground contact model for forward dynamics

Foot–ground contact models are an important part of forward dynamic biomechanic models, particularly those used to model gait, and have many challenges associated with them. Contact models can dramatically increase the complexity of the multibody system equations, especially if the contact surface is relatively large or conforming. Since foot–ground contact has a large potential contact area, creating a computationally efficient model is challenging. This is particularly problematic in predictive simulations, which may determine optimal performance by running a model simulation thousands of times. An ideal contact model must find a balance between accuracy for large, conforming surfaces, and computational efficiency.Volumetric contact modelling is explored as a computationally efficient model for foot–ground contact. Previous foot models have used volumetric contact before, but were limited to 2D motion and approximated the surfaces as spheres or 2D shapes. The model presented here improves on current work by using ellipsoid contact geometry and considering 3D motion and geometry. A gait experiment was used to parametrise and validate the model. The model ran over 100 times faster than real-time (in an inverse simulation at 128 fps) and matched experimental normal force and centre of pressure location with less than 7% root-mean-square error.In most gait studies, only the net reaction forces, centre of pressure, and body motions are recorded and used to identify parameters. In this study, contact pressure was also recorded and used as a part of the identification, which was found to increase parameter optimisation time from 10 to 164 s (due to the additional time needed to calculate the pressure distribution) but helped the results converge to a more realistic model. The model matched experimental pressures with 33–45% root-mean-square error, though some of this was due to measurement errors.The same parametrisation was done with friction included in the foot model. It was determined that the velocity-based friction model that was used was inappropriate for use in an inverse-dynamics simulation. Attempting to optimise the model to match experimental friction resulted in a poor match to the experimental friction forces, inaccurate values for the coefficient of friction, and a poorer match to the experimental normal force.     

Uniform regularity for a 3D time-dependent Ginzburg–Landau model in superconductivity

In this paper, we prove the uniform-in-$\sigma$ regularity for 3D time-dependent Ginzburg–Landau model in superconductivity in the case of Coulomb gauge. Here $\sigma$ is the normal conductivity of the superconducting material.