Full uncertainty analysis for Bayesian nonparametric mixture models

A full posterior analysis for nonparametric mixture models using Gibbs-type prior distributions is presented. This includes the well known Dirichlet process mixture (DPM) model. The random mixing distribution is removed enabling a simple-to-implement Markov chain Monte Carlo (MCMC) algorithm. The removal procedure takes away some of the posterior uncertainty and how it is replaced forms a novel aspect to the work. The removal, MCMC algorithm and replacement of the uncertainty only require the probabilities of a new or an old value associated with the corresponding Gibbs-type exchangeable sequence. Consequently, no explicit representations of the prior or posterior are required and instead only knowledge of the exchangeable sequence is needed. This allows the implementation of mixture models with full posterior uncertainty, not previously possible, including one introduced by Gnedin. Numerous illustrations are presented, as is an R-package called CopRe which implements the methodology, and other supplemental material.     

Variational Bayesian methods for spatial data analysis

With scientific data available at geocoded locations, investigators are increasingly turning to spatial process models for carrying out statistical inference. However, fitting spatial models often involves expensive matrix decompositions, whose computational complexity increases in cubic order with the number of spatial locations. This situation is aggravated in Bayesian settings where such computations are required once at every iteration of the Markov chain Monte Carlo (MCMC) algorithms. In this paper, we describe the use of Variational Bayesian (VB) methods as an alternative to MCMC to approximate the posterior distributions of complex spatial models. Variational methods, which have been used extensively in Bayesian machine learning for several years, provide a lower bound on the marginal likelihood, which can be computed efficiently. We provide results for the variational updates in several models especially emphasizing their use in multivariate spatial analysis. We demonstrate estimation and model comparisons from VB methods by using simulated data as well as environmental data sets and compare them with inference from MCMC.     

Deviance information criterion (DIC) in Bayesian multiple QTL mapping

Mapping multiple quantitative trait loci (QTL) is commonly viewed as a problem of model selection. Various model selection criteria have been proposed, primarily in the non-Bayesian framework. The deviance information criterion (DIC) is the most popular criterion for Bayesian model selection and model comparison but has not been applied to Bayesian multiple QTL mapping. A derivation of the DIC is presented for multiple interacting QTL models and calculation of the DIC is demonstrated using posterior samples generated by Markov chain Monte Carlo (MCMC) algorithms. The DIC measures posterior predictive error by penalizing the fit of a model (deviance) by its complexity, determined by the effective number of parameters. The effective number of parameters simultaneously accounts for the sample size, the cross design, the number and lengths of chromosomes, covariates, the number of QTL, the type of QTL effects, and QTL effect sizes. The DIC provides a computationally efficient way to perform sensitivity analysis and can be used to quantitatively evaluate if including environmental effects, gene-gene interactions, and/or gene-environment interactions in the prior specification is worth the extra parameterization. The DIC has been implemented in the freely available package R/qtlbim, which greatly facilitates the general usage of Bayesian methodology for genome-wide interacting QTL analysis.     

Dirichlet Process Gaussian Mixture Models: Choice of the Base Distribution

In the Bayesian mixture modeling framework it is possible to infer the necessary number of components to model the data and therefore it is unnecessary to explicitly restrict the number of components. Nonparametric mixture models sidestep the problem of finding the “correct” number of mixture components by assuming infinitely many components. In this paper Dirichlet process mixture (DPM) models are cast as infinite mixture models and inference using Markov chain Monte Carlo is described. The specification of the priors on the model parameters is often guided by mathematical and practical convenience. The primary goal of this paper is to compare the choice of conjugate and non-conjugate base distributions on a particular class of DPM models which is widely used in applications, the Dirichlet process Gaussian mixture model (DPGMM). We compare computational efficiency and modeling performance of DPGMM defined using a conjugate and a conditionally conjugate base distribution. We show that better density models can result from using a wider class of priors with no or only a modest increase in computational effort.     

A novel color image multilevel thresholding based segmentation using nature inspired optimization algorithms

Multilevel thresholding for segmentation is an essential task and indispensable process in various applications. Conventional color multilevel thresholding based image segmentations are computationally expensive, and lack accuracy and stability. To address this issue, this paper introduces the comparative performance study of different objective functions using cuckoo search and other optimization algorithms to solve the color image segmentation problem via multilevel thresholding. During the optimization process, solutions are evaluated using Otsu or Kapur's method. Performance of the proposed approach has been assessed using a variety of benchmark images, and compared against three other nature inspired algorithms namely differential evolution (DE), wind driven optimization (WDO) and particle swam optimization (PSO) algorithms. Results have been analyzed both qualitatively and quantitatively, based on the fitness values of obtained best solutions and four popular performance measures namely PSNR, MSE, SSIM and FSIM indices as well. According to statistical analysis of different nature inspired optimization algorithms, Kapur's entropy was found to be more accurate and robust for multilevel colored satellite image segmentation problem. On the other hand, cuckoo search was found to be most promising for colored satellite image segmentation.     

Convergence of concurrent Markov chain Monte-Carlo algorithms

We examine the parallel execution of a class of stochastic algorithms called Markov chain Monte-Carlo (MCMC) algorithms. We focus on MCMC algorithms in the context of image processing, using Markov random field models. Our parallelisation approach is based on several, concurrently running, instances of the same stochastic algorithm that deal with the whole data set. Firstly we show that the speed-up of the parallel algorithm is limited because of the statistical properties of the MCMC algorithm. We examine coupled MCMC as a remedy for this problem. Secondly, we exploit the parallel execution to monitor the convergence of the stochastic algorithms in a statistically reliable manner. This new convergence measure for MCMC algorithms performs well, and is an improvement on known convergence measures. We also link our findings with recent work in the statistical theory of MCMC.     

Comparison of Methods for Identifying Phenotype Subgroups Using Categorical Features Data With Application to Autism Spectrum Disorder

We evaluate the performance of the Dirichlet process mixture (DPM) and the latent class model (LCM) in identifying autism phenotype subgroups based on categorical autism spectrum disorder (ASD) diagnostic features from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition Text Revision. A simulation study is designed to mimic the diagnostic features in the ASD dataset in order to evaluate the LCM and DPM methods in this context. Likelihood based information criteria and DPM partitioning are used to identify the best fitting models. The Rand statistic is used to compare the performance of the methods in recovering simulated phenotype subgroups. Our results indicate excellent recovery of the simulated subgroup structure for both methods. The LCM performs slightly better than DPM when the correct number of latent subgroups is selected a priori. The DPM method utilizes a maximum a posteriori (MAP) criterion to estimate the number of classes, and yielded results in fair agreement with the LCM method. Comparison of model fit indices in identifying the best fitting LCM showed that adjusted Bayesian information criteria (ABIC) picks the correct number of classes over 90% of the time. Thus, when diagnostic features are categorical and there is some prior information regarding the number of latent classes, LCM in conjunction with ABIC is preferred.     

Data augmentation and parameter expansion for independent or spatially correlated ordinal data

Data augmentation and parameter expansion can lead to improved iterative sampling algorithms for Markov chain Monte Carlo (MCMC). Data augmentation allows for simpler and more feasible simulation from a posterior distribution. Parameter expansion accelerates convergence of iterative sampling algorithms by increasing the parameter space. Data augmentation and parameter-expanded data augmentation MCMC algorithms are proposed for fitting probit models for independent ordinal response data. The algorithms are extended for fitting probit linear mixed models for spatially correlated ordinal data. The effectiveness of data augmentation and parameter-expanded data augmentation is illustrated using the probit model and ordinal response data, however, the approach can be used broadly across model and data types.     

Parallel Implementation of a Class of Adaptive Signal Processing Applications

Recently, High Performance Computing (HPC) platforms have been employed to realize many computationally demanding applications in signal and image processing. These applications require real-time performance constraints to be met. These constraints include latency as well as throughput. In order to meet these performance requirements, efficient parallel algorithms are needed. These algorithms must be engineered to exploit the computational characteristics of such applications. In this paper we present a methodology for mapping a class of adaptive signal processing applications onto HPC platforms such that the throughput performance is optimized. We first define a new task model using the salient computational characteristics of a class of adaptive signal processing applications. Based on this task model, we propose a new execution model. In the earlier linear pipelined execution model, the task mapping choices were restricted. The new model permits flexible task mapping choices, leading to improved throughput performance compared with the previous model. Using the new model, a three-step task mapping methodology is developed. It consists of (1) a data remapping step, (2) a coarse resource allocation step, and (3) a fine performance tuning step. The methodology is demonstrated by designing parallel algorithms for modern radar and sonar signal processing applications. These are implemented on IBM SP2 and Cray T3E, state-of-the-art HPC platforms, to show the effectiveness of our approach. Experimental results show significant performance improvement over those obtained by previous approaches. Our code is written using C and the Message Passing Interface (MPI). Thus, it is portable across various HPC platforms. Received April 8, 1998; revised February 2, 1999.     

Efficient Markov chain Monte Carlo methods for decoding neural spike trains

Stimulus reconstruction or decoding methods provide an important tool for understanding how sensory and motor information is represented in neural activity. We discuss Bayesian decoding methods based on an encoding generalized linear model (GLM) that accurately describes how stimuli are transformed into the spike trains of a group of neurons. The form of the GLM likelihood ensures that the posterior distribution over the stimuli that caused an observed set of spike trains is log concave so long as the prior is. This allows the maximum a posteriori (MAP) stimulus estimate to be obtained using efficient optimization algorithms. Unfortunately, the MAP estimate can have a relatively large average error when the posterior is highly nongaussian. Here we compare several Markov chain Monte Carlo (MCMC) algorithms that allow for the calculation of general Bayesian estimators involving posterior expectations (conditional on model parameters). An efficient version of the hybrid Monte Carlo (HMC) algorithm was significantly superior to other MCMC methods for gaussian priors. When the prior distribution has sharp edges and corners, on the other hand, the "hit-and-run" algorithm performed better than other MCMC methods. Using these algorithms, we show that for this latter class of priors, the posterior mean estimate can have a considerably lower average error than MAP, whereas for gaussian priors, the two estimators have roughly equal efficiency. We also address the application of MCMC methods for extracting nonmarginal properties of the posterior distribution. For example, by using MCMC to calculate the mutual information between the stimulus and response, we verify the validity of a computationally efficient Laplace approximation to this quantity for gaussian priors in a wide range of model parameters; this makes direct model-based computation of the mutual information tractable even in the case of large observed neural populations, where methods based on binning the spike train fail. Finally, we consider the effect of uncertainty in the GLM parameters on the posterior estimators.     

A new R package for Bayesian estimation of multivariate normal mixtures allowing for selection of the number of components and interval-censored data

An R package mixAK is introduced which implements routines for a semiparametric density estimation through normal mixtures using the Markov chain Monte Carlo (MCMC) methodology. Besides producing the MCMC output, the package computes posterior summary statistics for important characteristics of the fitted distribution or computes and visualizes the posterior predictive density. For the estimated models, penalized expected deviance (PED) and deviance information criterion (DIC) is directly computed which allows for a selection of mixture components. Additionally, multivariate right-, left- and interval-censored observations are allowed. For univariate problems, the reversible jump MCMC algorithm has been implemented and can be used for a joint estimation of the mixture parameters and the number of mixture components. The core MCMC routines have been implemented in C++ and linked to R to ensure a reasonable computational speed. We briefly review the implemented algorithms and illustrate the use of the package on three real examples of different complexity.     

Two fully-unsupervised methods for MR brain image segmentation using SOM-based strategies

Image segmentation consists in partitioning an image into different regions. MRI image segmentation is especially interesting, since an accurate segmentation of the different brain tissues provides a way to identify many brain disorders such as dementia, schizophrenia or even the Alzheimer's disease. A large variety of image segmentation approaches have been implemented before. Nevertheless, most of them use a priori knowledge about the voxel classification, which prevents figuring out other tissue classes different from the classes the system was trained for. This paper presents two unsupervised approaches for brain image segmentation. The first one is based on the use of relevant information extracted from the whole volume histogram which is processed by using self-organizing maps (SOM). This approach is faster and computationally more efficient than previously reported methods. The second method proposed consists of four stages including MRI brain image acquisition, first and second order feature extraction using overlapping windows, evolutionary computing-based feature selection and finally, map units are grouped by means of a novel SOM clustering algorithm. While the first method is a fast procedure for the segmentation of the whole volume and provides a way to model tissue classes, the second approach is a more robust scheme under noisy or bad intensity normalization conditions that provides better results using high resolution images, outperforming the results provided by other algorithms in the state-of-the-art, in terms of the average overlap metric. The proposed algorithms have been successfully evaluated using the IBSR and IBSR 2.0 databases, as well as high-resolution MR images from the Nuclear Medicine Department of the “Virgen de las Nieves” Hospital, Granada, Spain (VNH), providing in any case good segmentation results.     

Semiparametric regression with shape-constrained penalized splines

In semiparametric regression models, penalized splines can be used to describe complex, non-linear relationships between the mean response and covariates. In some applications it is desirable to restrict the shape of the splines so as to enforce properties such as monotonicity or convexity on regression functions. We describe a method for imposing such shape constraints on penalized splines within a linear mixed model framework. We employ Markov chain Monte Carlo (MCMC) methods for model fitting, using a truncated prior distribution to impose the requisite shape restrictions. We develop a computationally efficient MCMC sampler by using a correspondingly truncated multivariate normal proposal distribution, which is a restricted version of the approximate sampling distribution of the model parameters in an unconstrained version of the model. We also describe a cheap approximation to this methodology that can be applied for shape-constrained scatterplot smoothing. Our methods are illustrated through two applications, the first involving the length of dugongs and the second concerned with growth curves for sitka spruce trees.     

Mixture model selection via hierarchical BIC

The Bayesian information criterion (BIC) is one of the most popular criteria for model selection in finite mixture models. However, it implausibly penalizes the complexity of each component using the whole sample size and completely ignores the clustered structure inherent in the data, resulting in over-penalization. To overcome this problem, a novel criterion called hierarchical BIC (HBIC) is proposed which penalizes the component complexity only using its local sample size and matches the clustered data structure well. Theoretically, HBIC is an approximation of the variational Bayesian (VB) lower bound when sample size is large and the widely used BIC is a less accurate approximation. An empirical study is conducted to verify this theoretical result and a series of experiments is performed on simulated and real data sets to compare HBIC and BIC. The results show that HBIC outperforms BIC substantially and BIC suffers from underestimation.     

Bayesian estimation of unrestricted and order-restricted association models for a two-way contingency table

In two-way contingency tables analysis, a popular class of models for describing the structure of the association between the two categorical variables are the so-called “association” models. Such models assign scores to the classification variables which can be either fixed and prespecified or unknown parameters to be estimated. Under the row-column (RC) association model, both row and column scores are unknown parameters without any restriction concerning their ordinality. It is natural to impose order restrictions on the scores when the classification variables are ordinal. The Bayesian approach for the RC (unrestricted and restricted) model is adopted. MCMC methods are facilitated in order the parameters to be estimated. Furthermore, an alternative parametrization of the association models is proposed. This new parametrization simplifies computation in the MCMC procedure and leads to a natural parameter space for the order constrained model. The proposed methodology is illustrated via a popular dataset.     

A Bayesian analysis of spherical pattern based on finite Langevin mixture

Parameter estimation is a cornerstone of most fundamental problems of statistical research and practice. In particular, finite mixture models have long been heavily relied on deterministic approaches such as expectation maximization (EM). Despite their successful utilization in wide spectrum of areas, they have inclined to converge to local solutions. An alternative approach is the adoption of Bayesian inference that naturally addresses data uncertainty while ensuring good generalization. To this end, in this paper we propose a fully Bayesian approach for Langevin mixture model estimation and selection via MCMC algorithm based on Gibbs sampler, Metropolis–Hastings and Bayes factors. We demonstrate the effectiveness and the merits of the proposed learning framework through synthetic data and challenging applications involving topic detection and tracking and image categorization.     

Segmentation of fuzzy images: a novel and fast two-step pseudo MAP method

This paper presents a new two-step pseudo maximum a posteriori (MAP) segmentation method for the Markov random field (MRF)-modeled image because the exact MAP estimation is hard to implement due to intractable complexity. The expectation maximization (EM) and Markov Chain Monte Carlo (MCMC) methods are adopted to estimate the parameters for the MRF model due to their comparatively good performance. Although the image segmentation algorithms via graph cuts have become very popular nowadays, our proposed algorithm still performs significantly better in automatic identification and segmentation of fuzzy images than them, which is shown by the quantitative results on synthesized images. In practical applications, the proposed two-step pseudo MAP method is superior in segmenting the fuzzy laser images reflected from the weld pool surfaces during the P-GMAW welding process.     

A novel view of the variational Bayesian clustering

We prove that the evaluation function of variational Bayesian (VB) clustering algorithms can be described as the log likelihood of given data minus the Kullback–Leibler (KL) divergence between the prior and the posterior of model parameters. In this novel formalism of VB, the evaluation functions can be explicitly interpreted as information criteria for model selection and the KL divergence imposes a heavy penalty on the posterior far from the prior. We derive the update process of the variational Bayesian clustering with finite mixture Student's t-distribution, taking the penalty term for the degree of freedoms into account.     

Model Selection for Unsupervised Learning of Visual Context

This study addresses the problem of choosing the most suitable probabilistic model selection criterion for unsupervised learning of visual context of a dynamic scene using mixture models. A rectified Bayesian Information Criterion (BICr) and a Completed Likelihood Akaike’s Information Criterion (CL-AIC) are formulated to estimate the optimal model order (complexity) for a given visual scene. Both criteria are designed to overcome poor model selection by existing popular criteria when the data sample size varies from small to large and the true mixture distribution kernel functions differ from the assumed ones. Extensive experiments on learning visual context for dynamic scene modelling are carried out to demonstrate the effectiveness of BICr and CL-AIC, compared to that of existing popular model selection criteria including BIC, AIC and Integrated Completed Likelihood (ICL). Our study suggests that for learning visual context using a mixture model, BICr is the most appropriate criterion given sparse data, while CL-AIC should be chosen given moderate or large data sample sizes.     

Morphable model space based face super-resolution reconstruction and recognition

Super-resolution image reconstruction is the process of producing a high-resolution image from a set of low-resolution images of the same scene. For the applications of performing face evaluation and/or recognition from low-resolution video surveillance, in the past, super-resolution image reconstruction was mainly used as a separate preprocessing step to obtain a high-resolution image in the pixel domain that is later passed to a face feature extraction and recognition algorithm. Such three-stage approach suffers a high degree of computational complexity. A low-dimensional morphable model space based face super-resolution reconstruction and recognition algorithm is proposed in this paper. The approach tries to construct the high-resolution information both required by reconstruction and recognition directly in the low dimensional feature space. We show that comparing with generic pixel domain algorithms, the proposed approach is more robust and more computationally efficient.