Pareto-Based Multiobjective Machine Learning: An Overview and Case Studies

Machine learning is inherently a multiobjective task. Traditionally, however, either only one of the objectives is adopted as the cost function or multiple objectives are aggregated to a scalar cost function. This can be mainly attributed to the fact that most conventional learning algorithms can only deal with a scalar cost function. Over the last decade, efforts on solving machine learning problems using the Pareto-based multiobjective optimization methodology have gained increasing impetus, particularly due to the great success of multiobjective optimization using evolutionary algorithms and other population-based stochastic search methods. It has been shown that Pareto-based multiobjective learning approaches are more powerful compared to learning algorithms with a scalar cost function in addressing various topics of machine learning, such as clustering, feature selection, improvement of generalization ability, knowledge extraction, and ensemble generation. One common benefit of the different multiobjective learning approaches is that a deeper insight into the learning problem can be gained by analyzing the Pareto front composed of multiple Pareto-optimal solutions. This paper provides an overview of the existing research on multiobjective machine learning, focusing on supervised learning. In addition, a number of case studies are provided to illustrate the major benefits of the Pareto-based approach to machine learning, e.g., how to identify interpretable models and models that can generalize on unseen data from the obtained Pareto-optimal solutions. Three approaches to Pareto-based multiobjective ensemble generation are compared and discussed in detail. Finally, potentially interesting topics in multiobjective machine learning are suggested.     

Cooperative spectrum sensing in cognitive radio network using multiobjective evolutionary algorithms and fuzzy decision making

The cognitive radio has emerged as a potential solution to the problem of spectrum scarcity. Spectrum sensing unit in cognitive radio deals with the reliable detection of primary user’s signal. Cooperative spectrum sensing exploits the spatial diversity between cognitive radios to improve sensing accuracy. The selection of the weight assigned to each cognitive radio and the global decision threshold can be formulated as a constrained multiobjective optimization problem where probabilities of false alarm and detection are the two conflicting objectives. This paper uses evolutionary algorithms to solve this optimization problem in a multiobjective framework. The simulation results offered by different algorithms are assessed and compared using three performance metrics. This study shows that our approach which is based on the concept of cat swarm optimization outperforms other algorithms in terms of quality of nondominating solutions and efficient computation. A fuzzy logic based strategy is used to find out a compromise solution from the set of nondominated solutions. Different tests are carried out to assess the stability of the simulation results offered by the heuristic evolutionary algorithms. Finally the sensitivity analysis of different parameters is performed to demonstrate their impact on the overall performance of the system.     

正交免疫克隆粒子群多目标优化算法

该文基于抗体克隆选择学说理论,提出了一种求解多目标优化问题的粒子群算法正交免疫克隆粒子群算法(Orthogonal Immune Clone Particle Swarm Optimization, OICPSO)。根据多目标的特点,提出了适合粒子群算法的克隆算子,免疫基因算子,克隆选择算子。免疫基因操作中采用了离散正交交叉算子来获得目标空间解的均匀采样,得到理想的Pareto解集,并引入拥挤距离来减少获得Pareto解集的大小,同时获得具有良好均匀性和宽广性的Pareto最优解集。实验中,与NSGA-II和MOPSO算法进行了比较,并对算法的性能指标进行了分析。结果表明,OICPSO不仅增加了种群解的多样性而且可以得到分布均匀的Pareto有效解集,对于多目标优化问题是有效地。     

Pareto Optimal Microwave Filter Design Using Multiobjective Differential Evolution

Microwave filters play an important role in modern wireless communications. A novel method for the design of multilayer dielectric and open loop ring resonator (OLRR) filters under constraints is presented. The proposed design method is based on generalized differential evolution (GDE3), which is a multiobjective extension of differential evolution (DE). GDE3 algorithm can be applied for global optimization to any engineering problem with an arbitrary number of objective and constraint functions. GDE3 is compared against other evolutionary multiobjective algorithms like nondominated sorting genetic algorithm-II (NSGA-II), multiobjective particle swarm optimization (MOPSO) and multiobjective particle swarm optimization with fitness sharing (MOPSO-fs) for a number of microwave filter design cases. In the multilayer dielectric filter design case a predefined database of low loss dielectric materials is used. The results indicate the advantages of this approach and the applicability of this design method.      

Novel chemical reaction optimization based unequal clustering and routing algorithms for wireless sensor networks

Energy conserving of sensor nodes is the most crucial issue in the design of wireless sensor networks (WSNs). In a cluster based routing approach, cluster heads (CHs) cooperate with each other to forward their data to the base station (BS) via multi-hop routing. In this process, CHs closer to the BS are burdened with heavier relay traffic and tend to die prematurely which causes network partition is popularly known as a hot spot problem. To mitigate the hot spot problem, in this paper, we propose unequal clustering and routing algorithms based on novel chemical reaction optimization (nCRO) paradigm, we jointly call these algorithms as novel CRO based unequal clustering and routing algorithms (nCRO-UCRA). In clustering, we partition the network into unequal clusters such that smaller size clusters near to the sink and larger size clusters relatively far away from the sink. For this purpose, we develop the CH selection algorithm based on nCRO paradigm and assign the non-cluster head sensor nodes to the CHs based on derived cost function. Then, a routing algorithm is presented which is also based on nCRO based approach. All these algorithms are developed with the efficient schemes of molecular structure encoding and novel potential energy functions. The nCRO-UCRA is simulated extensively on various scenarios of WSNs and varying number of sensors and the CHs. The results are compared with some existing algorithms and original CRO based algorithm called as CRO-UCRA to show the superiority in terms of various performance metrics like residual energy, network lifetime, number of alive nodes, data packets received by the BS and convergence rate.     

An alternative clustering approach for reconstructing cross cut shredded text documents

In this paper, we propose a clustering approach for solving the problem of reconstructing cross-cut shredded documents. This problem is important in the field of forensic science. Unlike other clustering approaches which are applied as a preprocessing step before the actual reconstruction algorithms, our clustering approach is part of the reconstruction process itself. We define a new cost function which mainly relies on black pixels to measure the cost of pairing two shreds together. The reconstruction algorithm creates multiple clusters which grow by adding additional shreds based on the cost function. Adding a shred may result in merging two or more clusters to produce a larger cluster. We, also, propose a way to involve the user in the reconstruction process. We compare our approach with a recent proposal and conclude that our approach gives better solutions in less time.     

Multiobjective gas turbine engine controller design using geneticalgorithms

This paper describes the use of multiobjective genetic algorithms (MOGAs) in the design of a multivariable control system for a gas turbine engine. The mechanisms employed to facilitate multiobjective search with the genetic algorithm are described with the aid of an example. It is shown that the MOGA confers a number of advantages over conventional multiobjective optimization methods by evolving a family of Pareto-optimal solutions rather than a single solution estimate. This allows the engineer to examine the trade-offs between the different design objectives and configurations during the course of an optimization. In addition, the paper demonstrates how the genetic algorithm can be used to search in both controller structure and parameter space thereby offering a potentially more general approach to optimization in controller design than traditional numerical methods. While the example in the paper deals with control system design, the approach described can be expected to be applicable to more general problems in the fields of computer aided design (CAD) and computer aided engineering (CAE)     

一种基于多样性信息和收敛度的多目标粒子群优化算法

为了提高多目标粒子群算法优化解的多样性和收敛性,提出了一种基于多样性信息和收敛度的多目标粒子群优化算法（Multiobjective Particle Swarm Optimization based on the Diversity Information and Convergence Degree,dicdMOPSO）.首先,利用非支配解多样性信息评估知识库中最优解的分布状态,设计出一种全局最优解选择机制,平衡了种群的进化过程,提高了非支配解的多样性和收敛性;其次,基于种群多样性信息设计出一种飞行参数调整机制,增强了粒子的全局探索能力和局部开发能力,获得了多样性和收敛性较好的种群.最后,将dicdMOPSO应用于标准测试函数测试,实验结果表明,dicdMOPSO与其他多目标算法相比不仅获得了多样性较高的可行解,而且能够较快的收敛到Pareto前沿.     

一种基于动态遗传算法的聚类新方法

 如何确定聚类数目一直是聚类分析中的难点问题.为此本文提出了一种基于动态遗传算法的聚类新方法,该方法采用最大属性值范围划分法克服划分聚类算法对初始值的敏感性,并运用两阶段的动态选择和变异策略,使选择概率和变异率跟随种群的聚类数目一致性变化,先进行不同聚类数目的并行搜索,再获取最优的聚类中心.七组数据聚类实验证明该方法能够实现数据集最佳划分的自动全局搜索,同时搜索到最佳聚类数目和最佳聚类中心.     

A Survey of Evolutionary Algorithms for Clustering

This paper presents a survey of evolutionary algorithms designed for clustering tasks. It tries to reflect the profile of this area by focusing more on those subjects that have been given more importance in the literature. In this context, most of the paper is devoted to partitional algorithms that look for hard clusterings of data, though overlapping (i.e., soft and fuzzy) approaches are also covered in the paper. The paper is original in what concerns two main aspects. First, it provides an up-to-date overview that is fully devoted to evolutionary algorithms for clustering, is not limited to any particular kind of evolutionary approach, and comprises advanced topics like multiobjective and ensemble-based evolutionary clustering. Second, it provides a taxonomy that highlights some very important aspects in the context of evolutionary data clustering, namely, fixed or variable number of clusters, cluster-oriented or nonoriented operators, context-sensitive or context-insensitive operators, guided or unguided operators, binary, integer, or real encodings, centroid-based, medoid-based, label-based, tree-based, or graph-based representations, among others. A number of references are provided that describe applications of evolutionary algorithms for clustering in different domains, such as image processing, computer security, and bioinformatics. The paper ends by addressing some important issues and open questions that can be subject of future research.      

Microarray Gene Cluster Identification and Annotation Through Cluster Ensemble and EM-Based Informative Textual Summarization

Generating high-quality gene clusters and identifying the underlying biological mechanism of the gene clusters are the important goals of clustering gene expression analysis. To get high-quality cluster results, most of the current approaches rely on choosing the best cluster algorithm, in which the design biases and assumptions meet the underlying distribution of the dataset. There are two issues for this approach: 1) usually, the underlying data distribution of the gene expression datasets is unknown and 2) there are so many clustering algorithms available and it is very challenging to choose the proper one. To provide a textual summary of the gene clusters, the most explored approach is the extractive approach that essentially builds upon techniques borrowed from the information retrieval, in which the objective is to provide terms to be used for query expansion, and not to act as a stand-alone summary for the entire document sets. Another drawback is that the clustering quality and cluster interpretation are treated as two isolated research problems and are studied separately. In this paper, we design and develop a unified system Gene Expression Miner to address these challenging issues in a principled and general manner by integrating cluster ensemble, text clustering, and multidocument summarization and provide an environment for comprehensive gene expression data analysis. We present a novel cluster ensemble approach to generate high-quality gene cluster. In our text summarization module, given a gene cluster, our expectation-maximization based algorithm can automatically identify subtopics and extract most probable terms for each topic. Then, the extracted top k topical terms from each subtopic are combined to form the biological explanation of each gene cluster. Experimental results demonstrate that our system can obtain high-quality clusters and provide informative key terms for the gene clusters.     

Designing resilient networks using multicriteria metaheuristics

The paper deals with the design of resilient networks that are fault tolerant against link failures. Usually, fault tolerance is achieved by providing backup paths, which are used in case of an edge failure on a primary path. We consider this task as a multiobjective optimization problem: to provide resilience in networks while minimizing the cost subject to capacity constraint. We propose a stochastic approach, which can generate multiple Pareto solutions in a single run. The feasibility of the proposed method is illustrated by considering several network design problems using a single weighted average of objectives and a direct multiobjective optimization approach using the Pareto dominance concept.     

态势觉察中目标分群技术的实现

对态势觉察中的目标分群技术进行了问题描述,指出了目标分群即是由群的递增建立和群的动态维护所组成的动态过程;对群的递增建立和动态维护进行了详细的讨论,分别提出了具体的算法,并给出了算法的实现描述。     

A Novel Approach for Discovering Overlapping Clusters in Gene Expression Data

Many existing clustering algorithms have been used to identify coexpressed genes in gene expression data. These algorithms are used mainly to partition data in the sense that each gene is allowed to belong only to one cluster. Since proteins typically interact with different groups of proteins in order to serve different biological roles, the genes that produce these proteins are therefore expected to coexpress with more than one group of genes. In other words, some genes are expected to belong to more than one cluster. This poses a challenge to gene expression data clustering as there is a need for overlapping clusters to be discovered in a noisy environment. For this task, we propose an effective information theoretical approach, which consists of an initial clustering phase and a second reclustering phase, in this paper. The proposed approach has been tested with both simulated and real expression data. Experimental results show that it can improve the performances of existing clustering algorithms and is able to effectively uncover interesting patterns in noisy gene expression data so that, based on these patterns, overlapping clusters can be discovered.     

Evolution-Based Tabu Search Approach to Automatic Clustering

Traditional clustering algorithms (e.g., the K-means algorithm and its variants) are used only for a fixed number of clusters. However, in many clustering applications, the actual number of clusters is unknown beforehand. The general solution to this type of a clustering problem is that one selects or defines a cluster validity index and performs a traditional clustering algorithm for all possible numbers of clusters in sequence to find the clustering with the best cluster validity. This is tedious and time-consuming work. To easily and effectively determine the optimal number of clusters and, at the same time, construct the clusters with good validity, we propose a framework of automatic clustering algorithms (called ETSAs) that do not require users to give each possible value of required parameters (including the number of clusters). ETSAs treat the number of clusters as a variable, and evolve it to an optimal number. Through experiments conducted on nine test data sets, we compared the ETSA with five traditional clustering algorithms. We demonstrate the superiority of the ETSA in finding the correct number of clusters while constructing clusters with good validity.     

数据流上基于K-median聚类的算法研究

文章研究和分析了数据流上的K-median聚类算法技术,包括:(1)流模型和K-median问题定义;(2)基于流的K-median聚类基本决策和内在机理;(3)理论上有性能保证的流算法。对于每一特征,这种技术能在没有实际保留任何数据流对象的情形下有效地确定聚类点。它通过一个聚类块的一分为二或相邻聚类块的合二为一来动态地生成聚类点,从而实现上述目标。作为结果,这种技术所确定的聚类点将比其他常规方法更准确。在数据流环境中,这种技术能够在产生高质量聚类结果的同时非常有效地执行。     

Optimization and FROC analysis of rule-based detection schemes using a multiobjective approach

Computerized detection schemes have the potential of increasing diagnostic accuracy in medical imaging by alerting radiologists to lesions that they initially overlooked. These schemes typically employ multiple parameters such as threshold values or filter weights to arrive at a detection decision. In order for the system to have high performance, the values of these parameters need to be set optimally. Conventional optimization techniques are designed to optimize a scalar objective function. The task of optimizing the performance of a computerized detection scheme, however, is clearly a multiobjective problem: we wish to simultaneously improve the sensitivity and false-positive rate of the system. In this work we investigate a multiobjective approach to optimizing computerized rule-based detection schemes. In a multiobjective optimization, multiple objectives are simultaneously optimized, with the objective now being a vector-valued function. The multiobjective optimization problem admits a set of solutions, known as the Pareto-optimal set, which are equivalent in the absence of any information regarding the preferences of the objectives. The performances of the Pareto-optimal solutions can be interpreted as operating points on an optimal free-response receiver operating characteristic (FROC) curve, greater than or equal to the points on any possible FROC curve for a given dataset and detection scheme. It is demonstrated that generating FROC curves in this manner eliminates several known problems with conventional FROC curve generation techniques for rule-based detection schemes. We employ the multiobjective approach to optimize a rule-based scheme for clustered microcalcification detection that has been developed in our laboratory.     

A novel weighted clustering algorithm in mobile ad hoc networks using discrete particle swarm optimization (DPSOWCA)

In this paper, a novel weighted clustering algorithm in mobile ad hoc networks using discrete particle swarm optimization (DPSOWCA) is proposed. The proposed algorithm shows how discrete particle swarm optimization can be useful in enhancing the performance of clustering algorithms in mobile ad hoc networks. Consequently, it results in the minimum number of clusters and hence minimum cluster heads. The goals of the algorithm are to minimize the number of cluster heads, to enhance network stability, to maximize network lifetime, and to achieve good end‐to‐end performance. Analysis and simulation of the algorithm have been implemented and the validity of the algorithm has been proved. Results show that the proposed algorithm performs better than the existing weight‐based clustering algorithm and adapts to different kinds of network conditions. Copyright © 2009 John Wiley & Sons, Ltd.     

Gaze prediction for first-person videos based on inverse non-negative sparse coding with determinant sparse measure

Gaze prediction is a significant approach for processing a large amount of incoming visual information of videos. Recent gaze prediction algorithms often employ sparse models with the assumption that every superpixel in the video frames can be represented as linear combinations of a few salient superpixels. However, they are not actuated enough because of the insufficient knowledge that video signals contain a non-negative request. Hence, we develop a novel gaze prediction based on an inverse sparse coding framework with a determinant sparse measure. By introducing this sparse measure, the solutions are non-negative and sparser than conventional sparse constraints. However, the proposed optimization problem becomes nonconvex, which is difficult to solve. To efficiently address the corresponding nonconvex optimization problem, we propose a novel algorithm based on the difference in convex function programming, which can yield the global solutions. Experimental results indicate the improved accuracy of the proposed approach compared with state-of-the-art algorithms.     

Multiobjective genetic optimization of diagnostic classifiers withimplications for generating receiver operating characteristiccurves

It is well understood that binary classifiers have two implicit objective functions (sensitivity and specificity) describing their performance. Traditional methods of classifier training attempt to combine these two objective functions (or two analogous class performance measures) into one so that conventional scalar optimization techniques can be utilized. This involves incorporating a priori information into the aggregation method so that the resulting performance of the classifier is satisfactory for the task at hand. We have investigated the use of a niched Pareto multiobjective genetic algorithm (GA) for classifier optimization. With niched Pareto GA's, an objective vector is optimized instead of a scalar function, eliminating the need to aggregate classification objective functions. The niched Pareto GA returns a set of optimal solutions that are equivalent in the absence of any information regarding the preferences of the objectives. The a priori knowledge that was used for aggregating the objective functions in conventional classifier training can instead be applied post-optimization to select from one of the series of solutions returned from the multiobjective genetic optimization. We have applied this technique to train a linear classifier and an artificial neural network (ANN), using simulated datasets. The performances of the solutions returned from the multiobjective genetic optimization represent a series of optimal (sensitivity, specificity) pairs, which can be thought of as operating points on a receiver operating characteristic (ROC) curve. All possible ROC curves for a given dataset and classifier are less than or equal to the ROC curve generated by the niched Pareto genetic optimization.