Feature selection in single and ensemble learning‐based bankruptcy prediction models

Feature selection is an important data preprocessing step for the construction of an effective bankruptcy prediction model. The prediction performance can be affected by the employed feature selection and classification techniques. However, there have been very few studies of bankruptcy prediction that identify the best combination of feature selection and classification techniques. In this study, two types of feature selection methods, including filter‐ and wrapper‐based methods, are considered, and two types of classification techniques, including statistical and machine learning techniques, are employed in the development of the prediction methods. In addition, bagging and boosting ensemble classifiers are also constructed for comparison. The experimental results based on three related datasets that contain different numbers of input features show that the genetic algorithm as the wrapper‐based feature selection method performs better than the filter‐based one by information gain. It is also shown that the lowest prediction error rates for the three datasets are provided by combining the genetic algorithm with the naïve Bayes and support vector machine classifiers without bagging and boosting.     

Testing Different Ensemble Configurations for Feature Selection

In recent years, ensemble learning has become a prolific area of study in pattern recognition, based on the assumption that using and combining different learning models in the same problem could lead to better performance results than using a single model. This idea of ensemble learning has traditionally been used for classification tasks, but has more recently been adapted to other machine learning tasks such as clustering and feature selection. We propose several feature selection ensemble configurations based on combining rankings of features from individual rankers according to the combination method and threshold value used. The performance of each proposed ensemble configuration was tested for synthetic datasets (to assess the adequacy of the selection), real classical datasets (with more samples than features), and DNA microarray datasets (with more features than samples). Five different classifiers were studied in order to test the suitability of the proposed ensemble configurations and assess the results.     

Feature mining and pattern classification for steganalysis of LSB matching steganography in grayscale images

In this paper, we present a scheme based on feature mining and pattern classification to detect LSB matching steganography in grayscale images, which is a very challenging problem in steganalysis. Five types of features are proposed. In comparison with other well-known feature sets, the set of proposed features performs the best. We compare different learning classifiers and deal with the issue of feature selection that is rarely mentioned in steganalysis. In our experiments, the combination of a dynamic evolving neural fuzzy inference system (DENFIS) with a feature selection of support vector machine recursive feature elimination (SVMRFE) achieves the best detection performance. Results also show that image complexity is an important reference to evaluation of steganalysis performance.     

A Bolasso based consistent feature selection enabled random forest classification algorithm: An application to credit risk assessment

Credit risk assessment has been a crucial issue as it forecasts whether an individual will default on loan or not. Classifying an applicant as good or bad debtor helps lender to make a wise decision. The modern data mining and machine learning techniques have been found to be very useful and accurate in credit risk predictive capability and correct decision making. Classification is one of the most widely used techniques in machine learning. To increase prediction accuracy of standalone classifiers while keeping overall cost to a minimum, feature selection techniques have been utilized, as feature selection removes redundant and irrelevant attributes from dataset. This paper initially introduces Bolasso (Bootstrap-Lasso) which selects consistent and relevant features from pool of features. The consistent feature selection is defined as robustness of selected features with respect to changes in dataset Bolasso generated shortlisted features are then applied to various classification algorithms like Random Forest (RF), Support Vector Machine (SVM), Naïve Bayes (NB) and K-Nearest Neighbors (K-NN) to test its predictive accuracy. It is observed that Bolasso enabled Random Forest algorithm (BS-RF) provides best results forcredit risk evaluation. The classifiers are built on training and test data partition (70:30) of three datasets (Lending Club’s peer to peer dataset, Kaggle’s Bank loan status dataset and German credit dataset obtained from UCI). The performance of Bolasso enabled various classification algorithms is then compared with that of other baseline feature selection methods like Chi Square, Gain Ratio, ReliefF and stand-alone classifiers (no feature selection method applied). The experimental results shows that Bolasso provides phenomenal stability of features when compared with stability of other algorithms. Jaccard Stability Measure (JSM) is used to assess stability of feature selection methods. Moreover BS-RF have good classification accuracy and is better than other methods in terms of AUC and Accuracy resulting in effectively improving the decision making process of lenders.     

A machine learning approach for feature selection traffic classification using security analysis

Class imbalance has become a big problem that leads to inaccurate traffic classification. Accurate traffic classification of traffic flows helps us in security monitoring, IP management, intrusion detection, etc. To address the traffic classification problem, in literature, machine learning (ML) approaches are widely used. Therefore, in this paper, we also proposed an ML-based hybrid feature selection algorithm named WMI_AUC that make use of two metrics: weighted mutual information (WMI) metric and area under ROC curve (AUC). These metrics select effective features from a traffic flow. However, in order to select robust features from the selected features, we proposed robust features selection algorithm. The proposed approach increases the accuracy of ML classifiers and helps in detecting malicious traffic. We evaluate our work using 11 well-known ML classifiers on the different network environment traces datasets. Experimental results showed that our algorithms achieve more than 95% flow accuracy results.     

A feature selection enabled hybrid‐bagging algorithm for credit risk evaluation

Hybrid models based on feature selection and machine learning techniques have significantly enhanced the accuracy of standalone models. This paper presents a feature selection‐based hybrid‐bagging algorithm (FS‐HB) for improved credit risk evaluation. The 2 feature selection methods chi‐square and principal component analysis were used for ranking and selecting the important features from the datasets. The classifiers were built on 5 training and test data partitions of the input data set. The performance of the hybrid algorithm was compared with that of the standalone classifiers: feature selection‐based classifiers and bagging. The hybrid FS‐HB algorithm performed best for qualitative dataset with less features and tree‐based unstable base classifier. Its performance on numeric data was also better than other standalone classifiers, whereas comparable to bagging with only selected features. Its performance was found better on 70:30 data partition and the type II error, which is very significant in risk evaluation was also reduced significantly. The improved performance of FS‐HB is attributed to the important features used for developing the classifier thereby reducing the complexity of the algorithm and the use of ensemble methodology, which added to the classical bias variance trade‐off and performed better than standalone classifiers.     

Dementia classification using MR imaging and clinical data with voting based machine learning models

Dementia is one of the leading causes of severe cognitive decline, it induces memory loss and impairs the daily life of millions of people worldwide. In this work, we consider the classification of dementia using magnetic resonance (MR) imaging and clinical data with machine learning models. We adapt univariate feature selection in the MR data pre-processing step as a filter-based feature selection. Bagged decision trees are also implemented to estimate the important features for achieving good classification accuracy. Several ensemble learning-based machine learning approaches, namely gradient boosting (GB), extreme gradient boost (XGB), voting-based, and random forest (RF) classifiers, are considered for the diagnosis of dementia. Moreover, we propose voting-based classifiers that train on an ensemble of numerous basic machine learning models, such as the extra trees classifier, RF, GB, and XGB. The implementation of a voting-based approach is one of the important contributions, and the performance of different classifiers are evaluated in terms of precision, accuracy, recall, and F1 score. Moreover, the receiver operating characteristic curve (ROC) and area under the ROC curve (AUC) are used as metrics for comparing these classifiers. Experimental results show that the voting-based classifiers often perform better compared to the RF, GB, and XGB in terms of precision, recall, and accuracy, thereby indicating the promise of differentiating dementia from imaging and clinical data.

     

Online Feature Selection of Class Imbalance via PA Algorithm

Imbalance classification techniques have been frequently applied in many machine learning application domains where the number of the majority (or positive) class of a dataset is much larger than that of the minority (or negative) class. Meanwhile, feature selection (FS) is one of the key techniques for the high-dimensional classification task in a manner which greatly improves the classification performance and the computational efficiency. However, most studies of feature selection and imbalance classification are restricted to off-line batch learning, which is not well adapted to some practical scenarios. In this paper, we aim to solve high-dimensional imbalanced classification problem accurately and efficiently with only a small number of active features in an online fashion, and we propose two novel online learning algorithms for this purpose. In our approach, a classifier which involves only a small and fixed number of features is constructed to classify a sequence of imbalanced data received in an online manner. We formulate the construction of such online learner into an optimization problem and use an iterative approach to solve the problem based on the passive-aggressive (PA) algorithm as well as a truncated gradient (TG) method. We evaluate the performance of the proposed algorithms based on several real-world datasets, and our experimental results have demonstrated the effectiveness of the proposed algorithms in comparison with the baselines.     

基于随机子空间核极端学习机集成的高光谱遥感图像分类

结合随机子空间和核极端学习机集成提出了一种新的高光谱遥感图像分类方法。首先利用随机子空间方法从高光谱遥感图像数据的整体特征中随机生成多个大小相同的特征子集;然后利用核极端学习机在这些特征子集上进行训练从而获得基分类器;最后将所有基分类器的输出集成起来,通过投票机制得到分类结果。在高光谱遥感图像数据集上的实验结果表明:所提方法能够提高分类效果,且其分类总精度要高于核极端学习机和随机森林方法。     

Feature selection in the Laplacian support vector machine

Traditional classifiers including support vector machines use only labeled data in training. However, labeled instances are often difficult, costly, or time consuming to obtain while unlabeled instances are relatively easy to collect. The goal of semi-supervised learning is to improve the classification accuracy by using unlabeled data together with a few labeled data in training classifiers. Recently, the Laplacian support vector machine has been proposed as an extension of the support vector machine to semi-supervised learning. The Laplacian support vector machine has drawbacks in its interpretability as the support vector machine has. Also it performs poorly when there are many non-informative features in the training data because the final classifier is expressed as a linear combination of informative as well as non-informative features. We introduce a variant of the Laplacian support vector machine that is capable of feature selection based on functional analysis of variance decomposition. Through synthetic and benchmark data analysis, we illustrate that our method can be a useful tool in semi-supervised learning.     

Feature selection considering weighted relevancy

Feature selection plays an important role in pattern recognition and machine learning. Feature selection based on information theory intends to preserve the feature relevancy between features and class labels while eliminating irrelevant and redundant features. Previous feature selection methods have offered various explanations for feature relevancy, but they ignored the relationships between candidate feature relevancy and selected feature relevancy. To fill this gap, we propose a feature selection method named Feature Selection based on Weighted Relevancy (WRFS). In WRFS, we introduce two weight coefficients that use mutual information and joint mutual information to balance the importance between the two kinds of feature relevancy terms. To evaluate the classification performance of our method, WRFS is compared to three competing feature selection methods and three state-of-the-art methods by two different classifiers on 18 benchmark data sets. The experimental results indicate that WRFS outperforms the other baselines in terms of the classification accuracy, AUC and F1 score.     

Improving dynamic facial expression recognition with feature subset selection

This paper addresses the dynamic recognition of basic facial expressions in videos using feature subset selection. Feature selection has been already used by some static classifiers where the facial expression is recognized from one single image. Past work on dynamic facial expression recognition has emphasized the issues of feature extraction and classification, however, less attention has been given to the critical issue of feature selection in the dynamic scenario. The main contributions of the paper are as follows. First, we show that dynamic facial expression recognition can be casted into a classical classification problem. Second, we combine a facial dynamics extractor algorithm with a feature selection scheme for generic classifiers.We show that the paradigm of feature subset selection with a wrapper technique can improve the dynamic recognition of facial expressions. We provide evaluations of performance on real video sequences using five standard machine learning approaches: Support Vector Machines, K Nearest Neighbor, Naive Bayes, Bayesian Networks, and Classification Trees.     

Hybridized approach of image segmentation in classification of fruit mango using BPNN and discriminant analyzer

In machine learning, image classification accuracy generally depends on image segmentation and feature extraction methods with the extracted features and its qualities. The main focus of this paper is to determine the defected area of mangoes using image segmentation algorithm for improving the classification accuracy. The Enhanced Fuzzy based K-means clustering algorithm is designed for increasing the efficiency of segmentation. Proposed segmentation method is compared with K-means and Fuzzy C-means clustering methods. The geometric, texture and colour based features are used in the feature extraction. Process of feature selection is done by Maximally Correlated Principal Component Analysis (MCPCA). Finally, in the classification step, severe portions of the affected area are analyzed by Backpropagation Based Discriminant Classifier (BBDC). Proposed classifier is compared with BPNN and Naive Bayes classifiers. The images are classified into three classes in final output like Class A –good quality mango, Class B-average quality mango, and Class C-poor quality mango. Finally, the evaluated results of the proposed model examine various defected and healthy mango images and prove that the proposed method has the highest accuracy when compared with existing methods.

     

Comparing techniques for authorship attribution of source code

Attributing authorship of documents with unknown creators has been studied extensively for natural language text such as essays and literature, but less so for non‐natural languages such as computer source code. Previous attempts at attributing authorship of source code can be categorised by two attributes: the software features used for the classification, either strings of n tokens/bytes (n‐grams) or software metrics; and the classification technique that exploits those features, either information retrieval ranking or machine learning. The results of existing studies, however, are not directly comparable as all use different test beds and evaluation methodologies, making it difficult to assess which approach is superior. This paper summarises all previous techniques to source code authorship attribution, implements feature sets that are motivated by the literature, and applies information retrieval ranking methods or machine classifiers for each approach. Importantly, all approaches are tested on identical collections from varying programming languages and author types. Our conclusions are as follows: (i) ranking and machine classifier approaches are around 90% and 85% accurate, respectively, for a one‐in‐10 classification problem; (ii) the byte‐level n‐gram approach is best used with different parameters to those previously published; (iii) neural networks and support vector machines were found to be the most accurate machine classifiers of the eight evaluated; (iv) use of n‐gram features in combination with machine classifiers shows promise, but there are scalability problems that still must be overcome; and (v) approaches based on information retrieval techniques are currently more accurate than approaches based on machine learning. Copyright © 2012 John Wiley & Sons, Ltd.     

Robust automatic target recognition using learning classifier systems

This work developed and demonstrated a machine learning approach for robust ATR. The primary innovation of this work was the development of an automated way of developing inference rules that can draw on multiple models and multiple feature types to make robust ATR decisions. The key realization is that this “meta learning” problem is one of structural learning, and that it can be conducted independently of parameter learning associated with each model and feature based technique. This was accomplished by using a learning classifier system, which is based on genetics-based machine learning, for the ill conditioned combinatorial problem of structural rule learning, while using statistical and mathematical techniques for parameter learning.This system was tested on MSTAR Public Release SAR data using standard and extended operation conditions. These results were also compared against two baseline classifiers, a PCA based distance classifier and a MSE classifier. The classifiers were evaluated for accuracy (via training set classification) and robustness (via testing set classification). In both cases, the LCS based robust ATR system performed well with accuracy over 99% and robustness over 80%.     

Handwritten digit recognition: benchmarking of state-of-the-art techniques

This paper presents the results of handwritten digit recognition on well-known image databases using state-of-the-art feature extraction and classification techniques. The tested databases are CENPARMI, CEDAR, and MNIST. On the test data set of each database, 80 recognition accuracies are given by combining eight classifiers with ten feature vectors. The features include chaincode feature, gradient feature, profile structure feature, and peripheral direction contributivity. The gradient feature is extracted from either binary image or gray-scale image. The classifiers include the k-nearest neighbor classifier, three neural classifiers, a learning vector quantization classifier, a discriminative learning quadratic discriminant function (DLQDF) classifier, and two support vector classifiers (SVCs). All the classifiers and feature vectors give high recognition accuracies. Relatively, the chaincode feature and the gradient feature show advantage over other features, and the profile structure feature shows efficiency as a complementary feature. The SVC with RBF kernel (SVC-rbf) gives the highest accuracy in most cases but is extremely expensive in storage and computation. Among the non-SV classifiers, the polynomial classifier and DLQDF give the highest accuracies. The results of non-SV classifiers are competitive to the best ones previously reported on the same databases.     

Integrating radiologist feedback with computer aided diagnostic systems for breast cancer risk prediction in ultrasonic images: An experimental investigation in machine learning paradigm

With advancements in machine learning algorithms and computer aided diagnostic (CAD) systems, the performance of automated analysis of radiological images has improved substantially in recent times. However, the lack of integration between the radiologist and CAD systems restrains the rate of progress as well as the reach of such advancements in clinical use. This article aims to improve the clinical efficiency of ultrasound based CAD systems for classification of breast lesions by integrating back-propagation artificial neural network (BPANN), support vector machine (SVM) and radiologist feedback. The acquired breast ultrasound images were subjected to wavelet based filtering in order to reduce speckle noise followed by feature extraction, feature selection and classification. Experiments on a database of 178 ultrasound images of breast anomalies (88 benign and 90 malignant) show that the proposed methodology achieves classification accuracy of 98.621% and 98.276%, respectively, when all 457 and 19 most relevant features selected by multi-criteria feature selection method were used for classification. The accuracy achieved is significantly higher than that using conventional classifiers based on BPANN and SVM. Further, it is found that integrating expert opinion in CAD systems improves its overall performance. The quantitative results obtained are discussed in light of some recently reported studies.     

基于ELM和证据理论的纹理图像分类

目前纹理图像分类有不同的方法,但对纹理的描述还不够全面,而且当有新方法提取的特征加入时,系统的可扩展性也不够,通用性不好。本文针对上述问题提出了一种将D-S证据理论与极限学习机相结合的决策级融合模型,用来对纹理图像进行分类。采用三种不同方法来提取特征以获得更多更全面的纹理表现形式,并对提取的每种特征向量用极限学习机建立相应的分类器,最后用D-S证据理论在不确定性表示、度量和组合方面有着的优势来进行决策级融合。对于证据理论中基本概率赋值函数(BPAF)难以有效获取的问题,由于极限学习机具有学习速度快,泛化性能好的优点并且产生唯一的最优解的优点,所以利用其来构造其基本概率赋值函数。实验结果表明这种方法比单个分类器具有更高的识别正确率,降低了识别的不确定性。     

Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms

Improving accuracies of machine learning algorithms is vital in designing high performance computer-aided diagnosis (CADx) systems. Researches have shown that a base classifier performance might be enhanced by ensemble classification strategies. In this study, we construct rotation forest (RF) ensemble classifiers of 30 machine learning algorithms to evaluate their classification performances using Parkinson's, diabetes and heart diseases from literature.While making experiments, first the feature dimension of three datasets is reduced using correlation based feature selection (CFS) algorithm. Second, classification performances of 30 machine learning algorithms are calculated for three datasets. Third, 30 classifier ensembles are constructed based on RF algorithm to assess performances of respective classifiers with the same disease data. All the experiments are carried out with leave-one-out validation strategy and the performances of the 60 algorithms are evaluated using three metrics; classification accuracy (ACC), kappa error (KE) and area under the receiver operating characteristic (ROC) curve (AUC).Base classifiers succeeded 72.15%, 77.52% and 84.43% average accuracies for diabetes, heart and Parkinson's datasets, respectively. As for RF classifier ensembles, they produced average accuracies of 74.47%, 80.49% and 87.13% for respective diseases.RF, a newly proposed classifier ensemble algorithm, might be used to improve accuracy of miscellaneous machine learning algorithms to design advanced CADx systems.