Observational Learning Algorithm for an Ensemble of Neural Networks

We propose Observational Learning Algorithm (OLA), an ensemble learning algorithm with T and O steps alternating. In the T-step, an ensemble of networks is trained with a training data set. In the O-step, ‘virtual’ data are generated in which each target pattern is determined by observing the member networks’ output for the input pattern. These virtual data are added to the training data and the two steps are repeatedly executed. The virtual data was found to play the role of a regularisation term as well as that of temporary hints having the auxiliary information regarding the target function extracted from the ensemble. From numerical experiments involving both regression and classification problems, the OLA was shown to provide better generalisation performance than simple committee, boosting and bagging approaches, when insufficient and noisy training data are given. We examined the characteristics of the OLA in terms of ensemble diversity and robustness to noise variance. The OLA was found to balance between ensemble diversity and the average error of individual networks, and to be robust to the variance of noise distribution. Also, OLA was applied to five real world problems from the UCI repository, and its performance was compared with bagging and boosting methods. Received: 15 November 2000, Received in revised form: 07 November 2001, Accepted: 13 November 2001     

Structured large margin machines: sensitive to data distributions

This paper proposes a new large margin classifier—the structured large margin machine (SLMM)—that is sensitive to the structure of the data distribution. The SLMM approach incorporates the merits of “structured” learning models, such as radial basis function networks and Gaussian mixture models, with the advantages of “unstructured” large margin learning schemes, such as support vector machines and maxi-min margin machines. We derive the SLMM model from the concepts of “structured degree” and “homospace”, based on an analysis of existing structured and unstructured learning models. Then, by using Ward’s agglomerative hierarchical clustering on input data (or data mappings in the kernel space) to extract the underlying data structure, we formulate SLMM training as a sequential second order cone programming. Many promising features of the SLMM approach are illustrated, including its accuracy, scalability, extensibility, and noise tolerance. We also demonstrate the theoretical importance of the SLMM model by showing that it generalizes existing approaches, such as SVMs and M⁴s, provides novel insight into learning models, and lays a foundation for conceiving other “structured” classifiers. Editor: Dale Schuurmans. This work was supported by the Hong Kong Research Grant Council under Grants G-T891 and B-Q519.     

Function in Device Representation

We explore the meanings of the terms ‘structure’, ‘behaviour’, and, especially, ‘function’ in engineering practice. Computers provide great assistance in calculation tasks in engineering practice, but they also have great potential for helping with reasoning tasks. However, realising this vision requires precision in representing engineering knowledge, in which the terms mentioned above play a central role. We start with a simple ontology for representing objects and causal interactions between objects. Using this ontology, we investigate a range of meanings for the terms of interest. Specifically, we distinguish between function as effect on the environment, and a device-centred view of device function. In the former view, function is seen as an intended or desired role that an artifact plays in its environment. We identify an important concept called mode of deployment that is often left implicit, but whose explicit representation is necessary for correct and complete reasoning. We discuss the task of design and design verification in this framework. We end with a discussion that relates needs in the world to functions of artifacts created to satisfy the needs.     

Roll-to-roll hot embossing of microstructures

In this paper we present a new roll-to-roll embossing process allowing the replication of micro patterns with feature sizes down to 0.5 μm. The embossing process can be run in ‘continuous mode’ as well as in ‘discontinuous mode’. Continuous hot embossing is suitable for the continuous output of micro patterned structures. Discontinuous hot embossing has the advantage that it is not accompanied by waste produced during the initial hot embossing phase. This is because in ‘discontinuous mode’, embossing does not start before the foil has reached the target temperature. The foil rests between two parallel heating plates and foil movement and embossing starts only after the part of the foil resting between the heating plates has reached a thermal steady state. A new type of embossing master is used which is based on flexible silicon substrates. The embossing pattern with sub-μm topographic resolution is prepared on silicon wafers by state of the art lithography and dry etching techniques. The wafers are thinned down to a thickness of 40 μm, which guarantees the mechanical flexibility of the embossing masters. Up to 20 individual chips with a size of 20 × 20 mm2 were assembled on a roller. Embossing experiments with COC foils showed a good replication of the silicon master structures in the foil. The maximum depth of the embossed holes was about 70% of the master height.     

Dynamic weighting ensemble classifiers based on cross-validation

Ensemble of classifiers constitutes one of the main current directions in machine learning and data mining. It is accepted that the ensemble methods can be divided into static and dynamic ones. Dynamic ensemble methods explore the use of different classifiers for different samples and therefore may get better generalization ability than static ensemble methods. However, for most of dynamic approaches based on KNN rule, additional part of training samples should be taken out for estimating “local classification performance” of each base classifier. When the number of training samples is not sufficient enough, it would lead to the lower accuracy of the training model and the unreliableness for estimating local performances of base classifiers, so further hurt the integrated performance. This paper presents a new dynamic ensemble model that introduces cross-validation technique in the process of local performances’ evaluation and then dynamically assigns a weight to each component classifier. Experimental results with 10 UCI data sets demonstrate that when the size of training set is not large enough, the proposed method can achieve better performances compared with some dynamic ensemble methods as well as some classical static ensemble approaches.     

On Optimal Bilinear Quadrilateral Meshes

The novelty of this work is in presenting interesting error properties of two types of asymptotically ‘optimal’ quadrilateral meshes for bilinear approximation. The first type of mesh has an error equidistributing property, where the maximum interpolation error is asymptotically the same over all elements. The second type has faster than expected ‘super-convergence’ property for certain saddle-shaped data functions. The ‘super-convergent’ mesh may be an order of magnitude more accurate than the error equidistributing mesh. Both types of mesh are generated by a coordinate transformation of a regular mesh of squares. The coordinate transformation is derived by interpreting the Hessian matrix of a data function as a metric tensor. The insights in this work may have application in mesh design near known corner or point singularities.     

Efficient distribution-free population learning of simple concepts

We consider a variant of the ‘population learning model’ proposed by Kearns and Seung [8], in which the learner is required to be ‘distribution-free’ as well as computationally efficient. A population learner receives as input hypotheses from a large population of agents and produces as output its final hypothesis. Each agent is assumed to independently obtain labeled sample for the target concept and output a hypothesis. A polynomial time population learner is said to PAC-learn a concept class, if its hypothesis is probably approximately correct whenever the population size exceeds a certain bound which is polynomial, even if the sample size for each agent is fixed at some constant. We exhibit some general population learning strategies, and some simple concept classes that can be learned by them. These strategies include the ‘supremum hypothesis finder’, the ‘minimum superset finder’ (a special case of the ‘supremum hypothesis finder’), and various voting schemes. When coupled with appropriate agent algorithms, these strategies can learn a variety of simple concept classes, such as the ‘high–low game’, conjunctions, axis-parallel rectangles and others. We give upper bounds on the required population size for each of these cases, and show that these systems can be used to obtain a speed up from the ordinary PAC-learning model [11], with appropriate choices of sample and population sizes. With the population learner restricted to be a voting scheme, what we have is effectively a model of ‘population prediction’, in which the learner is to predict the value of the target concept at an arbitrarily drawn point, as a threshold function of the predictions made by its agents on the same point. We show that the population learning model is strictly more powerful than the population prediction model. Finally, we consider a variant of this model with classification noise, and exhibit a population learner for the class of conjunctions in this model. This revised version was published online in June 2006 with corrections to the Cover Date.     

Pool-based active learning in approximate linear regression

The goal of pool-based active learning is to choose the best input points to gather output values from a ‘pool’ of input samples. We develop two pool-based active learning criteria for linear regression. The first criterion allows us to obtain a closed-form solution so it is computationally very efficient. However, this solution is not necessarily optimal in the single-trial generalization error analysis. The second criterion can give a better solution, but it does not have a closed-form solution and therefore some additional search strategy is needed. To cope with this problem, we propose a practical procedure which enables us to efficiently search for a better solution around the optimal solution of the first method. Simulations with toy and benchmark datasets show that the proposed active learning method compares favorably with other active learning methods as well as the baseline passive learning scheme. Furthermore, the usefulness of the proposed active learning method is also demonstrated in wafer alignment in semiconductor exposure apparatus.     

Quantifying counts and costs via classification

Many business applications track changes over time, for example, measuring the monthly prevalence of influenza incidents. In situations where a classifier is needed to identify the relevant incidents, imperfect classification accuracy can cause substantial bias in estimating class prevalence. The paper defines two research challenges for machine learning. The ‘quantification’ task is to accurately estimate the number of positive cases (or class distribution) in a test set, using a training set that may have a substantially different distribution. The ‘cost quantification’ variant estimates the total cost associated with the positive class, where each case is tagged with a cost attribute, such as the expense to resolve the case. Quantification has a very different utility model from traditional classification research. For both forms of quantification, the paper describes a variety of methods and evaluates them with a suitable methodology, revealing which methods give reliable estimates when training data is scarce, the testing class distribution differs widely from training, and the positive class is rare, e.g., 1% positives. These strengths can make quantification practical for business use, even where classification accuracy is poor.     

Large-scale network intrusion detection based on distributed learning algorithm

As network traffic bandwidth is increasing at an exponential rate, it’s impossible to keep up with the speed of networks by just increasing the speed of processors. Besides, increasingly complex intrusion detection methods only add further to the pressure on network intrusion detection (NIDS) platforms, so the continuous increasing speed and throughput of network poses new challenges to NIDS. To make NIDS usable in Gigabit Ethernet, the ideal policy is using a load balancer to split the traffic data and forward those to different detection sensors, which can analyze the splitting data in parallel. In order to make each slice contains all the evidence necessary to detect a specific attack, the load balancer design must be complicated and it becomes a new bottleneck of NIDS. To simplify the load balancer this paper put forward a distributed neural network learning algorithm (DNNL). Using DNNL a large data set can be split randomly and each slice of data is presented to an independent neural network; these networks can be trained in distribution and each one in parallel. Completeness analysis shows that DNNL’s learning algorithm is equivalent to training by one neural network which uses the technique of regularization. The experiments to check the completeness and efficiency of DNNL are performed on the KDD’99 Data Set which is a standard intrusion detection benchmark. Compared with other approaches on the same benchmark, DNNL achieves a high detection rate and low false alarm rate.