Learning criteria for training neural network classifiers

This paper presents a study of two learning criteria and two approaches to using them for training neural network classifiers, specifically a Multi-Layer Perceptron (MLP) and Radial Basis Function (RBF) networks. The first approach, which is a traditional one, relies on the use of two popular learning criteria, i.e. learning via minimising a Mean Squared Error (MSE) function or a Cross Entropy (CE) function. It is shown that the two criteria have different charcteristics in learning speed and outlier effects, and that this approach does not necessarily result in a minimal classification error. To be suitable for classification tasks, in our second approach an empirical classification criterion is introduced for the testing process while using the MSE or CE function for the training. Experimental results on several benchmarks indicate that the second approach, compared with the first, leads to an improved generalisation performance, and that the use of the CE function, compared with the MSE function, gives a faster training speed and improved or equal generalisation performance.Nomenclature x random input vector withd real number components x ₁ ...x _d ] - t random target vector withc binary components t ₁ ...t _c ] - y(·) neural network function or output vector - parameters of a neural model - learning rate - momentum - decay factor - O objective function - E mean sum-of-squares error function - L cross entropy function - n nth training pattern - N number of training patterns - (·) transfer function in a neural unit - z _j output of hidden unit-j - a _i activation of unit-j - W _ij weight from hidden unit-j to output unit-i - W _jl ⁰ weight from input unit-l to hidden unit-j - _j centre vector _{j 1} ... _jd ] of RBF unit-j - _j width vector _{j 1}, ... _jd ] of RBF unit-j - p( ·¦·) conditional probability function