-*- mode: org -*-
Focus strict-ish-ly on MLP, SLFN, RBFNN, or ANN learning

*  42(<-589): Multiobjective optimization of ensembles of multilayer perceptrons for pattern classification

Pattern classification seeks to minimize error of unknown patterns,
however, in many real world applications, type I (false positive) and
type II (false negative) errors have to be dealt with separately,
which is a complex problem since an attempt to minimize one of them
usually makes the other grow. Actually, a type of error can be more
important than the other, and a trade-off that minimizes the most
important error type must be reached. Despite the importance of
type-II errors, most pattern classification methods take into account
only the global classification error. In this paper we propose to
optimize both error types in classification by means of a
multiobjective algorithm in which each error type and the network size
is an objective of the fitness function. A modified version of the
GProp method (optimization and design of multilayer perceptrons) is
used, to simultaneously optimize the network size and the type I and
II errors.

2006



* 165(<-511): Pareto-based multiobjective machine learning: An overview and case studies

Machine learning is inherently a multiobjective task. Traditionally,
however, either only one of the objectives is adopted as the cost
function or multiple objectives are aggregated to a scalar cost
function. This can be mainly attributed to the fact that most
conventional learning algorithms can only deal with a scalar cost
function. Over the last decade, efforts on solving machine learning
problems using the Pareto-based multiobjective optimization
methodology have gained increasing impetus, particularly due to the
great success of multiobjective optimization using evolutionary
algorithms and other population-based stochastic search methods. It
has been shown that Pareto-based multiobjective learning approaches
are more powerful compared to learning algorithms with a scalar cost
function in addressing various topics of machine learning, such as
clustering, feature selection, improvement of generalization ability,
knowledge extraction, and ensemble generation. One common benefit of
the different multiobjective learning approaches is that a deeper
insight into the learning problem can be gained by analyzing the
Pareto front composed of multiple Pareto-optimal solutions. This paper
provides an overview of the existing research on multiobjective
machine learning, focusing on supervised learning. In addition, a
number of case studies are provided to illustrate the major benefits
of the Pareto-based approach to machine learning, e.g., how to
identify interpretable models and models that can generalize on unseen
data from the obtained Pareto-optimal solutions. Three approaches to
Pareto-based multiobjective ensemble generation are compared and
discussed in detail. Finally, potentially interesting topics in
multiobjective machine learning are suggested.

2008





* 241(<-405): Neural network ensembles: immune-inspired approaches to the diversity of components

This work applies two immune-inspired algorithms, namely opt-aiNet and
omni-aiNet, to train multi-layer perceptrons (MLPs) to be used in the
construction of ensembles of classifiers. The main goal is to
investigate the influence of the diversity of the set of solutions
generated by each of these algorithms, and if these solutions lead to
improvements in performance when combined in ensembles. omni-aiNet is
a multi-objective optimization algorithm and, thus, explicitly
maximizes the components' diversity at the same time it minimizes
their output errors. The opt-aiNet algorithm, by contrast, was
originally designed to solve single-objective optimization problems,
focusing on the minimization of the output error of the
classifiers. However, an implicit diversity maintenance mechanism
stimulates the generation of MLPs with different weights, which may
result in diverse classifiers. The performances of opt-aiNet and
omni-aiNet are compared with each other and with that of a
second-order gradient-based algorithm, named MSCG. The results
obtained show how the different diversity maintenance mechanisms
presented by each algorithm influence the gain in performance obtained
with the use of ensembles.

2010





* 242(<-504): The Q-norm complexity measure and the minimum gradient method: A novel approach to the machine learning structural risk minimization problem

This paper presents a novel approach for dealing with the structural
risk minimization (SRM) applied to a general setting of the machine
learning problem. The formulation is based on the fundamental concept
that supervised learning is a bi-objective optimization problem in
which two conflicting objectives should be minimized. The objectives
are related to the empirical training error and the machine
complexity. In this paper, one general Q-norm method to compute the
machine complexity is presented, and, as a particular practical case,
the minimum gradient method (MGM) is derived relying on the definition
of the fat-shattering dimension. A practical mechanism for parallel
layer perceptron (PLP) network training, involving only quasi-convex
functions, is generated using the aforementioned
definitions. Experimental results on 15 different benchmarks are
presented, which show the potential of the proposed ideas.

2008



* 243(<-543): Controlling the parallel layer perceptron complexity using a multiobjective learning algorithm

This paper deals with the parallel layer perceptron (PLP) complexity
control, bias and variance dilemma, using a multiobjective (MOBJ)
training algorithm. To control the bias and variance the training
process is rewritten as a bi-objective problem, considering the
minimization of both training error and norm of the weight vector,
which is a measure of the network complexity. This method is, applied
to regression and classification problems and compared with several
other training procedures and topologies. The results show that the
PLP MOBJ training algorithm presents good generalization results,
outperforming traditional methods in the tested examples.

2007





* 244(<-548): Improving generalization of MLPs with sliding mode control and the Levenberg-Marquardt algorithm

A variation of the well-known Levenberg-Marquardt for training neural
networks is proposed in this work. The algorithm presented restricts
the norm of the weights vector to a preestablished norm value and
finds the minimum error solution for that norm value. The norm
constrain controls the neural networks degree of freedom. The more the
norm increases, the more flexible is the neural model. Therefore, more
fitted to the training set. A range of different norm solutions is
generated and the best generalization solution is selected according
to the validation set error. The results show the efficiency of the
algorithm in terms of generalization performance. (c) 2006 Elsevier
B.V. All rights reserved.

2007




* 246(<-560): Many-objective training of a multi-layer perceptron

In this paper, a many-objective training scheme for a multi-layer
feed-forward neural network is studied. In this scheme, each training
data set, or the average over sub-sets of the training data, provides
a single objective. A recently proposed group of evolutionary
many-objective optimization algorithms based on the NSGA-II algorithm
have been examined with respect to the handling of such problem
cases. A modified NSGA-II algorithm, using the norm of an individual
as a secondary ranking assignment method, appeared to give the best
results, even for a large number of objectives (up to 50 in this
study). However, there was no notable increase in performance against
the standard backpropagation algorithm, and a remarkable drop in
performance for higher-dimensional feature spaces (dimension 30 in
this study).

2007





* 247(<-645): Training neural networks with a multi-objective sliding mode control algorithm

This paper presents a new sliding mode control algorithm that is able
to guide the trajectory of a multi-layer perceptron within the plane
formed by the two objective functions: training set error and norm of
the weight vectors. The results show that the neural networks obtained
are able to generate an approximation to the Pareto set, from which an
improved generalization performance model is selected. (C) 2002
Elsevier Science B.V. All rights reserved.

2003





* 248(<-661): Recent advances in the MOBJ algorithm for training artificial neural networks.

This paper presents a new scheme for training MLPs which employs a
relaxation method for multi-objective optimization. The algorithm
works by obtaining a reduced set of solutions, from which the one with
the best generalization is selected. This approach allows balancing
between the training error and norm of network weight vectors, which
are the two objective functions of the multi-objective optimization
problem. The method is applied to classification and regression
problems and compared with Weight Decay (WD), Support Vector Machines
(SVMs) and standard Backpropagation (BP). It is shown that the
systematic procedure for training proposed results on good
generalization neural models, and outperforms traditional methods.

2001





* 249(<-665): Improving generalization of MLPs with multi-objective optimization

This paper presents a new learning scheme for improving generalization
of multilayer perceptrons. The algorithm uses a multi-objective
optimization approach to balance between the error of the training
data and the norm of network weight vectors to avoid overfitting. The
results are compared with support vector machines and standard
backpropagation. (C) 2000 Elsevier Science B.V. All rights reserved.

2000





* 251(<- 85): Time series forecasting by neural networks: A knee point-based multiobjective evolutionary algorithm approach

In this paper, we investigate the problem of time series forecasting
using single hidden layer feedforward neural networks (SLFNs), which
is optimized via multiobjective evolutionary algorithms. By utilizing
the adaptive differential evolution (JADE) and the knee point
strategy, a nondominated sorting adaptive differential evolution
(NSJADE) and its improved version knee point-based NSJADE (KP-NSJADE)
are developed for optimizing SLFNs. JADE aiming at refining the search
area is introduced in nondominated sorting genetic algorithm II
(NSGA-II). The presented NSJADE shows superiority on multimodal
problems when compared with NSGA-II. Then NSJADE is applied to train
SLFNs for time series forecasting. It is revealed that individuals
with better forecasting performance in the whole population gather
around the knee point. Therefore, KP-NSJADE is proposed to explore the
neighborhood of the knee point in the objective space. And the
simulation results of eight popular time series databases illustrate
the effectiveness of our proposed algorithm in comparison with several
popular algorithms. (C) 2014 Elsevier Ltd. All rights reserved.

2014





* 252(<-124): An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection

This study examines the contribution of diversity under a
multi-objective context for the promotion of learners in an
evolutionary system that generates combinations of partially trained
learners. The examined system uses a grammar-driven genetic
programming to evolve hierarchical, multi-component combinations of
multilayer perceptrons and support vector machines for regression. Two
advances are studied. First, a ranking formula is developed for the
selection probability of the base learners. This formula incorporates
both a diversity measure and the performance of learners, and it is
tried over a series of artificial and real-world problems. Results
show that when the diversity of a learner is incorporated with equal
weights to the learner performance in the evolutionary selection
process, the system is able to provide statistically significantly
better generalization. The second advance examined is a substitution
phase for learners that are over-dominated, under a multi-objective
Pareto domination assessment scheme. Results here show that the
substitution does not improve significantly the system performance,
thus the exclusion of very weak learners, is not a compelling task for
the examined framework.

2014






* 255(<-649): Designing a phenotypic distance index for radial basis function neural networks

MultiObjective Evolutionary Algorithms (MOEAs) may cause a premature
convergence if the selective pressure is too large, so, MOEAs usually
incorporate a niche-formation procedure to distribute the population
over the optimal solutions and let the population evolve until the
Pareto-optimal region is completely explored. This niche-formation
scheme is based on a distance index that measures the similarity
between two solutions in order to decide if both may share the same
niche or not. The similarity criterion is usually based on a Euclidean
norm (given that the two solutions are represented with a vector),
nevertheless, as this paper will explain, this kind of metric is not
adequate for RBFNNs, thus being necessary a more suitable distance
index. The experimental results obtained show that a MOEA including
the proposed distance index is able to explore sufficiently the
Pareto-optimal region and provide the user a wide variety of
Pareto-optimal solutions.

2003





* 256(<-658): Hierarchical genetic algorithm for near optimal feedforward neural network design.

In this paper, we propose a genetic algorithm based design procedure
for a multi layer feed forward neural network. A hierarchical genetic
algorithm is used to evolve both the neural networks topology and
weighting parameters. Compared with traditional genetic algorithm
based designs for neural networks, the hierarchical approach addresses
several deficiencies, including a feasibility check highlighted in
literature. A multi objective cost function is used herein to optimize
the performance and topology of the evolved neural network
simultaneously. In the prediction of Mackey Glass chaotic time series,
the networks designed by the proposed approach prove to be
competitive, or even superior, to traditional learning algorithms for
the multi layer Perceptron networks and radial basis function
networks. Based upon the chosen cost function, a linear weight
combination decision making approach has been applied to derive an
approximated Pareto optimal solution set. Therefore, designing a set
of neural networks can be considered as solving a two objective
optimization problem.

2002





* 260(<-453): Parallel multiobjective memetic RBFNNs design and feature selection for function approximation problems

The design of radial basis function neural networks (RBFNNs) still
remains as a difficult task when they are applied to classification or
to regression problems. The difficulty arises when the parameters that
define an RBFNN have to be set, these are: the number of RBFs, the
position of their centers and the length of their radii. Another issue
that has to be faced when applying these models to real world
applications is to select the variables that the RBFNN will use as
inputs. The literature presents several methodologies to perform these
two tasks separately, however, due to the intrinsic parallelism of the
genetic algorithms, a parallel implementation will allow the algorithm
proposed in this paper to evolve solutions for both problems at the
same time. The parallelization of the algorithm not only consists in
the evolution of the two problems but in the specialization of the
crossover and mutation operators in order to evolve the different
elements to be optimized when designing RBFNNs. The subjacent genetic
algorithm is the non-sorting dominated genetic algorithm II (NSGA-II)
that helps to keep a balance between the size of the network and its
approximation accuracy in order to avoid overfitted networks. Another
of the novelties of the proposed algorithm is the incorporation of
local search algorithms in three stages of the algorithm:
initialization of the population, evolution of the individuals and
final optimization of the Pareto front. The initialization of the
individuals is performed hybridizing clustering techniques with the
mutual information (MI) theory to select the input variables. As the
experiments will show, the synergy of the different paradigms and
techniques combined by the presented algorithm allow to obtain very
accurate models using the most significant input variables. (C) 2009
Published by Elsevier B.V.

2009






* 261(<-544): A new hybrid methodology for cooperative-coevolutionary optimization of radial basis function networks

This paper presents a new multiobjective cooperative-coevolutive
hybrid algorithm for the design of a Radial Basis Function Network
(RBFN). This approach codifies a population of Radial Basis Functions
(RBFs) (hidden neurons), which evolve by means of cooperation and
competition to obtain a compact and accurate RBFN. To evaluate the
significance of a given RBF in the whole network, three factors have
been proposed: the basis function's contribution to the network's
output, the error produced in the basis function radius, and the
overlapping among RBFs. To achieve an RBFN composed of RBFs with
proper values for these quality factors our algorithm follows a
multiobjective approach in the selection process. In the design
process, a Fuzzy Rule Based System (FRBS) is used to determine the
possibility of applying operators to a certain RBF. As the time
required by our evolutionary algorithm to converge is relatively
small, it is possible to get a further improvement of the solution
found by using a local minimization algorithm (for example, the
Levenberg-Marquardt method). In this paper the results of applying our
methodology to function approximation and time series prediction
problems are also presented and compared with other alternatives
proposed in the bibliography.

2007





* 262(<-638): Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation

This paper presents a multiobjective evolutionary algorithm to
optimize radial basis function neural networks (RBFNNs) in order to
approach target functions from a set of input-output pairs. The
procedure allows the application of heuristics to improve the solution
of the problem at hand by including some new genetic operators in the
evolutionary process. These new operators are based on two well-known
matrix transformations: singular value decomposition (SVD) and
orthogonal least squares (OLS), which have been used to define new
mutation operators that produce local or global modifications in the
radial basis functions (RBFs) of the networks (the individuals in the
population in the evolutionary, procedure). After analyzing the
efficiency of the different operators, we have shown that the global
mutation operators yield an improved procedure to adjust the
parameters of the RBFNNs.

2003





* 263(<-204): Memetic multiobjective particle swarm optimization-based radial basis function network for classification problems

This paper presents a new multiobjective evolutionary algorithm
applied to a radial basis function (RBF) network design based on
multiobjective particle swarm optimization augmented with local search
features. The algorithm is named the memetic multiobjective particle
swarm optimization RBF network (MPSON) because it integrates the
accuracy and structure of an RBF network. The proposed algorithm is
implemented on two-class and multiclass pattern classification
problems with one complex real problem. The experimental results
indicate that the proposed algorithm is viable, and provides an
effective means to design multiobjective RBF networks with good
generalization capability and compact network structure. The accuracy
and complexity of the network obtained by the proposed algorithm are
compared with the memetic non-dominated sorting genetic algorithm
based RBF network (MGAN) through statistical tests. This study shows
that MPSON generates RBF networks coming with an appropriate balance
between accuracy and simplicity, outperforming the other algorithms
considered. (C) 2013 Elsevier Inc. All rights reserved.

2013





* 265(<-325): Memetic Elitist Pareto Differential Evolution algorithm based Radial Basis Function Networks for classification problems

This paper presents a new multi-objective evolutionary hybrid
algorithm for the design of Radial Basis Function Networks (RBFNs) for
classification problems. The algorithm, MEPDEN, Memetic Elitist Pareto
evolutionary approach based on the Non-dominated Sorting Differential
Evolution (NSDE) multiobjective evolutionary algorithm which has been
adapted to design RBFNs, where the NSDE algorithm is augmented with a
local search that uses the Back-propagation algorithm. The MEPDEN is
tested on two-class and multiclass pattern classification
problems. The results obtained in terms of Mean Square Error (MSE),
number of hidden nodes, accuracy (ACC), sensitivity (SEN), specificity
(SPE) and Area Under the receiver operating characteristics Curve
(AUC), show that the proposed approach is able to produce higher
prediction accuracies with much simpler network structures. The
accuracy and complexity of the network obtained by the proposed
algorithm are compared with Memetic Eilitist Pareto Non-dominated
Sorting Genetic Algorithm based RBFN (MEPGAN) through statistical
tests. This study showed that MEPDEN obtains RBFNs with an appropriate
balance between accuracy and simplicity, outperforming the other
method considered. (C) 2011 Elsevier B.V. All rights reserved.

2011





* 266(<-378): Memetic Pareto Evolutionary Artificial Neural Networks to determine growth/no-growth in predictive microbiology

The main objective of this work is to automatically design neural
network models with sigmoid basis units for binary classification
tasks. The classifiers that are obtained achieve a double objective: a
high classification level in the dataset and a high classification
level for each class. We present MPENSGA2, a Memetic Pareto
Evolutionary approach based on the NSGA2 multiobjective evolutionary
algorithm which has been adapted to design Artificial Neural Network
models, where the NSGA2 algorithm is augmented with a local search
that uses the improved Resilient Backpropagation with
backtracking-IRprop+ algorithm. To analyze the robustness of this
methodology, it was applied to four complex classification problems in
predictive microbiology to describe the growth/no-growth interface of
food-borne microorganisms such as Listeria monocytogenes, Escherichia
coli R31, Staphylococcus aureus and Shigella flexneri. The results
obtained in Correct Classification Rate (CCR), Sensitivity (S) as the
minimum of sensitivities for each class, Area Under the receiver
operating characteristic Curve (AUC), and Root Mean Squared Error
(RMSE), show that the generalization ability and the classification
rate in each class can be more efficiently improved within a
multiobjective framework than within a single-objective framework. (C)
2009 Elsevier B.V. All rights reserved.

2011






* 268(<-414): Sensitivity Versus Accuracy in Multiclass Problems Using Memetic Pareto Evolutionary Neural Networks

This paper proposes a multiclassification algorithm using multilayer
perceptron neural network models. It tries to boost two conflicting
main objectives of multiclassifiers: a high correct classification
rate level and a high classification rate for each class. This last
objective is not usually optimized in classification, but is
considered here given the need to obtain high precision in each class
in real problems. To solve this machine learning problem, we use a
Pareto-based multiobjective optimization methodology based on a
memetic evolutionary algorithm. We consider a memetic Pareto
evolutionary approach based on the NSGA2 evolutionary algorithm
(MPENSGA2). Once the Pareto front is built, two strategies or
automatic individual selection are used: the best model in accuracy
and the best model in sensitivity ( extremes in the Pareto
front). These methodologies are applied to solve 17 classification
benchmark problems obtained from the University of California at
Irvine (UCI) repository and one complex real classification
problem. The models obtained show high accuracy and a high
classification rate for each class.

2010





* 269(<-379): Radial basis function network based on time variant multi-objective particle swarm optimization for medical diseases diagnosis

This paper proposes an adaptive evolutionary radial basis function
(RBF) network algorithm to evolve accuracy and connections (centers
and weights) of RBF networks simultaneously. The problem of hybrid
learning of RBF network is discussed with the multi-objective
optimization methods to improve classification accuracy for medical
disease diagnosis. In this paper, we introduce a time variant
multi-objective particle swarm optimization(TVMOPSO) of radial basis
function (RBF) network for diagnosing the medical diseases. This study
applied RBF network training to determine whether RBF networks can be
developed using TVMOPSO, and the performance is validated based on
accuracy and complexity. Our approach is tested on three standard data
sets from UCI machine learning repository. The results show that our
approach is a viable alternative and provides an effective means to
solve multi-objective RBF network for medical disease diagnosis. It is
better than RBF network based on MOPSO and NSGA-II, and also
competitive with other methods in the literature. (C) 2010 Elsevier
B.V. All rights reserved.

2011





* 271(<-578): Applications of multi-objective structure optimization

We present applications of multi-objective evolutionary optimization
of feed-forward neural networks (NN) to two real world problems, car
and face classification. The possibly conflicting requirements on the
NNs are speed and classification accuracy, both of which can enhance
the embedding systems as a whole. We compare the results to the
outcome of a greedy optimization heuristic (magnitude-based pruning)
coupled with a multi-objective performance evaluation. For the car
classification problem, magnitude-based pruning yields competitive
results, whereas for the more difficult face classification, we find
that the evolutionary approach to NN design is clearly preferable. (c)
2006 Elsevier B.V. All rights reserved.

2006





* 274(<-113): Metrics to guide a multi-objective evolutionary algorithm for ordinal classification

Ordinal classification or ordinal regression is a classification
problem in which the labels have an ordered arrangement between
them. Due to this order, alternative performance evaluation metrics
are need to be used in order to consider the magnitude of errors. This
paper presents a study of the use of a multi-objective optimization
approach in the context of ordinal classification. We contribute a
study of ordinal classification performance metrics, and propose a new
performance metric, the maximum mean absolute error (MMAE). MMAE
considers per-class distribution of patterns and the magnitude of the
errors, both issues being crucial for ordinal regression problems. In
addition, we empirically show that some of the performance metrics are
competitive objectives, which justify the use of multi-objective
optimization strategies. In our case, a multi-objective evolutionary
algorithm optimizes an artificial neural network ordinal model with
different pairs of metric combinations, and we conclude that the pair
of the mean absolute error (MAE) and the proposed MMAE is the most
favourable. A study of the relationship between the metrics of this
proposal is performed, and the graphical representation in the
two-dimensional space where the search of the evolutionary algorithm
takes place is analysed. The results obtained show a good
classification performance, opening new lines of research in the
evaluation and model selection of ordinal classifiers. (C) 2014
Elsevier B.V. All rights reserved.

2014





* 276(<-334): Weighting Efficient Accuracy and Minimum Sensitivity for Evolving Multi-Class Classifiers

Recently, a multi-objective Sensitivity-Accuracy based methodology has
been proposed for building classifiers for multi-class problems. This
technique is especially suitable for imbalanced and multi-class
datasets. Moreover, the high computational cost of multi-objective
approaches is well known so more efficient alternatives must be
explored. This paper presents an efficient alternative to the Pareto
based solution when considering both Minimum Sensitivity and Accuracy
in multi-class classifiers. Alternatives are implemented by extending
the Evolutionary Extreme Learning Machine algorithm for training
artificial neural networks. Experiments were performed to select the
best option after considering alternative proposals and related
methods. Based on the experiments, this methodology is competitive in
Accuracy, Minimum Sensitivity and efficiency.

2011





* 277(<-561): A cooperative constructive method for neural networks for pattern recognition

In this paper, we propose a new constructive method, based on
cooperative coevolution, for designing automatically the structure of
a neural network for classification. Our approach is based on a
modular construction of the neural network by means of a cooperative
evolutionary process. This process benefits from the advantages of
coevolutionary computation as well as the advantages of constructive
methods. The proposed methodology can be easily extended to work with
almost any kind of classifier. The evaluation of each module that
constitutes the network is made using a multiobjective method. So,
each new module can be evaluated in a comprehensive way, considering
different aspects, such as performance, complexity, or degree of
cooperation with the previous modules of the network. In this way, the
method has the advantage of considering not only the performance of
the networks, but also other features. The method is tested on 40
classification problems from the UCI machine learning repository with
very good performance. The method is thoroughly compared with two
other constructive methods, cascade correlation and GMDH networks, and
other classification methods, namely, SVM, C4.5, and k
nearest-neighbours, and an ensemble of neural networks constructed
using four different methods. (c) 2006 Pattern Recognition
Society. Published by Elsevier Ltd. All rights reserved.

2007





* 278(<-599): Cooperative coevolution of artificial neural network ensembles for pattern classification

This paper presents a cooperative coevolutive approach for designing
neural network ensembles. Cooperative coevolution is a recent paradigm
in evolutionary computation that allows the effective modeling of
cooperative environments. Although theoretically, a single neural
network with a sufficient number of neurons in the hidden layer would
suffice to solve any problem, in practice many real-world problems are
too hard to construct the appropriate network that solve them. In such
problems, neural network ensembles are a successful
alternative. Nevertheless, the design of neural network ensembles is a
complex task. In this paper, we propose a general framework for
designing neural network ensembles by means of cooperative
coevolution. The proposed model has two main objectives: first, the
improvement of the combination of the trained individual networks;
second, the cooperative evolution of such networks, encouraging
collaboration among them, instead of a separate training of each
network. In order to favor the cooperation of the networks, each
network is evaluated throughout the evolutionary process using a
multiobjective method. For each network, different objectives are
defined, considering not only its performance in the given problem,
but also its cooperation with the rest of the networks. In addition, a
population of ensembles is evolved, improving the combination of
networks and obtaining subsets of networks to form ensembles that
perform better than the combination of all the evolved networks. The
proposed model is applied to ten real-world classification problems of
a very different nature from the UCI machine learning repository and
proben1 benchmark set. In all of them the performance of the model is
better than the performance of standard ensembles in terms of
generalization error. Moreover, the size of the obtained ensembles is
also smaller.

2005





* 283(<-569): Feature selection for ensembles applied to handwriting recognition

Feature selection for ensembles has shown to be an effective strategy
for ensemble creation clue to its ability of producing good subsets of
features, which make the classifiers of the ensemble disagree on
difficult cases. In this paper we present an ensemble feature
selection approach based on a hierarchical multi-objective genetic
algorithm. The underpinning paradigm is the "overproduce and
choose". The algorithm operates in two levels. Firstly, it performs
feature selection in order to generate a set of classifiers and then
it chooses the best team of classifiers. In order to show its
robustness, the method is evaluated in two different contexts:
supervised and unsupervised feature selection. In the former, we have
considered the problem of handwritten digit recognition and used three
different feature sets and multi-layer perceptron neural networks as
classifiers. In the latter, we took into account the problem of
handwritten month word recognition and used three different feature
sets and hidden Markov models as classifiers. Experiments and
comparisons with classical methods, such as Bagging and Boosting,
demonstrated that the proposed methodology brings compelling
improvements when classifiers have to work with very low error
rates. Comparisons have been done by considering the recognition rates
only.

2006







* 284(<-210): A new approach to radial basis function-based polynomial neural networks: analysis and design

In this study, we introduce a new topology of radial basis
function-based polynomial neural networks (RPNNs) that is based on a
genetically optimized multi-layer perceptron with radial polynomial
neurons (RPNs). This paper offers a comprehensive design methodology
involving various mechanisms of optimization, especially fuzzy C-means
(FCM) clustering and particle swarm optimization (PSO). In contrast to
the typical architectures encountered in polynomial neural networks
(PNNs), our main objective is to develop a topology and establish a
comprehensive design strategy of RPNNs: (a) The architecture of the
proposed network consists of radial polynomial neurons (RPN). These
neurons are fully reflective of the structure encountered in numeric
data, which are granulated with the aid of FCM clustering. RPN dwells
on the concepts of a collection of radial basis function and the
function-based nonlinear polynomial processing. (b) The PSO-based
design procedure being applied to each layer of the RPNN leads to the
selection of preferred nodes of the network whose local parameters
(such as the number of input variables, a collection of the specific
subset of input variables, the order of the polynomial, the number of
clusters of FCM clustering, and a fuzzification coefficient of the FCM
method) are properly adjusted. The performance of the RPNN is
quantified through a series of experiments where we use several
modeling benchmarks, namely a synthetic three-dimensional data and
learning machine data (computer hardware data, abalone data, MPG data,
and Boston housing data) already used in neuro-fuzzy modeling. A
comparative analysis shows that the proposed RPNN exhibits higher
accuracy in comparison with some previous models available in the
literature.

2013





* 287(<-209): A multi-objective micro genetic ELM algorithm

The extreme learning machine (ELM) is a methodology for learning
single-hidden layer feedforward neural networks (SLFN) which has been
proved to be extremely fast and to provide very good generalization
performance. ELM works by randomly choosing the weights and biases of
the hidden nodes and then analytically obtaining the output weights
and biases for a SLFN with the number of hidden nodes previously
fixed. In this work, we develop a multi-objective micro genetic ELM
(mu G-ELM) which provides the appropriate number of hidden nodes for
the problem being solved as well as the weights and biases which
minimize the MSE. The multi-objective algorithm is conducted by two
criteria: the number of hidden nodes and the mean square error
(MSE). Furthermore, as a novelty, mu G-ELM incorporates a regression
device in order to decide whether the number of hidden nodes of the
individuals of the population should be increased or decreased or
unchanged. In general, the proposed algorithm reaches better errors by
also implying a smaller number of hidden nodes for the data sets and
competitors considered. (C) 2013 Elsevier B.V. All rights reserved.

2013





* 288(<-424): A multi-objective memetic and hybrid methodology for optimizing the parameters and performance of artificial neural networks

The use of artificial neural networks implies considerable time spent
choosing a set of parameters that contribute toward improving the
final performance. Initial weights, the amount of hidden nodes and
layers, training algorithm rates and transfer functions are normally
selected through a manual process of trial-and-error that often fails
to find the best possible set of neural network parameters for a
specific problem. This paper proposes an automatic search methodology
for the optimization of the parameters and performance of neural
networks relying on use of Evolution Strategies, Particle Swarm
Optimization and concepts from Genetic Algorithms corresponding to the
hybrid and global search module. There is also a module that refers to
local searches, including the well-known Multilayer Perceptrons,
Back-propagation and the Levenberg-Marquardt training algorithms. The
methodology proposed here performs the search using the aforementioned
parameters in an attempt to optimize the networks and
performance. Experiments were performed and the results proved the
proposed method to be better than trial-and-error and other methods
found in the literature. Crown Copyright (C) 2009 Published by
Elsevier B.V. All rights reserved.

2010





* 295(<-659): Improving neural networks generalization with new constructive and pruning methods

This paper presents a new constructive method and pruning approaches
to control the design of Multi-Layer Perceptron (MLP) without loss in
performance. The proposed methods use a multi-objective approach to
guarantee generalization. The constructive approach searches for an
optimal solution according to the pareto set shape with increasing
number of hidden nodes. The pruning methods are able to simplify the
network topology and to identify linear connections between the inputs
and outputs of the neural model. Topology information and validation
sets are used.

2002





* 302(<-546): Fuzzy integral-based perceptron for two-class pattern classification problems

The single-layer perceptron with single output node is a well-known
neural network for two-class classification problems. Furthermore, the
sigmoid or logistic function is usually used as the activation
function in the output neuron. A critical step is to compute the sum
of the products of the connection weights with the corresponding
inputs, which indicates the assumption of additivity among individual
variables. Unfortunately, because the input variables are not always
independent of each other, an assumption of additivity may not be
reasonable enough. In this paper, the inner product can be replaced
with an aggregation value obtained by a useful fuzzy integral by
viewing each of the connection weights as a value of a lambda-fuzzy
measure for the corresponding variable. A genetic algorithm is then
employed to obtain connection weights by maximizing the number of
correctly classified training patterns and minimizing the errors
between the actual and desired outputs of individual training
patterns. The experimental results further demonstrate that the
proposed method outperforms the traditional single-layer perceptron
and performs well in comparison with other fuzzy or non-fuzzy
classification methods. (c) 2006 Elsevier Inc. All rights reserved.

2007





* 303(<-590): Training of multilayer perceptron neural networks by using cellular genetic algorithms

This paper deals with a method for training neural networks by using
cellular genetic algorithms (CGA). This method was implemented as
software, CGANN-Trainer, which was used to generate binary classifiers
for recognition of patterns associated with breast cancer images in a
multi-objective optimization problem. The results reached by the CGA
with the Wisconsin Breast Cancer Database, and the Wisconsin
Diagnostic Breast Cancer Database, were compared with some other
methods previously reported using the same databases, proving to be an
interesting alternative.

2006





* 329(<-271): Convergence analysis of sliding mode trajectories in multi-objective neural networks learning

The Pareto-optimality concept is used in this paper in order to
represent a constrained set of solutions that are able to trade-off
the two main objective functions involved in neural networks
supervised learning: data-set error and network complexity. The neural
network is described as a dynamic system having error and complexity
as its state variables and learning is presented as a process of
controlling a learning trajectory in the resulting state space. In
order to control the trajectories, sliding mode dynamics is imposed to
the network. It is shown that arbitrary learning trajectories can be
achieved by maintaining the sliding mode gains within their
convergence intervals. Formal proofs of convergence conditions are
therefore presented. The concept of trajectory learning presented in
this paper goes further beyond the selection of a final state in the
Pareto set, since it can be reached through different trajectories and
states in the trajectory can be assessed individually against an
additional objective function. (c) 2012 Elsevier Ltd. All rights
reserved.

2012





* 340(<-640): Speeding up backpropagation using multiobjective evolutionary algorithms

The use of backpropagation for training artificial neural networks
(ANNs) is usually associated with a long training process. The user
needs to experiment with a number of network architectures; with
larger networks, more computational cost in terms of training time is
required. The objective of this letter is to present an optimization
algorithm, comprising a multiobjective evolutionary algorithm and a
gradient-based local search. In the rest of the letter, this is
referred to as the memetic Pareto artificial neural network algorithm
for training ANNs. The evolutionary approach is used to train the
network and simultaneously optimize its architecture. The result is a
set of networks, with each network in the set attempting to optimize
both the training error and the architecture. We also present a
self-adaptive version with lower computational cost. We show
empirically that the proposed method is capable of reducing the
training time compared to gradient-based techniques.

2003





* 346(<-652): Multi-objective cooperative coevolution of artificial neural networks (multi-objective cooperative networks)

In this paper we present a cooperative coevolutive model for the
evolution of neural network topology and weights, called
MOBNET. MOBNET evolves subcomponents that must be combined in order to
form a network, instead of whole networks. The problem of assigning
credit to the subcomponents is approached as a multi-objective
optimization task. The subcomponents in a cooperative coevolutive
model must fulfill different criteria to be useful, these criteria
usually conflict with each other. The problem of evaluating the
fitness on an individual based on many criteria that must be optimized
together can be approached as a multi-criteria optimization problems,
so the methods from multi-objective optimization offer the most
natural way to solve the problem. In this work we show how using
several objectives for every subcomponent and evaluating its fitness
as a multi-objective optimization problem, the performance of the
model is highly competitive. MOBNET is compared with several standard
methods of classification and with other neural network models in
solving four real-world problems, and it shows the best overall
performance of all classification methods applied. It also produces
smaller networks when compared to other models. The basic idea
underlying MOBNET is extensible to a more general model of
coevolutionary computation, as none of its features are exclusive of
neural networks design. There are many applications of cooperative
coevolution that could benefit from the multi-objective optimization
approach proposed in this paper. (C) 2002 Elsevier Science Ltd. All
fights reserved.

2002





* 347(<-362): Learning in the feed-forward random neural network: A critical review

The Random Neural Network (RNN) has received, since its inception in
1989, considerable attention and has been successfully used in a
number of applications. In this critical review paper we focus on the
feed-forward RNN model and its ability to solve classification
problems. In particular, we paid special attention to the RNN
literature related with learning algorithms that discover the RNN
interconnection weights, suggested other potential algorithms that can
be used to find the RNN interconnection weights, and compared the RNN
model with other neural-network based and non-neural network based
classifier models. In review, the extensive literature review and
experimentation with the RNN feed-forward model provided us with the
necessary guidance to introduce six critical review comments that
identify some gaps in the RNN's related literature and suggest
directions for future research. (C) 2010 Elsevier B.V. All rights
reserved.

2011





* 352(<-662): Evolutionary optimization of RBF networks.

One of the main obstacles to the widespread use of artificial neural
networks is the difficulty of adequately defining values for their
free parameters. This article discusses how Radial Basis Function
(RBF) networks can have their parameters defined by genetic
algorithms. For such, it presents an overall view of the problems
involved and the different approaches used to genetically optimize RBF
networks. A new strategy to optimize RBF networks using genetic
algorithms is proposed, which includes new representation, crossover
operator and the use of a multiobjective optimization
criterion. Experiments using a benchmark problem are performed and the
results achieved using this model are compared to those achieved by
other approaches.

2001





* 356(<-594): Stopping criteria for ensemble of evolutionary artificial neural networks

The formation of ensemble of artificial neural networks has attracted
attentions of researchers in the machine learning and statistical
inference domains. It has been shown that combining different neural
networks could improve the generalization ability of the learning
machine. One challenge is when to stop the training or evolution of
the neural networks to avoid overfitting. In this paper, we show that
different early stopping criteria based on (i) the minimum validation
fitness of the ensemble, and (ii) the minimum of the average
population validation fitness could generalize better than the
survival population in the last generation. The proposition was tested
on four different ensemble methods: (i) a simple ensemble method,
where each individual of the population (created and maintained by the
evolutionary process) is used as a committee member, (ii) ensemble
with island model as a diversity promotion mechanism, (iii) a recent
successful ensemble method namely ensemble with negative correlation
learning and (iv) an ensemble formed by applying multi-objective
optimization. The experimental results suggested that using minimum
validation fitness of the ensemble as an early stopping criterion is
beneficial. (C) 2005 Elsevier B.V. All rights reserved.

2005





* 369(<-499): Hybrid multiobjective evolutionary design for artificial neural networks

Evolutionary algorithms are a class of stochastic search methods that
attempts to emulate the biological process of evolution, incorporating
concepts of selection, reproduction, and mutation. In recent years,
there has been an increase in the use of evolutionary approaches in
the training of artificial neural networks (ANNs). While evolutionary
techniques for neural networks have shown to provide superior
performance over conventional training approaches, the simultaneous
optimization of network performance and architecture will almost
always result in a slow training process due to the added algorithmic
complexity. In this paper, we present a geometrical measure based on
the singular value decomposition (SVD) to estimate the necessary
number of neurons to be used in training a single-hidden-layer
feedforward neural network (SLFN). In addition, we develop a new
hybrid multiobjective evolutionary approach that includes the features
of a variable length representation that allow for easy adaptation of
neural networks structures, an architectural recombination procedure
based on the geometrical measure that adapts the number of necessary
hidden neurons and facilitates the exchange of neuronal information
between candidate designs, and a microhybrid genetic algorithm (mu
HGA) with an adaptive local search intensity scheme for local
fine-tuning. In addition, the performances of well-known algorithms as
well as the effectiveness and contributions of the proposed approach
are analyzed and validated through a variety of data set types.

2008





* 476(<-647): Software verification of redundancy in neuro-evolutionary robotics

Evolutionary methods are now commonly used to automatically generate
autonomous controllers for physical robots as well as for virtually
embodied organisms. Although it is generally accepted that some amount
of redundancy may result from using an evolutionary approach, few
studies have focused on empirically testing the actual amount of
redundancy that is present in controllers generated using artificial
evolutionary systems. Network redundancy in certain application
domains such as defence, space, and safeguarding, is unacceptable as
it puts the reliability of the system at great risk. Thus, our aim in
this paper is to test and compare the redundancies of artificial
neural network (ANN) controllers that are evolved for a quadrupedal
robot using four different evolutionary methodologies. Our results
showed that the least amount of redundancy was generated using a
self-adaptive Pareto evolutionary multi-objective optimization (EMO)
algorithm compared to the more commonly used single-objective
evolutionary algorithm (EA) and weighted sum EMO algorithm. Finally,
self-adaptation was found to be highly beneficial in reducing
redundancy when compared against a hand-tuned Pareto EMO algorithm.

2003





* 527(<-270): Structure optimization of neural network for dynamic system modeling using multi-objective genetic algorithm

The problem of constructing an adequate and parsimonious neural
network topology for modeling non-linear dynamic system is studied and
investigated. Neural networks have been shown to perform function
approximation and represent dynamic systems. The network structures
are usually guessed or selected in accordance with the designer's
prior knowledge. However, the multiplicity of the model parameters
makes it troublesome to get an optimum structure. In this paper, an
alternative algorithm based on a multi-objective optimization
algorithm is proposed. The developed neural network model should
fulfil two criteria or objectives namely good predictive accuracy and
minimum model structure. The result shows that the proposed algorithm
is able to identify simulated examples correctly, and identifies the
adequate model for real process data based on a set of solutions
called the Pareto optimal set, from which the best network can be
selected.

2012





* 645(<-524): Multi-criteria optimization in nonlinear predictive control

The multi-criteria predictive control of nonlinear dynamical systems
based on Artificial Neural Networks (ANNs) and genetic algorithms
(GAs) are considered. The (ANNs) are used to determine process models
at each operating level; the control action is provided by minimizing
a set of control objective which is function of the future prediction
output and the future control actions in tacking in account
constraints in input signal. An aggregative method based on the
Non-dominated Sorting Genetic Algorithm (NSGA) is applied to solve the
multi-criteria optimization problem. The results obtained with the
proposed control scheme are compared in simulation to those obtained
with the multi-model control approach. (c) 2007 IMACS. Published by
Elsevier B.V. All rights reserved.

2008