Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools

Several grammar-based genetic programming algorithms have been proposed in the literature to automatically generate heuristics for hard optimization problems. These approaches specify the algorithmic building blocks and the way in which they can be combined in a grammar; the best heuristic for the problem being tackled is found by an evolutionary algorithm that searches in the algorithm design space defined by the grammar.In this work, we propose a novel representation of the grammar by a sequence of categorical, integer, and real-valued parameters. We then use a tool for automatic algorithm configuration to search for the best algorithm for the problem at hand. Our experimental evaluation on the one-dimensional bin packing problem and the permutation flowshop problem with weighted tardiness objective shows that the proposed approach produces better algorithms than grammatical evolution, a well-established variant of grammar-based genetic programming. The reasons behind such improvement lie both in the representation proposed and in the method used to search the algorithm design space.     

Crossover and mutation operators for grammar-guided genetic programming

This paper proposes a new grammar-guided genetic programming (GGGP) system by introducing two original genetic operators: crossover and mutation, which most influence the evolution process. The first, the so-called grammar-based crossover operator, strikes a good balance between search space exploration and exploitation capabilities and, therefore, enhances GGGP system performance. And the second is a grammar-based mutation operator, based on the crossover, which has been designed to generate individuals that match the syntactical constraints of the context-free grammar that defines the programs to be handled. The use of these operators together in the same GGGP system assures a higher convergence speed and less likelihood of getting trapped in local optima than other related approaches. These features are shown throughout the comparison of the results achieved by the proposed system with other important crossover and mutation methods in two experiments: a laboratory problem and the real-world task of breast cancer prognosis.     

Logic-based genetic programming with definite clause translation grammars

DCTG-GP is a genetic programming system that uses definite clause translation grammars. A DCTG is a logical version of an attribute grammar that supports the definition of context-free languages, and it allows semantic information associated with a language to be easily accommodated by the grammar. This is useful in genetic programming for defining the interpreter of a target language, or incorporating both syntactic and semantic problem-specific constraints into the evolutionary search. The DCTG-GP system improves on other grammar-based GP systems by permitting nontrivial semantic aspects of the language to be defined with the grammar. It also automatically analyzes grammar rules in order to determine their minimal depth and termination characteristics, which are required when generating random program trees of varied shapes and sizes. An application using DCTG-GP is described. Brian James Ross, Ph.D.: He is an associate professor of computer science at Brock University, where he has worked since 1992. He obtained his BCSc at the University of Manitoba, Canada, in 1984, his MSc at the University of British Columbia, Canada, in 1988, and his PhD at the University of Edinburgh, Scotland, in 1992. His research interests include evolutionary computation, machine learning, language induction, concurrency, and logic programming.     

Grammar-guided genetic programming and dimensional consistency: application to non-parametric identification in mechanics

Although genetic programming has often successfully been applied to non-parametric modeling, it is frequently impaired by the huge size of the search space explored. Domain knowledge is a powerful way to trim out the size of the space, by restricting the search to a priori relevant models. A most natural domain knowledge in scientific modeling is known as dimensional analysis, stipulating that the models must be consistent with regards to the variable measurement units.In this paper, it is shown that dimensional analysis can automatically be expressed as a context free grammar. Dimensionally-aware GP is thus achieved by employing the dimensional grammar within the grammar-guided GP framework first investigated by Gruau [On using syntactic constraints with genetic programming, in: P. Angeline, K.E. Kinnear Jr. (Eds.), Advances in Genetic Programming II, MIT Press, Cambridge, MA, 1996, pp. 377–394.].However, grammar-guided genetic programming encounters severe difficulties when it involves a complex grammar, which might explain why this approach has not been widely used so far. The drawback is blamed on the initialization step, which hardly constructs a sufficiently diversified initial population, thus hindering the success of evolution. This limitation is addressed by a new CFG compliant initialization procedure.The approach is validated on two problems related to the identification of mechanical properties of materials.     

Local and global optimization for Takagi–Sugeno fuzzy system by memetic genetic programming

This work presents a method to incorporate standard neuro-fuzzy learning for Takagi–Sugeno fuzzy systems that evolve under a grammar driven genetic programming (GP) framework. This is made possible by introducing heteroglossia in the functional GP nodes, enabling them to switch behavior according to the selected learning stage. A context-free grammar supports the expression of arbitrarily sized and composed fuzzy systems and guides the evolution. Recursive least squares and backpropagation gradient descent algorithms are used as local search methods. A second generation memetic approach combines the genetic programming with the local search procedures. Based on our experimental results, a discussion is included regarding the competitiveness of the proposed methodology and its properties. The contributions of the paper are: (i) introduction of an approach which enables the application of local search learning for intelligent systems evolved by genetic programming, (ii) presentation of a model for memetic learning of Takagi–Sugeno fuzzy systems, (iii) experimental results evaluating model variants and comparison with state-of-the-art models in benchmarking and real-world problems, (iv) application of the proposed model in control.     

Automated discovery of local search heuristics for satisfiability testing

The development of successful metaheuristic algorithms such as local search for a difficult problem such as satisfiability testing (SAT) is a challenging task. We investigate an evolutionary approach to automating the discovery of new local search heuristics for SAT. We show that several well-known SAT local search algorithms such as Walksat and Novelty are composite heuristics that are derived from novel combinations of a set of building blocks. Based on this observation, we developed CLASS, a genetic programming system that uses a simple composition operator to automatically discover SAT local search heuristics. New heuristics discovered by CLASS are shown to be competitive with the best Walksat variants, including Novelty+. Evolutionary algorithms have previously been applied to directly evolve a solution for a particular SAT instance. We show that the heuristics discovered by CLASS are also competitive with these previous, direct evolutionary approaches for SAT. We also analyze the local search behavior of the learned heuristics using the depth, mobility, and coverage metrics proposed by Schuurmans and Southey.     

A survey of semantic methods in genetic programming

Several methods to incorporate semantic awareness in genetic programming have been proposed in the last few years. These methods cover fundamental parts of the evolutionary process: from the population initialization, through different ways of modifying or extending the existing genetic operators, to formal methods, until the definition of completely new genetic operators. The objectives are also distinct: from the maintenance of semantic diversity to the study of semantic locality; from the use of semantics for constructing solutions which obey certain constraints to the exploitation of the geometry of the semantic topological space aimed at defining easy-to-search fitness landscapes. All these approaches have shown, in different ways and amounts, that incorporating semantic awareness may help improving the power of genetic programming. This survey analyzes and discusses the state of the art in the field, organizing the existing methods into different categories. It restricts itself to studies where semantics is intended as the set of output values of a program on the training data, a definition that is common to a rather large set of recent contributions. It does not discuss methods for incorporating semantic information into grammar-based genetic programming or approaches based on formal methods. The objective is keeping the community updated on this interesting research track, hoping to motivate new and stimulating contributions.     

Incremental Evolution of Fuzzy Grammar Fragments to Enhance Instance Matching and Text Mining

In many applications, it is useful to extract structured data from sections of unstructured text. A common approach is to use pattern matching (e.g., regular expressions) or more general grammar-based techniques. In cases where exact templates or grammar fragments are not known, it is possible to use machine learning approaches, based on words or n-grams, to identify the structured data. This is generally a two-stage (train/use) process that cannot easily cope with incremental extensions of the training set. In this paper, we combine a fuzzy grammar-based approach with incremental learning. This enables a set of grammar fragments to evolve incrementally, each time a new example is given, while guaranteeing that it can parse previously seen examples. We propose a novel measure of overlap between fuzzy grammar fragments that can also be used to determine the degree to which a string is parsed by a grammar fragment. This measure of overlap allows us to compare the range of two fuzzy grammar fragments (i.e., to estimate and compare the sets of strings that fuzzily conform to each grammar) without explicitly parsing any strings. A simple application shows the method's validity.      

A Hyper-Heuristic Framework for Lifetime Maximization in Wireless Sensor Networks With A Mobile Sink

Maximizing the lifetime of wireless sensor networks(WSNs) is an important and challenging research problem. Properly scheduling the movements of mobile sinks to balance the energy consumption of wireless sensor network is one of the most effective approaches to prolong the lifetime of wireless sensor networks. However, the existing mobile sink scheduling methods either require a great amount of computational time or lack effectiveness in finding high-quality scheduling solutions. To address the above issues, this paper proposes a novel hyperheuristic framework, which can automatically construct high-level heuristics to schedule the sink movements and prolong the network lifetime. In the proposed framework, a set of low-level heuristics are defined as building blocks to construct high-level heuristics and a set of random networks with different features are designed for training. Further, a genetic programming algorithm is adopted to automatically evolve promising high-level heuristics based on the building blocks and the training networks. By using the genetic programming to evolve more effective heuristics and applying these heuristics in a greedy scheme, our proposed hyper-heuristic framework can prolong the network lifetime competitively with other methods, with small time consumption. A series of comprehensive experiments, including both static and dynamic networks,are designed. The simulation results have demonstrated that the proposed method can offer a very promising performance in terms of network lifetime and response time.     

STOCHASTIC PARSING AND EVOLUTIONARY ALGORITHMS

This article aims to show the effectiveness of evolutionary algorithms in automatically parsing sentences of real texts. Parsing methods based on complete search techniques are limited by the exponential increase of the size of the search space with the size of the grammar and the length of the sentences to be parsed. Approximated methods, such as evolutionary algorithms, can provide approximate results, adequate to deal with the indeterminism that ambiguity introduces in natural language processing. This work investigates different alternatives to implement an evolutionary bottom-up parser. Different genetic operators have been considered and evaluated. We focus on statistical parsing models to establish preferences among different parses. It is not our aim to propose a new statistical model for parsing but a new algorithm to perform the parsing once the model has been defined. The training data are extracted from syntactically annotated corpora (treebanks) which provide sets of lexical and syntactic tags as well as the grammar in which the parsing is based. We have tested the system with two corpora: Susanne and Penn Treebank, obtaining very encouraging results.     

一种采用混合搜索的检测器生成算法

检测器生成算法中采用随机搜索生成的检测器会产生大量重叠,而采用进化搜索收敛速度较慢.将两种搜索方式相结合,提出一种采用混合搜索的检测器生成算法,该算法将随机搜索产生的检测器集作为进化搜索的初始种群,使用遗传算法进化产生成熟检测器.使用二雏人工数据测试算法.结果表明该算法能够以更少的检测器更精确地覆盖非自体空间,并能提升收敛速度.     

Single and parallel machine capacitated lotsizing and scheduling: New iterative MIP-based neighborhood search heuristics

We propose a general-purpose heuristic approach combining metaheuristics and mixed integer programming to find high quality solutions to the challenging single- and parallel-machine capacitated lotsizing and scheduling problem with sequence-dependent setup times and costs. Commercial solvers fail to solve even medium-sized instances of this NP-hard problem; therefore, heuristics are required to find competitive solutions. We develop construction, improvement and search heuristics all based on MIP formulations. We then compare the performance of these heuristics with those of two metaheuristics and other MIP-based heuristics that have been proposed in the literature, and to a state-of-the-art commercial solver. A comprehensive set of computational experiments shows the effectiveness and efficiency of the main approach, a stochastic MIP-based local search heuristic, in solving medium to large size problems. Our solution procedures are quite flexible and may easily be adapted to cope with model extensions or to address different optimization problems that arise in practice.     

Evolutionary-based heuristic generators for checkers and give-away checkers

Abstract: Two methods of genetic evolution of linear and non-linear heuristic evaluation functions for the game of checkers and give-away checkers are presented in the paper. The first method is based on the simplistic assumption that a relation 'close' to partial order can be defined over the set of evaluation functions. Hence an explicit fitness function is not necessary in this case and direct comparison between heuristics (a tournament) can be used instead. In the other approach a heuristic is developed step-by-step based on the set of training games. First, the end-game positions are considered and then the method gradually moves 'backwards' in the game tree up to the starting position and at each step the best fitted specimen from the previous step (previous game tree depth) is used as the heuristic evaluation function in the alpha-beta search for the current step. Experimental results confirm that both approaches lead to quite strong heuristics and give hope that a more sophisticated and more problem-oriented evolutionary process might ultimately provide heuristics of quality comparable to those of commercial programs.     

Multi-objective vehicle routing problem with time windows using goal programming and genetic algorithm

This paper presents a new model and solution for multi-objective vehicle routing problem with time windows (VRPTW) using goal programming and genetic algorithm that in which decision maker specifies optimistic aspiration levels to the objectives and deviations from those aspirations are minimized. VRPTW involves the routing of a set of vehicles with limited capacity from a central depot to a set of geographically dispersed customers with known demands and predefined time windows. This paper uses a direct interpretation of the VRPTW as a multi-objective problem where both the total required fleet size and total traveling distance are minimized while capacity and time windows constraints are secured. The present work aims at using a goal programming approach for the formulation of the problem and an adapted efficient genetic algorithm to solve it. In the genetic algorithm various heuristics incorporate local exploitation in the evolutionary search and the concept of Pareto optimality for the multi-objective optimization. Moreover part of initial population is initialized randomly and part is initialized using Push Forward Insertion Heuristic and λ-interchange mechanism. The algorithm is applied to solve the benchmark Solomon's 56 VRPTW 100-customer instances. Results show that the suggested approach is quiet effective, as it provides solutions that are competitive with the best known in the literature.     

AND/OR Branch-and-Bound search for combinatorial optimization in graphical models

This is the first of two papers presenting and evaluating the power of a new framework for combinatorial optimization in graphical models, based on AND/OR search spaces. We introduce a new generation of depth-first Branch-and-Bound algorithms that explore the AND/OR search tree using static and dynamic variable orderings. The virtue of the AND/OR representation of the search space is that its size may be far smaller than that of a traditional OR representation, which can translate into significant time savings for search algorithms. The focus of this paper is on linear space search which explores the AND/OR search tree. In the second paper we explore memory intensive AND/OR search algorithms. In conjunction with the AND/OR search space we investigate the power of the mini-bucket heuristics in both static and dynamic setups. We focus on two most common optimization problems in graphical models: finding the Most Probable Explanation in Bayesian networks and solving Weighted CSPs. In extensive empirical evaluations we demonstrate that the new AND/OR Branch-and-Bound approach improves considerably over the traditional OR search strategy and show how various variable ordering schemes impact the performance of the AND/OR search scheme.     

GRADIENT: Grammar-driven genetic programming framework for building multi-component,hierarchical predictive systems

This work presents the GRADIENT (GRAmmar-DrIven ENsemble sysTem) framework for the generation of hybrid multi-level predictors for function approximation and regression analysis tasks. The proposed model uses a context-free grammar guided genetic programming for the automatic building of multi-component prediction systems with hierarchical structures. A multi-population evolutionary algorithm together with resampling and cross-validatory approaches are used to increase component models’ diversity and facilitate more robust and efficient search for accurate solutions. The system has been tested on a number of synthetic and publicly available real-world regression and time series problems for a range of configurations in order to identify and subsequently illustrate and discuss its characteristics and performance. GRADIENT has been shown to be very competitive and versatile when compared to a number of state-of-the-art prediction methods.     

A genetic algorithm for the design space exploration of datapaths during high-level synthesis

High-level synthesis is comprised of interdependent tasks such as scheduling, allocation, and module selection. For today's very large-scale integration (VLSI) designs, the cost of solving the combined scheduling, allocation, and module selection problem by exhaustive search is prohibitive. However, to meet design objectives, an extensive design space exploration is often critical to obtaining superior designs. We present a framework for efficient design space exploration during high-level synthesis of datapaths for data-dominated applications. The framework uses a genetic algorithm (GA) to concurrently perform scheduling and allocation with the aim of finding schedules and module combinations that lead to superior designs while considering user-specified latency and area constraints. The GA uses a multichromosome representation to encode datapath schedules and module allocations and efficient heuristics to minimize functional and storage area costs, while minimizing circuit latencies. The framework provides the flexibility to perform resource-constrained scheduling, time-constrained scheduling, or a combination of the two, using a simple and fast list-scheduling technique. A graded penalty function is used as an objective function in evaluating the quality of designs to enable the GA to quickly reach areas of the search space where designs meeting user specified criteria are most likely to be found. Since GAs are population-based search heuristics, a unique feature of our framework is its ability to offer a large number of alternative datapath designs, all of which meet design specifications but differ in module, register, and interconnect configurations. Many experiments on well-known benchmarks show the effectiveness of our approach.     

Group-scheduling problems in electronics manufacturing

This paper addresses the flowshop group-scheduling problems typically encountered in the assembly of printed circuit boards in electronics manufacturing. A mathematical programming model is formulated to capture the characteristics inherent to group-scheduling problems experienced in electronics manufacturing as well as those common to a wide range of group-scheduling problems encountered in other production environments. Several heuristics, each incorporating different components that underlie the tabu search concept, are developed to solve this strongly NP-hard problem effectively in a timely manner. In order to investigate the quality of the heuristic solutions with respect to tight lower bounds, an effective and efficient decomposition approach is developed. The problem is decomposed into a master problem and single-machine subproblems, and a column generation algorithm is developed to solve the linear programming relaxation of the master problem. Branching schemes, compatible with the column generation subproblems, are employed to partition the solution space when the solution to the linear programming relaxation is not integral. Furthermore, tabu search based fast heuristics are implemented to solve the subproblems, and an effective stabilization method is developed to accelerate the column generation approach. An experimental design with both fixed and random factors accompanied by rigorous statistical analyses of computational tests conducted on randomly generated test problems as well as on a large size real industry problem confirm the high performance of the proposed approach in identifying quality lower bounds and strongly suggest its flexibility and applicability to a wide range of real problems.     

Regularization approach to inductive genetic programming

This paper presents an approach to regularization of inductive genetic programming tuned for learning polynomials. The objective is to achieve optimal evolutionary performance when searching high-order multivariate polynomials represented as tree structures. We show how to improve the genetic programming of polynomials by balancing its statistical bias with its variance. Bias reduction is achieved by employing a set of basis polynomials in the tree nodes for better agreement with the examples. Since this often leads to over-fitting, such tendencies are counteracted by decreasing the variance through regularization of the fitness function. We demonstrate that this balance facilitates the search as well as enables discovery of parsimonious, accurate, and predictive polynomials. The experimental results given show that this regularization approach outperforms traditional genetic programming on benchmark data mining and practical time-series prediction tasks