Reinforcement Learning Based Algorithms for Average Cost Markov Decision Processes

This article proposes several two-timescale simulation-based actor-critic algorithms for solution of infinite horizon Markov Decision Processes with finite state-space under the average cost criterion. Two of the algorithms are for the compact (non-discrete) action setting while the rest are for finite-action spaces. On the slower timescale, all the algorithms perform a gradient search over corresponding policy spaces using two different Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates. On the faster timescale, the differential cost function corresponding to a given stationary policy is updated and an additional averaging is performed for enhanced performance. A proof of convergence to a locally optimal policy is presented. Next, we discuss a memory efficient implementation that uses a feature-based representation of the state-space and performs TD(0) learning along the faster timescale. The TD(0) algorithm does not follow an on-line sampling of states but is observed to do well on our setting. Numerical experiments on a problem of rate based flow control are presented using the proposed algorithms. We consider here the model of a single bottleneck node in the continuous time queueing framework. We show performance comparisons of our algorithms with the two-timescale actor-critic algorithms of Konda and Borkar (1999) and Bhatnagar and Kumar (2004). Our algorithms exhibit more than an order of magnitude better performance over those of Konda and Borkar (1999).

Continuous relaxation and local maxima selection: conditions for equivalence

Relaxation labeling processes are a class of iterative algorithms for using contextual information to reduce local ambiguities. This paper introduces a new perspective toward relaxation-that of considering it as a process for reordering labels attached to nodes in a graph. This new perspective is used to establish the formal equivalence between relaxation and another widely used algorithm, local maxima selection. The equivalence specifies conditions under which a family of cooperative relaxation algorithms, which generalize the well-known ones, decompose into purely local ones. Since these conditions are also sufficient for guaranteeing the convergence of relaxation processes, they serve as stopping criteria. We feel that equivalences such as these are necessary for the proper application of relaxation and maxima selection in complex speech and vision understanding systems.     

Bathymetric study of Lake Hayq,Ethiopia

The generation of scientific information for improved understating of the physical dynamics of a lake is fundamental for guiding lake stakeholders and managers at the local level to implement best management practices and help design effective management strategies and policies at higher levels. Multitemporal bathymetric information on lakes is very important in hydrology and sediment studies to more clearly indicate environmental changes and to understand the effects of land processes on the hydrology of lakes. Accordingly, the purpose of this study was to map bathymetric charts of Lake Hayq ¹ in Ethiopia and to derive morphometric parameters, including depth, volume, area, width and length, and to plot curves illustrating the relationships between these parameters. The bathymetric survey was carried out using a combination of a SonarLite Portable Echo Sounder and Global Positioning System (GPS) to generate three‐dimensional (XYZ) hydrographic data. Surfer 8.01 and ArcGIS 9.3 software program were used for surface, gridding and morphometric analyses. Comparison of the results of this study with a previous study conducted in 1941 indicated the lake has experienced changes in depth and surface area. To reduce the negative impacts of human‐induced activities on the ecohydrology of the lake, and to maintain its ecological integrity, appropriate and integrated lake management practices must be adopted. This will necessitate policy formulation, active lake basin stakeholder involvement and implementation of basin‐wide lake management to ensure sustainable use of the lake and its basin resources.     

Monte Carlo simulation on tensile failure process of unidirectional carbon fiber reinforced nano-phased epoxy

This paper presents an analytical approach which combines the modified shear-lag model and Monte Carlo simulation technique to simulate the tensile failure process of unidirectional T700 carbon reinforced composite. Two kinds of matrix were investigated in the present paper, one is neat epoxy and the other one is SiC nano-particle filled epoxy. In the model, the strength of the fiber elements is randomly allocated by the Monte Carlo method, the elastic properties of the matrix elements and the friction after the interfaces breakage are definitely allocated. Using this model, the deformation, damage and failure process of the composite are simulated on the microscopic level, the tensile stress–strain relationship is well predicted. The relationship between mechanical properties of the fiber, matrix and composites was discussed. The analysis also shows that, compared with the neat system, nano-phased composite exhibits 10% improvement in tensile strength, which agrees with the experimental data.     

Characteristics of Metal Droplets in Slag Tapped from the Blast Furnace

Slag samples, hot‐metal samples and hot‐metal temperatures were obtained during tapping of two blast furnaces. Sampling was carried out at different time points during tapping of three separate heats. The size distribution and composition of metal droplets found in the slag were determined using scanning electron microscopy. Only metal droplets above 0.75 μm could be counted and analysed. All droplets were below 8 μm in diameter and the great majority of these droplets were found to be between 0.75 and 2 μm. The size distribution did not differ significantly for different slag samples. Iron was the main droplet component. Electron probe microanalysis showed that the droplets contained small amounts of carbon. The percentage of the area in a studied cross‐section that was covered with metal droplets varied between 0.01 and 0.07%. Calculations based on Stoke's law showed that the distance droplets travel in the slag is in the micron range. Slag samples taken in the beginning of slag tapping contained more droplets than those taken in the middle of slag tapping, an indication that most droplets can be found in the area near the furnace wall. Some droplets were determined to have magnesium enrichment at the external surface.     

Impact of the Boeing 767 Aircraft into the World Trade Center

A numerical simulation of the aircraft impact into the exterior columns of the World Trade Center (WTC) was done using LS-DYNA. For simplification, the fuselage was modeled as a thin-walled cylinder, the wings were modeled as box beams with a fuel pocket, and the engines were represented as rigid cylinders. The exterior columns of the WTC were represented as box beams. Actual masses, material properties and dimensions of the Boeing 767 aircraft and the exterior columns of the WTC were used in this analysis. It was found that about 46% of the initial kinetic energy of the aircraft was used to damage columns. The minimum impact velocity of the aircraft to just penetrate the exterior columns would be 130?m/s. It was also found that a Boeing 767 traveling at top speed would not penetrate exterior columns of the WTC if the columns were thicker than 20?mm.     

Mining Non-Redundant Association Rules

The traditional association rule mining framework produces many redundant rules. The extent of redundancy is a lot larger than previously suspected. We present a new framework for associations based on the concept of closed frequent itemsets. The number of non-redundant rules produced by the new approach is exponentially (in the length of the longest frequent itemset) smaller than the rule set from the traditional approach. Experiments using several “hard” as well as “easy” real and synthetic databases confirm the utility of our framework in terms of reduction in the number of rules presented to the user, and in terms of time.     

Study of the impact of a new hurdle technology composed of electro‐activated solution and low heat treatment on the canned pea and corn quality and microbial safety

The objective of this work was to evaluate the quality of sterilised canned pea and corn in electro‐activated brine solutions at moderate temperatures. The lowest change in vitamin C was associated with the lowest heat treatment, while the short treatment time resulted in significant changes in texture and colour of vegetables. Best texture profile was obtained with the acid electro‐activated brine solution for pea and corn. The neutral electro‐activated brine solution resulted in a less firm texture for pea and corn. The green colour and brightness of canned pea were higher when neutral solution was used (a* = ?8.4 ± 0.3) than for the acidic one (a* = ?3.7 ± 0.6). The yellowness of corn was better with the neutral electro‐activated brine solution (b* = 36.32 ± 1.24) than with the acidic one (b* = 28.44 ± 2.39). Thirty‐three percent decrease of energy consumption was obtained using the electro‐activation technology.     

Amino acid composition,foaming, emulsifying properties and surface hydrophobicity of mustard protein isolate as affected by pH and NaCl

Amino acid composition, protein hydrophobicity, foaming and emulsifying properties of mustard protein isolate at pH 3, 5, 7 and in 0.05 and 0.1 m NaCl were studied. Glutamic (19.18 ± 0.30%) and aspartic (7.49 ± 0.11%) acids were the dominants. Foaming ability was enhanced by NaCl. Time to reach 75 mL foam was 23% higher in water than in NaCl. Drained volume after 10 min was concentration dependent and was the lowest in 0.05 and 0.1 m NaCl at protein concentration of 2.5% and 5%. The emulsifying properties were pH and concentration dependent, and the best results were obtained at pH 3, corresponding to the highest positive charge density of the protein surface. The highest emulsion stability (90.22 ± 3.52%) was obtained in 0.05 m NaCl and 5% protein concentration, whereas the lowest (63.00 ± 1.06%) was in water at all protein concentrations. Protein hydrophobicity was low and depended of pH but not of NaCl.