A multiagent approach to managing air traffic flow

Intelligent air traffic flow management is one of the fundamental challenges facing the Federal Aviation Administration (FAA) today. FAA estimates put weather, routing decisions and airport condition induced delays at 1,682,700 h in 2007 (FAA OPSNET Data, US Department of Transportation website, ), resulting in a staggering economic loss of over $41 billion (Joint Economic Commission Majority Staff, Your flight has been delayed again, 2008). New solutions to the flow management are needed to accommodate the threefold increase in air traffic anticipated over the next two decades. Indeed, this is a complex problem where the interactions of changing conditions (e.g., weather), conflicting priorities (e.g., different airlines), limited resources (e.g., air traffic controllers) and heavy volume (e.g., over 40,000 flights over the US airspace) demand an adaptive and robust solution. In this paper we explore a multiagent algorithm where agents use reinforcement learning (RL) to reduce congestion through local actions. Each agent is associated with a fix (a specific location in 2D space) and has one of three actions: setting separation between airplanes, ordering ground delays or performing reroutes. We simulate air traffic using FACET which is an air traffic flow simulator developed at NASA and used extensively by the FAA and industry. Our FACET simulations on both artificial and real historical data from the Chicago and New York airspaces show that agents receiving personalized rewards reduce congestion by up to 80% over agents receiving a global reward and by up to 90% over a current industry approach (Monte Carlo estimation).     

Efficient evaluation functions for evolving coordination

Abstract This paper presents fitness evaluation functions that efficiently evolve coordination in large multi-component systems. In particular, we focus on evolving distributed control policies that are applicable to dynamic and stochastic environments. While it is appealing to evolve such policies directly for an entire system, the search space is prohibitively large in most cases to allow such an approach to provide satisfactory results. Instead, we present an approach based on evolving system components individually where each component aims to maximize its own fitness function. Though this approach sidesteps the exploding state space concern, it introduces two new issues: (1) how to create component evaluation functions that are aligned with the global evaluation function; and (2) how to create component evaluation functions that are sensitive to the fitness changes of that component, while remaining relatively insensitive to the fitness changes of other components in the system. If the first issue is not addressed, the resulting system becomes uncoordinated; if the second issue is not addressed, the evolutionary process becomes either slow to converge or worse, incapable of converging to good solutions. This paper shows how to construct evaluation functions that promote coordination by satisfying these two properties. We apply these evaluation functions to the distributed control problem of coordinating multiple rovers to maximize aggregate information collected. We focus on environments that are highly dynamic (changing points of interest), noisy (sensor and actuator faults), and communication limited (both for observation of other rovers and points of interest) forcing the rovers to evolve generalized solutions. On this difficult coordination problem, the control policy evolved using aligned and component-sensitive evaluation functions outperforms global evaluation functions by up to 400%. More notably, the performance improvements increase when the problems become more difficult (larger, noisier, less communication). In addition we provide an analysis of the results by quantifying the two characteristics (alignment and sensitivity discussed above) leading to a systematic study of the presented fitness functions.     

Analyzing and visualizing multiagent rewards in dynamic and stochastic domains

The ability to analyze the effectiveness of agent reward structures is critical to the successful design of multiagent learning algorithms. Though final system performance is the best indicator of the suitability of a given reward structure, it is often preferable to analyze the reward properties that lead to good system behavior (i.e., properties promoting coordination among the agents and providing agents with strong signal to noise ratios). This step is particularly helpful in continuous, dynamic, stochastic domains ill-suited to simple table backup schemes commonly used in TD(λ)/Q-learning where the effectiveness of the reward structure is difficult to distinguish from the effectiveness of the chosen learning algorithm. In this paper, we present a new reward evaluation method that provides a visualization of the tradeoff between the level of coordination among the agents and the difficulty of the learning problem each agent faces. This method is independent of the learning algorithm and is only a function of the problem domain and the agents’ reward structure. We use this reward property visualization method to determine an effective reward without performing extensive simulations. We then test this method in both a static and a dynamic multi-rover learning domain where the agents have continuous state spaces and take noisy actions (e.g., the agents’ movement decisions are not always carried out properly). Our results show that in the more difficult dynamic domain, the reward efficiency visualization method provides a two order of magnitude speedup in selecting good rewards, compared to running a full simulation. In addition, this method facilitates the design and analysis of new rewards tailored to the observational limitations of the domain, providing rewards that combine the best properties of traditional rewards.     

A case for trading risk in complex conceptual design trade studies

Complex conceptual system design trade studies traditionally consider risk after a conceptual design has been created. Further, one person is often tasked with collecting risk information and managing it from each subsystem. This paper proposes a method to explicitly consider and trade risk on the same level as other important system-level variables during the creation of conceptual designs in trade studies. The proposed risk trading method advocates putting each subsystem engineer in control of risk for each subsystem. A risk vector is proposed that organizes many different risk metrics for communication between subsystems. A method of coupling risk models to dynamic subsystem models is presented. Several risk visualization techniques are discussed. A trade study example is presented based upon a simplified spacecraft model. Results from introducing the risk trading methodology into a simulated Collaborative Design Center are presented. The risk trading method offers an approach to more thoroughly consider risk during the creation of conceptual designs in trade studies.     

A functional failure reasoning methodology for evaluation of conceptual system architectures

In this paper, we introduce a new methodology for reasoning about the functional failures during early design of complex systems. The proposed approach is based on the notion that a failure happens when a functional element in the system does not perform its intended task. Accordingly, a functional criticality is defined depending on the role of functionality in accomplishing designed tasks. A simulation-based failure analysis tool is then used to analyze functional failures and reason about their impact on overall system functionality. The analysis results are then integrated into an early stage system architecture analysis framework that analyzes the impact of functional failures and their propagation to guide system-level architectural design decisions. With this method, a multitude of failure scenarios can be quickly analyzed to determine the effects of architectural design decisions on overall system functionality. Using this framework, design teams can systematically explore risks and vulnerabilities during the early (functional design) stage of system development prior to the selection of specific components. Application of the presented method to the design of a representative aerospace electrical power system (EPS) testbed demonstrates these capabilities.     

A Comparison of the Drying Simulation Codes Transpore and Wood2D which are used for the Modelling of Two-Dimensional Wood Drying Processes

This study is devoted to investigating the heat and mass transfer phenomena that occur during the convective drying of wood at high temperatures. A comparison will be made between an existing two-dimensional computer code known as Transpore. which was developed by Perre in France, and another two-dimensional code which was developed independently by Turner in Australia. Both numerical codes use a comprehensive set of macroscopic equations to describe the drying process, and most importantly treat the anisotropic behaviour of the wood. The porous medium is defined by three state variables: temperature, moisture content and gaseous pressure and the numerical simulation codes allow the evolution of the distributions of these state variables to be analysed throughout the drying process. The numerical investigation presented in this research work will compare the results obtained from both simulation codes and comments will be made on their consistencies. The influence that the drying air characteristics (moist air and super-heated stream) have on the overall drying kinetics, together with the effect that varying the mesh structure or changing the relative permeability curves have on the results will be throughly scrutinised.     

Plants transformed with a mutant alfalfa mosaic virus coat protein gene are resistant to the mutant but not to wild-type virus

Transgenic tobacco plants expressing the wild-type (wt) coat protein (CP) gene of alfalfa mosaic virus (AIMV) have been shown to be resistant to infection with viral particles and RNAs or to infection with viral particles only. The difference in resistance of these plants to RNA inocula was found to correlate with a difference in the expression level of the transgene. Plants expressing a mutant AIMV CP with the N-terminal serine residue changed to glycine have been shown to be susceptible to infection with wt viral particles or RNAs. By site-directed mutagenesis of AIMV cDNA a viable mutant virus encoding CP with the same N-terminal mutation was obtained. Plants expressing wt or mutant CP were resistant to the mutant virus, demonstrating that a single amino acid substitution in CP did not permit the virus to overcome CP-mediated resistance. Although the mutant CP did not confer resistance to wt virus when expressed in transgenic plants, it was still effective in classical cross-protection: plants infected with the mutant virus were resistant to severe strain of AIMV. A model to explain the data is discussed.     

Chaperone protein GrpE and the GroEL/GroES complex promote the correct folding of tobacco mosaic virus coat protein for ribonucleocapsid assembly in vivo

Several prokaryotic chaperone proteins were shown to promote the correct folding and in vivo assembly of tobacco mosaic virus coat protein (TMV CP) using a chimaeric RNA packaging system in control or chaperone-deficient mutant strains of Escherichia coli. Mutations in groEL or dnaK reduced the amount of both total and soluble TMV CP, and the yield of assembled TMV-like particles, several-fold. Thus both GroEL and DnaK have significant direct or indirect effects on the overall expression, stability, folding and assembly of TMV CP in vivo. In contrast, while cells carrying a mutation in grpE expressed TMV CP to a higher overall level than control E. coli, the amounts of both soluble CP and assembled TMV-like particles were below control levels, suggesting a negative effect of GrpE on overall CP accumulation, but positive role(s) in CP folding and assembly. Curiously, cells with mutations in groES and, to a lesser extent, dnaJ expressed total, soluble and assembled forms of TMV CP significantly above control values, suggesting some form of negative control by these chaperone proteins. To avoid pleiotropic effects or artefacts in chaperone-null mutants, selected chaperone proteins were also over-expressed in control E. coli cells. Overproduction of GroEL or GroES alone had little effect. However, co-overexpression of GroEL and GroES resulted in a two-fold increase in soluble TMV CP and a four-fold rise in assembled TMV-like (pseudovirus) particles in vivo. Moreover, TMV CP was shown to interact directly with GroEL in vivo. Together, these results suggest that GrpE and the GroEL/GroES chaperone complex promote the correct folding and assembly of TMV CP into ribonucleocapsids in vivo.     

Pharmacokinetic features, immunogenicity, and toxicity of B43(anti-CD19)-pokeweed antiviral protein immunotoxin in cynomolgus monkeys

Withdrawal responses to mechanical and thermal stimuli applied to the plantar surface of the hindpaw were measured before and after bone damage. In separate groups of rats the bone was injured by scraping the periosteum of the tibia, drilling a hole through the tibia, aspirating bone marrow, or drilling a hole through the calcaneus. Scraping the periosteum did not alter withdrawal responses to the mechanical stimuli, or evoke nocifensive behavior. In contrast, secondary mechanical hyperalgesia and allodynia, and cold allodynia were observed after a hole was drilled through the tibia or calcaneus and after aspiration of bone marrow. The secondary hyperalgesia peaked at 2 h after injury. Drilling a hole through the calcaneus permitted primary hyperalgesia to be easily quantified. Primary hyperalgesia lasted up to 24 h after injury. Nocifensive behavior characterized by a lifting and guarding of the damaged limb was also observed after a hole was drilled through the tibia or calcaneus. Drilling a hole through the tibia or calcaneus should be a useful experimental model for investigating the mechanisms underlying bone pain.