A review of stochastic algorithms with continuous value function approximation and some new approximate policy iteration algorithms for multidimensional continuous applications

We review the literature on approximate dynamic programming, with the goal of better understanding the theory behind practical algorithms for solving dynamic programs with continuous and vector-valued states and actions and complex information processes. We build on the literature that has addressed the well-known problem of multidimensional (and possibly continuous) states, and the extensive literature on model-free dynamic programming, which also assumes that the expectation in Bellman’s equation cannot be computed. However, we point out complications that arise when the actions/controls are vector-valued and possibly continuous. We then describe some recent research by the authors on approximate policy iteration algorithms that offer convergence guarantees (with technical assumptions) for both parametric and nonparametric architectures for the value function.     

C-PSA: Constrained Pareto simulated annealing for constrained multi-objective optimization

In recent years, evolutionary algorithms (EAs) have been extensively developed and utilized to solve multi-objective optimization problems. However, some previous studies have shown that for certain problems, an approach which allows for non-greedy or uphill moves (unlike EAs), can be more beneficial. One such approach is simulated annealing (SA). SA is a proven heuristic for solving numerical optimization problems. But owing to its point-to-point nature of search, limited efforts has been made to explore its potential for solving multi-objective problems. The focus of the presented work is to develop a simulated annealing algorithm for constrained multi-objective problems. The performance of the proposed algorithm is reported on a number of difficult constrained benchmark problems. A comparison with other established multi-objective optimization algorithms, such as infeasibility driven evolutionary algorithm (IDEA), Non-dominated sorting genetic algorithm II (NSGA-II) and multi-objective Scatter search II (MOSS-II) has been included to highlight the benefits of the proposed approach.     

Free-living inferential modeling of blood glucose level using only noninvasive inputs

The goal of this work is to present a causation modeling methodology with the ability to accurately infer blood glucose levels using a large set of highly correlated noninvasive input variables over an extended period of time. These models can provide insight to improve glucose monitoring, and glucose regulation through advanced model-based control technologies. The efficacy of this approach is demonstrated using real data from a type 2 diabetic (T2D) subject collected under free-living conditions over a period of 25 consecutive days. The model was identified and tested using eleven variables that included three food variables as well as several activity and stress variables. The model was trained using 20 days of data and validated using 5 days of data. This gave a fitted correlation coefficient of 0.70 and an average absolute error (AAE) (i.e., the average of the absolute values for the measured glucose concentration minus modeled glucose concentration) of 13.3 mg/dL for the validation data. This AAE result was significantly better than the subject’s personal glucose meter AAE of 15.3 mg/dL for replicated measurements.     

Minimizing total tardiness in a stochastic single machine scheduling problem using approximate dynamic programming

This paper addresses the non-preemptive single machine scheduling problem to minimize total tardiness. We are interested in the online version of this problem, where orders arrive at the system at random times. Jobs have to be scheduled without knowledge of what jobs will come afterwards. The processing times and the due dates become known when the order is placed. The order release date occurs only at the beginning of periodic intervals. A customized approximate dynamic programming method is introduced for this problem. The authors also present numerical experiments that assess the reliability of the new approach and show that it performs better than a myopic policy.     

Multi-objective optimization of multicast overlays for collaborative applications

Current real-time collaborative application implementations use dedicated infrastructure to carry out all communication and synchronization activities. Specifically, they require all end nodes to communicate directly with and through the central server. In this paper, we investigate scenarios in which the most resource intensive functionality of continuous communication among collaborators to disseminate changes is decentralized, utilizing the end users themselves as relays. We observe that communication characteristics of real-time collaboration makes use of existing multicast mechanisms unsuitable. As collaborative editing sessions are typically long and repeated, it is possible to gather and leverage node behavior (their instabilities and frequency of sending updates) and communication links (latencies and average costs). Several criteria to determine the quality of a multicast tree can be identified: cost, latency and instability. In this paper, we analyze the complexity of the problem of finding optimal communication topologies, and propose approximate algorithms to optimize the same. We also consider the multiobjective problem in which we search for a topology that provides good trade-off between these sometimes conflicting measures. Validation of our proposed algorithms on numerous graphs shows that it is important to consider the multiobjective problem, as optimal solutions for one performance measure can be far from optimal for the other metrics. Finally, we briefly present an implementation of a communication library that uses the proposed algorithms to periodically adjust the dissemination tree.     

Two hybrid differential evolution algorithms for engineering design optimization

This paper presents two hybrid differential evolution algorithms for optimizing engineering design problems. One hybrid algorithm enhances a basic differential evolution algorithm with a local search operator, i.e., random walk with direction exploitation, to strengthen the exploitation ability, while the other adding a second metaheuristic, i.e., harmony search, to cooperate with the differential evolution algorithm so as to produce the desirable synergetic effect. For comparison, the differential evolution algorithm that the two hybrids are based on is also implemented. All algorithms incorporate a generalized method to handle discrete variables and Deb's parameterless penalty method for handling constraints. Fourteen engineering design problems selected from different engineering fields are used for testing. The test results show that: (i) both hybrid algorithms overall outperform the differential evolution algorithms; (ii) among the two hybrid algorithms, the cooperative hybrid overall outperforms the other hybrid with local search; and (iii) the performance of proposed hybrid algorithms can be further improved with some effort of tuning the relevant parameters.     

Trends in youth’s videogame playing, overall computer use, and communication technology use: The impact of self-esteem and the Big Five personality factors

The objective of the current research is to model trends in video game playing, overall computer use, and communication technology use in a longitudinal sample of youths, aged 11-16 over a 3-year interval. In addition, individual difference characteristics that may be predictive of these trends were included, namely, socio-demographic characteristics (gender, ethnicity, and parental income) and personality characteristics (self-esteem, the Big Five personality factors). Findings suggested that youth increased their overall computer and communication technology use but decreased their videogame playing over time. Many individual differences predicted mean levels of these technologies with fewer predicting slopes. Conclusions, implications, and limitations are discussed.     

What happens next? Predicting other road users' behaviour as a function of driving experience and processing time

Hazard perception is one of the most important facets of driving and if the appropriate diagnostic tool is used it can discriminate between novice and experienced drivers. In this study video clips of actual driving scenarios were shown to novice and experienced drivers. The clips were stopped just prior to hazard onset and either the screen went black or the final still image stayed on the screen. Participants were then asked five questions about what happened next. This variant of the hazard perception test allowed the influence of processing time to be included and the level of situation awareness to be measured. Experienced drivers significantly anticipated more correct hazardous outcomes than novice drivers when the screen went black. Novice drivers benefited from the extra processing time afforded by the image remaining on the screen and significantly anticipated more hazards when the image remained on the screen than when it went black. The findings indicate that when processing time is manipulated, hazard perception accuracy reveals experiential differences. These differences are discussed with reference to hazard perception and situation awareness. This research informs the current controversy over whether hazard perception is a good diagnostic tool for driving performance. It identifies potential confounds in previous work and demonstrates that experiential differences can be found if the appropriate tests are used. Further, it suggests improvements for new hazard perception tests.     

Using knowledge structures to redesign an instructor-operator station

Frequently, user interface (UI) designers must choose between modifying an established, but suboptimal and familiar, UI or to avoid such changes. Changing the UI’s, organization may frustrate users who have become familiar with the original design, whereas failing to make changes may force users to perform at an unsatisfactory level. This paper presents two studies that investigate whether users familiar with a poorly designed UI would find items faster, and prefer a reorganized UI that conformed to domain expert knowledge, or would their familiarity with the original UI yield faster performance and higher satisfaction.This paper describes activities to redesign a menu structure in a simulator instructor–operator station (IOS) using hierarchical card sorting and cluster analysis (Romesburg, 2004). This analysis was used to reorganize the menu structure to reflect the knowledge representations of domain experts in accordance with the principle of proximity compatibility (Wickens and Carswell, 1995, Rothrock et al., 2006). The new design was validated with a separate set of users by a reaction time experiment and preference selection.