Iterative augmentation of a medical training simulator: Effects of affective metacognitive scaffolding

Experiential training simulators are gaining increasing popularity for job-related training due to their potential to engage and motivate adult learners. They are designed to provide learning experiences that are directly connected to users' work environments and support self-regulated learning. Nevertheless, learners often fail to transfer the knowledge gained in the simulated environment to real-world contexts. The EU-funded ImREAL project aimed to bridge that gap by developing a suite of intelligent services designed to enhance existing training simulators. This paper presents work that was a subset of this research project, reporting the iterative development and evaluation of a scaffolding service, which was integrated into a simulator for training medical students to perform diagnostic interviews. The study comprises three evaluation phases, comparing the pure simulator to a first version with metacognitive scaffolding and then to a final version with affective metacognitive scaffolding and enriched user modelling. The scaffolding service provides the learner with metacognitive prompts; affective elements are realized by an integrated affect reporting tool and affective prompts. Using a mixed-method approach by analysing questionnaires (N = 106) and log-data (N = 426), the effects of the services were investigated with respect to real-world relevance, self-regulated learning support, learning experience, and integration. Despite some limitations, the outcomes of this study demonstrate the usefulness of affective metacognitive scaffolding in the context of experiential training simulators; significant post-simulation increases in perceived relevance of the simulator, reflective note-taking, overall motivation, and feeling of success could be identified. Perceived usability and flow of the simulation increased, whereas overall workload and frustration decreased. However, low response rates to specific functions of the simulation point to a need to further investigate how to raise users' awareness and understanding of the provided tools, to encourage interaction with the services, and to better convey the benefits of using them. Thus, future challenges concern not so much technological developments for personalizing learning experiences, but rather new ways to change user attitudes towards an open approach to learning systems that enables them to benefit from all offered features.     

Revealing the hidden structure of physiological states during metacognitive monitoring in collaborative learning

Using hidden Markov models (HMM), the current study looked at how learners' metacognitive monitoring is related to their physiological reactivity in the context of collaborative learning. The participants (N = 12, age 16–17 years, three females and nine males) in the study were high school students enrolled in an advanced physics course. The results show that during collaborative learning, the students engaged in monitoring in each self-regulated learning phase such as task understanding, planning and goal setting, task enactment, adaptation and reflection. The results of the HMM indicated that the learners' physiological reactivity was low when monitoring occurred. The associations between the states based on the HMM provide insights not only into how learners engage in metacognitive monitoring but also about their level of physiological reactivity in each state. In conclusion, exploring aspects of metacognitive monitoring in collaborative learning can be done with the help of physiological reactions.     

Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing

There is an agreement that perceived usability is important beyond actual effectiveness of software systems. Perceived usability is often obtained by self-reports provided after system use. Aiming to improve summative usability testing, we propose a methodology to enhance in-depth testing of users' performance and perceived usability at the task level. The metacognitive research approach allows detailed analysis of cognitive processes. Adapting its methodologies, we propose the Metacognitive Usability Profile (MUP) which includes a comprehensive set of measures based on collecting confidence in the success of each particular task and triangulating it with objective measures. We demonstrate using the MUP by comparing two versions of a project management system. Based on a task analysis we allocated tasks that differ between the versions and let participants (N = 100) use both versions. Although no difference was found between the versions in system-level perceived usability, the detailed task-level analysis exposed many differences. In particular, overconfidence was associated with low performance, which suggests that user interfaces better avoid illusions of knowing. Overall, the study demonstrates how the MUP exposes challenges users face. This, in turn, allows choosing the better task implementation among the examined options and to focus attempts for usability improvement.     

The impact of on-demand metacognitive help on effortful behaviour: A longitudinal study using task-related visual analytics

This longitudinal study investigates the differences in learners' effortful behaviour over time due to receiving metacognitive help—in the form of on-demand task-related visual analytics. Specifically, learners' interactions (N = 67) with the tasks were tracked during four self-assessment activities, conducted at four discrete points in time, over a period of 8 weeks. The considered and coded time points were: (a) prior to providing the metacognitive help; (b) while the task-related visual analytics were available (treatment); (c) after the removal of the treatment; and (d) while the option to receive metacognitive help was available again. To measure learners' effortful behaviour across the self-assessment activities, this study utilized learners' response-times to correctly/wrongly complete the tasks and on-task effort expenditure. The panel data analysis shown that the usage of metacognitive help caused statistically significant changes in learners' effortful behaviour, mostly in the third and fourth phase. Statistically significant changes were detected also in the usage of metacognitive help. These results provide empirical evidence on the benefits of task-related visual analytics to support learners' on-task engagement, and suggest relevant cues on how metacognitive help could be designed and prompted by focusing on the “task”, instead of the “self”.     

Effect of metacognitive strategies and verbal-imagery cognitive style on biology-based video search and learning performance

Videos have diverse content that can assist students in learning. However, because videos are linear media, video users may take a longer time than readers of text to evaluate the context. Therefore, the process of video search may vary from one user to another depending on the users' individual characteristics, and the effectiveness of video learning may also vary across individuals. This study evaluated 100 Taiwanese fifth graders searching for videos related to “understanding animals” on YouTube and examined the effects of the students' metacognitive strategies (planning, monitoring, and evaluating) and verbal-imagery cognitive style on their video searches. The observable indicators were quantitative (search behaviors, search performance, and learning performance) and qualitative (search process observations and interviews). The study concludes that metacognitive strategy is the primary influencer of video search. Students with better metacognitive skills used fewer keywords, browsed fewer videos, and spent less time evaluating videos, but they achieved higher learning performance. They reviewed the video metadata information on the user interface and did not attempt to watch videos on the video recommendation lists, particularly videos that were irrelevant to the task requirements. During the course of the searches, keyword usage had a significant influence on the students' search performance and learning performance. The fewer keywords the students used, the better search and learning performance they were able to achieve. Our results are different from those of previous studies on text, image, and map searches. Accordingly, users must adopt different search strategies when using various types of search engines.     

Evaluating intelligent knowledge systems: experiences with a user-adaptive assistant agent

This article examines experiences in evaluating a user-adaptive personal assistant agent designed to assist a busy knowledge worker in time management. We examine the managerial and technical challenges of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system. The CALO project was a seminal multi-institution effort to develop a personalized cognitive assistant. It included a significant attempt to rigorously quantify learning capability, which this article discusses for the first time, and ultimately the project led to multiple spin-outs including Siri. Retrospection on negative and positive experiences over the 6 years of the project underscores best practice in evaluating user-adaptive systems. Lessons for knowledge system evaluation include: the interests of multiple stakeholders, early consideration of evaluation and deployment, layered evaluation at system and component levels, characteristics of technology and domains that determine the appropriateness of controlled evaluations, implications of ‘in-the-wild’ versus variations of ‘in-the-lab’ evaluation, and the effect of technology-enabled functionality and its impact upon existing tools and work practices. In the conclusion, we discuss—through the lessons illustrated from this case study of intelligent knowledge system evaluation—how development and infusion of innovative technology must be supported by adequate evaluation of its efficacy.     

Integration of metacognitive skills in the design of learning objects

Recent studies have demonstrated that specific instruction about metacognitive strategies improves achievement, the accuracy of knowledge monitoring, and the application of learning strategies in hypermedia environments. However, there are no models to date for instructional designers who design and develop learning objects for the incorporation of specific scaffolds to aid student reflection about their metacognitive skills; thus making it difficult to identify tasks to orientate learners in improving such skills. In this paper, we propose the use of specific ontologies as the basis for incorporating information about metacognition in learning objects so that a Learning Management System can select and recommend tasks designed for the development and/or improvement of the learners’ metacognitive skills within the context of e-learning.     

Effects of the metacognitive computer-tool met.a.ware on the web search of laypersons

Today, laypersons often consult the Internet to inform themselves about health-related issues. However, the competent use of these often complex and heterogeneous information provisions cannot be taken for granted, because many Internet users are lacking the necessary metacognitive prerequisites. Therefore, we developed the metacognitive computer-tool met.a.ware, which supports laypersons’ Internet research for medical information by the means of metacognitive prompting and ontological classification. In an experimental investigation of met.a.ware a total of 118 participants with little medical knowledge were asked to conduct an Internet research on a medical topic. Participants were randomly assigned to four experimental groups that worked with met.a.ware and either received evaluation prompts, monitoring prompts, both types of prompts, or no prompts. All experimental conditions were additionally provided with ontological classification. One control group took paper and pencil notes. A further control group took notes using a blank text window. Results showed that laypersons receiving evaluation prompts outperformed controls in terms of knowledge about sources and produced more arguments commenting on the source of information in an essay task. In addition, laypersons receiving monitoring prompts acquired significantly more knowledge about facts, but did not perform better on a comprehension test than the controls. The availability of ontological categories helped to structure the notes laypersons in the conditions working with ontological classification took during Internet research. Analyses of the notes further demonstrated that the availability of ontological categories guided information search in direction of the selected categories. It is concluded, that met.a.ware is an effective tool that supports laypersons’ Internet research.     

基于强化学习算法的智能飞控开发系统

无人机控制器的设计开发是一项复杂的系统工程, 传统的基于代码编程的开发方式存在开发难度大、周期长及错误率高等缺点. 同时, 强化学习智能飞控算法虽在仿真中取得很好的性能, 但在实际中仍缺乏一套完备的开发系统. 本文提出一套基于模型的智能飞控开发系统, 使用模块化编程及自动代码生成技术, 将强化学习算法应用于飞控的嵌入式开发与部署. 该系统可以实现强化学习算法的训练仿真、测试及硬件部署, 旨在提升以强化学习为代表的智能控制算法的部署速度, 同时降低智能飞行控制系统的开发难度.     

Short- and long-term effects of students’ self-directed metacognitive prompts on navigation behavior and learning performance

This study seeks to promote learning in computer-based learning environments utilizing students’ self-directed metacognitive prompts. Such prompts are based on the idea of instructing students to design their own metacognitive scaffolds and learn with them afterward. In a pre-post experimental design, students in the experimental group (n = 35) were instructed to configure their own metacognitive prompts before learning whereas students in the control group (n = 35) learned without prompts. Log file analysis of navigation behavior indicates that students who learned with their individually designed, self-directed prompts visited relevant webpages significantly more often and spent a longer time on them compared with students in the control group. Moreover, participants in the experimental group attained better transfer performance immediately after learning. The long-term effect in transfer performance was even greater in a follow-up learning session conducted after three weeks without any instructional support in either group. These results are consistent with theories of metacognition and self-regulated learning and indicate that self-directed prompts can lead to sustainable effects.     

Real-Time Execution Monitoring

Today's programming methodology emphasizes the study of static aspects of programs. In practice, however, monitoring a program in execution, i.e., monitoring a process, is routinely done by any programmer whose task it is to produce a reliable piece of software. There are two reasons why one might want to examine the dynamic aspects of a program: first, to evaluate the performance of a program, and hence to assess its overall behavior; and second, to demonstrate the presence of programming errors, isolate erroneous program code, and correct it. This latter task is commonly called ``debugging a program' and requires a detailed insight into the innards of a program being executed. Today, many computer systems are being used to measure and control real-world processes. The pace of execution of these systems and their control programs is therefore bound to timing constraints imposed by the real-world process. As a step towards solving the problems associated with execution monitoring of real-time programs, we develop a set of appropriate concepts and define the basic requirements for a real-time monitoring facility. As a test case for the theoretical treatment of the topic, we design hardware and software for an experimental real-time monitoring system and describe its implementation.     

The effect of metacognitive training on mathematical word-problem solving

This study demonstrates how explicit metacognitive training influences the mathematical word-problem solving of forty 11–12-year-old low achievers in a cognitive-apprenticeship, computer-based environment. Results from the experimental and case study designs revealed that experimental students outperformed control students on ability to solve word-problems on their individual written measures; experimental students developed the ability to ascertain when to make metacognitive decisions, and elicit better regulated metacognitive decisions than control students; knowing when and how to use metacognitive strategies is an important determinant to successful word-problem solving; and the cognitive-apprenticeship-computer-based environment appears to amplify low achievers' metacognitive and cognitive behaviours during word-problem solving.     

Intelligent process monitoring by interfacing knowledge-based systems and multivariate statistical monitoring

An intelligent process monitoring and fault diagnosis environment has been developed by interfacing multivariate statistical process monitoring (MSPM) techniques and knowledge-based systems (KBS) for monitoring multivariable process operation. The real-time KBS developed in G2 is used with multivariate SPM methods based on canonical variate state space (CVSS) process models. Fault detection is based on T ² charts of state variables. Contribution plots in G2 are used for determining the process variables that have contributed to the out-of-control signal indicated by large T ² values, and G2 Diagnostic Assistant (GDA) is used to diagnose the source causes of abnormal process behavior. The MSPM modules developed in Matlab are linked with G2. This intelligent monitoring and diagnosis system can be used to monitor multivariable processes with autocorrelated, crosscorrelated, and collinear data. The structure of the integrated system is described and its performance is illustrated by simulation studies.     

Enhancing portability with multilingual ontology-based knowledge management

Information systems in multilingual environments, such as the EU, suffer from low portability and high deployment costs. In this paper we propose an ontology-based model for multilingual knowledge management in information systems. Our unique feature is a lightweight mechanism, dubbed context, that is associated with ontological concepts and specified in multiple languages. We use contexts to assist in resolving cross-language and local variation ambiguities. Equipped with such a model, we next provide a four-step procedure for overcoming the language barrier in deploying a new information system. We also show that our proposed solution can overcome differences that stem from local variations that may accompany multilingual information systems deployment. The proposed mechanism was tested in an actual multilingual eGovernment environment and by using real-world news syndication traces. Our empirical results serve as a proof-of-concept of the viability of the proposed model. Also, our experiments show that news items in different languages can be identified by a single ontology concept using contexts. We also evaluated the local interpretations of concepts of a language in different geographical locations.     

CitySpec with shield: A secure intelligent assistant for requirement formalization

An increasing number of monitoring systems have been developed in smart cities to ensure that a city’s real-time operations satisfy safety and performance requirements. However, many existing city requirements are written in English with missing, inaccurate, or ambiguous information. There is a high demand for assisting city policymakers in converting human-specified requirements to machine-understandable formal specifications for monitoring systems. To tackle this limitation, we build CitySpec (Chen et al., 2022), the first intelligent assistant system for requirement specification in smart cities. To create CitySpec, we first collect over 1,500 real-world city requirements across different domains (e.g., transportation and energy) from over 100 cities and extract city-specific knowledge to generate a dataset of city vocabulary with 3,061 words. We also build a translation model and enhance it through requirement synthesis and develop a novel online learning framework with shielded validation. The evaluation results on real-world city requirements show that CitySpec increases the sentence-level accuracy of requirement specification from 59.02% to 86.64%, and has strong adaptability to a new city and a new domain (e.g., the F1 score for requirements in Seattle increases from 77.6% to 93.75% with online learning). After the enhancement from the shield function, CitySpec is now immune to most known textual adversarial inputs (e.g., the attack success rate of DeepWordBug (Gao et al., 2018) after the shield function is reduced to 0% from 82.73%). We test the CitySpec with 18 participants from different domains. CitySpec shows its strong usability and adaptability to different domains, and also its robustness to malicious inputs.     

Parallel Factories for Smart Industrial Operations: From Big AI Models to Field Foundational Models and Scenarios Engineering

The rapid advancement of fundamental theories and computing capacity has brought artificial intelligence, internet of things, extended reality, and many other new intelligent technologies into our daily lives. Due to the lack of interpretability and reliability guarantees, it is extremely challenging to apply these technologies directly to real-world industrial systems. Here we present a new paradigm for establishing parallel factories in metaverses to accelerate the deployment of intelligent technologies in real-world industrial systems: QAII-1.0. Based on cyber-physical-social systems, QAII-1.0 incorporates complex social and human factors into the design and analysis of industrial operations and is capable of handling industrial operations involving complex social and human behaviors. In QAII-1.0, a field foundational model called EuArtisan combined with scenarios engineering is developed to improve the intelligence of industrial systems while ensuring industrial interpretability and reliability. Finally, parallel oil fields in metaverses are established to demonstrate the operating procedure of QAII-1.0.      

A performance model for analysis of heterogeneous multi-cluster systems

This paper addresses the problem of performance modeling for large-scale heterogeneous distributed systems with emphases on multi-cluster computing systems. Since the overall performance of distributed systems is often depends on the effectiveness of its communication network, the study of the interconnection networks for these systems is very important. Performance modeling is required to avoid poorly chosen components and architectures as well as discovering a serious shortfall during system testing just prior to deployment time. However, the multiplicity of components and associated complexity make performance analysis of distributed computing systems a challenging task. To this end, we present an analytical performance model for the interconnection networks of heterogeneous multi-cluster systems. The analysis is based on a parametric family of fat-trees, the m-port n-tree, and a deterministic routing algorithm, which is proposed in this paper. The model is validated through comprehensive simulation, which demonstrated that the proposed model exhibits a good degree of accuracy for various system organizations and under different working conditions.     

SeWDReSS: on the design of an application independent,secure, wide-area disaster recovery storage system

Distributed wide-area storage systems must tolerate both physical failure and logic errors. In particular, these functions are needed to enable the storage system to support remote disaster recovery. There are several solutions for distributed wide-area backup/archive systems implemented at application level, file system level or at storage subsystem level. However, they suffer from high deployment cost and security issues. Moreover, previous researches in literature only focus on any disk-related failures and ignore the fact that storage server linked predominantly to a Wide-Area-Network (WAN) which may be unavailable or owing to network failures. In this paper, we first model the efficiency and reliability of distributed wide area storage systems for all media, taking both network failures and disk failures into consideration. To provide higher performance, efficiency, reliability, and security to the wide-area disaster recovery storage systems, we present a configurable RAID-like data erasure-coding scheme referred to as Replication-based Snapshot Redundant Array of Independent Imagefiles (RSRAII). We argue that this scheme has benefits resulting from the consolidation of both erasure-coding and replication strategies. To this end, we propose a novel algorithm to improve the snapshot performance referred to as SMPDP (Snapshot based on Multi-Parallel Degree Pipeline). We also extend this study towards implementing a prototype system, called as SeWDReSS, which is shown to strike a tradeoff between reliability, storage space, security, and performance for distributed wide-area disaster recovery.     

Cross-paradigm analysis of autonomous agent architecture

Autonomous agent architectures are design methodologies—collections of knowledge and strategies which are applied to the problem of creating situated intelligence. This article attempts to integrate this knowledge across several architectural traditions, paying particular attention to features which have tended to be selected under the pressure of extensive use in real-world systems. We determine that the following strategies provide significant assistance in the design of autonomous intelligent agents: (i) modularity, which simplifies both design and control; (ii) hierarchically organized action selection, which focusses attention and provides prioritization when different modules conflict; and (iii) parallel environment monitoring which allows a system to be responsive and opportunistic by allowing attention to shift and priorities to be re-evaluated. We offer a review of four architectural paradigms: behaviour-based AI; two- and three-layered systems; belief, desire and intention architectures (particularly PRS); and Soar/ACT-R. By documenting trends within each of these communities towards establishing the components above, we argue that this convergent evolution is strong evidence for the components' utility. We then use this information to recommend specific strategies for researchers working under each paradigm to further exploit the knowledge and experience of the field as a whole.     

Further constructive results on interconnection and damping assignment control of mechanical systems: the Acrobot example

Interconnection and damping assignment passivity‐based control is a controller design methodology that achieves (asymptotic) stabilization of mechanical systems endowing the closed‐loop system with a Hamiltonian structure with a desired energy function—that qualifies as Lyapunov function for the desired equilibrium. The assignable energy functions are characterized by a set of partial differential equations that must be solved to determine the control law. A class of underactuation degree one systems for which the partial differential equations can be explicitly solved—making the procedure truly constructive—was recently reported by the authors. In this brief note, largely motivated by the interesting Acrobot example, we pursue this investigation for two degrees‐of‐freedom systems where a constant inertia matrix can be assigned. We concentrate then our attention on potential energy shaping and give conditions under which an explicit solution of the associated partial differential equation can be obtained. Using these results we show that it is possible to swing‐up the Acrobot from some configuration positions in the lower half plane, provided some conditions on the robot parameters are satisfied. Copyright © 2006 John Wiley & Sons, Ltd.