Expert system evaluation techniques: a selected bibliography

Abstract: The rapid growth in the use of expert systems has led to widespread interest in techniques for their evaluation, which includes verification and validation. Evaluation techniques in the software engineering community are well established but there are different opinions about the effectiveness of transferring software engineering methodologies to expert systems. This paper outlines some of the issues involved in evaluating expert systems and cites almost 200 significant papers on the topic. We present it with the expectation that it will help both new and established researchers become acquainted with the literature of an important and growing field.     

Interactive Line Art Rendering of Freeform Surfaces

In recent years, synthetically created line art renderings have reached quality levels that are aesthetically pleasing. Moreover, the sketch based approach was found to be quite capable at conveying geometrical information in an intuitive manner. While a growing interest in this type of rendering method has yielded successful and appealing results, the developed techniques were, for the most part, too slow to be embedded in real time interactive display. This paper presents a line art rendering method for freeform polynomial and rational surfaces that is capable of achieving real time and interactive display. A careful preprocessing stage that combines an a-priori construction of line art strokes with proper classification of these strokes, allows one to significantly alleviate the computational cost, in real time, of the sketch based rendering, and enables interactive line art display.     

On systems coaching

A skill central to requirements engineering is knowing which questions will uncover the real requirements. The author explains how we can apply techniques from family therapy to ask better, more relevant questions, which find its way into the professional business community as an activity known as coaching. The coaching model provides a good framework that can be used for discovering requirements. Requirements specification lies in the dialogue between the users and owners of the systems that the engineers produce. The challenge of using questions open up many views so we never make a final decision or actually freeze the specification. This is where the GROW (goal, reality, options, wrap-up) model from coaching guide us toward a possible solution.     

Survey on test data generation tools

Automating the process of software testing is a very popular research topic and of real interest to industry. Test automation can take part on different levels, e.g., test execution, test case generation, test data generation. This survey gives an overview of state-of-the art test data generation tools, either academic or commercial. The survey focuses on white- and gray-box techniques. The list of existing tools was filtered with respect to their public availability, their maturity, and activity. The remaining seven tools, i.e., AgitarOne, CodePro AnalytiX, AutoTest, C++test, Jtest, RANDOOP, and PEX, are briefly introduced and their evaluation results are summarized. For the evaluation we defined 31 benchmark tests, which check the tools capabilities to generate test data that satisfies a given specification: 24 primitive type benchmarks and 7 non-primitive type and more complex with respect to the specification benchmarks. Most of the commercial tools implement a test data strategy that uses constant values found in the method under test or values that are slightly modified by means of mathematical operations. This strategy turns out to be very effective. In general, all tools that combine multiple techniques perform very well. For example PEX uses constraint solving techniques, but in cases where the constraint solver reaches its limitations it uses random based techniques to overcome those limitations. Especially, the two commercial tools AgitarOne and PEX that combine multiple approaches to test data generation are able to pass all 31 tests. This survey reflects the status in 2011.     

Requirements and constructors for tailoring software processes: a systematic literature review

Organizations developing software-based systems or services often need to tailor process reference models—including product-oriented and project-oriented processes—to meet both their own characteristics and those of their projects. Existing process reference models, however, are often defined in a generic manner. They typically offer only limited mechanisms for adapting processes to the needs of organizational units, project goals, and project environments. This article presents a systematic literature review of peer-reviewed conference and journal articles published between 1990 and 2009. Our aim was both to identify requirements for process-tailoring notation and to analyze those tailoring mechanisms that are currently in existence and that consistently support process tailoring. The results show that the software engineering community has demonstrated an ever-increasing interest in software process tailoring, ranging from the consideration of theoretical proposals regarding how to tailor processes to the scrutiny of practical experiences in organizations. Existing tailoring mechanisms principally permit the modeling of variations of activities, artifacts, or roles by insertion or deletion. Two types of variations have been proposed: the individual modification of process elements and the simultaneous variation of several process elements. Resolving tailoring primarily refers to selecting or deselecting optional elements or to choosing between alternatives. It is sometimes guided by explicitly defined processes and supported by tools or mechanisms from the field of knowledge engineering. The study results show that tailoring notations are not as mature as the industry requires if they are to provide the kind of support for process tailoring that fulfills the requirements identified, i.e., including security policies for the whole process, or carrying out one activity rather than another. A notation must therefore be built, which takes these requirements into consideration in order to permit variant-rich processes representation and use this variability to consistently support process tailoring.     

Improving state class constructions for CTL* model checking of time Petri nets

The state space explosion is still one of the most challenging problems in formal verification using enumerative techniques. The challenge is even greater for real time systems whose state spaces are generally infinite due to time density. To use enumerative techniques with these systems, their state spaces need to be contracted into finite structures that preserve properties of interest. We propose in this paper an efficient approach to construct a contraction of the time Petri net model state space, which preserves its CTL* properties.     

Preview-based techniques for vehicle suspension control: a state-of-the-art review

Automotive suspension systems are key to ride comfort and handling performance enhancement. In the last decades semi-active and active suspension configurations have been the focus of intensive automotive engineering research, and have been implemented by the industry. The recent advances in road profile measurement and estimation systems make road-preview-based suspension control a viable solution for production vehicles. Despite the availability of a significant body of papers on the topic, the literature lacks a comprehensive and up-to-date survey on the variety of proposed techniques for suspension control with road preview, and the comparison of their effectiveness. To cover the gap, this literature review deals with the research conducted over the past decades on the topic of semi-active and active suspension controllers with road preview. The main formulations are reported for each control category, and the respective features are critically analysed, together with the most relevant performance indicators. The paper also discusses the effect of the road preview time on the resulting system performance, and identifies control development trends.     

Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review,two showcases and beyond

Explainable Artificial Intelligence (XAI) is an emerging research topic of machine learning aimed at unboxing how AI systems’ black-box choices are made. This research field inspects the measures and models involved in decision-making and seeks solutions to explain them explicitly. Many of the machine learning algorithms cannot manifest how and why a decision has been cast. This is particularly true of the most popular deep neural network approaches currently in use. Consequently, our confidence in AI systems can be hindered by the lack of explainability in these black-box models. The XAI becomes more and more crucial for deep learning powered applications, especially for medical and healthcare studies, although in general these deep neural networks can return an arresting dividend in performance. The insufficient explainability and transparency in most existing AI systems can be one of the major reasons that successful implementation and integration of AI tools into routine clinical practice are uncommon. In this study, we first surveyed the current progress of XAI and in particular its advances in healthcare applications. We then introduced our solutions for XAI leveraging multi-modal and multi-centre data fusion, and subsequently validated in two showcases following real clinical scenarios. Comprehensive quantitative and qualitative analyses can prove the efficacy of our proposed XAI solutions, from which we can envisage successful applications in a broader range of clinical questions.     

Reconfigurable Networks on Chip: DRNoC architecture

To cover the complexity of future systems, where thousands and hundreds of heterogeneous cores have to be interconnected, new on-chip communication solutions are being searched. In this context, Networks on Chip (NoCs) have been studied as bus alternative. However, the inclusion of NoCs’ broad design space increases even more the complexity of design flows. Additionally, today electronic industry demands drastic time to market reduction and improved device diversity. On the other side, reconfigurable devices have had an impressive evolution and now, they are complex heterogeneous platforms which include a broad variety of embedded cores. Furthermore, today it is possible to embed reconfigurable arrays in application-specific integrated circuits and thus create highly flexible systems. These tendencies provide support to the idea of reconfigurable on-chip communication, which can reduce the system design time and permit to adapt their characteristics to currently running applications. This paper overviews some reconfigurable NoCs’ state of the art solutions and describes a reconfigurable on-chip communication approach, called DRNoC, which explores the highest possible flexibility and is not limited to NoCs. An important aspect considered in this paper is real implementations and therefore, all the solutions discussed along it, including DRNoC, have been validated on FPGAs. The paper also highlights some technology restrictions of currently available FPGA reconfiguration techniques that do not permit to test real-live examples on such systems.     

An adaptive doctor-recommender system

ABSTRACT

Recommender systems use machine-learning techniques to make predictions about resources. The medical field is one where much research is currently being conducted on recommender system utility. In the last few years, the amount of information available online that relates to healthcare has increased tremendously. Patients nowadays are more aware and look for answers to healthcare problems online. This has resulted in a dire need of an effective reliable online system to recommend the physician that is best suited to a particular patient in a limited time. In this article, a hybrid doctor-recommender system is proposed, by combining different recommendation approaches: content base, collaborative and demographic filtering to effectively tackle the issue of doctor recommendation. The proposed system addresses the issue of personalization through analysing patient's interest towards selecting a doctor. It uses a novel adoptive algorithm to construct a doctor's ranking function. Moreover, this ranking function is used to translate patients’ criteria for selecting a doctor into a numerical base rating, which will eventually be used in the recommendation of doctors. The system has been evaluated thoroughly, and result show that recommendations are reasonable and can fulfil patient's demand for reliable doctor's selection effectively.     

State-rich model checking

In this paper we survey the area of formal verification techniques, with emphasis on model checking due to its wide acceptance by both academia and industry. The major approaches and their characteristics are presented, together with the main problems faced while trying to apply them. With the increased complexity of systems, as well as interest in software correctness, the demand for more powerful automatic techniques is pushing the theories and tools towards integration. We discuss the state of the art in combining formal methods tools, mainly model checking with theorem proving and abstract interpretation. In particular, we present our own recent contribution on an approach to integrate model checking and theorem proving to handle state-rich systems specified using a combination of Z and CSP.     

Parallel techniques in irregular codes: cloth simulation as case of study

When parallelizing irregular applications on ccNUMA machines several issues should be taken into account in order to achieve high code performance. These factors include locality exploitation and parallelism, as well as careful use of memory resources (memory overhead). An important number of numerical simulation codes are clear examples of irregular applications. Frequently these kinds of codes include reduction operations in their core, so that an important fraction of the computational time is spent on such operations. Specifically, cloth simulation belongs to this class of applications, being a topic of increasing interest in diverse areas, like in the multimedia industry. Moreover, when real time simulation is the aim, its parallelization becomes an important option.This paper discusses and compares different irregular reduction parallelization techniques on ccNUMA share memory machines. Broadly speaking, we may classify them into two groups: privatization-based and data partitioning-based methods. In this paper we describe a framework, based on data affinity, that permits to develop various algorithms inside the group of the data partitioning-based techniques. All these techniques and approaches are analyzed and adapted to the computational structure of a real, physically based, cloth simulator.     

Systems mapping workshops and their role in understanding medication errors in healthcare

Systems mapping workshops have been applied to the problem of medication errors in healthcare. The workshops were designed using experiential group work principles. They involved a range of stakeholders from within the health service as well as those who supply the health sector, including designers who may be able to enhance the safety of products and systems used in healthcare. Research has shown that the method encourages stakeholder participation, provides robust results within a limited time and enhances understanding across specialist interest groups. Additional, creative design workshops that considered the same topic showed significant promise in developing concepts from which potential solutions could be developed further.     

Post‐disaster assessment with unmanned aerial vehicles: A survey on practical implementations and research approaches

In this paper, we provide a review of the principal aspects related to search & rescue (SAR) with unmanned aerial vehicles (UAVs), with particular interest in the phase of post‐disaster assessment (PDA). Some areas of interest related to this topic have been chosen for the analysis: the aerial platforms used in the field, multirobot software architectures, onboard sensors and simultaneous localization and mapping approaches, terrain coverage algorithms, autonomous navigation techniques, and human‐swarm interfaces. All these aspects have been analyzed with respect to the state‐of‐the‐art, and also in relation to the project PRISMA, which focuses on the development and deployment of robots and autonomous systems that can operate in emergency scenarios, with a specific reference to monitoring and real‐time intervention.     

Uses of accelerometer data collected from a wearable system

This paper presents work, assessing the use of accelerometers in wearable systems for a number of applications. It discusses and demonstrates how body mounted accelerometers can be used in context aware computing systems and for measuring aspects of human performance, which may be used for teaching and demonstrating skill acquisition, coaching sporting activities, sports and human movement research, and teaching subjects such as physics and physical education. Analysis is restricted to considerations as to how raw data can be used, and how simple calculations of quantities of data in the time domain, can be used. The limitations of the use of such data are discussed.     

Aeroelastic tailoring and scaling using Bacterial Foraging Optimisation

Bacterial Foraging Optimisation (BFO) is investigated in an attempt to evaluate its use in solving complex optimisation problems for aeronautical structures. A hybrid variant of BFOA, which incorporates meta-modelling techniques, is also proposed and employed. The efficiency and effectiveness of the methods are tested for tailoring a rectangular composite wing, aiming to maximise the flutter speed and for scaling a joined-wing aircraft, targeting to match aeroelastic responses between the physical prototype and wind tunnel model. The obtained results are compared with those found using a range of other biologically inspired optimisation methods (GA, PSO, ACO), proving that the social foraging behavior of motile bacteria is an effective tool for aeroelastic optimisation.     

The State of the Art in Multilayer Network Visualization

Modelling relationship between entities in real‐world systems with a simple graph is a standard approach. However, reality is better embraced as several interdependent subsystems (or layers). Recently, the concept of a multilayer network model has emerged from the field of complex systems. This model can be applied to a wide range of real‐world data sets. Examples of multilayer networks can be found in the domains of life sciences, sociology, digital humanities and more. Within the domain of graph visualization, there are many systems which visualize data sets having many characteristics of multilayer graphs. This report provides a state of the art and a structured analysis of contemporary multilayer network visualization, not only for researchers in visualization, but also for those who aim to visualize multilayer networks in the domain of complex systems, as well as those developing systems across application domains. We have explored the visualization literature to survey visualization techniques suitable for multilayer graph visualization, as well as tools, tasks and analytic techniques from within application domains. This report also identifies the outstanding challenges for multilayer graph visualization and suggests future research directions for addressing them.     

A novel model usability evaluation framework (MUsE) for explainable artificial intelligence

When it comes to complex machine learning models, commonly referred to as black boxes, understanding the underlying decision making process is crucial for domains such as healthcare and financial services, as well as when they are used in connection with safety critical systems such as autonomous vehicles. As a result, interest in explainable artificial intelligence (xAI) tools and techniques has increased in recent years. However, the user experience (UX) effectiveness of existing xAI frameworks, especially concerning algorithms that work with data as opposed to images, is still an open research question. In order to address this gap, we examine the UX effectiveness of the Local Interpretable Model-Agnostic Explanations (LIME) xAI framework, one of the most popular model agnostic frameworks found in the literature, with a specific focus on its performance in terms of making tabular models more interpretable. In particular, we apply several state of the art machine learning algorithms on a tabular dataset, and demonstrate how LIME can be used to supplement conventional performance assessment methods. Based on this experience, we evaluate the understandability of the output produced by LIME both via a usability study, involving participants who are not familiar with LIME, and its overall usability via a custom made assessment framework, called Model Usability Evaluation (MUsE), which is derived from the International Organisation for Standardisation 9241-11:2018 standard.     

时空嵌入式生成对抗网络的地点预测方法*

定位技术的广泛使用可以积累大量的用户轨迹信息,为挖掘用户的行为轨迹提供便利.地点预测任务是众多基于位置服务的基础,学者们更关注如何有效利用这些轨迹数据进行地点预测.已有的方法或关注对长期模式(数天或数月)的预测,或致力于实时轨迹预测.文中研究的问题基于上述两者之间,即对弱实时条件下(数分钟或数小时)用户下一步的访问行为进行预测.为此,提出时空嵌入式的生成对抗网络模型(ST-GAN),在序列生成对抗网络的基础上,提出时空嵌入式长短时记忆生成模型(ST-LSTM)和时空嵌入式卷积神经网络判别模型(ST-CNN).ST-LSTM利用时空信息引导LSTM训练门机制,缓解数据的稀疏性.ST-CNN利用时空信息增强判别真伪访问序列的能力.此外,ST-GAN的训练优化机制使模型可以生成更多逼近真实的数据以引导模型学习,从而得到更好的预测效果.最后在真实的轨迹数据集上的实验验证ST-GAN的有效性.     

Preprocessing techniques for context recognition from accelerometer data

The ubiquity of communication devices such as smartphones has led to the emergence of context-aware services that are able to respond to specific user activities or contexts. These services allow communication providers to develop new, added-value services for a wide range of applications such as social networking, elderly care and near-emergency early warning systems. At the core of these services is the ability to detect specific physical settings or the context a user is in, using either internal or external sensors. For example, using built-in accelerometers, it is possible to determine whether a user is walking or running at a specific time of day. By correlating this knowledge with GPS data, it is possible to provide specific information services to users with similar daily routines. This article presents a survey of the techniques for extracting this activity information from raw accelerometer data. The techniques that can be implemented in mobile devices range from classical signal processing techniques such as FFT to contemporary string-based methods. We present experimental results to compare and evaluate the accuracy of the various techniques using real data sets collected from daily activities.