Evaluating performance and quality of knowledge-based systems:foundation and methodology

A survey of knowledge-based system (KBS) evaluation methods is presented. The authors argue that these methods are partial, poorly systematic, and not easily applicable. An approach to KBS evaluation that comprises a precise definition of the concepts of performance and quality, a general evaluation methodology, and a set of criteria to support its practical application is presented. The proposed approach has been tried only partially and with rather simple test cases     

A framework for the design and verification of software measurement methods

At the core of any engineering discipline is the use of measures, based on ISO standards or on widely recognized conventions, for the development and analysis of the artifacts produced by engineers. In the software domain, many alternatives have been proposed to measure the same attributes, but there is no consensus on a framework for how to analyze or choose among these measures. Furthermore, there is often not even a consensus on the characteristics of the attributes to be measured.In this paper, a framework is proposed for a software measurement life cycle with a particular focus on the design phase of a software measure. The framework includes definitions of the verification criteria that can be used to understand the stages of software measurement design. This framework also integrates the different perspectives of existing measurement approaches. In addition to inputs from the software measurement literature the framework integrates the concepts and vocabulary of metrology. This metrological approach provides a clear definition of the concepts, as well as the activities and products, related to measurement. The aim is to give an integrated view, involving the practical side and the theoretical side, as well as the basic underlying concepts of measurement.     

Playability: analysing user experience in video games

Currently, few studies focus on analysing the degree of the Player eXperience (PX) in video games. Video games have now become interactive entertainment systems with a high economic impact on society; these are interactive systems characterised by their subjectivity, which differ from other systems in that their main objective is to entertain and amuse the user (player). This work discusses the analysis and evaluation of the User eXperience (UX) in interactive entertainment systems, exploring how usability, given its definition, objectives and the fact it is one of the main dimensions of UX, is not sufficient to characterise the PX, giving rise to a new concept: Playability. In this paper, we present a framework for the analysis and evaluation of the UX in video games. The results show the need and importance of a framework to help us understand and measure the experience that players feel using these types of interactive systems, in order to improve the experience during play time. The proposed framework characterises the experience using attributes to identify and properties to measure UX. It thus provides a multifaceted analysis mechanism to assess the impact of the gaming experience and its relationship with the elements of a video game. We therefore present a system to represent UX based on this framework, with the aim of ensuring and measuring a satisfactory experience of the entertainment system. Finally, we discuss a practical experiment in which an evaluation of the playability of a commercial video game was carried out using the methods proposed in this work.     

A complexity metric approach to software quality measurement

Traditional statistical quality control for dealing with quality variation during mass production is not appropriate for software quality assessment. Therefore determination and measurement of user desirable software attributes become a subject of research interest. In this paper, software attributes are classified into attributes of subjective judgment, attributes based on complexity metrics, and attributes with rigorous mathematical definition. Methods of measuring those attributes are proposed in the paper. To measure the quality of software as a whole, model that integrates qualitative software attributes and quantitative complexity metrics is also given. Finally, distribution of software life cycle costs in relation to the weighting of software quality attributes is discussed.     

.基于知识系统的不确定性模型

研究基于知识系统（KBS）中知识的不确定性是开发KBS的重要问题,但现有模型化KBS几乎都是基于确定性知识的。以经典的CommonKADS模型为背景,采用模型化工程中的不确定性技术,研究KBS中不确定性知识的表示方法。首先在基于值系统的值集概念上引入假设函数集合的评估函数,定义静态不确定性领域知识;而后采用因果模型描述动态的不确定性推理知识和任务知识;最后将三类不确定性知识映射至CommonKADS模型。由此给出了描述不确定KBS的通用模型,扩展了KBS的可用性。     

A refinement framework to support validation and maintenance of knowledge-based systems

In this article, we argue that a refinement tool capable of assisting the maintainer of a Knowledge-Based System (KBS) with updating and upgrading of its knowledge base can substantially expand the scope of his activities and prolong the life of the system. We show that refinement tasks involved in KBS maintenance are similar to those taking place at the development stage, and thus a uniform refinement framework can be designed to support both activities. After defining refinement tasks expected to take place at different stages of a KBS's life cycle, we present a refinement framework capable of supporting them. It utilizes a small number of domain-independent heuristics to search for possible rule revisions which are expected to eliminate errors detected during KBS validation. An extended example is given to illustrate how different refinement tasks are carried out.     

Assessing information quality of e-learning systems: a web mining approach

E-learning systems provide a promising solution as an information exchanging channel. Improved technologies could mean faster and easier access to information but do not necessarily ensure the quality of this information; for this reason it is essential to develop valid and reliable methods of quality measurement and carry out careful information quality evaluations. This paper proposes an assessment model for information quality in e-learning systems based on the quality framework we proposed previously: the proposed framework consists of 14 quality dimensions grouped in three quality factors: intrinsic, contextual representation and accessibility. We use the relative importance as a parameter in a linear equation for the measurement scheme. Formerly, we implemented a goal-question-metrics approach to develop a set of quality metrics for the identified quality attributes within the proposed framework. In this paper, the proposed metrics were computed to produce a numerical rating indicating the overall information quality published in a particular e-learning system. The data collection and evaluation processes were automated using a web data extraction technique and results on a case study are discussed. This assessment model could be useful to e-learning systems designers, providers and users as it provides a comprehensive indication of the quality of information in such systems.     

Incorporating affective customer needs for luxuriousness into product design attributes

In a highly competitive market, customers' product affection is a critical factor to product success. However, understanding customers' affective needs is difficult to grasp; product design practitioners often misunderstand what customers really want. In this study we report our experience in developing and using an affective design framework that identified critical affective features customers have on products and are systematically incorporated into product design attributes. To identify key affective features such as luxuriousness, we utilized the Kansei engineering methodology. This approach consists of three steps: (1) selecting related affective features and product design attributes through a comprehensive literature survey, expert panel opinion, and focus group interviews; (2) conducting evaluation experiments; and (3) developing Kansei models using multivariate statistical analysis and analyzing critical product design attributes. To demonstrate applicability of the proposed affective design framework, 30 customers and 30 product design practitioners participated in an evaluation experiment for car crash pads, and 44 customers and 20 designers participated in an evaluation experiment for two interior room products (wallpapers and flooring materials). The evaluation experiments were conducted via systematically developed questionnaires consisting of a 7‐point semantic differential scale and a 100‐point magnitude estimation scale. The results of the experiments were analyzed using principal component regression and quantification theory type I method. Using the analyzed survey data, the relationship between luxuriousness and related affective features and product design attributes were identified. This relationship indicated that there was a significant difference in the perception of luxuriousness between customers and designers. Consequently, it is expected that the results of this study could provide a foundation for developing affective products. © 2009 Wiley Periodicals, Inc.     

基于透视投影的激光跟踪测量系统可视性分析

针对现有激光跟踪测量系统可视性分析方法在计算效率方面的不足，提出一种基于透视投影的激光跟踪测量系统可视性分析方法。给出了激光跟踪仪可视性分析的数学定义，利用测点与站位点之间的关系对可视性分析进行描述；利用包络盒对障碍物进行简化，同时根据障碍物、测点、站位点之间的关系，剔除了不参与可视性分析运算的障碍物；在此基础上，利用透视投影的原理，对障碍物在跟踪仪可布置平面上的投影进行了求解，并利用该投影与站位点之间是否包含的关系，得到了测点与站位点之间的可视性矩阵，结合提出的可视性势能的概念，对跟踪仪满足可视性的区域进行了求解。最后，以某飞机舱门检验工装的测量为例，验证了该方法和模型的正确性和有效性。     

KBS-aided design of tube bending processes

Cold bending of metal tube products is a metal forming process widely used in industry. With today's demand on high quality tube-dependent products, bending of tubing parts has become a precision metalworking operation that requires sophisticated knowledge and experience. In this work, a knowledge-based system (KBS) has been developed to aid the design of tube bending processes, including bending methods selection, tool/die design, and process parameters setup. The object-oriented programming techniques and the goal-driven search mechanism, featured by an interactive graphic user interface, have been applied in development of the proposed KBS. The system developed has shown effective in tube parts production with significantly reduced number of potential defects and failures, such as inaccuracy of bend angles or linear dimensions, wall thinning, flattening, wrinkling, twisting, dents or cracks, etc.     

Agent based intelligent search framework for product information using ontology mapping

The Semantic Web and Web services provide many opportunities in various applications such as product search and comparison in electronic commerce. We implemented an intelligent meta-search and recommendation system for products through consideration of multiple attributes by using ontology mapping and Web services. Under the assumption that each shopping site offers product ontology and product search service with Web services, we proposed a meta-search framework to configure a customer’s search intent, make and dispatch proper queries to each shopping site, evaluate search results from shopping sites, and show the customer the relevant product list with associated rankings. Ontology mapping is used for generating proper queries for shopping sites that have different product categories. We also implemented our framework and performed empirical evaluation of our approach with two leading shopping sites in the world.     

Managing software process measurement: A metamodel-based approach

The evaluation of software processes is nowadays a very important issue due to the growing interest of software companies in the improvement of the productivity and quality of delivered products. Software measurement plays a fundamental role here. Given the great diversity of entities which are candidates for measurement in the software process improvement context (process models, projects, resources, products) this measurement must be performed in a consistent and integrated way. This will facilitate the making of decisions in process improvement. In this paper, a proposal for the integrated management of the software measurement is presented. The goal is to provide companies with a generic and flexible environment for software measurement which facilitates and establishes the basis for a common and effective measurement process and which is not restricted to only one kind of software entity or to a single quality or evaluation model. In order to achieve this, the proposal adopts the Model Driven Engineering philosophy and provides: a metamodel for the definition of software measurement models; a flexible method to measure any kind of software entity represented by its corresponding metamodel and GenMETRIC, which is the software tool that supports the framework.     

The importance of detailed evaluation for KBS implementation in the engineering industry

The evaluation of a knowledge based system(s) (KBS) is discussed, with particular relevance to the development of KBS intended for implementation in the engineering industry.This discussion is based on the practical experiences of the authors when evaluating a KBS for use in the conceptual design of bridges. Consequently, detailed references are made to the actual evaluation carried out and the results found.The way in which systems have previously been evaluated is also discussed. The deficiencies of these past evaluation techniques are identified and ways in which the evaluation process could be improved are suggested.     

产品可视化需求建模系统下的交互式产品开发

在公理化设计理论的基础上,建立了需求产品的需求商机模型、需求参考模型与需求定义模型。建立产品知识库,应用九象限法对敏捷获取的需求商机的市场吸引力和企业实力多因素模型进行综合评估决策;对产品需求进行功能分解与整合,集成为需求产品,对其进行一致性检查;通过VRML完成需求产品的可视化表达。并以电冰箱的设计过程为例,讨论并验证了产品可视化需求建模环境下交互式需求创新产品开发系统RADAR的有效性。     

Augmented reality-based design customization of footwear for children

This paper presents a systematic framework for design customization of footwear for children and identifies three modules related to shoe styling: shoe surface, shoe bottom, and accessory. A new module, shoe cloth, is created to allow a quick change of shoe appearance. Consumers can specify various design attributes in each module, including color, texture, embroidery, and shape. A prototype design system is implemented using augmented reality and sensing technologies based on the framework. This system consists of novel functions that support customization, design evaluation, and pattern development. The user can virtually put a customized shoe model on his/her foot in a video stream. The proposed framework not only facilitates evaluating products that are highly interactive with users, but also helps engage them in the design process. This work realizes the concept of human-centric design for mass customization.     

用于软件构架评估的质量需求刻画

软件构架评估对于较大系统的开发具有重要意义，搞清质量需求是构架评估的基础和依据。本文介绍了用以进行质量需求刻画的一种框架，以性能这一质量属性为例对这一框架的应用做了说明，并指出了应用这种框架进行质量需求刻画时应注意的问题。     

Quality attribute modeling and quality aware product configuration in software product lines

In software product line engineering, the customers mostly concentrate on the functionalities of the target product during product configuration. The quality attributes of a target product, such as security and performance, are often assessed until the final product is generated. However, it might be very costly to fix the problem if it is found that the generated product cannot satisfy the customers’ quality requirements. Although the quality of a generated product will be affected by all the life cycles of product development, feature-based product configuration is the first stage where the estimation or prediction of the quality attributes should be considered. As we know, the key issue of predicting the quality attributes for a product configured from feature models is to measure the interdependencies between functional features and quality attributes. The current existing approaches have several limitations on this issue, such as requiring real products for the measurement or involving domain experts’ efforts. To overcome these limitations, we propose a systematic approach of modeling quality attributes in feature models based on domain experts’ judgments using the analytic hierarchical process (AHP) and conducting quality aware product configuration based on the captured quality knowledge. Domain experts’ judgments are adapted to avoid generating the real products for quality evaluation, and AHP is used to reduce domain experts’ efforts involved in the judgments. A prototype tool is developed to implement the concepts of the proposed approach, and a formal evaluation is carried out based on a large-scale case study.     

Using knowledge-based systems for estimating risks inherent in a proposed KBS project

Abstract: This article describes the development of a knowledge-based system (KBS) for guiding both technical and non-technical managers in estimating the risks inherent in proposed KBS development projects. The work described here is part of an ongoing research project studying the use of KBS both to teach and to do KBS development. So far, as part of this research project, over the past two years close to 150 non-technical and technical full-time business managers have developed some 75 KBS prototypes in conjunction with an MBA course in strategic planning. Based on replies to a survey of this test group, 17% of the KBS developed were reportedly used at work, 14% led to promotions, pay raises, or new jobs, and 14% led to participation in other KBS development projects at work. All but two of the survey respondents reported that their work on the KBS development project led to a substantial increase in their job knowledge or performance. The KBS work described here extends research work described by Cullen and Bryman [1988], Slagle and Wick [1988], Cohen and Howe [1988], Dologite [1988, 1989], and Mockler [1989(A) and (B)].