Developing usability heuristics with PROMETHEUS: A case study in virtual learning environments

Heuristic evaluation is one of the most widely-used methods for evaluating the usability of a software product. Proposed in 1990 by Nielsen and Molich, it consists in having a small group of evaluators performing a systematic revision of a system under a set of guiding principles known as usability heuristics. Although Nielsen’s 10 usability heuristics are used as the de facto standard in the process of heuristic evaluation, recent research has provided evidence not only for the need of custom domain specific heuristics, but also for the development of methodological processes to create such sets of heuristics. In this work we apply the PROMETHEUS methodology, recently proposed by the authors, to develop the VLEs heuristics: a novel set of usability heuristics for the domain of virtual learning environments. In addition to the development of these heuristics, our research serves as further empirical validation of PROMETHEUS. To validate our results we performed an heuristic evaluation using both VLEs and Nielsen’s heuristics. Our design explicitly controls the effect of evaluator variability by using a large number of evaluators. Indeed, for both sets of heuristics the evaluation was performed independently by 7 groups of 5 evaluators each. That is, there were 70 evaluators in total, 35 using VLEs and 35 using Nielsen’s heuristics. In addition, we perform rigorous statistical analyses to establish the validity of the novel VLEs heuristics. The results show that VLEs perform better than Nielsen’s heuristics, finding more problems, which are also more relevant to the domain, as well as satisfying other quantitative and qualitative criteria. Finally, in contrast to evaluators using Nielsen’s heuristics, evaluators using VLEs heuristics reported greater satisfaction regarding utility, clarity, ease of use, and need of additional elements.     

The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods

Computer professionals have a need for robust, easy-to-use usability evaluation methods (UEMs) to help them systematically improve the usability of computer artifacts. However, cognitive walkthrough (CW), heuristic evaluation (HE), and thinking- aloud study (TA)-3 of the most widely used UEMs-suffer from a substantial evaluator effect in that multiple evaluators evaluating the same interface with the same UEM detect markedly different sets of problems. A review of 11 studies of these 3 UEMs reveals that the evaluator effect exists for both novice and experienced evaluators, for both cosmetic and severe problems, for both problem detection and severity assessment, and for evaluations of both simple and complex systems. The average agreement between any 2 evaluators who have evaluated the same system using the same UEM ranges from 5% to 65%, and no 1 of the 3 UEMs is consistently better than the others. Although evaluator effects of this magnitude may not be surprising for a UEM as informal as HE, it is certainly notable that a substantial evaluator effect persists for evaluators who apply the strict procedure of CW or observe users thinking out loud. Hence, it is highly questionable to use a TA with 1 evaluator as an authoritative statement about what problems an interface contains. Generally, the application of the UEMs is characterized by (a) vague goal analyses leading to variability in the task scenarios, (b) vague evaluation procedures leading to anchoring, or (c) vague problem criteria leading to anything being accepted as a usability problem, or all of these. The simplest way of coping with the evaluator effect, which cannot be completely eliminated, is to involve multiple evaluators in usability evaluations.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

A Study of the Evaluator Effect in Usability Testing

ABSTRACT

The evaluator effect names the observation that usability evaluators in similar conditions identify substantially different sets of usability problems. Yet little is known about the factors involved in the evaluator effect. We present a study of 50 novice evaluators' usability tests and subsequent comparisons, in teams and individually, of the resulting usability problems. The same problems were analyzed independently by 10 human–computer interaction experts. The study shows an agreement between evaluators of about 40%, indicating a substantial evaluator effect. Team matching of problems following the individual matching appears to improve the agreement, and evaluators express greater satisfaction with the teams' matchings. The matchings of individuals, teams, and independent experts show evaluator effects of similar sizes; yet individuals, teams, and independent experts fundamentally disagree about which problems are similar. Previous claims in the literature about the evaluator effect are challenged by the large variability in the matching of usability problems; we identify matching as a key determinant of the evaluator effect. An alternative view of usability problems and evaluator agreement is proposed in which matching is seen as an activity that helps to make sense of usability problems and where the existence of a correct matching is not assumed.     

Survey instrument for the universal design of consumer products

Universal design is a process intended to include all user groups in product or environmental design. The objective of this study was to develop a usability testing survey instrument to inform how well consumer products complied with established principles of universal design. Thirty-six adults, aging adults and adult wheelchair users performed standardized tasks with pens, food storage containers, pliers and calculators, and for each task responded to a preliminary set of survey items and rated task difficulty. Factor analysis of the survey responses produced an eleven-factor solution that accounted for 67% of the variance in scores and corresponded fairly closely to the principles of universal design. Analysis of scale scores developed from each factor showed that some of the scales were sensitive to product feature and user group differences, and were negatively associated with perceived task difficulty. Such a tool may aid designers who intend their products for user groups of diverse abilities and preferences.     

What you get is what you see: revisiting the evaluator effect in usability tests

Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it.     

A systematic examination of universal design resources: part 1, heuristic evaluation

This paper presents the evaluation of eight published Universal Design Resources (UDRs) to measure how effectively they support typical design processes and design psychology. New heuristics and principles to evaluate the UDRs from the point of view of designers who were universal design novices were created. Established methodologies for heuristic evaluation were used with the new heuristics. The evaluators found numerous problems in seven of the eight UDRs, providing evidence in support of the hypothesis that the content UDRs does not facilitate the design process and is not commensurate with what is known about typical design psychology.     

认知协作遍历检查在可用性评估中的应用

在协作任务仿真模型的基础上,提出一种认知协作遍历检查技术,给出其遍历过程算法。通过第1阶段的评估对比实验,证明该可用性评估技术适用于设计早期的协同软件系统,且评估成本较低,效率较高。在第2阶段实验中增加了评估者的数量,用以分析评估者数量及其个体表现对评估结果的影响,并针对一般情况下获得最高收益代缴比的样本数量给出建议。     

User values based evaluation model to assess product universality

Unlike the conventional design methodology, the universal design aims to integrate various user groups within the design space to benefit the user of all ages with distinct abilities. The objective of this work is to ensure universality of the designed product with the help of user-values based evaluation model. In the proposed model, users are classified into three groups based on their abilities and needs. This classification analyses the values associated with the individual needs of a different group of users. The universality of the product is analyzed by using the proposed model and accordingly the product can be improved by incorporating the values that have been overlooked by the designer. The frequently neglected values can be identified to modify universal design principles.Relevance to industry.Providing functional and usable products to all sets of users is the most important goal of industry, in today's marketplace. To achieve this goal, universal product design evaluation model has been proposed in this work. With the help of evaluation model, design firms and designers can ensure the universality of the product at an early phase of design and also while redesigning of the existing products.     

Towards efficient partial evaluation in logic programming

Partial evaluation is a symbolic manipulation technique used to produce efficient algorithms when part of the input to the algorithm is known. Other applications of partial evaluators such as universal compilation and compiler generation are also known to be possible. A partial evaluator receives as input a program and partially known input to that program, and outputs a residual program which should run at least as efficient as the input program with restricted input. In this paper we study the case where both the input and residual programs are logic programs, being the partial evaluator itself a logic program. Up to now, partial evaluators have failed to process large “non=toy” examples. Here we present extensions to partial evaluators whic will allow us to produce more efficient residual programs using less computing resources, during partial evaluation. First, the introduced extensions allow the processing of large examples, which is not possible with the previous techniques. This is now possible since the extensions use less CPU time and memory consumption during the partial evaluation process. Second, the extended partial evaluator produces smaller residual programs, producing important CPU time optimizing effects. With the standard techniques, a partial evaluator will most probably act as a pessimizer, not as an optimizer. Examples are given.     

An analysis of expected-outcome

Abstract

Static evaluators have been used in every game program ever written. These heuristic functions attempt to differentiate between strong and weak moves by assigning them values based on directly detectable game features. Despite their ubiquity, evaluation functions are not well understood; the development of a theory of evaluator design has been too long coming. In fact, the general consensus is that no theory is possible, because expertise is required to develop even a simple evaluator.

One recently introduced ‘generic’ evaluation function, the expected-outcome model, proposed evaluating a node as the expected value of a game's outcome, given random play from that node on. Experimental studies conducted on evaluators designed under this model yielded encouraging results in tac-tac-toe, Othello, and chess. This paper analyzes the expected-outcome model on a simple class of game trees, and shows that the moves recommended under the assumptions of random play and perfect play are identical. This vindicates what appeared to have been an overly naive assumption, and furthers the claim that statistically interpreted evaluation functions are powerful, as well as elegant.     

Determining the effectiveness of the usability problem inspector: a theory-based model and tool for finding usability problems

Despite the increased focus on usability and on the processes and methods used to increase usability, a substantial amount of software is unusable and poorly designed. Much of this is attributable to the lack of cost-effective usability evaluation tools that provide an interaction-based framework for identifying problems. We developed the user action framework and a corresponding evaluation tool, the usability problem inspector (UPI), to help organize usability concepts and issues into a knowledge base. We conducted a comprehensive comparison study to determine if our theory-based framework and tool could be effectively used to find important usability problems in an interface design, relative to two other established inspection methods (heuristic evaluation and cognitive walkthrough). Results showed that the UPI scored higher than heuristic evaluation in terms of thoroughness, validity, and effectiveness and was consistent with cognitive walkthrough for these same measures. We also discuss other potential advantages of the UPI over heuristic evaluation and cognitive walkthrough when applied in practice. Potential applications of this work include a cost-effective alternative or supplement to lab-based formative usability evaluation during any stage of development.     

通用设计,因你而变——通用设计原则在产品设计中的应用

通过对通用设计的源起和设计原则的阐释,结合其在实例中的应用,说明该原则能够对产品设计产生很大的影响。通用设计在产品设计中的应用不仅能改变所有人生活、工作中的不便,同时也能给企业带来良好的声誉和经济效益。     

A Collaborative Usability Evaluation (CUE) Model for Health IT Design and Implementation

ABSTRACT

In this article, the authors describe a collaborative usability evaluation (CUE) model based on a participatory ergonomics framework. The model was applied to an organization’s health IT design and implementation processes. Individuals received education on usability evaluation principles and methods and had an opportunity for hands-on experience with usability testing. Actual usability evaluations increased participant confidence in their ability to independently design and lead such evaluations. The CUE model has since been applied to the organization’s iterative process of health IT design, has been institutionalized by the health IT department, and is integrated in its health IT implementation timelines.     

SUE inspection: an effective method for systematic usabilityevaluation of hypermedia

SUE inspection is a novel usability evaluation method for hypermedia, which falls into the category of inspection methods. Its primary goal is to favor a systematic usability evaluation (SUE), for supplying hypermedia usability inspectors with a structured flow of activities, allowing them to obtain more reliable, comparable, and cost-effective evaluation results. This is obtained especially due to the use of evaluation patterns, called abstract tasks (AT), which describe in details the activities that evaluators must carry out during inspection. AT helps share and transfer evaluation know-how among different evaluators, thus making it easier to learn the SUE inspection method by newcomers. A further notable advantage provided by the SUE inspection over other existent approaches is that it focuses on navigation and information structures, making evident some problems that other "surface-oriented" approaches might not reveal. This paper describes the SUE inspection for hypermedia. It also reports on a validation experiment, involving 28 evaluators, that has shown the major effectiveness and efficiency of the SUE inspection for hypermedia, with respect to traditional heuristic evaluation techniques     

Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites

This study was conducted to compare CHE between Human-Computer Interaction (HCI) experts and novices in evaluating the Smartphone app for the cultural heritage site. It uses the Smartphone Mobile Application heuRisTics (SMART), focusing on smartphone applications and traditional Nielsen heuristics, focusing on a wider range of interactive systems. Six experts and six novices used the severity rating scale to categorise the severity of the usability issues. These issues were mapped to both heuristics. The study found that experts and novice evaluators identified 19 and 14 usability issues, respectively, with ten as the same usability issues. However, these same usability issues have been rated differently. Although the t-test indicates no significant differences between experts and novices in their ratings for usability issues, these results nevertheless indicate the need for both evaluators in CHE to provide a more comprehensive perspective on the severity of the usability issues. Furthermore, the mapping of the usability issues for Nielsen and SMART heuristics concluded that more issues with the smartphone app could be addressed through smartphone-specific heuristics than general heuristics, indicating a better tool for heuristic evaluation of the smartphone app. This study also provides new insight into the required number of evaluators needed for CHE.

     

Interpretation of a cross-cultural usability evaluation: A case study based on a hypermedia system for rare species management in Namibia

We present the results of a usability evaluation of a locally developed hypermedia information system aiming at conservation biologists and wildlife managers in Namibia. Developer and end user come from different ethnic backgrounds, as is common to software development in Namibia and many developing countries. To overcome both the cultural and the authoritarian gap between usability evaluator and user, the evaluation was held as a workshop with usability evaluators who shared the target users’ ethnic and social backgrounds. Different data collection methods were used and results as well as specific incidences recorded. Results suggest that it is difficult for Namibian computer users to evaluate functionality independently from content. Users displayed evidence of a passive search strategy and an expectation that structure is provided rather than self generated. The comparison of data collection methods suggests that questionnaires are inappropriate in Namibia because they do not elicit a truthful response from participants who tend to provide answers they think are “expected”. The paper concludes that usability goals and methods have to be determined and defined within the target users’ cultural context.     

Human Factors Analysis,Design, and Evaluation of Engage,a Consumer Health IT Application for Geriatric Heart Failure Self-Care

ABSTRACT

Human factors and ergonomics (HFE) and related approaches can be used to enhance research and development of consumer-facing health information technology (IT) systems, including technologies supporting the needs of people with chronic disease. A multiphase HFE study of health IT supporting self-care of chronic heart failure by older adults is described. The study was based on HFE frameworks of “patient work” and incorporated the three broad phases of user-centered design: study or analysis, design, and evaluation. In the study phase, data from observations, interviews, surveys, and other methods were analyzed to identify gaps in and requirements for supporting heart failure self-care. The design phase applied findings from the study phase throughout an iterative process, culminating in the design of the Engage application, a product intended for continuous use over 30 days to stimulate self-care engagement, behavior, and knowledge. During the evaluation phase, a variety of usability issues through expert heuristic evaluation and laboratory-based usability testing were identified. The implications of our findings regarding heart failure self-care in older adults and the methodological challenges of rapid translational field research and development in this domain are discussed.     

Do patterns help novice evaluators? A comparative study

Evaluating e-learning systems is a complex activity which requires considerations of several criteria addressing quality in use as well as educational quality. Heuristic evaluation is a widespread method for usability evaluation, yet its output is often prone to subjective variability, primarily due to the generality of many heuristics. This paper presents the pattern-based (PB) inspection, which aims at reducing this drawback by exploiting a set of evaluation patterns to systematically drive inspectors in their evaluation activities. The application of PB inspection to the evaluation of e-learning systems is reported in this paper together with a study that compares this method to heuristic evaluation and user testing. The study involved 73 novice evaluators and 25 end users, who evaluated an e-learning application using one of the three techniques. The comparison metric was defined along six major dimensions, covering concepts of classical test theory and pragmatic aspects of usability evaluation. The study showed that evaluation patterns, capitalizing on the reuse of expert evaluators know-how, provide a systematic framework which reduces reliance on individual skills, increases inter-rater reliability and output standardization, permits the discovery of a larger set of different problems and decreases evaluation cost. Results also indicated that evaluation in general is strongly dependent on the methodological apparatus as well as on judgement bias and individual preferences of evaluators, providing support to the conceptualisation of interactive quality as a subjective judgement, recently brought forward by the UX research agenda.     

Developing SMASH: A set of SMArtphone's uSability Heuristics

The smartphone market is nowadays highly competitive. When buying a new device, users focus on visual esthetics, ergonomics, performance, and user experience, among others. Assessing usability issues allows improving these aspects. One popular method for detecting usability problems is heuristic evaluation, in which evaluators employ a set of usability heuristics as guide. Using proper heuristics is highly relevant. In this paper we present SMASH, a set of 12 usability heuristics for smartphones and mobile applications, developed iteratively. SMASH (previously named TMD: Usability heuristics for Touchscreen-based Mobile Devices) was experimentally validated. The results support its utility and effectiveness.