The Impact of Geographic Location on the Subjective Assessment of System Usability

Marketing researchers use geography to identify specific user groups for studies to more effectively describe their potential customer base. Since usability professionals often recruit users employing similar selection criteria as their marketing peers, the use of geographic information might also be relevant when selecting usability test participants. In total, 3,168 participants from across the United States rated the usability of different hardware, software, and web-based products using the System Usability Scale (SUS). SUS scores were compared across geographic divisions to determine if usability assessments differ by location. SUS scores were also compared across rural and urban areas to determine if usability assessment scores change with population density. There was a lack of evidence to support significant differences in usability scores across both US geographic areas and zones of population density. The findings suggest that people make similar system usability assessments regardless of the area of the United States in which they live.     

The Relationship Between System Effectiveness and Subjective Usability Scores Using the System Usability Scale

This article examines the relationship between users’ subjective usability assessments, as measured using the System Usability Scale (SUS), and the ISO metric of effectiveness, using task success as the measure. The article reports the results of two studies designed to explore the relationship between SUS scores and user success rates for a variety of interfaces. The first study was a field study, where stereotypical usability assessments on a variety of products and services were performed. The second study was a well-controlled laboratory study where the level of success that users were able to achieve was controlled. For both studies, the relationship between SUS scores and their attendant performance were examined at both the individual level and the average system level. Although the correlations are far from perfect, there are reliable and reasonably strong positive correlations between subjective usability measures and task success rates, for both the laboratory and field studies at both the individual and system level.     

Measuring Perceived Usability: SUS,UMUX, and CSUQ Ratings for Four Everyday Products

This research continued previous investigation of the relationships among measures of perceived usability: the System Usability Scale (SUS), three metrics derived from the Usability Metric for User Experience (UMUX), and the Computer System Usability Questionnaire (CSUQ), this time with ratings of four everyday products (Excel, Word, Amazon, and Gmail). SUS ratings of these products were generally consistent with previous reports. Significant differences in SUS means across studies could be due to differences in frequency of use, with implications for using these data as usability benchmarks. Correspondence among the various measures of perceived usability was also consistent with previous research. Considering frequency of use, mean differences ranged from -2.0 to 1.8 (average shift in Sauro-Lewis grade range from -0.6 to 0.8). When SUS scores were above average, the range restriction of the UMUX-LITEr led to relatively large discrepancies with SUS, suggesting it might not always be better than unadjusted UMUXLITE.     

The Impact of Personality on the Subjective Assessment of Usability

The system usability scale (SUS) has been widely employed in both the field and the laboratory as a valid and reliable measure of system usability. Although its psychometric properties are relatively well understood, the impact that differences in users’ personality traits have on their perceived usability of products, services, and systems has not been deeply explored—even though people’s scores on personality traits have been shown to be reliable and predict a staggering array of societally important outcomes in work, school, and life domains. In this study, 268 users assessed the usability of 20 different products retrospectively with the SUS. Five broad personality traits were then measured using the Mini-IPIP scale. Results indicated that measured personality traits do correlate with the rated usability of products, where measures of Openness to Experience and Agreeableness have the strongest positive correlations with subjective usability assessment. Implications for practitioners, designers, and researchers are discussed.     

Measuring the Usability of Mobile Applications for Phones and Tablets

The use of applications on mobile devices has reached historic levels. Using the System Usability Scale (SUS), data were collected on the usability of applications used on two kinds of mobile platforms—phones and tablets—across two general classes of operating systems, iOS and Android. Over 4 experiments, 3,575 users rated the usability of 10 applications that had been selected based on their popularity, as well as 5 additional applications that users had identified as using frequently. The average SUS rating for the top 10 apps across all platforms was 77.7, with a nearly 20-point spread (67.7–87.4) between the highest and lowest rated apps. Overall, applications on phone platforms were judged to be more usable than applications on the tablet platforms. Practitioners can use the information in this article to make better design decisions and benchmark their progress against a known universe of apps for their specific mobile platform.     

Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study

This article describes the psychometric properties of the Emotional Metric Outcomes (EMO) questionnaire and the System Usability Scale (SUS) using data collected as part of a large-sample unmoderated usability study (n = 471). The EMO is a concise multifactor standardized questionnaire that provides an assessment of transaction-driven personal and relationship emotional outcomes, both positive and negative. The SUS is a well-known standardized usability questionnaire designed to assess perceived usability. In previous research, psychometric evaluation using data from a series of online surveys showed that the EMO and its component scales had high reliability and concurrent validity with loyalty and overall experience metrics but did not find the expected four-factor structure. Previous structural analyses of the SUS have had mixed results. Analysis of the EMO data from the usability study revealed the expected four-factor structure. The factor structure of the SUS appeared to be driven by item tone. The estimated reliability of the SUS (.90) was consistent with previous estimates. The EMO and its subscales were also quite reliable, with the estimates of reliability for the various EMO scales ranging from .86 to .96. Regression analysis using SUS, EMO, and Effort as predictors revealed different key drivers for the outcome metrics of Satisfaction and Likelihood-to-Recommend. The key recommendations are to include the EMO as part of the battery of poststudy standardized questionnaires, along with the SUS (or similar questionnaire), but to be cautious in reporting SUS subscales such as Usable and Learnable.     

A Slovene Translation of the System Usability Scale: The SUS-SI

The System Usability Scale (SUS) is a widely adopted and studied questionnaire for usability evaluation. It is technology independent and has been used to evaluate the perceived usability of a broad range of products, including hardware, software, and websites. In this article we present a Slovene translation of the SUS (the SUS-SI) along with the procedure used in its translation and psychometric evaluation. The results indicated that the SUS-SI has properties similar to the English version. Slovene usability practitioners should be able to use the SUS-SI with confidence when conducting user research.     

The System Usability Scale: Past,Present, and Future

The System Usability Scale (SUS) is the most widely used standardized questionnaire for the assessment of perceived usability. This review of the SUS covers its early history from inception in the 1980s through recent research and its future prospects. From relatively inauspicious beginnings, when its originator described it as a “quick and dirty usability scale,” it has proven to be quick but not “dirty.” It is likely that the SUS will continue to be a popular measurement of perceived usability for the foreseeable future. When researchers and practitioners need a measure of perceived usability, they should strongly consider using the SUS.     

Usability of RPG-based Learning Framework

The study assessed the usability of the Role-Playing Game-based Learning Framework (RPG LF) using the System Usability Scale (SUS). The test was conducted on a group of 35 undergraduates who played the sample implementation of the RPG LF. The study concluded with the respondents giving positive feedback on the use of the RPG LF for learning. Detailed probing into gender comparison and academic age group comparison revealed more details about the usability of RPG LF. There was not much difference between the sexes. Younger academic age group demonstrated more uncertainty compared to their seniors in terms of the usability of RPG LF. The study also factorized the original SUS data into 8-question Usability scale and 2-question Learnability scale, respectively. The findings were in line with the findings by other researchers that the 8-question Usability scale is a good substitute for the original 10-question SUS and is a better candidate in comparison to the 2-question Learnability scale. The study concluded that the RPG LF is a good framework to implement game into enhancing learning experience and suggested for longitudinal study to be conducted to assess the effectiveness of RPG LF.     

Perspective-based Usability Inspection: An Empirical Validation of Efficacy

Inspection is a fundamental means of achieving software usability. Past research showed that the current usability inspection techniques were rather ineffective. We developed perspective-based usability inspection, which divides the large variety of usability issues along different perspectives and focuses each inspection session on one perspective. We conducted a controlled experiment to study its effectiveness, using a post-test only control group experimental design, with 24 professionals as subjects. The control group used heuristic evaluation, which is the most popular technique for usability inspection. The experimental design and the results are presented, which show that inspectors applying perspective-based inspection not only found more usability problems related to their assigned perspectives, but also found more overall problems. Perspective-based inspection was shown to be more effective for the aggregated results of multiple inspectors, finding about 30% more usability problems for 3 inspectors. A management implication of this study is that assigning inspectors more specific responsibilities leads to higher performance. Internal and external threats to validity are discussed to help better interpret the results and to guide future empirical studies.     

Measuring Perceived Usability: The SUS,UMUX-LITE,and AltUsability

The purpose of this research was to investigate various measurements of perceived usability, in particular, to assess (a) whether a regression formula developed previously to bring Usability Metric for User Experience LITE (UMUX-LITE) scores into correspondence with System Usability Scale (SUS) scores would continue to do so accurately with an independent set of data; (b) whether additional items covering concepts such as findability, reliability, responsiveness, perceived use by others, effectiveness, and visual appeal would be redundant with the construct of perceived usability or would align with other potential constructs; and (c) the dimensionality of the SUS as a function of self-reported frequency of use and expertise. Given the broad use of and emerging interpretative norms for the SUS, it was encouraging that the regression equation for the UMUX-LITE worked well with this independent set of data, although there is still a need to investigate its efficacy with a broader set of products and methods. Results from a series of principal components analyses indicated that most of the additional concepts, such as findability, familiarity, efficiency, control, and visual appeal covered the same statistical ground as the other more standard metrics for perceived usability. Two of the other items (Reliable and Responsive) made up a reliable construct named System Quality. None of the structural analyses of the SUS as a function of frequency of use or self-reported expertise produced the expected components, indicating the need for additional research in this area and a need to be cautious when using the Usable and Learnable components described in previous research.     

An Empirical Evaluation of the System Usability Scale

This article presents nearly 10 year's worth of System Usability Scale (SUS) data collected on numerous products in all phases of the development lifecycle. The SUS, developed by Brooke (1996) Brooke, J. 1996. “SUS: A “quick and dirty” usability scale”. In Usability evaluation in industry, Edited by: Jordan, P. W., Thomas, B. A. Weerdmeester and McClelland, I. L. 189–194. London: Taylor & Francis.  [Google Scholar], reflected a strong need in the usability community for a tool that could quickly and easily collect a user's subjective rating of a product's usability. The data in this study indicate that the SUS fulfills that need. Results from the analysis of this large number of SUS scores show that the SUS is a highly robust and versatile tool for usability professionals. The article presents these results and discusses their implications, describes nontraditional uses of the SUS, explains a proposed modification to the SUS to provide an adjective rating that correlates with a given score, and provides details of what constitutes an acceptable SUS score.     

Usability evaluation of a virtual reality smartphone app for a living museum

This paper elaborates the empirical evidence of a usability evaluation of a VR and non-VR virtual tour application for a living museum. The System Usability Scale (SUS) was used in between participants experiments (Group 1: non-VR version and Group 2: VR version) with 40 participants. The results show that the mean scores of all components for the VR version are higher compared to the non-VR version, overall SUS score (72.10 vs 68.10), usability score (75.50 vs 71.70), and learnability (58.40 vs 57.00). Further analysis using a two-tailed independent t test showed no difference between the non-VR and VR versions. Additionally, no significant difference was observed between the groups in the context of gender, nationality, and prior experience (other VR tour applications) for overall SUS score, usability score, and learnability score. Α two-tailed independent t test indicated no significant difference in the usability score between participants with VR experience and no VR experience. However, a significant difference was found between participants with VR experience and no VR experience for both SUS score (t(38) = 2.17, p = 0.037) and learnability score (t(38) = 2.40, p = 0.021). The independent t test results indicated a significant difference between participant with and without previous visits to SCV for the usability score (t(38) = −2.31, p = 0.027), while there was no significant differences observed in other components. It can be concluded that both versions passed based on the SUS score. However, the sub-scale usability and learnability scores indicated some usability issue.

     

Measuring Perceived Usability: The CSUQ,SUS, and UMUX

The primary purpose of this research was to investigate the relationship between two widely used questionnaires designed to measure perceived usability: the Computer System Usability Questionnaire (CSUQ) and the System Usability Scale (SUS). The correlation between concurrently collected CSUQ and SUS scores was 0.76 (over 50% shared variance). After converting CSUQ scores to a 0–100-point scale (to match the range of the SUS scores), there was a small but statistically significant difference between CSUQ and SUS means. Although this difference (just under 2 scale points out of a possible 100) was statistically significant, it did not appear to be practically significant. Although usability practitioners should be cautious pending additional independent replication, it appears that CSUQ scores, after conversion to a 0–100-point scale, can be interpreted with the Sauro–Lewis curved grading scale. As a secondary research goal, investigation of variations of the Usability Metric for User Experience (UMUX) replicated previous findings that the regression-adjusted version of the UMUX-LITE (UMUX-LITEr) had the closest correspondence with concurrently collected SUS scores. Thus, even though these three standardized questionnaires were independently developed and have different item content and formats, they largely appear to be measuring the same thing, presumably, perceived usability.     

Towards a Standard Arabic System Usability Scale: Psychometric Evaluation using Communication Disorder App

This research aims to customize the System Usability Scale (SUS) so it can be used as a standard usability measure on native Arabic-language speakers. A process of rigor translation followed by psychometric evaluation has been administered through a questionnaire in the Arabic-language that reflects a reasonable level of reliability, validity and sensitivity when compared to the original English-language SUS questionnaire. The final result of the Arabic-System Usability Scale (A-SUS) is then applied on students majoring in Communication Disorders Sciences (Kuwait University) to measure the usability of a mobile application. A-SUS encapsulates the essence of the original SUS and provides an adequate tool for professionals to use with Arabic native speaking users to evaluate systems usability.     

The Inference of Perceived Usability From Beauty

A review of 15 papers reporting 25 independent correlations of perceived beauty with perceived usability showed a remarkably high variability in the reported coefficients. This may be due to methodological inconsistencies. For example, products are often not selected systematically, and statistical tests are rarely performed to test the generality of findings across products. In addition, studies often restrict themselves to simply reporting correlations without further specification of underlying judgmental processes.

The present study's main objective is to re-examine the relation between beauty and usability, that is, the implication that “what is beautiful is usable.” To rectify previous methodological shortcomings, both products and participants were sampled in the same way and the data aggregated both by averaging over participants to assess the covariance across ratings of products and by averaging over products to assess the covariance across participants. In addition, we adopted an inference perspective to qualify underlying processes to examine the possibility that, under the circumstances pertaining in most studies of this kind where participants have limited experience of using a website or product, the relationship between beauty and usability is mediated by goodness.

A mediator analysis of the relationship between beauty, the overall evaluation (i.e., “goodness”) and pragmatic quality (as operationalization of usability) suggests that the relationship between beauty and usability has been overplayed as the correlation between pragmatic quality and beauty is wholly mediated by goodness. This pattern of relationships was consistent across four different data sets and different ways of data aggregation. Finally, suggestions are made regarding methodologies that could be used in future studies that build on these results.     

Generation,Adoption, and Tuning of Usability Evaluation Multimethods

The selection and customization of usability evaluation methods, given the peculiarities of their application domains, still remains a critical issue; this especially when dealing with complex products and/or nonexpert usability evaluators. Moreover, as time goes by, the quality of the evaluation results has a heavier impact on the product design process. Starting from classic usability evaluation methods, the research described in this article generates multimethods semiautomatically. It allows quantitative characterization of these multimethods before their application in the field and exploits the comparison between this prior assessment and a final estimate, made after adoption, to update the information used by the method selection process. The most critical issue related to usability, subjectivity, is considered and dealt with throughout the entire research. A case study, done at the end of the development phase, helps validate the proposed approach to usability evaluation.     

Images of Usability

The term usability is ubiquitous in human–computer interaction, so much so that it is commonly used without definition. Rather than one established meaning of usability, there are, however, multiple images of usability. Although each image provides a partial view, the partiality remains implicit unless confronted with alternative images. This study delineates six images of usability: universal usability, situational usability, perceived usability, hedonic usability, organizational usability, and cultural usability. The different foci of the images provide opportunities for becoming sensitized to manifold aspects of the use of a system and thereby acquiring a genuine understanding of its usability. The six images differ, for example, in the extent to which they include aspects of the outcome of the process of using a system or merely the process of use, whether they involve collaborative use or merely individual use, and in their view of usability as perceived by individuals or shared by groups. Several challenges result from recognizing that usability is a set of images rather than a coherent concept, including a risk of misunderstandings in discussions of usability because participants may assume different images of usability and a need for supplementary methods addressing the collaborative and long-term aspects of usability. Moreover, the images call for extending the scope of practical usability work to include the effects achieved by users during their use of systems for real work.     

Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes

To better support usability practice, most usability research focuses on evaluation methods. New ideas in usability research are mostly proposed as new evaluation methods. Many publications describe experiments that compare methods. Comparisons may indicate that some methods have important deficiencies, and thus often advise usability practitioners to prefer a specific method in a particular situation. An expectation persists in human–computer interaction (HCI) that results about evaluation methods should be the standard “unit of contribution” rather than favoring larger units (e.g., usability work as a whole) or smaller ones (e.g., the impact of specific aspects of a method). This article argues that these foci on comparisons and method innovations ignore the reality that usability evaluation methods are loose incomplete collections of resources, which successful practitioners configure, adapt, and complement to match specific project circumstances. Through a review of existing research on methods and resources, resources associated with specific evaluation methods, and ones that can complement existing methods, or be used separately, are identified. Next, a generic classification scheme for evaluation resources is developed, and the scheme is extended with project specific resources that impact the effective use of methods. With these reviews and analyses in place, implications for research, teaching, and practice are derived. Throughout, the article draws on culinary analogies. A recipe is nothing without its ingredients, and just as the quality of what is cooked reflects the quality of its ingredients, so too does the quality of usability work reflect the quality of resources as configured and combined. A method, like a recipe, is at best a guide to action for those adopting approaches to usability that are new to them. As with culinary dishes, HCI needs to focus more on what gets cooked, and how it gets cooked, and not just on how recipes suggest that it could be cooked.     

DEEP: Design-Oriented Evaluation of Perceived Usability

Existing perceived usability questionnaires detect the appearance of usability issues rather than the underlying design generating those issues. This limits the capability of existing instruments to directly inform design recommendations. To address this problem, a usability questionnaire structured around the analytical composition of the design was created and validated. A four-stage process was followed. First, 3 usability experts refined 54 questions from highly cited usability questionnaires and structured them around 6 design dimensions. Second, 12 raters scored the questions by their relevance to assess usability. Third, questions and dimensions were then improved through exploratory and confirmatory factor analysis (N?=?196) and, fourth, further enhanced through confirmatory factor analysis (N?=?362). The result is DEEP, a 19-question usability questionnaire based on 5 main design dimensions (content, information architecture, navigation, layout, and visual guidance). DEEP can be used to capture detailed usability feedback that more directly relates to specific aspects of design requirements.