Seeking and providing assistance while learning to use information systems

Throughout their lives, people are faced with various learning situations, for example when they learn how to use new software, services or information systems. However, research in the field of Interactive Learning Environments shows that learners needing assistance do not systematically seek or use help, even when it is available. The aim of the present study is to explore the role of some factors from research in Interactive Learning Environments in another situation: using a new technology not as a means of acquiring knowledge but to realize a specific task. Firstly, we present the three factors included in this study (1) the role of the content of assistance, namely operative vs. function-oriented help; (2) the role of the user’s prior knowledge; (3) the role of the trigger of assistance, i.e. help provided after the user’s request vs. help provided by the system. In this latter case, it is necessary to detect the user’s difficulties. On the basis of research on problem-solving, we list behavioral criteria expressing the user’s difficulties. Then, we present two experiments that use “real” technologies developed by a large company and tested by “real” users. The results showed that (1) even when participants had reached an impasse, most of them never sought assistance, (2) operative assistance that was automatically provided by the system was effective for novice users, and (3) function-oriented help that was automatically provided by the system was effective for expert users. Assistance can support deadlock awareness and can also focus on deadlock solving by guiding task. Assistance must be adapted to prior knowledge, progress and goals of learners to improve learning.     

Collaborative control for a robotic wheelchair: evaluation of performance, attention, and workload

Powered wheelchair users often struggle to drive safely and effectively and, in more critical cases, can only get around when accompanied by an assistant. To address these issues, we propose a collaborative control mechanism that assists users as and when they require help. The system uses a multiple-hypothesis method to predict the driver's intentions and, if necessary, adjusts the control signals to achieve the desired goal safely. The main emphasis of this paper is on a comprehensive evaluation, where we not only look at the system performance but also, perhaps more importantly, characterize the user performance in an experiment that combines eye tracking with a secondary task. Without assistance, participants experienced multiple collisions while driving around the predefined route. Conversely, when they were assisted by the collaborative controller, not only did they drive more safely but also they were able to pay less attention to their driving, resulting in a reduced cognitive workload. We discuss the importance of these results and their implications for other applications of shared control, such as brain-machine interfaces, where it could be used to compensate for both the low frequency and the low resolution of the user input.     

Adapting the human plausible reasoning theory to a graphical user interface

This paper describes the adaptation of a cognitive theory, called Human Plausible Reasoning (HPR), for the purposes of an intelligent graphical user interface (GUI). The GUI is called intelligent file manipulator (IFM) and manages files and folders in a similar way as the Windows 98/NT Explorer. However, IFM also incorporates intelligence, which aims at rendering the interaction more human-like than in a standard explorer in terms of assistance to users' errors. IFM constantly reasons about users' actions, goals, plans, and possible errors, and offers automatic assistance in case of a problematic situation. HPR is used in IFM to simulate the reasoning of users in its user modeling component and the reasoning of human expert helpers when they try to provide assistance to users. The adaptation of HPR in IFM has focused on the domain representation, statement transforms, and certainty parameters. The certainty parameters of HPR have been combined in a novel way with user stereotypes and the simple additive weighting theory. IFM has been evaluated and the evaluation results showed that IFM could generate plausible hypotheses about users' errors and helpful advice to a satisfactory extent; hence, HPR seemed to have fulfilled the purpose for which it was incorporated in IFM.     

Using redundancies to find errors

Programmers generally attempt to perform useful work. If they performed an action, it was because they believed it served some purpose. Redundant operations violate this belief. However, in the past, redundant operations have been typically regarded as minor cosmetic problems rather than serious errors. This paper demonstrates that, in fact, many redundancies are as serious as traditional hard errors (such as race conditions or pointer dereferences). We experimentally test this idea by writing and applying five redundancy checkers to a number of large open source projects, finding many errors. We then show that, even when redundancies are harmless, they strongly correlate with the presence of traditional hard errors. Finally, we show how flagging redundant operations gives a way to detect mistakes and omissions in specifications. For example, a locking specification that binds shared variables to their protecting locks can use redundancies to detect missing bindings by flagging critical sections that include no shared state.     

HR Contribution to IT Innovation Implementation: Results of Three Case Studies

The theoretical and empirical investigation of information technologies (IT) innovation implementations reveals that they lack conceptual and practical support from HRM professionals. The HRM practices undertaken by HR professionals are not even discussed. We argue that HR departments should contribute to IT innovation projects by being responsible for explicitly defining job tasks that have to be automated, establishing a rewards system for those who have to learn and use a new technology and analysing the training needs of the users and providing them with adequate training. HRM should more actively intervene in IT innovation projects. Such intervention would, we believe, foster compatibility between the prior intentions of information technology, the strategies and practices, the individual users’ needs in it and the adoption of IT by the users.     

Seeking and providing assistance while learning to use information systems

Throughout their lives, people are faced with various learning situations, for example when they learn how to use new software, services or information systems. However, research in the field of Interactive Learning Environments shows that learners needing assistance do not systematically seek or use help, even when it is available. The aim of the present study is to explore the role of some factors from research in Interactive Learning Environments in another situation: using a new technology not as a means of acquiring knowledge but to realize a specific task. Firstly, we present the three factors included in this study (1) the role of the content of assistance, namely operative vs. function-oriented help; (2) the role of the user’s prior knowledge; (3) the role of the trigger of assistance, i.e. help provided after the user’s request vs. help provided by the system. In this latter case, it is necessary to detect the user’s difficulties. On the basis of research on problem-solving, we list behavioral criteria expressing the user’s difficulties. Then, we present two experiments that use “real” technologies developed by a large company and tested by “real” users. The results showed that (1) even when participants had reached an impasse, most of them never sought assistance, (2) operative assistance that was automatically provided by the system was effective for novice users, and (3) function-oriented help that was automatically provided by the system was effective for expert users. Assistance can support deadlock awareness and can also focus on deadlock solving by guiding task. Assistance must be adapted to prior knowledge, progress and goals of learners to improve learning.     

The effects of errors on system trust, self-confidence, and the allocation of control in route planning

The concept of trust is believed by some to compensate for feelings of uncertainty. Therefore, trust is considered to be crucial in people's decision to rely on a complex automated system to perform tasks for them. This experiment aimed to study the effects of errors on control allocation, and the mediating role of trust and self-confidence in the domain of route planning. Using a computer-based route planner, participants completed 10 route-planning trials in manual mode, and 10 in automatic mode, allowing participants to become equally experienced in operating both modes. During these so-called fixed trials, the numbers of errors in automatic as well as manual mode were systematically varied. Subsequently, participants completed six free trials, during which they were free to choose between modes. Our results showed that high automation error rates (AERs) decreased levels of system trust compared to low AERs. Conversely, high manual error rates (MERs) resulted in lower levels of self-confidence compared to low MERs, although to a lesser extent. Moreover, the difference between measures of trust and self-confidence proved to be highly predictive of the number of times automatic mode was selected during the six free trials. Additionally, results suggest a fundamental bias to trust one's own abilities over those of the system. Finally, evidence indicating a relationship between trust and self-confidence is discussed.     

A life story in three parts: the use of triptychs to make sense of personal digital data

Many social media platforms support the curation of personal digital data, and, more recently, the use of that data for review and reflection. We explored the process of reflection by asking users to create a meaningful ‘triptych’ of photographs drawn from their Facebook accounts. In a first study, we asked participants to manually trawl their own accounts and select three relevant images, which we then framed and used as an interview probe. In a second study, we designed an automated triptych generation system and assessed participants’ experiences of using this system. We conducted qualitative analyses of participant interviews from both studies. Consistent with other ‘slow technology’ work, we found the act of creating a physical artefact from social media data gave that data new meaning, albeit with notable differences between manual versus automatically generated triptychs. We conclude by discussing possible improvements to the design of the automated triptych system.     

Investigating the Effect of System Reliability,Risk, and Role on Users' Emotions and Attitudes toward a Safety-Critical Drone System

In safety-critical systems, it is essential to communicate relevant information to facilitate decision-making, promote trust, and improve performance without overloading users. To explore the effect of system performance information on rational and emotional processing by users, we performed a between-subject experiment in which participants were asked to imagine themselves as a drone operator or system administrator in a high-, medium-, or low-risk scenario. Then, based on their imagined scenario and role, participants rated the relevance of four aspects of system reliability to decision-making with the system, as well as the expected intensity of the GREAT emotions. Results indicate that system performance information affected participants’ reasoning differently depending on risk level. Moreover, participants had different perspectives depending on their role in the system. Those in administrator roles indicated higher respect ratings for those with a similar role. These findings demonstrate that contextual risk and a user’s role can influence emotions and attitudes toward safety-critical computer systems.     

Enabling effective design of multimodal interfaces for speech-to-speech translation system: An empirical study of longitudinal user behaviors over time and user strategies for coping with errors

The study provides an empirical analysis of long-term user behavioral changes and varying user strategies during cross-lingual interaction using the multimodal speech-to-speech (S2S) translation system of USC/SAIL. The goal is to inform user adaptive designs of such systems. A 4-week medical-scenario-based study provides the basis for our analysis. The data analyzed includes user interviews, post-session surveys, and the extensive system logs that were post-processed and annotated. The annotations measured the meaning transfer rates using human evaluations and a scale defined here called the concept matching score.First, qualitative data analysis investigates user strategies in dealing with errors, such as repeat, rephrase, change topic, start over, and the participants’ self-reported longitudinal adaptation to errors. Post-session surveys explore participant experience with the system and point to a trend of user-perceived increased performance over time.The log data analysis provides further insightful results. Users chose to allow some degradation (84% of original concepts) of their intended meaning to proceed through the system, even after they observed potential errors in the visual output from the speech recognizer. The rejected utterances, on average, had only 25% of the original concepts. This user-filtered outcome, after the complete channel transfer through the S2S system, is that 91% of the successful turns result in transfer of at least half the intended concepts while 90% of the user rejected turns would have conveyed less than half the intended meaning.The multimodal interface results in 24% relative improvement in the confirmation mode and in 31% relative improvement in the choice mode compared to the speech-only modality. Analysis also showed that users of the multimodal interface temporally change their strategies by accepting more system-produced choices. This user behavior can expedite communication seeking an operating balance between user strategies and system performance factors. Lastly, user utterance length is analyzed. Longer utterances in general imply more information delivered per utterance but potentially at the cost of increased processing degradation. The analysis demonstrates that users reduce their utterance length after unsuccessful turns and increase it after successful turns and that there is a learning effect that increases this behavior over the duration of the study.     

Error explanation with distance metrics

In the event that a system does not satisfy a specification, a model checker will typically automatically produce a counterexample trace that shows a particular instance of the undesirable behavior. Unfortunately, the important steps that follow the discovery of a counterexample are generally not automated. The user must first decide if the counterexample shows genuinely erroneous behavior or is an artifact of improper specification or abstraction. In the event that the error is real, there remains the difficult task of understanding the error well enough to isolate and modify the faulty aspects of the system. This paper describes a (semi-)automated approach for assisting users in understanding and isolating errors in ANSI C programs. The approach, derived from Lewis’ counterfactual approach to causality, is based on distance metrics for program executions. Experimental results show that the power of the model checking engine can be used to provide assistance in understanding errors and to isolate faulty portions of the source code.     

A knowledge and collaboration-based CBR process to improve network performance-related support activities

In a context characterized by a growing demand for networked services, users of advanced applications sometimes face network performance troubles that may actually prevent them from completing their tasks. Therefore, providing assistance for user communities that have difficulties using the network has been identified as one of the major issues of performance-related support activities. Despite the advances network management has made over the last years, there is a lack of guidance services to provide users with information that goes beyond merely presenting network properties. In this light, the research community has been highlighting the importance of User-Perceived Quality (UPQ) scores during the evaluation of network services for network applications, such as Quality of Experience (QoE) and Mean Opinion Score (MOS). However, despite their potential to assist end-users to deal with network performance troubles, only few types of network applications have well established UPQ scores. Besides that, they are defined through experiments essentially conducted in laboratory, rather than actual usage. This paper thus presents a knowledge and Collaboration-based Network Users’ Support (CNUS) Case-Based Reasoning (CBR) Process that predicts UPQ scores to assist users by focusing on the collaboration among them through the sharing of their experiences in using network applications. It builds (i) a knowledge base that includes not only information about network performance problems, but also applications’ characteristics, (ii) a case base that contains users’ opinions, and (iii) a user database that stores users’ profiles. By processing them, CNUS benefits users through the indication of the degree of satisfaction they may achieve based on the general opinion from members of their communities in similar contexts. In order to evaluate the suitability of CNUS, a CBR system was built and validated through an experimental study conducted in laboratory with a multi-agent system that simulated scenarios where users request for assistance. The simulation was supported by an ontology of network services and applications and reputation scheme implemented through the PageRank algorithm. The results of the study pointed to the effectiveness of CNUS, and its resilience to users’ collusive and incoherent behaviors. Besides that, they showed the influence of the knowledge about network characteristics, users’ profiles and application features on computer-based support activities.     

<Emphasis Type="SmallCaps">ProB</Emphasis>: an automated analysis toolset for the B method

We present ProB, a validation toolset for the B method. ProB’s automated animation facilities allow users to gain confidence in their specifications. ProB also contains a model checker and a refinement checker, both of which can be used to detect various errors in B specifications. We describe the underlying methodology of ProB, and present the important aspects of the implementation. We also present empirical evaluations as well as several case studies, highlighting that ProB enables users to uncover errors that are not easily discovered by existing tools. This research was carried out as part of the EU research project IST 511599 RODIN (Rigorous Open Development Environment for Complex Systems).     

Semi-automated CCTV surveillance: The effects of system confidence,system accuracy and task complexity on operator vigilance,reliance and workload

Recent advances in computer vision technology have lead to the development of various automatic surveillance systems, however their effectiveness is adversely affected by many factors and they are not completely reliable. This study investigated the potential of a semi-automated surveillance system to reduce CCTV operator workload in both detection and tracking activities. A further focus of interest was the degree of user reliance on the automated system. A simulated prototype was developed which mimicked an automated system that provided different levels of system confidence information. Dependent variable measures were taken for secondary task performance, reliance and subjective workload. When the automatic component of a semi-automatic CCTV surveillance system provided reliable system confidence information to operators, workload significantly decreased and spare mental capacity significantly increased. Providing feedback about system confidence and accuracy appears to be one important way of making the status of the automated component of the surveillance system more ‘visible’ to users and hence more effective to use.     

Evaluation of a traceability approach for informal freehand sketches

Most engineers and designers prefer to use large drawing boards such as whiteboards or flip charts for the initial collaborative sketching of a system’s models. Large interactive displays have recently begun to replace these physical drawing boards, blurring the line between freehand sketching and toolkit-aided modeling. While digital boards offer more flexibility in drawing and navigating models, they must also provide appropriate cognitive support for frequent shifts of focus and navigation between related artifacts. Furthermore, automated assistance in uncovering potential inconsistencies and contradictions between model sketches would be beneficial so that users do not get lost amid their sketches. In this paper, we discuss an approach to create relationships between the elements of informal hand-drawn sketches on large interactive displays by combining fuzzy search with classic information retrieval techniques. The identification and maintenance of relationships is particularly challenging because we are working with hand-drawn and hand-lettered model sketches rather than the syntactically clean models created with digital modeling toolkits. We evaluated our approach by analyzing 89 model sketches from 16 industry projects and found that it identifies relations between sketched model elements with high precision and recall.     

Highlighting items as means of adaptive assistance

Providing adaptive help during interaction with the system can be used to assist users in accomplishing their tasks. We propose providing guidance by highlighting the steps required for performing a task that the user intends to complete according to the prediction of a system. We present a study aimed at examining whether highlighting intended user steps in menus and toolbars as a means of assisting users in performing tasks is useful in terms of user response and performance. We also examined the effects of different accuracy levels of help and the control format on user response and performance. An experiment was conducted in which 64 participants performed tasks using menus and toolbars of a simulated email application. Participants were offered a highlighted guidance of the required steps in varying levels of accuracy (100%, 80%, 60% and no guidance). Our results support the benefits of highlighted help both in user performance times and in user satisfaction from receiving such assistance. Users found the assistance necessary and helpful and by the same token not unduly intrusive. Additionally, users felt that such assistance generally helped in reducing performance time on tasks. We did not find a significant difference when users receiving help at 80% accuracy was compared to those receiving help at 100% accuracy; however, such a difference does appear for those receiving 60% accuracy. In such cases, we found that the user's satisfaction level, perceived usefulness and trust in the system decreased while their notion of perceived intrusiveness increased. We conclude that assisting users by highlighting the required steps is useful so long as the minimal accuracy level of help is higher than 60%. Our study has implications on the implementation of highlighting next steps as a means of adaptive help and on integrating probability-based algorithms such as intention prediction to adaptive assistance systems.     

Images of self and others as computer users: the role of gender and experience

Abstract Gender differences in the pursuit of technology careers are a current issue of concern. We report on two studies that use surveys, drawings and interviews to examine sixth- and eighth-grade students' perceptions of knowledgeable computer users and their self-perception as a computer-type person. In Study 1, participants were asked to generate representations of computer users in pictures or words. The results indicate that the majority of representations were of male users and they frequently wore glasses. Students of both genders were more likely to draw males. Eighth-grade students' representations included more stereotypical features than those of sixth-grade students. In Study 2, students were asked whether they believed that there was such a thing as a computer-type person and whether they perceived themselves to be one. Eighty per cent of students rejected this characterization. They differed from students who accepted it in their levels of past experience, their confidence, and the probability that they shared their knowledge with others. The results of both studies suggest that while there is a male image of computer science in general, it is not overly negative and students' self-perception is not governed by their own gender as much as by other variables.     

Misuse of automated decision aids: Complacency, automation bias and the impact of training experience

The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment (N=24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behavior, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids.     

The effects of self-awareness on body movement indicators of the intention to deceive

A study was conducted to investigate the body movements of participants waiting to be interviewed in one of two conditions: preparing to answer questions truthfully or preparing to lie. The effects of increased self-awareness were also investigated, with half of the participants facing a mirror; the other half facing a blank wall. Analysis of covertly obtained video footage showed a significant interaction for the duration of hand/arm movements between deception level and self-awareness. Without a mirror, participants expecting to lie spent less time moving their hands than those expecting to tell the truth; the opposite was seen in the presence of a mirror. Participants expecting to lie also had higher levels of anxiety and thought that they were left waiting for less time than those expecting to tell the truth. These findings led to the identification of further research areas with the potential to support deception detection in security applications.     

Is talking to an automated teller machine natural and fun?

Usability and affective issues of using automatic speech recognition technology to interact with an automated teller machine (ATM) are investigated in two experiments. The first uncovered dialogue patterns of ATM users for the purpose of designing the user interface for a simulated speech ATM system. Applying the Wizard-of-Oz methodology, multiple mapping and word spotting techniques, the speech driven ATM accommodates bilingual users of Bahasa Melayu and English. The second experiment evaluates the usability of a hybrid speech ATM, comparing it with a simulated manual ATM. The aim is to investigate how natural and fun can talking to a speech ATM be for these first-time users. Subjects performed the withdrawal and balance enquiry tasks. The ANOVA was performed on the usability and affective data. The results showed significant differences between systems in the ability to complete the tasks as well as in transaction errors. Performance was measured on the time taken by subjects to complete the task and the number of speech recognition errors that occurred. On the basis of user emotions, it can be said that the hybrid speech system enabled pleasurable interaction. Despite the limitations of speech recognition technology, users are set to talk to the ATM when it becomes available for public use.