This computer responds to user frustration:: Theory,design, and results

Use of technology often has unpleasant side effects, which may include strong, negative emotional states that arise during interaction with computers. Frustration, confusion, anger, anxiety and similar emotional states can affect not only the interaction itself, but also productivity, learning, social relationships, and overall well-being. This paper suggests a new solution to this problem: designing human–computer interaction systems to actively support users in their ability to manage and recover from negative emotional states. An interactive affect–support agent was designed and built to test the proposed solution in a situation where users were feeling frustration. The agent, which used only text and buttons in a graphical user interface for its interaction, demonstrated components of active listening, empathy, and sympathy in an effort to support users in their ability to recover from frustration. The agent's effectiveness was evaluated against two control conditions, which were also text-based interactions: (1) users’ emotions were ignored, and (2) users were able to report problems and ‘vent’ their feelings and concerns to the computer. Behavioral results showed that users chose to continue to interact with the system that had caused their frustration significantly longer after interacting with the affect–support agent, in comparison with the two controls. These results support the prediction that the computer can undo some of the negative feelings it causes by helping a user manage his or her emotional state.     

Extracting relevance and affect information from physiological text annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to (1) indicate perceived relevance and then to (2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction.     

Dynamic emotion modelling and anomaly detection in conversation based on emotional transition tensor

Conversational data in social media contain a great deal of useful information, and conversation anomaly detection is an important research direction in the field of sentiment analysis. Each user has his or her own specific emotional characteristic, and by studying the distribution and sampling the users’ emotional transitions, we can simulate specific emotional transitions in the conversations. Anomaly detection in conversation data refers to detecting users’ abnormal opinions and sentiment patterns as well as special temporal aspects of such patterns. This paper proposes a hybrid model that combines the convolutional neural network long short-term memory (CNN-LSTM) with a Markov chain Monte Carlo (MCMC) method to identify users’ emotions, sample users’ emotional transition and detect anomalies according to the transition tensor. The emotional transition sampling is implemented by improving the MCMC algorithm and the anomalies are detected by calculating the similarity between the normal transition tensor and the current transition tensor of the user. The experiment was carried on four corpora, and the results show that emotions can be well sampled to conform to user’s characteristics and anomaly can be detected by the proposed method. The model proposed can be used in intelligent conversation systems, such as simulating the emotional transition and detecting the abnormal emotions.     

面向虚实融合的人机交互

面向虚实融合的人机交互涉及计算机科学、认知心理学、人机工程学、多媒体技术和虚拟现实等领域，旨在提高人机交互的效率，同时响应人类认知与情感的需求，在办公教育、机器人和虚拟/增强现实设备中都有广泛应用。本文从人机交互涉及感知计算、人与机器人交互及协同、个性化人机对话和数据可视化等4个维度系统阐述面向虚实融合人机交互的发展现状。对国内外研究现状进行对比，展望未来的发展趋势。本文认为兼具可迁移与个性化的感知计算、具备用户行为深度理解的人机协同、用户自适应的对话系统等是本领域的重要研究方向。     

Affective recommender systems in online news industry: how emotions influence reading choices

Recommender systems have become ubiquitous over the last decade, providing users with personalized search results, video streams, news excerpts, and purchasing hints. Human emotions are widely regarded as important predictors of behavior and preference. They are a crucial factor in decision making, but until recently, relatively little has been known about the effectiveness of using human emotions in personalizing real-world recommender systems. In this paper we introduce the Emotion Aware Recommender System (EARS), a large scale system for recommending news items using user’s self-assessed emotional reactions. Our original contribution includes the formulation of a multi-dimensional model of emotions for news item recommendations, introduction of affective item features that can be used to describe recommended items, construction of affective similarity measures, and validation of the EARS on a large corpus of real-world Web traffic. We collect over 13,000,000 page views from 2,700,000 unique users of two news sites and we gather over 160,000 emotional reactions to 85,000 news articles. We discover that incorporating pleasant emotions into collaborative filtering recommendations consistently outperforms all other algorithms. We also find that targeting recommendations by selected emotional reactions presents a promising direction for further research. As an additional contribution we share our experiences in designing and developing a real-world emotion-based recommendation engine, pointing to various challenges posed by the practical aspects of deploying emotion-based recommenders.

     

人机交互中的智能体情感计算研究

机器的情感是通过融入具有情感能力的智能体实现的,虽然目前在人机交互领域已经有大量研究成果,但有关智能体情感计算方面的研究尚处起步阶段,深入开展这项研究对推动人机交互领域的发展具有重要的科学和应用价值。本文通过检索Scopus数据库选择有代表性的文献,重点关注情感在智能体和用户之间的双向流动,分别从智能体对用户的情绪感知和对用户情绪调节的角度开展分析总结。首先梳理了用户情绪的识别方法,即通过用户的表情、语音、姿态、生理信号和文本信息等多通道信息分析用户的情绪状态,归纳了情绪识别中的一些机器学习方法。其次从用户体验角度分析具有情绪表现力的智能体对用户的影响,总结了智能体的情绪生成和表现技术,指出智能体除了通过表情之外,还可以通过注视、姿态、头部运动和手势等非言语动作来表现情绪。并且梳理了典型的智能体情绪架构,举例说明了强化学习在智能体情绪设计中的作用。同时为了验证模型的准确性,比较了已有的情感评估手段和评价指标。最后指出智能体情感计算急需解决的问题。通过对现有研究的总结,智能体情感计算研究是一个很有前景的研究方向,希望本文能够为深入开展相关研究提供借鉴。     

Product design,semantics and emotional response

This paper explores theoretical issues in ergonomics related to semantics and the emotional content of design. The aim is to find answers to the following questions: how to design products triggering ‘happiness’ in one's mind; which product attributes help in the communication of positive emotions; and finally, how to evoke such emotions through a product. In other words, this is an investigation of the ‘meaning’ that could be designed into a product in order to ‘communicate’ with the user at an emotional level. A literature survey of recent design trends, based on selected examples of product designs and semantic applications to design, including the results of recent design awards, was carried out in order to determine the common attributes of their design language. A review of Good Design Award winning products that are said to convey and/or evoke emotions in the users has been done in order to define good design criteria. These criteria have been discussed in relation to user emotional responses and a selection of these has been given as examples.     

Data-driven models for timing feedback responses in a Map Task dialogue system

Traditional dialogue systems use a fixed silence threshold to detect the end of users’ turns. Such a simplistic model can result in system behaviour that is both interruptive and unresponsive, which in turn affects user experience. Various studies have observed that human interlocutors take cues from speaker behaviour, such as prosody, syntax, and gestures, to coordinate smooth exchange of speaking turns. However, little effort has been made towards implementing these models in dialogue systems and verifying how well they model the turn-taking behaviour in human–computer interactions. We present a data-driven approach to building models for online detection of suitable feedback response locations in the user's speech. We first collected human–computer interaction data using a spoken dialogue system that can perform the Map Task with users (albeit using a trick). On this data, we trained various models that use automatically extractable prosodic, contextual and lexico-syntactic features for detecting response locations. Next, we implemented a trained model in the same dialogue system and evaluated it in interactions with users. The subjective and objective measures from the user evaluation confirm that a model trained on speaker behavioural cues offers both smoother turn-transitions and more responsive system behaviour.     

Investigating the Effect of System Reliability,Risk, and Role on Users' Emotions and Attitudes toward a Safety-Critical Drone System

In safety-critical systems, it is essential to communicate relevant information to facilitate decision-making, promote trust, and improve performance without overloading users. To explore the effect of system performance information on rational and emotional processing by users, we performed a between-subject experiment in which participants were asked to imagine themselves as a drone operator or system administrator in a high-, medium-, or low-risk scenario. Then, based on their imagined scenario and role, participants rated the relevance of four aspects of system reliability to decision-making with the system, as well as the expected intensity of the GREAT emotions. Results indicate that system performance information affected participants’ reasoning differently depending on risk level. Moreover, participants had different perspectives depending on their role in the system. Those in administrator roles indicated higher respect ratings for those with a similar role. These findings demonstrate that contextual risk and a user’s role can influence emotions and attitudes toward safety-critical computer systems.     

Computer anxiety and anger: the impact of computer use, computer experience, and self-efficacy beliefs

Past research has demonstrated that the level of computer experience users have is the most valuable predictor in whether or not they will suffer computer anxiety symptoms, but this was not the case in the present study. No research was found which examined the correlates of computer anger symptoms. In the current study, the relationship between the computer use (frequency and duration), computer experience, and self-efficacy beliefs of users were analyzed as predictors for computer anxiety and anger symptoms. Questionnaire data from a sample of 242 university students were analyzed. The results indicated that computer self-efficacy beliefs, not computer experience or use, had the largest significant relationship with both computer anxiety, and anger. It is suggested that self-efficacy beliefs be increased so that users may experience lower levels of anxiety and anger. These findings are contrary to the trend of training computer users in specific computer domains. As computer anxiety and anger are negative psychological “states”, an immediate method to deal with these negative emotions should be developed. One possibility that is explored is the application of computer-based therapy that can be used while a user is experiencing negative emotional symptoms.     

Cross-generational analysis of predictive factors of addictive behavior in smartphone usage

Smartphone addiction has been widely researched in recent years, and the effects of various demographic, personality-linked, psychological, and emotional variables, have been found. Our research goal was to examine this phenomenon from the cross-generational perspective and compare the factors that can predict smartphone addiction for different age groups. We conducted a study with 216 Israeli smartphone users, representing three generations of smartphone users: Generation X, Generation Y, and Generation Z, who filled in an 80-item questionnaire. The factors examined included the social environment pressure to use smartphone, emotional gain from smartphones, personality, daily usage time, various mobile apps and user needs. The main finding of the study is that a significantly higher level of addictive behavior was found for Generation Y compared to the other two generations. The strongest predictive factors in the computed hierarchical regression model for all three generations were social environment pressure and emotional gain. Interestingly, emotional gain from smartphone use, which reflects users' enjoyment and positive emotions along with relief of negative emotions and psychological states, was significantly higher for generation Z than for the older generations. In addition, neuroticism and the daily usage time appeared as predictive factors for the younger generations, and for Generation Z alone the WhatsApp app usage was found as a significant predictive factor as well. This study contributes to understanding the factors of smartphone addictive behavior for different generations, which might lead to more effective educational measures and explanatory campaigns on technology effects on psychological well-being.     

Modeling context-aware and intention-aware in-car infotainment systems

It is fundamental to understand users’ intentions to support them when operating a computer system with a dynamically varying set of functions, e.g., within an in-car infotainment system. The system needs to have sufficient information about its own and the user’s context to predict those intentions. Although the development of current in-car infotainment systems is already model-based, explicitly gathering and modeling contextual information and user intentions is currently not supported. However, manually creating software that understands the current context and predicts user intentions is complex, error-prone and expensive. Model-based development can help in overcoming these issues. In this paper, we present an approach for modeling a user’s intention based on Bayesian networks. We support developers of in-car infotainment systems by providing means to model possible user intentions according to the current context. We further allow modeling of user preferences and show how the modeled intentions may change during run-time as a result of the user’s behavior. We demonstrate feasibility of our approach using an industrial case study of an intention-aware in-car infotainment system. Finally, we show how modeling of contextual information and modeling user intentions can be combined by using model transformation.     

Interacting meaningfully with machine learning systems: Three experiments

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple “right/wrong” judgments. If the users themselves could work hand-in-hand with machine learning systems, the users’ understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users’ willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich human–computer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence.     

How emotion is made and measured

How we design and evaluate for emotions depends crucially on what we take emotions to be. In affective computing, affect is often taken to be another kind of information—discrete units or states internal to an individual that can be transmitted in a loss-free manner from people to computational systems and back. While affective computing explicitly challenges the primacy of rationality in cognitivist accounts of human activity, at a deeper level it often relies on and reproduces the same information-processing model of cognition. Drawing on cultural, social, and interactional critiques of cognition which have arisen in human–computer interaction (HCI), as well as anthropological and historical accounts of emotion, we explore an alternative perspective on emotion as interaction: dynamic, culturally mediated, and socially constructed and experienced. We demonstrate how this model leads to new goals for affective systems—instead of sensing and transmitting emotion, systems should support human users in understanding, interpreting, and experiencing emotion in its full complexity and ambiguity. In developing from emotion as objective, externally measurable unit to emotion as experience, evaluation, too, alters focus from externally tracking the circulation of emotional information to co-interpreting emotions as they are made in interaction.     

Touch gestures in communicating emotional intention via vibrotactile stimulation

Remote communication between people typically relies on audio and vision although current mobile devices are increasingly based on detecting different touch gestures such as swiping. These gestures could be adapted to interpersonal communication by using tactile technology capable of producing touch stimulation to a user's hand. It has been suggested that such mediated social touch would allow for new forms of emotional communication. The aim was to study whether vibrotactile stimulation that imitates human touch can convey intended emotions from one person to another. For this purpose, devices were used that converted touch gestures of squeeze and finger touch to vibrotactile stimulation. When one user squeezed his device or touched it with finger(s), another user felt corresponding vibrotactile stimulation on her device via four vibrating actuators. In an experiment, participant dyads comprising a sender and receiver were to communicate variations in the affective dimensions of valence and arousal using the devices. The sender's task was to create stimulation that would convey unpleasant, pleasant, relaxed, or aroused emotional intention to the receiver. Both the sender and receiver rated the stimulation using scales for valence and arousal so that the match between sender's intended emotions and receiver's interpretations could be measured. The results showed that squeeze was better at communicating unpleasant and aroused emotional intention, while finger touch was better at communicating pleasant and relaxed emotional intention. The results can be used in developing technology that enables people to communicate via touch by choosing touch gesture that matches the desired emotion.     

Environments to support context and emotion aware visual interaction

The interaction with software systems is often affected by many types of hurdles that induce users to make errors and mistakes, and to break the continuity of their reasoning while carrying out a working task with the computer. As a consequence, negative emotional states, such as frustration, dissatisfaction, and anxiety, may arise. In this paper, we illustrate how the Software Shaping Workshop (SSW) methodology can represent a solution to the problem of developing interactive systems that are correctly perceived and interpreted by end-users, thus becoming more acceptable and favouring positive emotional states. In the methodology, a key role is played by domain-expert users, that is, experts in a specific domain, not necessarily experts in computer science. Domain-expert users’ skills and background, including their knowledge of the domain and users’ needs and habits, are exploited to create context and emotion aware visual interactive systems. Examples of these systems are illustrated by referring to a case study in the automation field.     

EmoTales: creating a corpus of folk tales with emotional annotations

Emotions are inherent to any human activity, including human–computer interactions, and that is the reason why recognizing emotions expressed in natural language is becoming a key feature for the design of more natural user interfaces. In order to obtain useful corpora for this purpose, the manual classification of texts according to their emotional content has been the technique most commonly used by the research community. The use of corpora is widespread in Natural Language Processing, and the existing corpora annotated with emotions support the development, training and evaluation of systems using this type of data. In this paper we present the development of an annotated corpus oriented to the narrative domain, called EmoTales, which uses two different approaches to represent emotional states: emotional categories and emotional dimensions. The corpus consists of a collection of 1,389 English sentences from 18 different folk tales, annotated by 36 different people. Our model of the corpus development process includes a post-processing stage performed after the annotation of the corpus, in which a reference value for each sentence was chosen by taking into account the tags assigned by annotators and some general knowledge about emotions, which is codified in an ontology. The whole process is presented in detail, and revels significant results regarding the corpus such as inter-annotator agreement, while discussing topics such as how human annotators deal with emotional content when performing their work, and presenting some ideas for the application of this corpus that may inspire the research community to develop new ways to annotate corpora using a large set of emotional tags.     

Understanding Key Antecedents of User Loyalty toward Mobile Messenger Applications: An Integrative View of Emotions and the Dedication-Constraint Model

Given the growth of and competition among mobile messenger applications (MMAs), attracting users’ attention and enhancing their loyalty have become large challenges for MMA service providers. This study provides a theoretical view for understanding the mechanisms that lead to user loyalty toward MMAs. Although emotions and the dedication-constraint model are the two main research disciplines via which the formation of user loyalty has been investigated, few studies have unified these two disciplines. A theoretical model is developed by synthesizing emotional responses and the dedication-constraint model. Based on the ambivalent view of emotions, we examine the exact effects of positive and negative emotions on user loyalty to MMA. Moreover, we identify an encompassing set of antecedents to affective and calculative commitments in the MMA context. A structural equation modeling (SEM) method is used to test the research model based on a sample of 300 KakaoTalk users in South Korea. Our findings reveal that user loyalty to MMAs is jointly shaped by dedication- and constraint-based mechanisms and emotional responses. The findings indicate that affective commitment significantly influences user loyalty, both directly and indirectly, through positive emotions. However, calculative commitment has significant positive effects on positive emotions and user loyalty, but it is also positively related to negative emotions. Perceived usefulness, perceived enjoyment, and trust significantly influence affective commitment to MMAs, while social norms significantly affect calculative commitment to MMAs. Theoretical and managerial implications and future research directions are subsequently discussed.     

A Novel framework of EEG-based user identification by analyzing music-listening behavior

This paper introduces a novel framework for user identification by analyzing neuro-signals. Studies regarding Electroencephalography (EEG) revealed that such bio-signals are sensitive, hard to forge, confidential, and unique which the conventional biometric systems like face, speaker, signature and voice lack. Traditionally, researchers investigated the neuro-signal patterns by asking users to perform various imaginary, visual or calculative tasks. In this work, we have analyzed this neuro-signal pattern using audio as stimuli. The EEG signals are recorded simultaneously while user is listening to music. Four different genres of music are considered as users have their own preference and accordingly they respond with different emotions and interests. The users are also asked to provide music preference which acts as a personal identification mechanism. The framework offers the benefit of uniqueness in neuro-signal pattern even with the same music preference by different users. We used two different classifiers i.e. Hidden Markov Model (HMM) based temporal classifier and Support Vector Machine (SVM) for user identification system. A dataset of 2400 EEG signals while listening to music was collected from 60 users. User identification performance of 97.50 % and 93.83 % have been recorded with HMM and SVM classifiers, respectively. Finally, the performance of the system is also evaluated on various emotional states after showing different emotional videos to users.     

Affective design of products using an audio-based protocol: Application to eyeglass frame

The measurement and understanding of user emotions elicited by product appearance are critical elements of the product development process. This paper proposes a new emotion measurement method, called Auditory Parameter Method. It is a non-verbal technique that uses auditory stimuli (music samples) and association tests for evaluating a set of products, given by their pictures. From user-tests, it provides an assessment of these products according to a series of emotional dimensions. We present the methodological framework used to build the links between user's emotional responses and geometrical features of the products. The method is described on an application case, an eyeglass frame. Analysis of Variance models are employed to examine how various shape factors influence users' emotional responses. To demonstrate the effectiveness of our protocol, we compare the proposed method with the conventional Semantic Differential using Principal Component Analysis and Generalized Procrustes Analysis. The new protocol demonstrates interesting qualities for collecting the intuitive emotions of users and for providing a discriminant measurement of emotions. It can also be used by designers to stimulate creativity.