Data-driven user simulation for automated evaluation of spoken dialog systems

This paper proposes a novel integrated dialog simulation technique for evaluating spoken dialog systems. A data-driven user simulation technique for simulating user intention and utterance is introduced. A novel user intention modeling and generating method is proposed that uses a linear-chain conditional random field, and a two-phase data-driven domain-specific user utterance simulation method and a linguistic knowledge-based ASR channel simulation method are also presented. Evaluation metrics are introduced to measure the quality of user simulation at intention and utterance. Experiments using these techniques were carried out to evaluate the performance and behavior of dialog systems designed for car navigation dialogs and a building guide robot, and it turned out that our approach was easy to set up and showed similar tendencies to real human users.     

Active acquisition of user models: Implications for decision-theoretic dialog planning and plan recognition

This article investigates the implications ofactive user model acquisition upon plan recognition, domain planning, and dialog planning in dialog architectures. A dialog system performs active user model acquisition by querying the user during the course of the dialog. Existing systems employ passive strategies that rely on inferences drawn from passive observation of the dialog. Though passive acquisition generally reduces unnecessary dialog, in some cases the system can effectively shorten the overall dialog length by selectively initiating subdialogs for acquiring information about the user.We propose a theory identifying conditions under which the dialog system should adoptactive acquisition goals. Active acquisition imposes a set ofrationality requirements not met by current dialog architectures. To ensure rational dialog decisions, we propose significant extensions to plan recognition, domain planning, and dialog planning models, incorporating decision-theoretic heuristics for expected utility. The most appropriate framework for active acquisition is a multi-attribute utility model wherein plans are compared along multiple dimensions of utility. We suggest a general architectural scheme, and present an example from a preliminary implementation.The author will be at the Department of Computer Science, University of Toronto, untilThe author will be at the Department of Computer Science, University of Toronto, untilThe author will be at the Department of Computer Science, University of Toronto, untilThe author will be at the Department of Computer Science, University of Toronto, untilThe author will be at the Department of Computer Science, University of Toronto, untilThe author will be at the Department of Computer Science, University of Toronto, until     

Demonstrating the Cognitive Plausibility of Interactive System Specifications

Much of the behaviour of an interactive system is determined by its user population. This paper describes how assumptions about the user can be brought into system models in order to reason about their behaviour. We describe a system model containing reasonable assumptions about the user as being ‘cognitively plausible’. Before asserting the plausibility of a model however we must first be able to make the assumptions made in that model inspectable. There is a tension between the inspectability of user assumptions and the tractability of models; inspectable models tend to not be very tractable and vice versa. We describe how we can get round this tension, by deriving tractable models from explicit user assumptions. The resulting models may not of themselves be very inspectable to human-factors workers, but the process by which they are derived is inspectable. Hence we claim that we can have both tractability and inspectability. We exemplify our claims using a simple cognitive model and ‘Meeting Maker’, an interactive electronic diary system. Received March 2000 / Accepted in revised form July 2000     

Using dynamic user models in the recognition of the plans of the user

This paper is concerned with information-seeking dialogues in a restricted domain (we consider a consultation system for a Computer Science Department, delivering information about the various tasks that the users may want to perform: for example, how to access the library, get information about the courses of the Department, etc.) and presents a framework where a plan recognition and a user modeling component are integrated to cooperate in the task of identifying the user's plans and goals. The focus of the paper is centered on the techniques used for building the user model and exploiting it in the determination of the user's intentions. For this task, we use stereotypes and we propose some inference rules for expanding the user model by inferring the user's beliefs from both the sentences s/he utters and the information stored in the plan library of the system, that describes the actions in the domain. Moreover, we introduce some disambiguation rules that are applied to the information in the user model for restricting the set of ambiguous hypotheses on the user's plans and goals to the most plausible ones. This also simplifies a further clarification dialogue if it is necessary for a precise identification of the user's intentions.     

The user modeling shell system BGP-MS

BGP-MS is a user modeling shell system that can assist interactive software systems in adapting to their current users by taking the users' presumed knowledge, beliefs, and goals into account. It offers applications several methods for communicating observations concerning the user to BGP-MS, and for obtaining information on currently held assumptions about the user from BGP-MS. It provides a choice of two integrated formalisms for representing beliefs and goals, and includes several types of inferences for drawing additional assumptions based on an initial interview, observed user actions, and stereotypical knowledge about pre-defined user subgroups. BGP-MS is a customizable software system that is independent from applications, operates concurrently with them, and interacts with them through inter-process communication. For tailoring BGP-MS to a specific application domain, the developer must select those components of BGP-MS that are needed in this domain and fill them with relevant domain-dependent user modeling knowledge. This paper first summarizes the user modeling services that BGP-MS provides to application programs at runtime. It discusses the representational and inferential foundations that determine the scope and the limits of these services, and also gives a detailed example illustrating the interaction between the various system components. It describes interfaces that are available to application developers for tailoring BGP-MS to the specific user modeling needs of their application domains. Finally, it compares the system with all other major user modeling shell systems, and describes a first application that employs BGP-MS for adapting hypertext to users' terminological knowledge.The managing UMUAI editor for this paper was Sandra Carberry, University of Delaware.     

User simulation in a stochastic dialog system

We present a new methodology of user simulation applied to the evaluation and refinement of stochastic dialog systems. Common weaknesses of these systems are the scarceness of the training corpus and the cost of an evaluation made by real users. We have considered the user simulation technique as an alternative way of testing and improving our dialog system. We have developed a new dialog manager that plays the role of the user. This user dialog manager incorporates several knowledge sources, combining statistical and heuristic information in order to define its dialog strategy. Once the user simulator is integrated into the dialog system, it is possible to enhance the dialog models by an automatic strategy learning. We have performed an extensive evaluation, achieving a slight but clear improvement of the dialog system.     

User modeling in dialog systems: Potentials and hazards

In order to be capable of exhibiting a wide range of cooperative behavior, a computer-based dialog system must have available assumptions about the current user's goals, plans, background knowledge and (false) beliefs, i.e., maintain a so-called user model. Apart from cooperativity aspects, such a model is also necessary for intelligent coherent dialog behavior in general. This article surveys recent research on the problem of how such a model can be constructed, represented and used by a system during its interaction with the user. Possible applications, as well as potential problems concerning the advisability of application, are then discussed. Finally, a number of guidelines are presented which should be observed in future research to reduce the risk of a potential misuse of user modeling technology.     

Foundations of secure deductive databases

We develop a formal logical foundation for secure deductive databases. This logical foundation is based on an extended logic involving several modal operators. We develop two models of interaction between the user and the database called “yes-no” dialogs, and “yes-no-don't know” dialogs. Both dialog frameworks allow the database to lie to the user. We develop an algorithm for answering queries using yes-no dialogs and prove that secure query processing using yes-no dialogs is NP-complete. Consequently, the degree of computational intractability of query processing with yes-no dialogs is no worse than for ordinary databases. Furthermore, the algorithm is maximally cooperative to user in the sense that lying is resorted to only when absolutely necessary. For Horn databases, we show that secure query processing can be achieved in linear time-hence, this is no more intractable than the situation in ordinary databases. Finally, we identify necessary and sufficient conditions for the database to be able to preserve security. Similar results are also obtained for yes-no-don't know dialogs     

基于人物角色的用户建模研究

在以用户为中心的软件设计中,需要通过对用户建模来明确和分析用户,了解用户的特定需求.介绍用户建模的相关概念,阐述基于人物角色的用户建模原理、详细介绍基于人物角色的用户建模过程,指出人物角色在以用户为中心的软件设计过程中的应用.     

A visual tool for user-interface development

VITUID is a visual tool for user interface development. It aims to help user-interface designers, who may be non-programmers, to specify and create interactive, graphical user interfaces through graphical interaction. VITUID lets designers specify the user command set and the human-computer dialogs by building a tree and specifying the behavior of each dialog via a dialog-control word, adopting the object-oriented approach. The main strategy of VITUID is to separate a user interface into an application-dependent part and an application-independent part and then let the designer specify only the application-dependent part.     

The AT&T spoken language understanding system

Spoken language understanding (SLU) aims at extracting meaning from natural language speech. Over the past decade, a variety of practical goal-oriented spoken dialog systems have been built for limited domains. SLU in these systems ranges from understanding predetermined phrases through fixed grammars, extracting some predefined named entities, extracting users' intents for call classification, to combinations of users' intents and named entities. In this paper, we present the SLU system of VoiceTone/spl reg/ (a service provided by AT&T where AT&T develops, deploys and hosts spoken dialog applications for enterprise customers). The SLU system includes extracting both intents and the named entities from the users' utterances. For intent determination, we use statistical classifiers trained from labeled data, and for named entity extraction we use rule-based fixed grammars. The focus of our work is to exploit data and to use machine learning techniques to create scalable SLU systems which can be quickly deployed for new domains with minimal human intervention. These objectives are achieved by 1) using the predicate-argument representation of semantic content of an utterance; 2) extending statistical classifiers to seamlessly integrate hand crafted classification rules with the rules learned from data; and 3) developing an active learning framework to minimize the human labeling effort for quickly building the classifier models and adapting them to changes. We present an evaluation of this system using two deployed applications of VoiceTone/spl reg/.     

Improving accessibility with user-tailored interfaces

The first stage in the design of a user interface is the quest for its ‘typical user’, an abstract generalization of each user of the application. However, in web systems and other scenarios where the application can be used by dozens of different kinds of users, the identification of this ‘typical user’ is quite difficult, if not impossible. Our proposal is to avoid the construction of interactive dialogs during the design stage, building them dynamically once the specific cognitive, perceptual and motor requirements of the current user are known: that is, during the execution stage. This is the approach used by GADEA, an intelligent user interface management system (UIMS) able to separate the functionality of an application from its interface in real time. The system adapts the components of the interface depending on the information stored in a user model which is continuously updated by a small army of data-gathering agents.     

A probabilistic framework for dialog simulation and optimal strategy learning

The design of Spoken Dialog Systems cannot be considered as the simple combination of speech processing technologies. Indeed, speech-based interface design has been an expert job for a long time. It necessitates good skills in speech technologies and low-level programming. Moreover, rapid development and reusability of previously designed systems remains uneasy. This makes optimality and objective evaluation of design very difficult. The design process is therefore a cyclic process composed of prototype releases, user satisfaction surveys, bug reports and refinements. It is well known that human intervention for testing is time-consuming and above all very expensive. This is one of the reasons for the recent interest in dialog simulation for evaluation as well as for design automation and optimization. In this paper we expose a probabilistic framework for a realistic simulation of spoken dialogs in which the major components of a dialog system are modeled and parameterized thanks to independent data or expert knowledge. Especially, an Automatic Speech Recognition (ASR) system model and a User Model (UM) have been developed. The ASR model, based on articulatory similarities in language models, provides task-adaptive performance prediction and Confidence Level (CL) distribution estimation. The user model relies on the Bayesian Networks (BN) paradigm and is used both for user behavior modeling and Natural Language Understanding (NLU) modeling. The complete simulation framework has been used to train a reinforcement-learning agent on two different tasks. These experiments helped to point out several potentially problematic dialog scenarios.     

The promise of a voice-enabled Web

Many computing professionals have heard of XML, and some use it to describe text, images and other data with rich structure. The author discusses an innovative use of XML, called VoiceXML, to support human-computer dialogs via spoken input and audio output. VoiceXML defines dialogs between humans and machines in terms of audio files to be played, text-to-speech synthesis and speech recognition capabilities, and touch-tone input. The author reviews the existing architectures for World Wide Web and telephone services, describes how VoiceXML enables consolidation of service logic for Web and phone, and summarizes the features of the VoiceXML 1.0 specification. Implementation of VoiceXML clients and VoiceXML services has begun in many of the VoiceXML Forum's member companies and will soon be available in the marketplace. The World Wide Web Voice Browser working group has adopted VoiceXML 1.0 as the basis for the dialog markup language that is part of its speech user interface framework     

Do (and Say) as I Say: Linguistic Adaptation in Human–Computer Dialogs

There is strong research evidence showing that people naturally align to each other’s vocabulary, sentence structure, and acoustic features in dialog, yet little is known about how the alignment mechanism operates in the interaction between users and computer systems let alone how it may be exploited to improve the efficiency of the interaction. This article provides an account of lexical alignment in human–computer dialogs, based on empirical data collected in a simulated human–computer interaction scenario. The results indicate that alignment is present, resulting in the gradual reduction and stabilization of the vocabulary-in-use, and that it is also reciprocal. Further, the results suggest that when system and user errors occur, the development of alignment is temporarily disrupted and users tend to introduce novel words to the dialog. The results also indicate that alignment in human–computer interaction may have a strong strategic component and is used as a resource to compensate for less optimal (visually impoverished) interaction conditions. Moreover, lower alignment is associated with less successful interaction, as measured by user perceptions. The article distills the results of the study into design recommendations for human–computer dialog systems and uses them to outline a model of dialog management that supports and exploits alignment through mechanisms for in-use adaptation of the system’s grammar and lexicon.     

网络号百用户兴趣模型挖掘算法

为了实现在线推荐信息服务,要对网络号百用户的访问行为进行分析,荻取用户访问聚类模型,从而在聚类模型的基础上进行在线推荐.介绍获取用户访问路径信息的方法,对用户访问路径信息建立相似度矩阵,基于相似度矩阵改进K-means算法,据此进行用户模型聚类,给出分析案例,并说明算法实现过程.     

AN AUTOMATIC DIALOG SIMULATION TECHNIQUE TO DEVELOP AND EVALUATE INTERACTIVE CONVERSATIONAL AGENTS

During recent years, conversational agents have become a solution to provide straightforward and more natural ways of retrieving information in the digital domain. In this article, we present an agent-based dialog simulation technique for learning new dialog strategies and evaluating conversational agents. Using this technique, the effort necessary to acquire data required to train the dialog model and then explore new dialog strategies is considerably reduced. A set of measures has also been defined to evaluate the dialog strategy that is automatically learned and to compare different dialog corpora. We have applied this technique to explore the space of possible dialog strategies and evaluate the dialogs acquired for a conversational agent that collects monitored data from patients suffering from diabetes. The results of the comparison of these measures for an initial corpus and a corpus acquired using the dialog simulation technique show that the conversational agent reduces the time needed to complete the dialogs and improve their quality, thereby allowing the conversational agent to tackle new situations and generate new coherent answers for the situations already present in an initial model.     

Towards real-time GOMS: a model of expert behaviour in a highly interactive task

We present an analysis of an expert performing a highly interactive computer task. The analysis uses GOMS models, specifying the Goals, Operators, Methods, and Selection rules used by the expert. Two models are presented, one with function-level operators which perform high-level functions in the domain, and one with keystroke-level operators which describe hand movements. For a segment of behaviour in which the expert accomplished about 30 functions in about 30 s, the function-level model predicted the observed behaviour well, while the keystroke-level model predicted only about half of the observed hand movements. These results, including the discrepancy between the models, are discussed.