Pattern‐based GUI testing: Bridging the gap between design and quality assurance

Software systems with a graphical user interface (GUI) front end are typically designed using user interface (UI) Patterns, which describe generic solutions (with multiple possible implementations) for recurrent GUI design problems. However, existing testing techniques do not take advantage of this fact to test GUIs more efficiently. In this paper, we present a new pattern‐based GUI testing (PBGT) approach that formalizes the notion of UI Test Patterns, which are generic test strategies to test UI patterns over their different implementations. The PBGT approach is evaluated via 2 case studies. The first study involves 2 fielded Web application subjects; findings show that PBGT is both practical and useful, as testing teams were able to find real bugs in a reasonable time interval. The second study allows deeper analysis by studying software subjects seeded with artificial faults; the findings show that PBGT is more effective than a manual model‐based test case generation approach.     

Generating transition probabilities to support model‐based software testing

Markov chain usage models support test planning, test automation, and analysis of test results. In practice, transition probabilities for Markov chain usage models are often specified using a cycle of assigning, verifying, and revising specific values for individual transition probabilities. For large systems, such an approach can be difficult for a variety of reasons. We describe an improved approach that represents transition probabilities by explicitly preserving the information concerning test objectives and the relationships between transition probabilities in a format that is easy to maintain and easy to analyze. Using mathematical programming, transition probabilities are automatically generated to satisfy test management objectives and constraints. A more mathematical treatment of this approach is given in References [ 1 ] (Poore JH, Walton GH, Whittaker JA. A constraint‐based approach to the representation of software usage models. Information and SoftwareTechnology 2000; at press) and [ 2 ] (Walton GH. Generating transition probabilities for Markov chain usage models. PhD Thesis, University of Tennessee, Knoxville, TN, May 1995.). In contrast, this paper is targeted at the software engineering practitioner, software development manager, and test manager. This paper also adds to the published literature on Markov chain usage modeling and model‐based testing by describing and illustrating an iterative process for usage model development and optimization and by providing some recommendations for embedding model‐based testing activities within an incremental development process. Copyright © 2000 John Wiley & Sons, Ltd.     

基于事件处理函数的GUI测试用例集约简技术

GUI测试用例集约简是降低GUI软件测试成本的有效手段.GUI软件的消息循环机制以及事件驱动特性,导致传统的基于控制流和数据流的测试用例集约简技术难以直接应用于GUI测试用例集约简.如何在尽可能保持原有测试用例集缺陷发现能力的基础上,尽可能地降低GUI测试用例集规模,是GUI测试用例集约简的一个挑战.以事件处理函数为核心,结合控制流和数据流技术,根据事件处理函数代码结构特征以及事件处理函数之间的数据依赖关系定义测试冗余规则,制定并实现了3种测试用例集约简技术.实验结果表明:与已有技术相比,其中两种根据事件处理函数之间的数据依赖关系制定的测试用例集约简技术达到了较好的约简效果.     

影响GUI测试性能的两个因素:测试用例及测试预测

GUI测试在软件开发和测试中具有重要的地位.GUI自动化测试的实际应用强调了测试的成本和效率.在GUI测试中,测试用例和测试预测是两个与图形用户界面测试性能相关的主要因素.根据GUI的事件流图,事件关联图等特性,通过分析GUI事件的代码复杂度,将事件分配到各个长度不同的测试用例中,并定义测试预测的级别.通过实例,研究测试用例和测试预测对于图形用户界面测试性能的影响,从而为进一步完善GUI测试框架提供了依据.     

Evolutionary algorithms for the multi‐objective test data generation problem

Automatic test data generation is a very popular domain in the field of search‐based software engineering. Traditionally, the main goal has been to maximize coverage. However, other objectives can be defined, such as the oracle cost, which is the cost of executing the entire test suite and the cost of checking the system behavior. Indeed, in very large software systems, the cost spent to test the system can be an issue, and then it makes sense by considering two conflicting objectives: maximizing the coverage and minimizing the oracle cost. This is what we did in this paper. We mainly compared two approaches to deal with the multi‐objective test data generation problem: a direct multi‐objective approach and a combination of a mono‐objective algorithm together with multi‐objective test case selection optimization. Concretely, in this work, we used four state‐of‐the‐art multi‐objective algorithms and two mono‐objective evolutionary algorithms followed by a multi‐objective test case selection based on Pareto efficiency. The experimental analysis compares these techniques on two different benchmarks. The first one is composed of 800 Java programs created through a program generator. The second benchmark is composed of 13 real programs extracted from the literature. In the direct multi‐objective approach, the results indicate that the oracle cost can be properly optimized; however, the full branch coverage of the system poses a great challenge. Regarding the mono‐objective algorithms, although they need a second phase of test case selection for reducing the oracle cost, they are very effective in maximizing the branch coverage. Copyright © 2011 John Wiley & Sons, Ltd.     

基于事件权重的GUI测试路径生成方法

针对GUI事件驱动特性产生无限多个测试路径的问题,提出在事件流图中考虑事件权重的GUI测试路径生成方法。分析了事件流图模型,采用层次分析法计算事件的权重,通过遍历事件流图生成GUI测试路径。该方法减少了测试路径的数目,生成的测试路径更具有效性,提高了测试效率。     

一种基于事件处理函数的GUI测试方法

事件处理函数响应用户GUI(graphic user interface)操作并完成软件预定义功能,事件处理函数以及事件处理函数之间的关系实现是否与规约一致,是GUI 测试的重点.针对现有的基于模型GUI 测试用例自动生成过程中面临的测试用例规模庞大以及生成的测试用例无效问题,从分析事件处理函数的角度出发,提出了一种GUI 测试模型EHG.针对此模型,结合事件处理函数及其代码结构,提出了两个测试覆盖准则：完整最短路径覆盖准则和完整最短路径定义-引用对覆盖准则;利用基于反馈的测试用例生成技术生成测试用例.实验结果表明,针对较为复杂的应用,该方法不仅能够有效控制测试用例规模,消除无效测试用例,而且生成的测试用例能有效提高事件处理函数的代码结构覆盖率.     

Automatic test case generation from Simulink/Stateflow models using model checking

Model‐based test generation techniques based on random input generation and guided simulation do not satisfy the demands of high test coverage and completeness guarantees as required by safety‐critical applications. Recently, test generation techniques based on model checking have been reported to bridge this gap. To evaluate the effectiveness of these techniques, an in‐house tool suite, AutoMOTGen, has been developed for Simulink/Stateflow and applied on real‐life case studies at General Motors. This paper outlines the test generation methodology of AutoMOTGen and gives a comparative study with a commercial, primarily random input‐based, test generation tool on the same set of examples. The results indicate that in terms of coverage, model checking‐based techniques complement the random input‐based techniques. In addition, they provide proofs for unreachability that can aid in debugging the models. Therefore, it is recommended that model checking‐based tools be utilized to complement and enhance the effectiveness of model‐based testing methods in safety‐critical systems engineering. Copyright © 2013 John Wiley & Sons, Ltd.     

A holistic approach to model‐based testing of Web service compositions

The behavior of composed Web services depends on the results of the invoked services; unexpected behavior of one of the invoked services can threat the correct execution of an entire composition. This paper proposes an event‐based approach to black‐box testing of Web service compositions based on event sequence graphs, which are extended by facilities to deal not only with service behavior under regular circumstances (i.e., where cooperating services are working as expected) but also with their behavior in undesirable situations (i.e., where cooperating services are not working as expected). Furthermore, the approach can be used independently of artifacts (e.g., Business Process Execution Language) or type of composition (orchestration/choreography). A large case study, based on a commercial Web application, demonstrates the feasibility of the approach and analyzes its characteristics. Test generation and execution are supported by dedicated tools. Especially, the use of an enterprise service bus for test execution is noteworthy and differs from other approaches. The results of the case study encourage to suggest that the new approach has the power to detect faults systematically, performing properly even with complex and large compositions. Copyright © 2012 John Wiley & Sons, Ltd.     

Automating test oracles from restricted natural language agile requirements

Manual testing of software requirements written in natural language for agile or any other methodology requires more time and human resources. This leaves the testing process error prone and time consuming. For satisfied end users with bug‐free software delivered on time, there is a need to automate the test oracle process for natural language or informal requirements. The automation of the test oracle is relatively easier with formal requirements, but this task is difficult to achieve with natural language requirements. This study proposes an approach called Restricted Natural Language Agile Requirements Testing (ReNaLART) to automate the test oracle from restricted natural language agile requirements. For this purpose, it uses an existing user story template with some modifications for writing user stories. This helps in identifying test input and expected output for a user story. For comparison of expected and observed outputs it makes use of a regex pattern and string distance functions. It is capable of assigning different types of verdicts automatically depending upon the similarity/dissimilarity between observed and expected outputs of user stories. ReNaLART is validated using several case studies of different domains, namely, OLX Pakistan, Mental Health Tests, McDelivery Pakistan, BlueStacks, Power Searching with Google, TensorFlow Playground, w3Schools 2018 offline and Touch'D. It revealed several faults in five of the above listed eight applications. Plus, the proposed test oracle on an average took 0.02 s for test data generation, expected output generation and verdict assignment. Both these facts show the fault revealing effectiveness and efficiency of ReNaLART.     

Software testing using model programs

A strategy described as ‘testing using M model programs’ (abbreviated to ‘M‐mp testing’) is investigated as a practical alternative to software testing based on manual outcome prediction. A model program implements suitably selected parts of the functional specification of the software to be tested. The M‐mp testing strategy requires that M (M ≥ 1) model programs as well as the program under test, P, should be independently developed. P and the M model programs are then subjected to the same test data. Difference analysis is conducted on the outputs and appropriate corrective action is taken. P and the M model programs jointly constitute an approximate test oracle. Both M‐mp testing and manual outcome prediction are subject to the possibility of correlated failure. In general, the suitability of M‐mp testing in a given context will depend on whether building and maintaining model programs is likely to be more cost effective than manually pre‐calculating P's expected outcomes for given test data. In many contexts, M‐mp testing could also facilitate the attainment of higher test adequacy levels than would be possible with manual outcome prediction. A rigorous experiment in an industrial context is described in which M‐mp testing (with M = 1) was used to test algorithmically complex scheduling software. In this case, M‐mp testing turned out to be significantly more cost effective than testing based on manual outcome prediction. Copyright © 2001 John Wiley & Sons, Ltd.     

Event‐driven model predictive control of timed hybrid Petri nets

Hybrid Petri nets represent a powerful modeling formalism that offers the possibility of integrating, in a natural way, continuous and discrete dynamics in a single net model. Usual control approaches for hybrid nets can be divided into discrete‐time and continuous‐time approaches. Continuous‐time approaches are usually more precise, but can be computationally prohibitive. Discrete‐time approaches are less complex, but can entail mode‐mismatch errors due to fixed time discretization. This work proposes an optimization‐based event‐driven control approach that applies on continuous time models and where the control actions change when discrete events occur. Such an approach is computationally feasible for systems of interest in practice and avoids mode‐mismatch errors. In order to handle modelling errors and exogenous disturbances, the proposed approach is implemented in a closed‐loop strategy based on event‐driven model predictive control. Copyright © 2013 John Wiley & Sons, Ltd.     

Generating test data from state‐based specifications

Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system‐level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated. This paper presents general criteria for generating test inputs from state‐based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults. Copyright © 2003 John Wiley & Sons, Ltd.     

Information Ordering with an Event‐Enriched Vector Space Model for Multi‐Document News Summarization

Information ordering is a nontrivial task in multi‐document summarization (MDS), which typically relies on the traditional vector space model (VSM) notorious for semantic deficiency. In this article, we propose a novel event‐enriched VSM to alleviate the problem by building event semantics into sentence representations. The mediation of event information between sentence and term, especially in the news domain, has an intuitive appeal as well as technical advantage in common sentence‐level operations such as sentence similarity computation. Inspired by the block‐style writing by humans, we base the sentence ordering algorithm on sentence clustering. To accommodate the complexity introduced by event information, we adopt a soft‐to‐hard clustering strategy on the event and sentence levels, using expectation–maximization clustering and K‐means, respectively. For the purpose of cluster‐based sentence ordering, the event‐enriched VSM enables us to design an ordering algorithm to enhance event coherence computed between sentence and sentence–context pairs. Drawing on the findings of earlier research, we also incorporate topic continuity measures and time information into the scheme. We evaluate the performance of the model and its variants automatically and manually, with experimental results showing clear advantage of the event‐based model over baseline and non‐event‐based models in information ordering for multi‐document news summarization. We are confident that the event‐enriched VSM has even greater potential in summarization and beyond, which awaits further research. © 2014 Wiley Periodicals, Inc.     

Model‐based event‐triggered predictive control for networked systems with communication delays compensation

This paper addresses the model‐based event‐triggered predictive control problem for networked control systems (NCSs). Firstly, we propose a discrete event‐triggered transmission scheme on the sensor node by introducing a quadratic event‐triggering function. Then, on the basis of the aforementioned scheme, a novel class of model‐based event‐triggered predictive control algorithms on the controller node is designed for compensating for the communication delays actively and achieving the desired control performance while using less network resources. Two cases, that is, the value of the communication delay of the first event‐triggered state is less or bigger than the sampling period, are considered separately for certain NCSs, regardless of the communication delays of the subsequent event‐triggered states. The codesign problems of the controller and event‐triggering parameter for the two cases are discussed by using the linear matrix inequality approach and the (switching) Lyapunov functional method. Furthermore, we extended our results to the NCSs with systems uncertainties. Finally, a practical ball and beam system is studied numerically to demonstrate the compensation effect for the communication delays with the proposed novel model‐based event‐triggered predictive control scheme. Copyright © 2014 John Wiley & Sons, Ltd.     

基于EDPN的面向对象的系统测试用例生成技术的研究

面向对象的软件测试技术研究的主要领域之一就是基于规约的系统测试,该测试技术在系统测试过程中会产生不可预测、事件静止的“死锁”和路径爆炸等问题。论文将事件驱动的Petri网(Event-DrivenPetriNetwork,简称EDPN)模型直观可靠的特点和较好的耦合性运用到系统测试技术中,探讨了解决问题的途径,提出了一种基于EDPN的唯一输入输出(UIO)测试用例的生成方法,并设计了基于深度优先搜索方法的自动生成测试用例的算法。     

Model‐based security testing: a taxonomy and systematic classification

Model‐based security testing relies on models to test whether a software system meets its security requirements. It is an active research field of high relevance for industrial applications, with many approaches and notable results published in recent years. This article provides a taxonomy for model‐based security testing approaches. It comprises filter criteria (i.e. model of system security, security model of the environment and explicit test selection criteria) as well as evidence criteria (i.e. maturity of evaluated system, evidence measures and evidence level). The taxonomy is based on a comprehensive analysis of existing classification schemes for model‐based testing and security testing. To demonstrate its adequacy, 119 publications on model‐based security testing are systematically extracted from the five most relevant digital libraries by three researchers and classified according to the defined filter and evidence criteria. On the basis of the classified publications, the article provides an overview of the state of the art in model‐based security testing and discusses promising research directions with regard to security properties, coverage criteria and the feasibility and return on investment of model‐based security testing. Copyright © 2015 John Wiley & Sons, Ltd.     

An empirical evaluation of several test‐a‐few strategies for testing particular conditions

Existing specification‐based testing techniques often generate comprehensive test suites to cover diverse combinations of test‐relevant aspects. Such a test suite can be prohibitively expensive to execute exhaustively because of its large size. A pragmatic strategy often adopted in practice, called test‐once strategy, is to identify certain particular conditions from the specification and to test each such condition once only. This strategy is implicitly based on the uniformity assumption that the implementation will process a particular condition uniformly, regardless of other parameters or inputs. As the decision of adopting the test‐once strategy is often based on the specification, whether the uniformity assumption actually holds in the implementation needs to be critically assessed, or else the risk of inadequate testing could be non‐negligible. As viable alternatives to reduce such a risk, a family of test‐a‐few strategies for the testing of particular conditions is proposed in this paper. Two rounds of experiments that evaluate the effectiveness of the test‐a‐few strategies as compared with the test‐once strategy are further reported. Our experiments do the following: (1) provide clear evidence that the uniformity assumption often, but not always, holds and that the assumption usually fails to hold when the implementation is faulty; (2) demonstrate that all our proposed test‐a‐few strategies are statistically more reliable than the test‐once strategy in revealing faulty programs; (3) show that random sampling is already substantially more effective than the test‐once strategy; and (4) indicate that, compared with other test‐a‐few strategies under study, choice coverage seems to achieve a better trade‐off between test effort and effectiveness. Copyright © 2011 John Wiley & Sons, Ltd.     

From Object‐Z specifications to ClassBench test suites

This paper describes a method for specification‐based class testing that incorporates test case generation, execution, and evaluation based on formal specifications. This work builds on previous achievements in the areas of specification‐based testing and class testing by integrating the two within a single framework. The initial step of the method is to generate test templates for individual operations from a specification written in the Object‐Z specification language. These test templates are combined to produce a finite state machine for the class that is used as the basis for test case execution using the ClassBench test execution framework. An oracle derived from the Object‐Z specification is used to evaluate the outputs. The method is explained using a simple example and its application to a more substantial case study is also discussed. Copyright © 2000 John Wiley & Sons, Ltd.     

Search‐based software test data generation: a survey

The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that, in general, test data generation is an undecidable problem. Metaheuristic search techniques offer much promise in regard to these problems. Metaheuristic search techniques are high‐level frameworks, which utilize heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey‐box properties, for example safety constraints; and also non‐functional properties, such as worst‐case execution time. This paper surveys some of the work undertaken in this field, discussing possible new future directions of research for each of its different individual areas. Copyright © 2004 John Wiley & Sons, Ltd.