Incremental approach to the technology of test design for industrial projects

The paper presents a technology to effort reduction in developing test suites for industrial software products based on an incremental technology. The main problems to be solved by incremental technology are full automation design of test scenarios and significant reducing of test explosion. The proposed approach provides solutions to the mentioned problems through joint co-working of the designer and the customer with usage of formal models; and through methods of symbolic verification with automatic generation of test suites with the toolset VRS/TAT.     

Avoiding the pitfalls of installing systems management suites

Systems management suites offer the promise of a single, centralized solution for businesses that performs disparate enterprise management functions, including electronic software distribution, network management, alert management, help desk, inventory management, performance monitoring, metering, and other tasks essential to maintaining enterprise productivity. The unfulfilled promise of systems management suites has become critical for organizations seeking control of the distributed enterprise. The article reports on the results of a survey of IT professionals in collaboration with a recognized expert on the distributed enterprise. This systems management suite survey: assesses organizations' commitment to suite technology and/or best-of-breed point solutions and their ability to implement suite technology throughout the enterprise; examines the costs associated with suite deployment; and identifies the essential steps and key obstacles to suite deployment and management. Although the survey found that these complex suites still left a lot to be desired, it also pointed out pitfalls some organizations were encountering along the way to early adoption     

基于变异分析的测试用例约简方法

针对回归测试过程中由于测试需求的变更导致测试用例规模不断扩大、测试成本不断增加的问题,提出一种基于变异分析的测试用例约简方法（RTM）。首先,以测试用例能否检测到指定变异体为依据,对测试用例进行划分并创建二进制数值形式的变异体事务集矩阵;然后,应用改进的关联挖掘算法获取测试用例间的关联关系;最后,根据这些关联关系有效约简测试用例。6个经典程序仿真实验结果表明,RTM能够使约简后的测试用例约简率达到37%,与传统贪心算法和启发式算法相比,测试用例约简率提高了6%,且在提高测试用例约简率的同时,保证了测试覆盖率,单个测试用例的测试覆盖率平均提高了11%。所提方法能够利用尽可能少的测试用例满足更多的测试需求,有效提高了测试效率,降低了测试成本。     

Improving the effectiveness of test suite reduction for user-session-based testing of web applications

ContextTest suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a reduced test suite can be time consuming.ObjectiveWe propose ordering the tests in a reduced suite to increase its rate of fault detection. The ordered reduced test suite can be executed in time constrained situations, where, even if test execution is stopped early, the best test cases from the reduced suite will already be executed.MethodIn this paper, we present several approaches to order reduced test suites using experimentally verified prioritization criteria for the domain of web applications. We conduct an empirical study with three subject applications and user-session-based test cases to demonstrate how ordered reduced test suites often make a practical contribution. To enable comparison between test suites of different sizes, we develop Mod_APFD_C, a modification of the traditional prioritization effectiveness measure.ResultsWe find that by ordering the reduced suites, we create test suites that are more effective than unordered reduced suites. In each of our subject applications, there is at least one ordered reduced suite that outperforms the best unordered reduced suite and the best prioritized original suite.ConclusionsOur results show that when a tester does not have enough time to execute the entire reduced suite, executing an ordered reduced suite often improves the rate of fault detection. By coupling the underlying system’s characteristics with observations from our study on the criteria that produce the best ordered reduced suites, a tester can order their reduced test suites to obtain increased testing effectiveness.     

IEEE754标准浮点测试向量的生成

介绍了在IEEE754标准的规定下生成用于浮点功能部件的测试向量的方法，讨论了测试向量在数据通路上的差错覆盖率，并给出了对该方法的一些改进措施。     

基于漏洞库的fuzzing测试技术

鉴于主流fuzzing(模糊)测试技术生成的测试用例随机性和针对性无法兼顾的问题,提出一种基于漏洞库的fuzzing测试技术.根据漏洞产生的原因和重现的方法对漏洞库进行整理分类,构造出测试用例集用于fuzzing测试.这样生成的测试用例集直接与漏洞相关,具有更强的针对性;同时扩展的测试用例集可以用于检测未知的同类漏洞,覆盖面更广,因此保证了一定的随机性.以ftp服务器测试为例,选取了W indow s平台下的4款ftp服务器进行测试,共发现了3款软件的6个新漏洞,提交Se-curityFocus并通过,其中5个漏洞得到了国际权威漏洞数据库CVE的认可.     

A probabilistic alternative to regression suites

Automated regression suites are essential in developing large applications, while maintaining reasonable quality and timetables. The main argument against the automation of regression suites, in addition to the cost of creation and maintenance, is the observation that if you run the same test many times, it becomes increasingly less likely to find bugs. To alleviate such problems, a new regression suite practice, using random test generators to create regression suites on-the-fly, is becoming more common. In this practice, instead of maintaining tests, we generate test suites on-the-fly by choosing several specifications and generating a number of tests from each specification.     

Standardized TTCN-3 specifications for SIP-ISUP/ISDN interworking testing

ETSI technical committee TISPAN currently funds the development of TTCN-3 network integration test suites using TTCN-3. Specialist task forces (STF) have already been initiated to implement and validate abstract test suites focusing on, e.g., SIP/ISUP and SIP/ISDN interworking units. In this paper, we report on our implementation approach and experiences made in the STF on testing interworking between SIP and ISDN/ISUP. Special attention has been given to the reuse of know-how and of existing TTCN-3 code available due to the existing SIP conformance test suite and the TTCN-3 test framework developed by ETSI for IPv6. Validation has been done using the test laboratory of a telecom network provider.     

Empirical Validation of Three Software Metrics Suites to Predict Fault-Proneness of Object-Oriented Classes Developed Using Highly Iterative or Agile Software Development Processes

Empirical validation of software metrics suites to predict fault proneness in object-oriented (OO) components is essential to ensure their practical use in industrial settings. In this paper, we empirically validate three OO metrics suites for their ability to predict software quality in terms of fault-proneness: the Chidamber and Kemerer (CK) metrics, Abreu's Metrics for Object-Oriented Design (MOOD), and Bansiya and Davis' Quality Metrics for Object-Oriented Design (QMOOD). Some CK class metrics have previously been shown to be good predictors of initial OO software quality. However, the other two suites have not been heavily validated except by their original proposers. Here, we explore the ability of these three metrics suites to predict fault-prone classes using defect data for six versions of Rhino, an open-source implementation of JavaScript written in Java. We conclude that the CK and QMOOD suites contain similar components and produce statistical models that are effective in detecting error-prone classes. We also conclude that the class components in the MOOD metrics suite are not good class fault-proneness predictors. Analyzing multivariate binary logistic regression models across six Rhino versions indicates these models may be useful in assessing quality in OO classes produced using modern highly iterative or agile software development processes.     

Testing communication protocols

The authors describe a unifying, common intermediate model that lets Lotos specifications and TTCN test suites be translated into extended finite-state machines, providing a single medium for conformance testing. The paper covers specifying a protocol in Lotos, specifying test suites in TTCN, translating specifications to CIM, and the generation, verification, selection, parametrization, and execution of tests     

组合测试中用例集的选择策略

组合测试用例众多,软件测试人员都希望用最少的测试用例集达到最好的测试覆盖。研究了组合测试的常规算法,比较了配对测试法和正交表测试法的特点,并借助于PICT测试工具重点研究了2因子模型的配对组合测试,在此基础上,总结出在组合测试中测试用例集的选择策略,进而达到满意的测试覆盖。     

A density‐based greedy algorithm for higher strength covering arrays

Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for t‐way interactions with t≥3. This study extends an efficient density‐based algorithm for pairwise coverage to generate t‐way interaction test suites and shows that it guarantees a logarithmic upper bound on the size of the test suites as a function of the number of factors. To complement this theoretical guarantee, an implementation is outlined and some practical improvements are made. Computational comparisons with other published methods are reported. Many of the results improve upon those in the literature. However, limitations on the ability of one‐test‐at‐a‐time algorithms are also identified. Copyright © 2008 John Wiley & Sons, Ltd.     

基于一致性测试理论的Statechart描述的测试用例自动生成

本文研究Statechart描述的测试语义和测试用例的自动生成.基于Tretmans的从标记转换系统描述自动生成测试用例的方法,我们研究如何从Statechart描述自动生成测试用例.本文的主要贡献在于建立了基于Statechart描述的一致性测试和测试用例生成的形式化基础.为Statechart描述建立了形式化测试语...     

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction     

基于弱变异准则的测试用例约简方法

针对为数较多的测试用例增加了回归测试成本的问题，提出一种基于弱变异准则的测试用例约简方法。首先，基于弱变异准则获得测试用例和变异分支关系矩阵；然后，重复约简4种无效测试需求和子集测试用例；最后，结合人工鱼群算法选择当前最优测试用例，并且交替执行简化和测试用例选择操作直至覆盖所有测试需求。该方法针对6个经典程序与贪心算法和HGS算法相比，基于弱变异准则并且不改变或稍微改变变异评分的条件下，约简率分别提高了73.4%和8.2%，且耗时分别降低了25.3%和56.1%。实验结果表明，所提方法在回归测试中可有效约简测试用例，降低测试代价。     

On the order of test goals in specification-based testing

Model-based testing techniques often select test cases according to test goals such as coverage criteria or mutation adequacy. Complex criteria and large models lead to large test suites, and a test case created for one coverage item usually covers several other items as well. This can be problematic if testing is expensive and resources are limited. Therefore, test case generation can be optimized in order to avoid unnecessary test cases and minimize the test generation and execution costs. Because of this optimization the order in which test goals are selected is expected to have an impact on both the performance of the test case generation and the size of resulting test suites, although finding the optimal order is not feasible in general. In this paper we report on experiments to determine the effects of the order in which test goals are selected on performance and the size of resulting test suites, and evaluate different heuristics to select test goals such that the time required to generate test suites as well as their size are minimized. The test case generation approach used for experimentation uses model checkers, and experimentation shows that good results can be achieved with any random ordering, but some improvement is still possible with simple heuristics.     

Standardization and testing of implementations of mathematical functions in floating point numbers

Requirements definition and test suites development for implementations of mathematical functions in floating point arithmetic in the framework of the IEEE 754 standard are considered. A method based on this standard is proposed for defining requirements for such functions. This method can be used for the standardization of implementations of such functions; this kind of standardization extends IEEE 754. A method for designing test suites for the verification of those requirements is presented. The proposed methods are based on specific properties of the representation of floating point numbers and on some features of the functions under examination.     

Model-based testing of internet e-mail protocols

This paper considers Model-Based Testing of e-mail protocols: a method for modeling mail protocols is proposed, specific features of mail protocols are considered in the context of testing, and results of testing of popular open-source mail servers are presented. As an example, test suites for the SMTP and POP3 protocols developed in JavaTESK (a Java extension for testing with formal methods) are presented. The test suites consist of two parts: independent conformance tests for protocols and a joint test imitating the operation of mail protocols in a network.     

Secure and Light Weight Elliptic Curve Cipher Suites in SSL/TLS

In the current circumstance, e-commerce through an online banking system plays a significant role. Customers may either buy goods from E-Commerce websites or use online banking to move money to other accounts. When a user participates in these types of behaviors, their sensitive information is sent to an untrustworthy network. As a consequence, when transmitting data from an internal browser to an external E-commerce web server using the cryptographic protocol SSL/TLS, the E-commerce web server ensures the security of the user’s data. The user should be pleased with the confidentiality, authentication, and authenticity properties of the SSL/TLS on both the user’s web browser and the remote E-commerce web server. E-Commerce web servers should choose the best SSL/TLS cipher suites for negotiating the user in order to attain such optimistic scenarios, as the cipher suite used in SSL/TLS plays an important role in securing E-Commerce web servers. The paper primarily focuses on analyzing the SSL/TLS cipher and elliptic curves. The paper also recommends the best elliptic curve cipher suites for E-Commerce and online banking servers, based on their power consumption, handshake execution time, and key exchange and signature verification time.     

Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication failure, and timing. Furthermore, they do not provide the engineer with a means to evaluate the relative effectiveness of different criteria, nor the relative effectiveness of adequate test suites satisfying a given criterion. This paper makes three contributions to the development and use of test adequacy criteria for distributed systems: (1) a testing method based on discrete-event simulations; (2) a fault-based analysis technique for evaluating test suites and adequacy criteria; and (3) a series of case studies that validate the method and technique. The testing method uses a discrete-event simulation as an operational specification of a system, in which the behavioral effects of distribution are explicitly represented. Adequacy criteria and test cases are then defined in terms of this simulation-based specification. The fault-based analysis involves mutation of the simulation-based specification to provide a foil against which test suites and the criteria that formed them can be evaluated. Three distributed systems were used to validate the method and technique, including DNS, the Domain Name System.