多路径测试数据自动生成方法及应用

以程序结构测试自动生成为研究背景,提出了一种重叠路径结构用以描述程序路径,并以此为基础设计了一种多路径测试数据生成适应值算法,实现了一次搜索完成多条路径的测试数据生成。算法通过目标路径间共享遗传算法产生的中间个体减少单一路径搜索始于随机产生的无序个体的初期迭代,从而加快搜索收敛的速度。应用于常用的基准程序和取自实际项目的程序,该算法与典型的分支谓词距离算法相比平均消耗时间缩短了70.6%。     

Generating test data for both paths coverage and faults detection using genetic algorithms: multi-path case

Generating test data that can expose the faults of the program is an important issue in software testing. Although previous methods of covering path can generate test data to traverse target path, the test data generated by these methods are difficult in detecting some low-probabilistic faults that lie on the covered paths. We present a method of generating test data for covering multiple paths to detect faults in this study. First, we transform the problem of covering multiple paths and detecting faults into a multi-objective optimization problem with constraint, and construct a mathematical model for it. Then, we give a strategy of solving the model based on a weighted genetic algorithm. Finally, we apply our method to several real-world programs, and compare it with several methods. The experimental results confirm that the proposed method can more efficiently generate test data that not only traverse the target paths but also detect faults lying in them than other methods.     

基于路径引导的回归测试用例集扩增方法

为了全面测试演化软件,回归测试通常需要生成新的测试用例。concolic测试是一种沿着具体执行路径进行符号执行的软件验证技术,通过生成测试数据来执行程序的所有可行路径。回归测试中,由于concolic测试关注于程序本身,没有利用已有测试用例和软件演化信息,导致生成大量无效测试数据,浪费资源和时间。为解决此问题,提出一种基于路径引导的回归测试用例集扩增方法。该方法将目标路径作为引导,根据软件演化信息选择有利于覆盖目标路径的测试用例,利用已有测试用例跳过重叠初始子路径,对后续目标子路径进行concolic测试并生成覆盖目标路径的测试数据。案例分析表明,本文方法相比传统concolic测试,本方法在覆盖程序可行路径的同时,可有效减少concolic测试路径,提高测试数据生成效率。     

A novel strategy for automatic test data generation using soft computing technique

Software testing is one of the most crucial and analytical aspect to assure that developed software meets prescribed quality standards. Software development process invests at least 50% of the total cost in software testing process. Optimum and efficacious test data design of software is an important and challenging activity due to the nonlinear structure of software. Moreover, test case type and scope determines the quality of test data. To address this issue, software testing tools should employ intelligence based soft computing techniques like particle swarm optimization (PSO) and genetic algorithm (GA) to generate smart and efficient test data automatically. This paper presents a hybrid PSO and GA based heuristic for automatic generation of test suites. In this paper, we described the design and implementation of the proposed strategy and evaluated our model by performing experiments with ten container classes from the Java standard library. We analyzed our algorithm statistically with test adequacy criterion as branch coverage. The performance adequacy criterion is taken as percentage coverage per unit time and percentage of faults detected by the generated test data. We have compared our work with the heuristic based upon GA, PSO, existing hybrid strategies based on GA and PSO and memetic algorithm. The results showed that the test case generation is efficient in our work.     

Robustness testing for software components

Component-based development allows one to build software from existing components and promises to improve software reuse and reduce costs. For critical applications, the user of a component must ensure that it fits the requirements of the application. To achieve this, testing is a well-suited means when the source code of the components is not available. Robustness testing is a testing methodology to detect the vulnerabilities of a component under unexpected inputs or in a stressful environment. As components may fail differently in different states, we use a state machine based approach to robustness testing. First, a set of paths is generated to cover transitions of the state machine, and it is used by the test cases to bring the component into a specific control state. Second, method calls with invalid inputs are fed to the component in different states to test the robustness. By traversing the paths, the test cases cover more states and transitions compared to stateless API testing. We apply our approach to several components, including open source software, and compare our results with existing approaches.     

Dynamic stopping criteria for search-based test data generation for path testing

ContextEvolutionary algorithms have proved to be successful for generating test data for path coverage testing. However in this approach, the set of target paths to be covered may include some that are infeasible. It is impossible to find test data to cover those paths. Rather than searching indefinitely, or until a fixed limit of generations is reached, it would be desirable to stop searching as soon it seems likely that feasible paths have been covered and all remaining un-covered target paths are infeasible.ObjectiveThe objective is to develop criteria to halt the evolutionary test data generation process as soon as it seems not worth continuing, without compromising testing confidence level.MethodDrawing on software reliability growth models as an analogy, this paper proposes and evaluates a method for determining when it is no longer worthwhile to continue searching for test data to cover un-covered target paths. We outline the method, its key parameters, and how it can be used as the basis for different decision rules for early termination of a search. Twenty-one test programs from the SBSE path testing literature are used to evaluate the method.ResultsCompared to searching for a standard number of generations, an average of 30–75% of total computation was avoided in test programs with infeasible paths, and no feasible paths were missed due to early termination. The extra computation in programs with no infeasible paths was negligible.ConclusionsThe method is effective and efficient. It avoids the need to specify a limit on the number of generations for searching. It can help to overcome problems caused by infeasible paths in search-based test data generation for path testing.     

Automatic generation of basis test paths using variable length genetic algorithm

Path testing is the strongest coverage criterion in white box testing. Finding target paths is a key challenge in path testing. Genetic algorithms have been successfully used in many software testing activities such as generating test data, selecting test cases and test cases prioritization. In this paper, we introduce a new genetic algorithm for generating test paths. In this algorithm the length of the chromosome varies from iteration to another according to the change in the length of the path. Based on the proposed algorithm, we present a new technique for automatically generating a set of basis test paths which can be used as testing paths in any path testing method. The proposed technique uses a method to verify the independency of the generated paths to be included in the basis set of paths. In addition, this technique employs a method for checking the feasibility of the generated paths. We introduce new definitions for the key concepts of genetic algorithm such as chromosome representation, crossover, mutation, and fitness function to be compatible with path generation. In addition, we present a case study to show the efficiency of our technique. We conducted a set of experiments to evaluate the effectiveness of the proposed path generation technique. The results showed that the proposed technique causes substantial reduction in path generation effort, and that the proposed GA algorithm is effective in test path generation.     

基于分支覆盖的回归测试路径选择

回归测试是迭代式软件开发的重要环节,测试数据生成是回归测试的前提.传统的回归测试方法,从已有的测试数据中选择部分测试数据,并生成一些新的测试数据,以验证程序的正确性.但是,该方法容易生成冗余的测试数据,从而降低了回归测试的效率.研究了回归测试的分支覆盖问题,通过利用已有测试数据的路径覆盖信息,并选择一定个数的路径,以覆盖所有的目标分支.首先,以若干路径形成的集合作为决策变量,以路径最少、覆盖的分支最多以及包含的未覆盖路径最少为目标,建立路径选择问题的3目标优化模型;然后,采用遗传算法求解上述模型时,设计了基于目标重要性的个体评价策略;最后,基于已有的测试数据与选择的路径之间的覆盖关系,确定需要生成的测试数据.将所提方法应用于6个基准工业程序测试中,并与其他方法比较.实验结果表明,采用该方法选择的路径,能够覆盖更多的分支,需要生成的测试数据更少,回归测试消耗的时间更短.     

基于PSO的软件结构测试数据自动生成方法

测试数据自动生成是软件测试过程中一个关键的问题。现有的结构测试数据自动生成,多采用基于遗传算法的方法。这些方法存在算法复杂、参数不易设置问题。该文提出一种基于粒子群算法的软件结构测试数据自动生成方法,以分支函数叠加法作为适应值函数。针对三角形判别程序的结构测试数据生成实验结果表明,与基于遗传算法的方法相比,可以更高效地生成测试数据,在粒子数目与种群个数相同的情况下,生成所需测试数据的迭代次数仅是遗传算法的1/16左右。     

The Economics of Unit Testing

Conventional wisdom and anecdote suggests that testing takes between 30 to 50% of a project's effort. However testing is not a monolithic activity as it consists of a number of different phases such as unit testing, integration testing and finally system and acceptance test. Unit testing has received a lot of criticism in terms of the amount of time that it is perceived to take and its perceived costs. However it still remains an important verification activity being an effective means to test individual software components for boundary value behavior and ensure that all code has been exercised adequately. We examine the available data from three safety-related, industrial software projects that have made use of unit testing. Using this information we argue that the perceived costs of unit testing may be exaggerated and that the likely benefits in terms of defect detection are quite high in relation to those costs. We also discuss the different issues that have been found applying the technique at different phases of the development and using different methods to generate those tests. We also compare results we have obtained with empirical results from the literature and highlight some possible weakness of research in this area.     

面向云测试的并行测试用例自动生成方法

为实现云计算环境下的高效软件测试,提出一种Web 应用系统功能测试的并行测试用例自动生成方法。该方法首先根据场景流图采用深度优先遍历算法生成并行测试路径,录制组合产生并行测试脚本,并进行参数化处理;然后,使用基于搜索的软件测试(SBST)方法自动生成可经过目标路径的有效测试数据集,脚本与数据耦合形成大量可并行部署的自动化测试用例。为验证方法的有效性设计了自动化云测试原型系统。实验结果表明,该并行测试用例自动生成方法可以高效地为云平台提供测试用例输入,提高测试效率。     

Evolutionary mutation testing for IoT with recorded and generated events

Mutation testing is a testing technique that has been applied successfully to several programming languages. Despite its benefits for software testing, the high computational cost of mutation testing has kept it from being widely used. Several refinements have been proposed to reduce its cost by reducing the number of generated mutants; one of those is evolutionary mutation testing (EMT). Evolutionary mutation testing aims at generating a reduced set of mutants with an evolutionary algorithm, which searches for potentially equivalent and difficult to kill mutants that help improve the test suite. Evolutionary mutation testing has been evaluated in two contexts so far, ie, web service compositions and object-oriented C++ programmes. This study explores its performance when applied to event processing language queries of various domains. This study also considers the impact of the test data, since a lack of events or the need to have specific values in them can hinder testing. The effectiveness of evolutionary mutation testing with the original test data generators and the new internet of things test event generator tool is compared in multiple case studies.     

基于退火遗传算法的多路径测试用例生成*

针对遗传算法在多路径测试用例自动生成中的不足,提出基于退火遗传算法的生成方法。对遗传算法的适应度函数进行优化设计,以满足多路径测试用例生成。同时在算法中引入路径存储机制,从而增强测试用例自动生成的功效。在提高算法的局部搜索能力方面,对遗传算法的两点交叉算子进行改进,并引入模拟退火算法到变异操作。三角形判断程序的测试用例生成实验结果,验证了该方法的有效性。     

DeltaFuzz: Historical Version Information Guided Fuzz Testing

With the widespread use of agile software development methods,such as agile and scrum,software is iteratively updated more frequently.To ensure the quality of the software,regression testing is conducted before new versions are released.Moreover,to improve the efficiency of regression testing,testing efforts should be concentrated on the modified and impacted parts of a program.However,the costs of manually constructing new test cases for the modified and impacted parts are relatively expensive.Fuzz testing is an effective method for generating test data automatically,but it is usually devoted to achieving higher code coverage,which makes fuzz testing unsuitable for direct regression testing scenarios.For this reason,we propose a fuzz testing method based on the guidance of historical version information.First,the differences between the program being tested and the last version are analyzed,and the results of the analysis are used to locate change points.Second,change impact analysis is performed to find the corresponding impacted basic blocks.Finally,the fitness values of test cases are calculated according to the execution traces,and new test cases are generated iteratively by the genetic algorithm.Based on the proposed method,we implement a prototype tool DeltaFuzz and conduct experiments on six open-source projects.Compared with the fuzzing tool AFLGo,AFLFast and AFL,DeltaFuzz can reach the target faster,and the time taken by DeltaFuzz was reduced by 20.59%,30.05%and 32.61%,respectively.     

基于GA的多路径测试数据生成器适配函数设计

提出了基于GA的多路径测试数据生成的概念和实现方法。讨论了为了实现多路径测试数据生成,设计有效的和高效的适配函数应考虑的因素,用实际案例针对提出的适配函数进行了功能性能验证。结果表明了适配函数设计的有效性。     

Data Generation for Path Testing

We present two stochastic search algorithms for generating test cases that execute specified paths in a program. The two algorithms are: a simulated annealing algorithm (SA), and a genetic algorithm (GA). These algorithms are based on an optimization formulation of the path testing problem which include both integer- and real-value test cases. We empirically compare the SA and GA algorithms with each other and with a hill-climbing algorithm, Korel's algorithm (KA), for integer-value-input subject programs and compare SA and GA with each other on real-value subject programs. Our empirical work uses several subject programs with a number of paths. The results show that: (a) SA and GA are superior to KA in the number of executed paths, (b) SA tends to perform slightly better than GA in terms of the number of executed paths, and (c) GA is faster than SA; however, KA, when it succeeds in finding the solution, is the fastest.     

Bio-inspired metaheuristics: evolving and prioritizing software test data

Software testing is both a time and resource-consuming activity in software development. The most difficult parts of software testing are the generation and prioritization of test data. Principally these two parts are performed manually. Hence introducing an automation approach will significantly reduce the total cost incurred in the software development lifecycle. A number of automatic test case generation (ATCG) and prioritization approaches have been explored. In this paper, we propose two approaches: (1) a pathspecific approach for ATCG using the following metaheuristic techniques: the genetic algorithm (GA), particle swarm optimization (PSO) and artificial bee colony optimization (ABC); and (2) a test case prioritization (TCP) approach using PSO. Based on our experimental findings, we conclude that ABC outperforms the GA and PSO-based approaches for ATC.G Moreover, the results for PSO on TCP arguments demonstrate biased applicability for both small and large test suites against random, reverse and unordered prioritization schemes. Therefore, we focus on conducting a comprehensive and exhaustive study of the application of metaheuristic algorithms in solving ATCG and TCP problems in software engineering.     

Harmony search-based test data generation for branch coverage in software structural testing

Test data generation is always a key task in the field of software testing. In recent years, meta-heuristic search techniques have been considered as an effective way to assist test data generation in software structural testing. In this way, some representative test cases with high-coverage capability can be picked out from program input space. Harmony search (HS) is a recently developed algorithm and has been vigorously applied to various optimization problems. In the paper, we attempt to apply harmony search algorithm to generate test data satisfying branch coverage. At the preprocessing stage, the probes used for gathering coverage information are inserted into all branches via program static analysis. At the same time, the encoding and decoding styles between a test case and a harmony are also determined in advance. At the stage of test data searching, the subset of test data that has much stronger covering ability is stored in harmony memory. During the evolution process, one part of test suite is selected and adjusted from the harmony memory, and the other part is randomly generated from input space. Once a test suite is yielded after one-round search, its coverage can be measured by fitness function in our search algorithm. In our work, a new fitness function for branch coverage is constructed by comprehensively considering branch distance and branch weight. Here, the branch weight is determined by branch information in program, that is, the nesting level of a specific branch and the predicate types in it. Subsequently, the computed coverage metric is used for updating the test suite in the next round of searching. In order to validate the effectiveness of our proposed method, eight well-known programs are used for experimental evaluation. Experimental results show that the coverage of HS-based method is usually higher than those of other search algorithms, such as simulated annealing (SA) and genetic algorithm (GA). Meanwhile, HS demonstrates greater stability than SA and GA when varying the population size or performing repeated trials. That is to say, music-inspired HS algorithm is more suitable to generate test data for branch coverage in software structural testing.     

Observations in using parallel and sequential evolutionary algorithms for automatic software testing

In this paper we analyze the application of parallel and sequential evolutionary algorithms (EAs) to the automatic test data generation problem. The problem consists of automatically creating a set of input data to test a program. This is a fundamental step in software development and a time consuming task in existing software companies. Canonical sequential EAs have been used in the past for this task. We explore here the use of parallel EAs. Evidence of greater efficiency, larger diversity maintenance, additional availability of memory/CPU, and multi-solution capabilities of the parallel approach, reinforce the importance of the advances in research with these algorithms. We describe in this work how canonical genetic algorithms (GAs) and evolutionary strategies (ESs) can help in software testing, and what the advantages are (if any) of using decentralized populations in these techniques. In addition, we study the influence of some parameters of the proposed test data generator in the results. For the experiments we use a large benchmark composed of twelve programs that includes fundamental algorithms in computer science.