一个划算的回归测试用例排序的成本-收益模型

回归测试是一个昂贵的测试过程,它验证修改的软件,确认是否给已经测试的代码引进了新的错误.因为在软件开发的过程中,测试人员给测试包中增加了新的测试用例,使软件确认更加困难.为了使回归测试更经济划算,提出了很多技术和方法.测试用例选择,测试包缩减,测试用例排序可以帮助我们减少需要运行的测试用例数目,同时能够更快地满足测试目标.测试用例排序是在满足某个性能目标的基础上,保证在测试过程中首先执行更重要的功能.以前的研究提出了一个性能目标:缺陷发现率(APFD),以及很多改善缺陷发现率的技术.但是这些技术运用起来非常昂贵,而且可能不会减少回归测试的成本.在以往研究的基础上,提出了一个成本-收益模型,为比较各种排序算法提供了一定依据.并在此基础上,提供了一个测试用例排序算法.     

基于历史数据和多目标优化的测试用例排序方法

针对如何提高测试用例序列的揭错效率和回归测试效益问题,提出一种基于历史数据和多目标优化的测试用例排序方法。首先,根据测试用例的文本主题相似性和代码覆盖相似性对测试用例集进行聚类,并根据历史执行信息对测试用例间的执行失败关系进行关联规则挖掘,从而为后续过程做准备;然后,利用多目标优化算法对每个类簇内的测试用例进行排序,在此之后生成最终排序序列,使相似的测试用例分隔开;最后,利用测试用例间的关联规则,动态调整测试用例执行次序,从而使可能失败的测试用例优先执行,以进一步提高缺陷检测效率。与随机排序方法、基于聚类的排序方法、基于主题模型的排序方法、基于关联规则和多目标优化的排序方法相比,所提方法的平均故障检测率（APFD）平均值分别提高了12.59%、5.98%、3.01%和2.95%,基于成本的平均故障检测率（APFDc）平均值分别提高了17.17%、5.04%、5.08%和8.21%。实验结果表明,所提方法能有效提高回归测试效益。     

基于定值-引用链的测试用例优先级排序算法

测试用例优先级排序作为一种高效实用的回归测试技术,通常以测试用例的覆盖度作为优先级排序的量化指标,忽略了测试用例的其他测试性能。针对该问题,提出一种基于DU链的测试用例优先级排序算法。该算法 综合考虑 测试用例的DU链覆盖度和回归测试的错误检测能力,对测试用例优先级进行量化。与已有算法相比,该算法基于数据流覆盖,充分利用了测试执行的历史信息和程序模块的耦合信息,在排序过程中动态计算测试用例的优先级量化值。实验结果表明,采用优先级排序算法的测试用例集能在测试过程中以较短的时间发现更多的错误,有效地提高了回归测试的检错效率。     

运用变异测试的并行程序测试用例最小化算法

在并行程序测试中,测试输入和线程交互时序是影响并行错误检测的两个关键因素。以缩减并行错误检测的输入空间为目标,给出一种基于变异测试的测试用例最小化算法。首先对并行程序进行研究,选取与并行错误密切相关的9个变异算子,并以此为基础为待测程序生成多种变异体;采用JPF作为线程调度工具来执行测试用例,根据变异评分与平均时间成本对测试用例进行排序,在优化后的测试用例集中选取检测能力不重复的测试用例,从而得到面向并行错误检测的最小测试用例集。实验结果证明,该方法能有效减小测试用例集的规模,并大幅缩短运行时间,从而提高了并行程序的测试效率。     

面向CPU+GPU异构计算的多目标测试用例优先排序

测试用例优先排序是一种基于整个测试用例集以寻找最优测试用例执行序列的软件回归测试技术.由于其能够尽早地发现错误,同时应用灵活度高、不会漏掉重要测试用例等,在实际软件测试过程中可以有效提高测试效率.多目标测试用例优化排序是寻找同时覆盖多个测试准则的用例执行序列,通常采用演化算法优化求解,但执行时间较长,严重影响了在实际软件测试中的应用.采用先进的GPU图形卡通用并行计算技术,提出了面向CPU+GPU异构计算下的多目标测试用例优先排序技术,在NSGA-II算法中,实现了基于序列编码的适应度函数计算和交叉操作的GPU并行计算,在近6万行有效代码的工业界开源程序上实现了30倍的计算效率提升.同时,实验验证了不同并行策略的计算加速比,提出了切实可行的CPU+GPU异构计算模式,并提供了相应的原形工具.     

基于函数调用路径的回归测试用例选择排序方法研究

针对在回归测试过程中,因为不断修复软件中存在的缺陷所造成的测试工作量大、测试效率低等问题,论文将测试用例选择与优先级排序技术相结合,以面向函数调用的路径覆盖生成方法为基础,提出了一种面向函数调用路径（Functions Calling Path, FCP）的测试用例选择与排序方法。首先根据函数调用关系图,对程序中被修改函数与其他函数的关联性进行分析,从初始测试用例集中选择测试用例,形成回归测试用例集;然后对这些测试用例进行优先级排序,并动态地调整优先级排序结果;最后,对优先级排序结果进行再次选择,确定最小的回归测试用例集。实验结果表明,测试用例选择与排序方法对优化回归测试用例是有效的,大大减少了回归测试用例数量,降低了回归测试成本。     

回归测试中测试用例集优化方法的研究

回归测试是软件演化过程中频繁进行的且开销巨大的一项任务,测试用例集的优化程度直接影响着测试的成本和效率。针对回归测试过程的特点,提出一种对测试用例集优化的新方法,即通过对测试用例集进行必要的消除冗余和调整排序,完成了对初始测试用例集的精简以及执行顺序的确定过程,使得有限的测试资源得到科学合理的分配。实验结果表明,相对于以往的测试用例集优化方法,新方法的效率和资源分配的合理性均有了显著的提高。     

基于函数调用路径的测试用例优先级排序

基于覆盖的优先级排序技术通常以代码覆盖信息作为测试用例的特征加以度量,忽略了其他优先级的影响因素,缺乏全面性和动态性。针对该问题,提出基于函数调用路径的测试用例优先级排序方法。以函数调用路径为基础,通过对源代码新旧版本的对比,分析回归测试影响域,确定回归测试用例集的范围。将测试用例函数调用路径覆盖能力、单元测试时函数中检测出缺陷的个数,以及函数的扇入系数等影响因素应用于优先级排序,确定测试用例优先级量化方法,并在测试执行过程中,通过调整算法实现优先级的动态调整,优化优先级排序。实验结果表明,优先级量化方法能提高测试的缺陷检测率,发现程序中的缺陷,降低测试成本。     

基于PSO的多目标测试用例预优化

随着软件规模的增大,在软件回归测试中,重复执行庞大的全部测试用例集已不再现实。在这种情况下,对测试用例集进行预处理就尤为重要。测试用例预优化是寻找最佳测试用例执行序列的一种技术。在实际的软件回归测试中,基于多目标的测试用例优化技术已逐步取代了单目标优化;应用进化算法解决多目标测试用例预优化是当前研究的热点。但由于进化算法主要是基于种群进行遗传迭代,种群间的交互机制相对复杂,算法的执行效率会随着种群及测试用例集规模的增大而显著下降。针对上述情况,提出了一种基于粒子群优化算法(PSO)的测试用例预优化方法,设计了粒子的表示和状态更新方式,研究了不同粒子更新方式和迭代次数及粒子群大小对多目标测试用例预优化结果的影响。实验结果显示,同基于NSGA-Ⅱ的方法相比,所提方法的执行效率显著提高,可以解决实际回归测试中的多目标测试用例预优化问题。     

软件回归测试中的自动测试生成方法

软件回归测试一般使用现有的测试用例集进行测试,如何有效利用这些用例成为回归测试的关键。研究自动对现有测试用例集的扩展,包括自动改变测试用例的执行顺序、自动进行测试用例数目的增减等,提出基于现有测试用例随机生成新的测试用例集的方法。实验证明,该方法生成的测试文件能在原有环境下成功执行,能实现自动测试范围的扩大或缩小,以及测试时间的增加或减少,为软件回归测试提供了有用的自动化工具。     

Improving test efficiency through system test prioritization

Software testing is an expensive process consuming at least 50% of the total development cost. Among the types of testing, system testing is the most expensive and complex. Companies are frequently faced with budgetary constraints, which may limit their ability to effectively complete testing efforts before delivering a software product. We build upon prior test case prioritization research and present a system-level approach to test case prioritization called Prioritization of Requirements for Test (PORT). PORT prioritizes system test cases based on four factors for each requirement: customer priority, implementation complexity, fault proneness, and requirements volatility. Test cases for requirements with higher priority based upon a weighted average of these factors are executed earlier in system test. An academic feasibility study and three post hoc industrial studies were conducted. Results indicate that PORT can be used to improve the rate of failure detection when compared with a random and operational profile-driven random approach. Furthermore, we investigated the contribution of the prioritization factors towards the improved rate of failure detection and found customer priority was the most significant contributor. Tool support is provided for the PORT scheme which allows for automatic collection of the four factor values and the resultant test case prioritization.     

基于测试用例设计信息的回归测试优先级算法

优先级技术是一种高效实用的回归测试技术.文中针对现有优先级技术未能有效使用测试用例设计信息的不足,提出了一组新的回归测试优先级动态调整算法.与已有方法相比,新算法充分考虑了测试用例的设计信息,能够通过及时捕捉和利用测试执行信息对测试用例优先级进行动态调整,具有时间复杂度低、检错效率高等优点.将其应用于Windows平台下应用软件的回归测试结果表明,新算法有益于在短时间内检测出更多的错误.     

基于函数占用和需求分析测试代码用例自适应优先排序

在采用函数占用（覆盖）的测试代码用例优先级排序过程中,其特征一般是以函数占用代码率信息表示,容易忽视其他优先排序影响因素,对此提出基于函数占用和需求分析的测试代码用例自适应优先排序方式。首先,以测试代码用例优先级排序过程中的调用函数存在路径为参照,利用源代码信息变更影响,进行影响域的回归测试分析,并对测试代码用例集进行回归范围确定。其次,考虑测试代码用例优先级排序的需求影响因素,并确定其需求优先级评价指标,然后与调用函数源代码变更影响指标进行权重自适应融合,并设计基于函数占用和需求分析评价优先级的动态调整算法。最后,通过仿真对比显示,所提方法可提高缺陷检测正确率,降低测试成本。     

基于聚类算法的测试用例排序

在软件开发过程中,回归测试是一项重要而复杂耗时的工作。测试用例排序技术作为提高测试效率的一种有效手段,是近年的研究热点之一。测试用例排序技术根据测试目标对测试用例进行排序,使得最优的测试用例能够优先执行。同时,各种软件信息的底层结构及关联信息有助于提高测试效率。不同的代码单元在测试过程中拥有不同的测试需求。因此,利用聚类算法能够有效地分析测试用例间的结构信息,从而指导测试用例的排序过程。实验结果表明,新的测试用例排序策略有效地提高了测试用例排序的效果,能够更早地检测到软件错误。     

Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites. In particular, Java and the JUnit testing framework are being used extensively to build software systems in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm. To investigate the practical implications of these results, we present a set of cost-benefits models for test case prioritization, and show how the effectiveness differences observed can result in savings in practice, but vary substantially with the cost factors associated with particular testing processes.     

Improving the efficiency of functional verification based on test prioritization

Functional verification has become the key bottleneck that delays time-to-market during the embedded system design process. And simulation-based verification is the mainstream practice in functional verification due to its flexibility and scalability. In practice, the success of the simulation-based verification highly depends on the quality of functional tests in use which is usually evaluated by coverage metrics. Since test prioritization can provide a way to simulate the more important tests which can improve the coverage metrics evidently earlier, we propose a test prioritization approach based on the clustering algorithm to obtain a high coverage level earlier in the simulation process. The k-means algorithm, which is one of the most popular clustering algorithms and usually used for the test prioritization, has some shortcomings which have an effect on the effectiveness of test prioritization. Thus we propose three enhanced k-means algorithms to overcome these shortcomings and improve the effectiveness of the test prioritization. Then the functional tests in the simulation environment can be ordered with the test prioritization based on the enhanced k-means algorithms. Finally, the more important tests, which can improve the coverage metrics evidently, can be selected and simulated early within the limited simulation time. Experimental results show that the enhanced k-means algorithms are more accurate and efficient than the standard k-means algorithm for the test prioritization, especially the third enhanced k-means algorithm. In comparison with simulating all the tests randomly, the more important tests, which are selected with the test prioritization based on the third enhanced k-means algorithm, achieve almost the same coverage metrics in a shorter time, which achieves a 90% simulation time saving.     

Selecting a Cost-Effective Test Case Prioritization Technique

Regression testing is an expensive testing process used to validate modified software and detect whether new faults have been introduced into previously tested code. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One goal of prioritization is to increase a test suite's rate of fault detection. Previous empirical studies have shown that several prioritization techniques can significantly improve rate of fault detection, but these studies have also shown that the effectiveness of these techniques varies considerably across various attributes of the program, test suites, and modifications being considered. This variation makes it difficult for a practitioner to choose an appropriate prioritization technique for a given testing scenario. To address this problem, we analyze the fault detection rates that result from applying several different prioritization techniques to several programs and modified versions. The results of our analyses provide insights into which types of prioritization techniques are and are not appropriate under specific testing scenarios, and the conditions under which they are or are not appropriate. Our analysis approach can also be used by other researchers or practitioners to determine the prioritization techniques appropriate to other workloads.     

基于蚁群算法的测试用例优先排序

测试用例优先排序技术通过优化测试用例的执行次序来提高软件测试的效率,是增强型软件测试和回归测试的重要研究课题。针对基于需求的测试用例优先排序问题,提出了一种基于蚁群算法的求解方法,采用不同的测试用例间距离及用例序列评价策略,给出了该方法的2种不同实现方式。首先,针对黑盒测试特点,设计了基于需求的一般性测试用例序列评价指标;其次,提出测试用例吸引度概念,基于测试用例吸引度定义了测试用例间的距离;然后,给出了信息素更新策略、最优解集更新策略、局部最优解突变策略等主要设计策略,分别实现了该方法基于距离和基于指标的2种实现方式。实验结果表明,该方法具有很好的全局寻优能力,整体效果上优于粒子群算法、遗传算法和随机测试。     

基于搜索算法的测试用例优化问题研究

测试用例优化是软件测试领域的研究热点之一,已有优化算法的评估多以代码覆盖为基础,这并不能完全反应程序的错误状况,高代码覆盖率的测试用例并不一定具有高的揭错能力.本文结合变异测试,从新的角度-———错误覆盖出发,对两种贪心算法、一种启发式算法、及遗传算法的优化效果进行了评估,实验结果表明额外贪心算法优于其它三种算法.该评估结果的置信度更高,为此类算法的评估提供了新的研究思路.