Empirical studies of a prediction model for regression testselection

Regression testing is an important activity that can account for a large proportion of the cost of software maintenance. One approach to reducing the cost of regression testing is to employ a selective regression testing technique that: chooses a subset of a test suite that was used to test the software before the modifications; then uses this subset to test the modified software. Selective regression testing techniques reduce the cost of regression testing if the cost of selecting the subset from the test suite together with the cost of running the selected subset of test cases is less than the cost of rerunning the entire test suite. Rosenblum and Weyuker (1997) proposed coverage-based predictors for use in predicting the effectiveness of regression test selection strategies. Using the regression testing cost model of Leung and White (1989; 1990), Rosenblum and Weyuker demonstrated the applicability of these predictors by performing a case study involving 31 versions of the KornShell. To further investigate the applicability of the Rosenblum-Weyuker (RW) predictor, additional empirical studies have been performed. The RW predictor was applied to a number of subjects, using two different selective regression testing tools, Deja vu and TestTube. These studies support two conclusions. First, they show that there is some variability in the success with which the predictors work and second, they suggest that these results can be improved by incorporating information about the distribution of modifications. It is shown how the RW prediction model can be improved to provide such an accounting     

Reduction-based methods and metrics for selective regression testing

In corrective maintenance, modified software is regression tested using selected test cases in order to ensure that the modifications have not caused adverse effects. This activity of selective regression testing involves regression test selection, which refers to selecting test cases from the previously run test suite, and test-coverage identification. In this paper, we propose three test-selection methods and two coverage identification metrics. The three methods aim to reduce the number of selected test cases for retesting the modified software. The first method, referred to as modification-based reduction version 1 (MBR1), selects a reduced number of test cases based on the modification made and its effects in the software. The second method, referred to as modification-based reduction version 2 (MBR2) improves MBR1 by further omitting tests that do not cover the modification. The third method, referred to as precise reduction (PR), reduces the number of test cases selected by omitting non-modification-revealing tests from the initial test suite. The two coverage metrics are McCabe-based regression test metrics, which are referred to as the Reachability regression Test selection McCabe-based metric (RTM), and data-flow Slices regression Test McCabe-based metric (STM). These metrics aim to assist the regression tester in monitoring test-coverage adequacy, reveal any shortage or redundancy in the test suite, and assist in identifying, where additional tests may be required for regression testing.We empirically compare MBR1, MBR2, and PR with three reduction and precision-oriented methods on 60 test-problems. The results show that PR selects the least number of test cases most of the time and omits non-modification-revealing tests. We also demonstrate the applicability of our proposed methods to object-oriented regression testing at the class level. Further, we illustrate typical application of the RTM and STM metrics using the 60 test-problems and two coverage-oriented selective regression-testing methods.     

多维标度法选择回归测试子集

在软件改动较大且资源限制严格时,回归测试目前的方法难以满足实际需要。因此,提出利用多维标度法选择回归测试子集的方法。该方法使用测试执行剖面表示测试集,通过计算剖面数据,将测试集合按照测试效果可视化分类,综合考虑了软件变更及测试用例的典型性,适用于大规模软件更改较多时获取有代表性的测试子集。实验表明,多维标度法能够全面反映原测试集对变更的测试程度,从而使得回归子集的获取及测试实施更有针对性,能够满足限制严格的回归测试要求。     

一个划算的回归测试用例排序的成本-收益模型

回归测试是一个昂贵的测试过程,它验证修改的软件,确认是否给已经测试的代码引进了新的错误.因为在软件开发的过程中,测试人员给测试包中增加了新的测试用例,使软件确认更加困难.为了使回归测试更经济划算,提出了很多技术和方法.测试用例选择,测试包缩减,测试用例排序可以帮助我们减少需要运行的测试用例数目,同时能够更快地满足测试目标.测试用例排序是在满足某个性能目标的基础上,保证在测试过程中首先执行更重要的功能.以前的研究提出了一个性能目标:缺陷发现率(APFD),以及很多改善缺陷发现率的技术.但是这些技术运用起来非常昂贵,而且可能不会减少回归测试的成本.在以往研究的基础上,提出了一个成本-收益模型,为比较各种排序算法提供了一定依据.并在此基础上,提供了一个测试用例排序算法.     

用遗传算法的测试用例最小化

回归测试就是验证对程序的修改是否迭到了预期的目的，同时检查修改是否损害了程序原有的正常功能。随着程序新版本的不断推出，测试用例集不断扩大，回归测试成本越来越高。测试用例最小化就是从已有的用例集中，找到一个测试运行代价最小的用例子集用于回归测试，并保持原来的测试覆盖率。本文主要研究用遗传算法解决测试用例最小化问题：基于测试历史数据，设计基因编码并构建初始种群；利用测试覆盖率和测试运行代价设计适应度函数；通过遗传算子完成进化过程并找到最优或近似最优解。最后本文给出了对算法进行实例研究的结果。结果表明，本文提出的用例最小化技术能有效缩减回归测试用例集，大幅度降低回归测试费用。     

构件软件回归测试用例选择策略

软构件技术虽被广泛应用于软件系统的开发中，但其测试问题并未得到很好地解决．系统构建者对外部提供的构件内部结构及其变更信息缺乏了解，很难选择出与构件变更相关的用例用于下一轮的测试．分析已有回归测试技术的不足，提出了两种改进的回归测试策略：一种是基于增强的构件版本变更信息的方法；另一种则是基于内建式测试设计的方法．通过对几个实例程序的实验分析，初步证实了所提出的方法在实际应用中的可行性与有效性．     

基于“Good”测试套件的GUI回归测试优化

回归测试在软件测试过程中是非常重要的,同时也是非常费时费力的。为了适应软件测试的需求,提高回归测试的效率,降低测试成本,针对目前GUI回归测试的困难,依据Atif M.Memon等人提出的新的回归测试方法,根据一个"Good"的测试套件对测试成本和"fault-detection effectiveness"的影响,突出一个"Good"的测试套件的特点,给出一个新的回归测试方法。该方法根据测试套件的特点,有针对性地选择合适的测试用例来构建回归测试套件。这样不仅有针对性,同时也优化了测试套件,提高了测试效率。     

A safe regression testing approach for safety critical systems

Regression testing is important activity during the software maintenance to deal with adverse effects of changes. Our approach is important for safety critical system as usually formal methods are preferred and highly recommended for the safety critical systems but they are also applied for the systems development of other than critical system. Our approach is based on Regression testing using VDM++ which takes two VDM++ specifications, one baseline and other delta (Changed) along with test suite for the baseline version. It compares both versions by using comparator module, identifies the change. By analyzing the change we classify the test cases from original test suite into obsolete, re-testable, and reusable test cases. Our scope is at unit level i.e. at class level. Our approach gets two versions of VDM++ specification and returns regression test suite for the delta version. Our approach distinguishes test cases which are still effective for the delta version of VDM++ specification and it differs from re-test all strategy as it can distinguish the test cases and identifies test cases which are useful for delta version. Test cases reusability and test case reduction is the main objective of our approach. Our approach presents how to perform regression testing using VDM++ specification during the maintenance of systems.     

测试用例集约简问题研究及其进展

测试用例集约简问题是软件测试中的关键问题之一,其目的是使用尽可能少的测试用例充分满足给定的测试目标,从而提高测试效率、降低测试成本。在简要介绍了测试用例集约简问题基本概念的基础上,总结了求解该问题的几种主要方法,分析比较了这些方法的效率和特性。随后探讨了与测试用例集约简问题强相关的测试用例集错误检测效率的问题,并研究了测试用例优先级技术。最后指出了测试用例集约简问题的下一步研究方向。     

A learning algorithm for optimizing continuous integration development and testing practice

Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large-scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault-detection effectiveness of continuous integration tests using historical test records and combining this information with coverage-based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest-all approach. The results further demonstrate no significant reduction in fault-detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.     

基于多优化目标的软件测试用例约简方法研究

软件测试是软件工程的一个重要组成部分,其目标是能够及时发现软件中的错误,确保软件高质量。测试用例是软件测试的基础,覆盖度较高且精简的测试用例集可以提高测试效率和降低成本。软件测试覆盖标准较多,一个好的测试用例评价指标也存在多种,为了能够在约简测试用例集规模的同时获取较高的测试能力,本文提出了一种基于多优化目标的测试用例集约简算法,该算法旨在根据测试用例需求,构建多优化目标的测试用例模型,使用该模型获取一个最优解的测试用例子集,使用最小化用例集方法最小化测试用例,迭代执行直到测试用例集覆盖所有的测试需求,实验结果表明该算法可以约简测试用例集,获取较高的综合测试效果。     

XML-manipulating test case prioritization for XML-manipulating services

A web service may evolve autonomously, making peer web services in the same service composition uncertain as to whether the evolved behaviors are compatible with its original collaborative agreement. Although peer services may wish to conduct regression testing to verify the agreed collaboration, the source code of the former service may be inaccessible to them. Owing to the black-box nature of peer services, traditional code-based approaches to regression testing are inapplicable. In addition, traditional techniques assume that a regression test suite for verifying a web service is available. The location to store a regression test suite is also a problem. On the other hand, we note that the rich interface specifications of a web service provide peer services with a means to formulate black-box testing strategies. In this paper, we provide a strategy for black-box service-oriented testing. We also formulate new test case prioritization strategies using tags embedded in XML messages to reorder regression test cases, and reveal how the test cases use the interface specifications of web services. We experimentally evaluate the effectiveness of these black-box strategies in revealing regression faults in modified WS-BPEL programs. The results show that the new techniques can have a high chance of outperforming random ordering. Moreover, our experiment shows that prioritizing test cases based on WSDL tag coverage can achieve a smaller variance than that based on the number of tags in XML messages in regression test cases, even though their overall fault detection rates are similar.     

回归测试中测试用例集优化方法的研究

回归测试是软件演化过程中频繁进行的且开销巨大的一项任务,测试用例集的优化程度直接影响着测试的成本和效率。针对回归测试过程的特点,提出一种对测试用例集优化的新方法,即通过对测试用例集进行必要的消除冗余和调整排序,完成了对初始测试用例集的精简以及执行顺序的确定过程,使得有限的测试资源得到科学合理的分配。实验结果表明,相对于以往的测试用例集优化方法,新方法的效率和资源分配的合理性均有了显著的提高。     

测试用例集启发式约简算法分析与评价

在软件开发和维护过程中,为了提高对源程序变更部分的信心并且保证源程序变更部分没有对未变更部分造成负面影响,需要对软件系统进行回归测试。回归测试是一个昂贵的测试过程。测试用例集约简算法是在仍然满足测试准则的前提下,通过删除所有冗余测试用例得到测试用例集的最小约简测试用例集,用以优化回归测试过程。综述了文献中主要的测试用例集启发式约简算法,通过统一的框架和术语定义了这些算法,分析和比较了这些算法的效率和优劣,指出了未来进一步研究的方向。     

Provable improvements on branch testing

This paper compares the fault-detecting ability of several software test data adequacy criteria. It has previously been shown that if C₁ properly covers C₂, then C₁ is guaranteed to be better at detecting faults than C₂, in the following sense: a test suite selected by independent random selection of one test case from each subdomain induced by C₁ is at least as likely to detect a fault as a test suite similarly selected using C₂. In contrast, if C₁ subsumes but does not properly cover C₂, this is not necessarily the case. These results are used to compare a number of criteria, including several that have been proposed as stronger alternatives to branch testing. We compare the relative fault-detecting ability of data flow testing, mutation testing, and the condition-coverage techniques, to branch testing, showing that most of the criteria examined are guaranteed to be better than branch testing according to two probabilistic measures. We also show that there are criteria that can sometimes be poorer at detecting faults than substantially less expensive criteria     

Empirical studies of a safe regression test selection technique

Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression test selection techniques select subsets that, under certain well-defined conditions, exclude no tests (from the original test suite) that if executed would reveal faults in the modified software. Many regression test selection techniques, including several safe techniques, have been proposed, but few have been subjected to empirical validation. This paper reports empirical studies on a particular safe regression test selection technique, in which the technique is compared to the alternative regression testing strategy of running all tests. The results indicate that safe regression test selection can be cost-effective, but that its costs and benefits vary widely based on a number of factors. In particular, test suite design can significantly affect the effectiveness of test selection, and coverage-based test suites may provide test selection results superior to those provided by test suites that are not coverage-based     

Test-suite reduction and prioritization for modified condition/decision coverage

Software testing is particularly expensive for developers of high-assurance software, such as software that is produced for commercial airborne systems. One reason for this expense is the Federal Aviation Administration's requirement that test suites be modified condition/decision coverage (MC/DC) adequate. Despite its cost, there is evidence that MC/DC is an effective verification technique and can help to uncover safety faults. As the software is modified and new test cases are added to the test suite, the test suite grows and the cost of regression testing increases. To address the test-suite size problem, researchers have investigated the use of test-suite reduction algorithms, which identify a reduced test suite that provides the same coverage of the software according to some criterion as the original test suite, and test-suite prioritization algorithms, which identify an ordering of the test cases in the test suite according to some criteria or goals. Existing test-suite reduction and prioritization techniques, however, may not be effective in reducing or prioritizing MC/DC-adequate test suites because they do not consider the complexity of the criterion. This paper presents new algorithms for test-suite reduction and prioritization that can be tailored effectively for use with MC/DC. The paper also presents the results of empirical studies of these algorithms.     

由需求得到回归测试用例排序技术

为了提高回归测试用例集的测试效率和有效性,提出由需求得到回归测试用例排序技术及其实现算法。由需求得到回归测试用例排序技术,将与软件需求相关的需求描述度、需求实现复杂度、需求稳定度和需求覆盖度等因素应用于测试用例排序,以缺陷检测加权平均百分比作为度量标准。通过实验,比较排序后用例和未排序用例缺陷检测情况,实验结果表明该技术排序后的回归测试用例集,能够尽早地发现更多的软件错误,有效提高回归测试效率,保证软件质量。     

嵌入式API测试套生成方法和技术

随着嵌入式计算机系统应用的不断扩展,嵌入式系统的可靠性引起了学术界和工业界的广泛关注,也提出了很多增进可靠性的方法和技术.然而,现有的方法和技术在测试套生成方面论述不多,所以在处理大批量嵌入式系统测试工作中遇到了挑战.讨论抽象测试套生成方法和适配技术,提出了LTS(labeled transition system)到BT(behavior tree)的转换算法,从而使TTCN(test and testing control notation)测试套可以通过转换嵌入式软件的LTS描述产生.还介绍了基于上述转换算法的嵌入式软件测试工具包,以及一个嵌入式物联网识读器测试案例研究.     

ALL-Uses覆盖引导的回归测试用例生成方法

基于扩展有限状态机(EFSM)的回归测试过程需要根据依赖关系变化对软件所做修改的影响域进行分析。为了针对软件某一功能进行修复,通常需要对多处代码进行同步修改,已有依赖分析方法在这种情况下暴露模型中触发条件和行为语句错误的效率不高。提出以ALL-Uses覆盖准则引导回归测试的方法,引入依赖关系变化因素的概念,修改待覆盖子路径的产生规则,对已有测试用例集中能有效覆盖子路径的用例进行选择、排序。针对已有测试用例无法覆盖的子路径,利用AOE活动图中求关键路径的方法将其补充为一条完整的迁移执行序列。选取三个软件进行实验,结果表明,本文方法可以在减小测试用例集规模的前提下有效提升ALL-Uses和植入错误的覆盖率,提高回归测试效率。