Improving test efficiency through system test prioritization

Software testing is an expensive process consuming at least 50% of the total development cost. Among the types of testing, system testing is the most expensive and complex. Companies are frequently faced with budgetary constraints, which may limit their ability to effectively complete testing efforts before delivering a software product. We build upon prior test case prioritization research and present a system-level approach to test case prioritization called Prioritization of Requirements for Test (PORT). PORT prioritizes system test cases based on four factors for each requirement: customer priority, implementation complexity, fault proneness, and requirements volatility. Test cases for requirements with higher priority based upon a weighted average of these factors are executed earlier in system test. An academic feasibility study and three post hoc industrial studies were conducted. Results indicate that PORT can be used to improve the rate of failure detection when compared with a random and operational profile-driven random approach. Furthermore, we investigated the contribution of the prioritization factors towards the improved rate of failure detection and found customer priority was the most significant contributor. Tool support is provided for the PORT scheme which allows for automatic collection of the four factor values and the resultant test case prioritization.