Ranking of software engineering metrics by fuzzy‐based matrix methodology

This research paper presents a framework for ranking of software engineering metrics based on expert opinion elicitation and fuzzy‐based matrix methodology. The proposed methodology is able to accommodate the imprecise and inexact data involved in the problem of ranking of software engineering metrics, vagueness and ambiguity occurring during expert (human) decision making and to depart from the complexity of formulation of the objective and the constraint function. The matrices lend themselves to mechanical manipulations and are useful for analyzing and deriving systems functions expeditiously to meet the objectives. The current research is based on software engineering metrics identified in an earlier study conducted by Lawrence Livermore National Laboratory. A set of ranking criteria were identified. Software engineering metrics are then ranked in ascending order using experts' opinion in accordance with the value of Permanent function on their criteria matrix. The proposed methodology has also been compared with other known methodologies. Copyright © 2011 John Wiley & Sons, Ltd.     

The impact of institutional forces on software metrics programs

Software metrics programs are an important part of a software organization's productivity and quality initiatives as precursors to process-based improvement programs. Like other innovative practices, the implementation of metrics programs is prone to influences from the greater institutional environment the organization exists in. In this paper, we study the influence of both external and internal institutional forces on the assimilation of metrics programs in software organizations. We use previous case-based research in software metrics programs as well as prior work in institutional theory in proposing a model of metrics implementation. The theoretical model is tested on data collected through a survey from 214 metrics managers in defense-related and commercial software organizations. Our results show that external institutions, such as customers and competitors, and internal institutions, such as managers, directly influence the extent to which organizations change their internal work-processes around metrics programs. Additionally, the adaptation of work-processes leads to increased use of metrics programs in decision-making within the organization. Our research informs managers about the importance of management support and institutions in metrics programs adaptation. In addition, managers may note that the continued use of metrics information in decision-making is contingent on adapting the organization's work-processes around the metrics program. Without these investments in metrics program adaptation, the true business value in implementing metrics and software process improvement is not realized.     

Tradeoff and Sensitivity Analysis in Software Architecture Evaluation Using Analytic Hierarchy Process

Software architecture evaluation involves evaluating different architecture design alternatives against multiple quality-attributes. These attributes typically have intrinsic conflicts and must be considered simultaneously in order to reach a final design decision. AHP (Analytic Hierarchy Process), an important decision making technique, has been leveraged to resolve such conflicts. AHP can help provide an overall ranking of design alternatives. However it lacks the capability to explicitly identify the exact tradeoffs being made and the relative size of these tradeoffs. Moreover, the ranking produced can be sensitive such that the smallest change in intermediate priority weights can alter the final order of design alternatives. In this paper, we propose several in-depth analysis techniques applicable to AHP to identify critical tradeoffs and sensitive points in the decision process. We apply our method to an example of a real-world distributed architecture presented in the literature. The results are promising in that they make important decision consequences explicit in terms of key design tradeoffs and the architecture's capability to handle future quality attribute changes. These expose critical decisions which are otherwise too subtle to be detected in standard AHP results. Liming Zhu is a PHD candidate in the School of Computer Science and Engineering at University of New South Wales. He is also a member of the Empirical Software Engineering Group at National ICT Australia (NICTA). He obtained his BSc from Dalian University of Technology in China. After moving to Australia, he obtained his MSc in computer science from University of New South Wales. His principle research interests include software architecture evaluation and empirical software engineering. Aybüke Aurum is a senior lecturer at the School of Information Systems, Technology and Management, University of New South Wales. She received her BSc and MSc in geological engineering, and MEngSc and PhD in computer science. She also works as a visiting researcher in National ICT, Australia (NICTA). Dr. Aurum is one of the editors of “Managing Software Engineering Knowledge”, “Engineering and Managing Software Requirements” and “Value-Based Software Engineering” books. Her research interests include management of software development process, software inspection, requirements engineering, decision making and knowledge management in software development. She is on the editorial boards of Requirements Engineering Journal and Asian Academy Journal of Management. Ian Gorton is a Senior Researcher at National ICT Australia. Until Match 2004 he was Chief Architect in Information Sciences and Engineering at the US Department of Energy's Pacific Northwest National Laboratory. Previously he has worked at Microsoft and IBM, as well as in other research positions. His interests include software architectures, particularly those for large-scale, high-performance information systems that use commercial off-the-shelf (COTS) middleware technologies. He received a PhD in Computer Science from Sheffield Hallam University. Dr. Ross Jeffery is Professor of Software Engineering in the School of Computer Science and Engineering at UNSW and Program Leader in Empirical Software Engineering in National ICT Australia Ltd. (NICTA). His current research interests are in software engineering process and product modeling and improvement, electronic process guides and software knowledge management, software quality, software metrics, software technical and management reviews, and software resource modeling and estimation. His research has involved over fifty government and industry organizations over a period of 15 years and has been funded from industry, government and universities. He has co-authored four books and over one hundred and twenty research papers. He has served on the editorial board of the IEEE Transactions on Software Engineering, and the Wiley International Series in Information Systems and he is Associate Editor of the Journal of Empirical Software Engineering. He is a founding member of the International Software Engineering Research Network (ISERN). He was elected Fellow of the Australian Computer Society for his contribution to software engineering research.     

Toward objective software process information: experiences from a case study

A critical problem in software development is the monitoring, control and improvement in the processes of software developers. Software processes are often not explicitly modeled, and manuals to support the development work contain abstract guidelines and procedures. Consequently, there are huge differences between ‘actual’ and ‘official’ processes: “the actual process is what you do, with all its omissions, mistakes, and oversights. The official process is what the book, i.e., a quality manual, says you are supposed to do” (Humphrey in A discipline for software engineering. Addison-Wesley, New York, 1995). Software developers lack support to identify, analyze and better understand their processes. Consequently, process improvements are often not based on an in-depth understanding of the ‘actual’ processes, but on organization-wide improvement programs or ad hoc initiatives of individual developers. In this paper, we show that, based on particular data from software development projects, the underlying software development processes can be extracted and that automatically more realistic process models can be constructed. This is called software process mining (Rubin et al. in Process mining framework for software processes. Software process dynamics and agility. Springer Berlin, Heidelberg, 2007). The goal of process mining is to better understand the development processes, to compare constructed process models with the ‘official’ guidelines and procedures in quality manuals and, subsequently, to improve development processes. This paper reports on process mining case studies in a large industrial company in The Netherlands. The subject of the process mining is a particular process: the change control board (CCB) process. The results of process mining are fed back to practice in order to subsequently improve the CCB process.     

结合多元度量指标软件缺陷预测研究进展

软件缺陷预测可帮助开发人员提前预测缺陷程序,合理分配有限的测试资源。软件缺陷预测的准确度不仅依赖于预测方法的选择,更依赖于软件的度量指标。因此,结合多元度量指标进行软件缺陷预测已成为当前的研究热点。从度量指标出发,对传统度量指标、多元度量指标以及结合多元度量指标的缺陷预测的研究进展进行了系统介绍。主要工作包含：介绍了传统的代码和过程度量指标、基于传统度量指标的软件缺陷预测模型以及影响数据质量的因素;阐述了语义结构度量指标;分析列举了当前用于软件缺陷预测的评价指标;结合预测粒度、传统度量指标、语义结构度量指标、跨项目软件缺陷预测对多元度量指标软件缺陷预测未来的研究趋势进行了展望。     

Cultivation and engineering of a software metrics program

Abstract. This paper reports from a case study of an organization that implements a software metrics program to measure the effects of its improvement efforts. The program measures key indicators of all completed projects and summarizes progress information in a quarterly management report. The implementation turns out to be long and complex, as the organization is confronted with dilemmas based on contradictory demands and value conflicts. The process is interpreted as a combination of a rational engineering process in which a metrics program is constructed and put into use, and an evolutionary cultivation process in which basic values of the software organization are confronted and transformed. The analysis exemplifies the difficulties and challenges that software organizations face when bringing known principles for software metrics programs into practical use. The article discusses the insights gained from the case in six lessons that may be used by Software Process Improvement managers in implementing a successful metrics program.     

Comparing distributed and face-to-face meetings for software architecture evaluation: A controlled experiment

Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation meetings. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of one of the key activities, developing scenario profiles, of the proposed groupware-supported process of evaluating software architecture. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenario profiles developed by distributed teams using a groupware tool were significantly better than the quality of scenario profiles developed by face-to-face teams (p < 0.001). However, questionnaires indicated that most participants preferred the face-to-face arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings for developing scenario profiles are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.

Empirical studies to assess the understandability of data warehouse schemas using structural metrics

Data warehouses are powerful tools for making better and faster decisions in organizations where information is an asset of primary importance. Due to the complexity of data warehouses, metrics and procedures are required to continuously assure their quality. This article describes an empirical study and a replication aimed at investigating the use of structural metrics as indicators of the understandability, and by extension, the cognitive complexity of data warehouse schemas. More specifically, a four-step analysis is conducted: (1) check if individually and collectively, the considered metrics can be correlated with schema understandability using classical statistical techniques, (2) evaluate whether understandability can be predicted by case similarity using the case-based reasoning technique, (3) determine, for each level of understandability, the subsets of metrics that are important by means of a classification technique, and assess, by means of a probabilistic technique, the degree of participation of each metric in the understandability prediction. The results obtained show that although a linear model is a good approximation of the relation between structure and understandability, the associated coefficients are not significant enough. Additionally, classification analyses reveal respectively that prediction can be achieved by considering structure similarity, that extracted classification rules can be used to estimate the magnitude of understandability, and that some metrics such as the number of fact tables have more impact than others.

Lightweight Software Metrics: The P10 Framework

Software metrics programs must provide value. Metrics need attachment to what's important about a software project. A thorough and comprehensive way to help ensure that connectedness is the Goal Question Metric. GQM proceeds in an orderly fashion to consider goals and specific questions so that when the metrics are defined they relate directly to project goals. The P10 software metrics framework better reflects the contemporary software development environment. P10 is a simple way to help ensure that the software project metrics you use will provide answers to the important questions you face in creating on-time, high-quality software     

Applying empirical software engineering to software architecture: challenges and lessons learned

In the last 15 years, software architecture has emerged as an important software engineering field for managing the development and maintenance of large, software-intensive systems. Software architecture community has developed numerous methods, techniques, and tools to support the architecture process (analysis, design, and review). Historically, most advances in software architecture have been driven by talented people and industrial experience, but there is now a growing need to systematically gather empirical evidence about the advantages or otherwise of tools and methods rather than just rely on promotional anecdotes or rhetoric. The aim of this paper is to promote and facilitate the application of the empirical paradigm to software architecture. To this end, we describe the challenges and lessons learned when assessing software architecture research that used controlled experiments, replications, expert opinion, systematic literature reviews, observational studies, and surveys. Our research will support the emergence of a body of knowledge consisting of the more widely-accepted and well-formed software architecture theories.     

Identifying and qualifying reusable software components

Identification and qualification of reusable software based on software models and metrics is explored. Software metrics provide a way to automate the extraction of reusable software components from existing systems, reducing the amount of code that experts must analyze. Also, models and metrics permit feedback and improvement to make the extraction process fit a variety of environments. Some case studies are described to validate the experimental approach. They deal with only the identification phase and use a very simple model of a reusable code component, but the results show that automated techniques can reduce the amount of code that a domain expert needs to evaluate to identify reusable parts     

软件质量的度量问题

度量的目的是为软件产品提供量化的测量结果，以避免主观论断的差错。软件度量是软件维护的基础。本文对软件度量的定义、度量的过程、面向对象软件的度量、度量与维护的关系等进行了讨论。     

A practical approach to software metrics

Metrics programs that create meaningful change in software practice must start with business goals in mind. Software metrics are quantitative standards of measurement for various aspects of software projects. A well-designed metrics program will support decision making by management and enhance return on the IT investment. There are many aspects of software projects that can be measured, but not all aspects are worth measuring. Starting a new metrics program or improving a current program consists of five steps: identify business goals; select metrics; gather historical data; automate measurement procedures; and use metrics in decision making     

Deriving complexity information from a formal communication protocol specification

Communication software systems have become very large and complex. Recognizing the complexity of such software systems is a key element in their development activities. Software metrics are useful quantitative indicators for assessing and predicting software quality attributes, like complexity. However, most of existing metrics are extracted from source programs at the implementation phase of the software life cycle. They cannot provide early feedback during the specification phase; and subsequently it is difficult and expensive to make changes to the system, if so indicated by the metrics. It is therefore important to be able to measure system complexity at the specification phase. However, most software specifications are written in natural languages from which metrics information is very hard to extract. In this paper, we describe how complexity information can be derived from a formal communication protocol specification written in Estelle so that it is possible to predict the complexity of its implementation and subsequently its development can be better managed. © 1998 John Wiley & Sons, Ltd.     

一种个体软件过程能力度量方法

个体软件过程(PSP)是由卡内基×梅隆大学软件工程研究所的Humphrey领导开发的.它是一种可用于控制、管理和改进个人工作方式的自我持续改进过程.随着工业界对软件过程改进需求的日益增长,PSP成为了软件组织为达成完全(从宏观到微观)量化过程管理研究中的一个热点课题.软件过程研究表明,高水平的个体软件过程能力是软件项目成功的关键,如何进行有效的个体软件过程能力度量是PSP中的一个核心问题.现有方法不能同时有效处理个体软件过程能力度量中的可变规模收益、多变量输入/输出以及决策者偏好问题.提出了一种综合了数据包络分析(DEA)和层次分析法(AHP)的个体软件过程能力评价方法——PSPADA,介绍了PSPADA的个体软件过程能力评价模型和核心算法(集成决策者偏好和估计规模收益).实验结果显示,PSPADA能够在考虑决策者偏好的同时,有效地进行多指标、规模收益可变的量化评估.     

Building measure-based prediction models for UML class diagram maintainability

The usefulness of measures for the analysis and design of object oriented (OO) software is increasingly being recognized in the field of software engineering research. In particular, recognition of the need for early indicators of external quality attributes is increasing. We investigate through experimentation whether a collection of UML class diagram measures could be good predictors of two main subcharacteristics of the maintainability of class diagrams: understandability and modifiability. Results obtained from a controlled experiment and a replica support the idea that useful prediction models for class diagrams understandability and modifiability can be built on the basis of early measures, in particular, measures that capture structural complexity through associations and generalizations. Moreover, these measures seem to be correlated with the subjective perception of the subjects about the complexity of the diagrams. This fact shows, to some extent, that the objective measures capture the same aspects as the subjective ones. However, despite our encouraging findings, further empirical studies, especially using data taken from real projects performed in industrial settings, are needed. Such further study will yield a comprehensive body of knowledge and experience about building prediction models for understandability and modifiability.

Replicating studies on cross- vs single-company effort models using the ISBSG Database

In 2001 the ISBSG database was used by Jeffery et al. (Using public domain metrics to estimate software development effort. Proceedings Metrics’01, London, pp 16–27, 2001; S1) to compare the effort prediction accuracy between cross- and single-company effort models. Given that more than 2,000 projects were later volunteered to this database, in 2005 Mendes et al. (A replicated comparison of cross-company and within-company effort estimation models using the ISBSG Database, in Proceedings of Metrics’05, Como, 2005; S2) replicated S1 but obtained different results. The difference in results could have occurred due to legitimate differences in data set patterns; however, they could also have occurred due to differences in experimental procedure given that S2 was unable to employ exactly the same experimental procedure used in S1 because S1’s procedure was not fully documented. Recently, we applied S2’s experimental procedure to the ISBSG database version used in S1 (release 6) to assess if differences in experimental procedure would have contributed towards different results (Lokan and Mendes, Cross-company and single-company effort models using the ISBSG Database: a further replicated study, Proceedings of the ISESE’06, pp 75–84, 2006; S3). Our results corroborated those from S1, suggesting that differences in the results obtained by S2 were likely caused by legitimate differences in data set patterns. We have since been able to reconstruct the experimental procedure of S1 and therefore in this paper we present both S3 and also another study (S4), which applied the experimental procedure of S1 to the data set used in S2. By applying the experimental procedure of S2 to the data set used in S1 (study S3), and the experimental procedure of S1 to the data set used in S2 (study S4), we investigate the effect of all the variations between S1 and S2. Our results for S4 support those of S3, suggesting that differences in data preparation and analysis procedures did not affect the outcome of the analysis. Thus, the different results of S1 and S2 are very likely due to fundamental differences in the data sets.

Evaluating multifactor risks under conceptual uncertainty

A tool is proposed to evaluate multifactor risks during the operation of an innovation system of technological forecast. A modified BOCR method of analytic hierarchy process (AHP) is developed. It allows: integrating the situation and force majeur risk evaluation into an overall structure of decision making with the help of AHP along with evaluating benefits, costs, and opportunities for each alternative; processing expert judgments in the form of fuzzy preference relations; taking into account a time parameter, when decision factors and alternatives may be corrected or fundamentally changed during some period. Indices of the risk of subjective judgment (information risk) evaluation are developed for given point, interval, and fuzzy expert judgments and probability distribution of expert judgments. Translated from Kibernetika i Sistemnyi Analiz, No. 2, pp. 72–82, March–April 2009.     

应用软件质量工程中过程度量的研究

过程度量是软件开发过程中实施软件质量保证(SQA)的一个重要课题。文章对过程度量的概念进行了一些介绍,讨论了在开发应用软件过程中常用的几种度量,可供软件开发部门在实践SQA时参考。     

软件过程度量的过程模型及其应用研究

软件度量是软件工程中最活跃的一个研究领域。本文给出了一个软件过程度量的过程模型,定义了实施软件过程度量的角色、内容、主要活动及相关的支持环境,重点阐述软件过程度量的数据采集、验证和分析活动的目标、任务和方法。通过一个软件项目的度量过程实例,研究并确认该模型在过程评估和改进方面的作用和意义。本文的研究对于规范和改善组织的软件过程度量,提高组织的过程能力成熟度,具有一定的指导意义和广泛的应用价值。