An Extended Replication of an Experiment for Assessing Methods for Software Requirements Inspections

We have performed an extended replication of the Porter-Votta-Basili experiment comparing the Scenario method and the Checklist method for inspecting requirements specifications using identical instruments. The experiment has been conducted in our educational context represented by a more general definition of a defect compared to the original defect list. Our study involving 24 undergraduate students manipulated three independent variables: detection method, requirements specification, and the order of the inspections. The dependent variable measured is the defect detection rate. We found the requirements specification inspected and not the detection method to be the most probable explanation for the variance in defect detection rate. This suggests that it is important to gather knowledge of how a requirements specification can convey an understandable view of the product and to adapt inspection methods accordingly. Contrary to the original experiment, we can not significantly support the superiority of the Scenario method. This is in accordance with a replication conducted by Fusaro, Lanubile and Visaggio, and might be explained by the lack of individual defect detection skill of our less experienced subjects.     

Further Experiences with Scenarios and Checklists

Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. This replication is broadly supportive of the results from the original experiment, namely, that the Scenario approach is superior to the Checklist approach; and that the meeting component of a software inspection is not an effective defect detection mechanism. This experiment also tentatively proposes additional relationships between general academic performance and individual inspection performance; and between meeting loss and group inspection performance.     

The role of replications in Empirical Software Engineering

Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.

Understanding replication of experiments in software engineering: A classification

ContextReplication plays an important role in experimental disciplines. There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment?ObjectiveTo improve our understanding of SE experiment replication, in this work we propose a classification which is intend to provide experimenters with guidance about what types of replication they can perform.MethodThe research approach followed is structured according to the following activities: (1) a literature review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE.ResultsWe propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment.ConclusionThe aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.     

Collusion-Proof Result Inference in Crowdsourcing

在传统的 crowdsourcing,工人们被期望提供独立答案给任务以便保证答案的差异。然而,最近的研究证明人群不是许多独立工人,但是相反工人们与对方一起交流并且协作。与小努力追求更多的报酬,一些工人可以共谋勾结提供重复答案,它将损坏聚集的结果的质量。尽管如此,就在 crowdsourcing 的结果推理上的串通的否定影响而言有很少努力。在这份报纸,我们特殊在公共平台为一般 crowdsourcing 任务担心防串通的结果推理问题。到那个目的,我们设计一个度量标准,工人表演变化率,由在移开重复答案前后计算吝啬的工人表演的差别识别共谋勾结的答案。然后,我们把串通察觉结果合并到存在结果推理方法甚至与串通行为的出现保证聚集的结果的质量。与真实世界、合成的数据集,我们进行了我们的途径的评估的一个广泛的集合。试验性的结果与最先进的方法比较表明我们的途径的优势。     

An experiment to assess the cost-benefits of code inspections inlarge scale software development

We conducted a long term experiment to compare the costs and benefits of several different software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development effort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. The article has several goals: (1) to describe the experiment's design and show how we used simulation techniques to optimize it; (2) to present our results and discuss their implications for both software practitioners and researchers; and (3) to discuss several new questions raised by our findings. For each inspection, we randomly assigned three independent variables: (1) the number of reviewers on each inspection team (1, 2, or 4); (2) the number of teams inspecting the code unit (1 or 2); and (3) the requirement that defects be repaired between the first and second team's inspections. The reviewers for each inspection were randomly selected without replacement from a pool of 11 experienced software developers. The dependent variables for each inspection included inspection interval (elapsed time), total effort, and the defect detection rate. Our results showed that these treatments did not significantly influence the defect detection effectiveness, but that certain combinations of changes dramatically increased the inspection interval     

A process for managing interaction between experimenters to get useful similar replications

ContextA replication is the repetition of an experiment. Several efforts have been made to adopt replication as a common practice in software engineering. There are different types of replications, depending on their purpose. Similar replications keep the experimental conditions as alike as possible to the original ones. External similar replications, where the replicating experimenters are not the same people as the original experimenters, have been a stumbling block. Several attempts at combining the results of replications have resulted in failure. Software engineering does not appear to be well suited to such replications, because it works with complex experimentally immature contexts. Software engineering settings have a large number of variables, and the role that many of them play is unknown. A successful (or useful) similar replication helps to better understand the phenomenon under study by verifying results and/or identifying contextual variables that could influence (or not) the results, through the combination of experimental results.ObjectiveTo be able to get successful similar replications, there needs to be interaction between original and replicating experimenters. In this paper, we propose an interaction process for achieving successful similar replications.MethodThis process consists of: an adaptation meeting, where experimenters tailor the experiment to the new setting; querying, to settle occasional inquiries while the experiment is being run; and a combination meeting, where experimenters meet to discuss the combination of replication outcomes with previous results. To check its effectiveness, the process has been tested on three different replications of the same experiment.ResultsThe proposed interaction process has helped to identify new contextual variables that could potentially influence (or not) the experimental results in the three replications run. Additionally, the interaction process has helped to uncover certain problems and deviations that occurred during some of the replications that we would have not been aware of otherwise.ConclusionsThere are signs that suggest that it is possible to get successful similar replications in software engineering experimentation, when there is appropriate interaction among experimenters.     

新的基于图像显著性区域特征的织物疵点检测算法

鉴于织物疵点类型的多样性和传统人工检测方法的低效率,为更有效地检测织物疵点,提出一种新的基于图像显著性特征的织物疵点检测方法——SGE。将原织物图分成相同两份:一份利用改进的基于频率的显著性区域(FSR)方法提取区域特征,粗定位疵点位置。另一份先Gabor滤波,取Gabor模图为输出特征;再利用基于像素的显著性区域(PSR)方法进行区域特征提取,细定位疵点位置;然后利用最大熵分别对粗细定位的疵点图进行分割,再融合;最后描绘轮廓,计算周长和面积,去除孤立点,得最终检测结果。采用OpenCV算法库,选取了4种具有代表的织物疵点图片进行验证。实验结果表明,这种粗细定位疵点的方法能够获得较好的检测结果,无需事先学习,能够满足实时性要求。     

Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects

Software requirements specifications (SRS) are often validated manually. One such process is inspection, in which several reviewers independently analyze all or part of the specification and search for faults. These faults are then collected at a meeting of the reviewers and author(s).Usually, reviewers use Ad Hoc or Checklist methods to uncover faults. These methods force all reviewers to rely on nonsystematic techniques to search for a wide variety of faults. We hypothesize that a Scenario-based method, in which each reviewer uses different, systematic techniques to search for different, specific classes of faults, will have a significantly higher success rate.In previous work we evaluated this hypothesis using 48 graduate students in computer science as subjects.We now have replicated this experiment using 18 professional developers from Lucent Technologies as subjects. Our goals were to (1) extend the external credibility of our results by studying professional developers, and to (2) compare the performances of professionals with that of the graduate students to better understand how generalizable the results of the less expensive student experiments were.For each inspection we performed four measurements: (1) individual fault detection rate, (2) team fault detection rate, (3) percentage of faults first identified at the collection meeting (meeting gain rate), and (4) percentage of faults first identified by an individual, but never reported at the collection meeting (meeting loss rate).For both the professionals and the students the experimental results are that (1) the Scenario method had a higher fault detection rate than either Ad Hoc or Checklist methods, (2) Checklist reviewers were no more effective than Ad Hoc reviewers, (3) Collection meetings produced no net improvement in the fault, and detection rate—meeting gains were offset by meeting losses,Finally, although specific measures differed between the professional and student populations, the outcomes of almost all statistical tests were identical. This suggests that the graduate students provided an adequate model of the professional population and that the much greater expense of conducting studies with professionals may not always be required.     

Effectiveness for detecting faults within and outside the scope of testing techniques: an independent replication

The verification and validation activity plays a fundamental role in improving software quality. Determining which the most effective techniques for carrying out this activity are has been an aspiration of experimental software engineering researchers for years. This paper reports a controlled experiment evaluating the effectiveness of two unit testing techniques (the functional testing technique known as equivalence partitioning (EP) and the control-flow structural testing technique known as branch testing (BT)). This experiment is a literal replication of Juristo et al. (2013). Both experiments serve the purpose of determining whether the effectiveness of BT and EP varies depending on whether or not the faults are visible for the technique (InScope or OutScope, respectively). We have used the materials, design and procedures of the original experiment, but in order to adapt the experiment to the context we have: (1) reduced the number of studied techniques from 3 to 2; (2) assigned subjects to experimental groups by means of stratified randomization to balance the influence of programming experience; (3) localized the experimental materials and (4) adapted the training duration. We ran the replication at the Escuela Politécnica del Ejército Sede Latacunga (ESPEL) as part of a software verification & validation course. The experimental subjects were 23 master’s degree students. EP is more effective than BT at detecting InScope faults. The session/program and group variables are found to have significant effects. BT is more effective than EP at detecting OutScope faults. The session/program and group variables have no effect in this case. The results of the replication and the original experiment are similar with respect to testing techniques. There are some inconsistencies with respect to the group factor. They can be explained by small sample effects. The results for the session/program factor are inconsistent for InScope faults. We believe that these differences are due to a combination of the fatigue effect and a technique x program interaction. Although we were able to reproduce the main effects, the changes to the design of the original experiment make it impossible to identify the causes of the discrepancies for sure. We believe that further replications closely resembling the original experiment should be conducted to improve our understanding of the phenomena under study.     

Assessing multiview framework (MF) comprehensibility and efficiency: A replicated experiment

Goal oriented quality models have become an important means for assessing and improving software quality. In previous papers, the authors have proposed an approach called multiview framework, for guiding quality managers in designing and managing a goal oriented quality model. This approach has been validated through a controlled experiment carried out with university students.In this paper, the authors discuss a replication of the controlled experiment, carried out with 28 university graduates attending a master degree course in an Italian university. Although research hypotheses are the same, context differs. In the replication, experimental subjects were more representative of practitioners, because their master degree course required project work with industrial partners.Using a cross-over experimental design we found that subjects using the multiview framework made significantly fiewer errors (p<0.05, effect size=1.08) and took significantly less time (p<0.51, effect size=1.82) to review the status of a project than when they used a standard GQM approach. This result was consistent with the results of our original experiment.     

A family of experiments to investigate the effects of groupware for software inspection

It is widely accepted that the inspection of software artifacts can find defects early in the development process and gather information on the quality of the evolving product. However, the inspection process is resource-intensive and involves tedious tasks, such as searching, sorting, and checking. Tool support for inspections can help accelerating these tasks and allows inspectors to concentrate on tasks particularly needing human attention. Only few tools are available for inspections. We have thus developed a set of groupware tools for both individual defect detection and inspection meetings to lower the effort of inspections and to increase their efficiency. This paper presents the Groupware-supported Inspection Process (GrIP) and describes tools for inspecting software requirements. As only little empirical work exists that directly compares paper-based and tool-based software inspection, we conducted a family of experiments in an academic environment to empirically investigate the effect of tool support regarding defect detection and inspection meetings. The main results of our family of experiments regarding individual defect detection are promising: The effectiveness of inspectors and teams is comparable to paper-based inspection without tool support; the inspection effort and defect overlap decreases significantly with tool support, while the efficiency of inspection teams increases considerably. Regarding tool support for inspection meetings the main findings of the experiments are that tool support considerably lowers the meeting effort, supports inspectors in identifying false positives, and reduces the number of true defects lost during a meeting. The number of unidentified false positives is still quite high.     

Automatic defect detection of metro tunnel surfaces using a vision-based inspection system

Due to the impact of the surrounding environment changes, train-induced vibration, and human interference, damage to metro tunnel surfaces frequently occurs. Therefore, accidents caused by the tunnel surface damage may happen at any time, since the lack of adequate and efficient maintenance. To our knowledge, effective maintenance heavily depends on the all-round and accurate defect inspection, which is a challenging task, due to the harsh environment (e.g., insufficient illumination, the limited time window for inspection, etc.). To address these problems, we design an automatic Metro Tunnel Surface Inspection System (MTSIS) for the efficient and accurate defect detection, which covers the design of hardware and software parts. For the hardware component, we devise a data collection system to capture tunnel surface images with high resolution at high speed. For the software part, we present a tunnel surface image pre-processing approach and a defect detection method to recognize defects with high accuracy. The image pre-processing approach includes image contrast enhancement and image stitching in a coarse-to-fine manner, which are employed to improve the quality of raw images and to avoid repeating detection for overlapped regions of the captured tunnel images respectively. To achieve automatic tunnel surface defect detection with high precision, we propose a multi-layer feature fusion network, based on the Faster Region-based Convolutional Neural Network (Faster RCNN). Our image pre-processing and the defect detection methods also promising performance in terms of recall and precision, which is demonstrated through a series of practical experimental results. Moreover, our MTSIS has been successfully applied on several metro lines.     

Controlled experiment replication in evaluation of e-learning system’s educational influence

We believe that every effectiveness evaluation should be replicated at least in order to verify the original results and to indicate evaluated e-learning system’s advantages or disadvantages. This paper presents the methodology for conducting controlled experiment replication, as well as, results of a controlled experiment and an internal replication that investigated the effectiveness of intelligent authoring shell eXtended Tutor–Expert System (xTEx-Sys). The initial and the replicated experiment were based on our approach that combines classical two-group experimental design and with factoral design. A trait that distinguishes this approach from others is the existence of arbitrary number of checkpoint-tests to determine the effectiveness in intermediate states. We call it a pre-and-post test control group experimental design with checkpoint-tests. The gained results revealed small or even negative effect sizes, which could be explained by the fact that the xTEx-Sys’s domain knowledge presentation is rather novel for students and therefore difficult to grasp and apply in earlier phases of the experiment. In order to develop and improve the xTEx-Sys, further experiments must be conducted.     

电压频控中抗强干扰软件关联缺陷检测

针对传统电压频控软件缺陷检测技术未考虑软件缺陷分类,存在检测精度低的问题,提出一种电压频控中抗强干扰软件关联缺陷检测技术。对软件关联缺陷检测原理进行分析,采用判别函数对待测软件样本进行识别,引入统计模式识别算法处理软件原始数据,依据关联缺陷概率分配,确定关联缺陷类别,计算缺陷特征值,利用贝叶斯分类器对关联缺陷进行划分,完成抗强干扰软件关联缺陷的分类,从而实现关联缺陷的高精度检测。实验结果表明,该检测技术对软件缺陷进行准确分类,在保证强抗干扰性的前提下,有效提高了检测精度。     

基于视觉传感器的PCB缺陷检测系统的研究与实现

为了实现PCB缺陷的在线自动检测,设计了一种PCB缺陷自动检测系统,该系统主要由机器臂、电气控制系统以及视觉传感器系统等组成。通过可编程控制的图像采集系统获取高质量的原始视觉图像,利用图像处理实现对缺陷目标的自动检测及识别。实验结果验证了该系统检测PCB板缺陷的高效性和实时性。     

Seeking reproducibility: Assessing a multimodal study of the testing effect

Low‐cost devices have widened the use of multimodal data in experiments providing a more complete picture of behavioural effects. However, the accurate collection and combination of multimodal and behavioural data in a manner that enables reproducibility is challenging and often requires researchers to refine their approaches. This paper presents a direct replication of a multimodal wordlist experiment. Specifically, we use a low‐cost Emotiv EPOC® to acquire electrophysiological measures of brain activity to investigate whether retrieval during learning facilitates the encoding of subsequent learning as measured by performance on recall tests and reflected by changes in alpha wave oscillations. Behavioural results of the wordlist experiment were replicated, but physiological results were not. We conclude the paper by highlighting the challenges faced in terms of replicating the previous work and in attempting to facilitate the reproducibility of our own experiment.     

扫描激光线源技术检测金属表面缺陷位置实验研究

利用扫描激光线源(Scaing Laser Line Source,SLLS)技术,提出一种金属表面缺陷位置检测方法,并搭建实验平台,实验研究该检测方法对金属表面缺陷位置的检测效果。方法中利用扫描激光线源技术对样品表面进行扫查,在样品内部激发超声信号,采用双波混合干涉方法实现对激发信号的探测,根据激发的瑞利波和反射回波与缺陷位置的关系,确定激发点和探测点到缺陷的距离,从而确定缺陷的位置。实验研究中分别以线源位置和探测点位置为基点确定缺陷位置,结果表明以探测点位置确定缺陷位置时,定位相对误差为0.23%。