An objective approach to exploring skill differences in strategies of computer program comprehension

An experiment was conducted to examine skill differences in the control strategy for computer program comprehension. A computer program along with its hierarchy of program plans was provided to 10 intermediate and 10 novice computer programmers. Each program plan is known as a program segment to the subjects. A random list of plan goals was also provided to the subjects. The subjects were asked to match each program segment with its goal while they were comprehending the program. Several measures of the subjects' performance and control strategy were collected and analysed. The results indicated the use of an overall top-down strategy by both intermediates and novices for program comprehension. Novices' control strategies involved more opportunistic elements than experts' in the overall top-down process of program comprehension. Those differences in the control strategy between intermediates and novices result in better performance in intermediates than novices.     

An objective approach to exploring skill differences in strategies of computer program comprehension

An experiment was conducted to examine skill differences in the control strategy for computer program comprehension. A computer program along with its hierarchy of program plans was provided to 10 intermediate and 10 novice computer programmers. Each program plan is known as a program segment to the subjects. A random list of plan goals was also provided to the subjects. The subjects were asked to match each program segment with its goal while they were comprehending the program. Several measures of the subjects' performance and control strategy were collected and analysed. The results indicated the use of an overall top-down strategy by both intermediates and novices for program comprehension. Novices' control strategies involved more opportunistic elements than experts' in the overall top-down process of program comprehension. Those differences in the control strategy between intermediates and novices result in better performance in intermediates than novices.     

Object-Oriented Program Comprehension: Effect of Expertise, Task and Phase

The goal of our study is to evaluate the effect on program comprehension of three factors that have not previously been studied in a single experiment. These factors are programmer expertise (expert versus novice), programming task (documentation versus reuse), and the development of understanding over time (phase 1 versus phase 2). This study is carried out in the context of the mental model approach to comprehension based on van Dijk and Kintsch's model [(1983) Strategies of Discourse Comprehension. New York: Academic]. One key aspect of this model is the distinction between two kinds of representation the reader might construct from a text: (1) the textbase, which refers to what is said in the text and how it is said, and (2) the situation model, which represents the situation referred to by the text. We have evaluated the effect of the three factors mentioned above on the development of both the textbase (or program model) and the situation model in object-oriented program comprehension. We found a four-way interaction of expertise, phase, task and type of model. For the documentation group we found that experts and novices differ in the elaboration of their situation model but not their program model. There was no interaction of expertise with phase and type of model in the documentation group. For the reuse group, there was a three-way interaction between phase, expertise and type of model. For the novice reuse group, the effect of the phase was to increase the construction of the situation model but not the program model. With respect to the task, our results show that novices do not spontaneously construct a strong situation model but are able to do so if the task demands it.     

An empirical investigation into the design of auditory cues to enhance computer program comprehension

Decades of research have led to notable improvements in the representations used to aid human comprehension of computer programs. Much of this research has focused on visual representations, which leaves open the question of how best to design auditory representations of computer programs. While this question has particular relevance for visually impaired programmers, sighted programmers might also benefit from enhanced auditory representations of their programs. In order to investigate this question empirically, first, we introduce artifact encoding, a novel approach to rigorously measuring the comprehensibility of auditory representations of computer programs. Using this approach as a foundation, we present an experimental study that compared the comprehensibility of two alternative auditory program representations: one with lexical scoping cues that convey the nesting level of program statements, and another without such scoping cues. The results of our first experiment validate both artifact encoding and the scoping cues we used. To see whether auditory cues validated through our paradigm can aid program comprehension in a realistic task scenario, we experimentally compared programmers' ability to debug programs using three alternative environments: (1) an auditory execution environment with our empirically derived auditory cues; (2) an auditory execution environment with the current state-of-the-art auditory cues generated by a screen reader running on top of Microsoft Visual Studio; and (3) a visual version of the execution environment. The results of our second experiment showed that our comprehensible auditory cues are significantly better than the state-of-the-art, affording human performance approaching the effectiveness of visual representations within the statistical margin of error. This research contributes a novel methodology and foundational empirical data that can guide the design of effective auditory representations of computer programs.     

Teaching self‐regulation strategies via an intelligent tutoring system (TuinLECweb): Effects for low‐skilled comprehenders

The use of documents to perform tasks is a continuous task demand in the current knowledge‐based society that involves making a series of decisions to self‐regulate the use of text information. Low‐skilled comprehenders have serious problems monitoring and self‐regulating their decisions in these task‐oriented reading situations, which has a negative impact on performance and merits instructional interventions in school settings. Intelligent tutoring systems have emerged as effective tools to teach strategic skills. Herein, we present TuinLECweb, an intelligent tutoring system that teaches monitoring and self‐regulation strategies to young adolescents. Study 1 explores whether high‐skilled and low‐skilled comprehenders respond to the intervention with TuinLECweb differently in terms of task‐oriented reading performance and monitoring accuracy of nonsearch decisions. The results show that the intervention with TuinLECweb primarily benefits low‐skilled comprehenders. Study 2 compares the efficacy of TuinLECweb over a classroom intervention for low‐skilled comprehenders. The results reveal that TuinLECweb training improves monitoring accuracy of nonsearch decisions over and above classroom instruction. Both studies also reveal that low‐skilled comprehenders use the strategies learned beyond the time of training. Our results provide arguments in favour of using computer‐assisted programs in strategy instruction.     

Are visual programming languages better? The role of imagery in program comprehension

This paper presents one experiment to explain why and under which circumstances visual programming languages would be easier to understand than textual programming languages. Towards this goal we bring together research from psychology of programming and image processing. According to current theories of imagery processing imagery facilitates a quicker access to semantic information. Thus, visual programming languages should allow for quicker construction of a mental representation based on data flow relationships of a program than procedural languages. To test this hypothesis the mental models of C and spreadsheet programmers were assessed in different program comprehension situations. The results showed that spreadsheet programmers developed data flow based mental representations in all situations, while C programmers seemed to access first a control flow and then data flow based mental representations. These results could help to expand theories of mental models from psychology of programming to account for the effect of imagery.     

VinEd — Asystem for programmanipulation through user-definable simultaneous views

Programmers’ mental representations of programs do not obey the order and immediate content of program texts. Multiple visible representations, or views, of programs help programmers to construct mental representations needed in, e.g., maintenance tasks. We have implemented VinEd, a language-independent editor framework that supports an unlimited number of user-definable and editable views. Views are defined by transformation programs that construct views from the original representation. VinEd extends the notion of views to any activity that can be based on the original program, e.g., program compilation. Thus VinEd can be extended to include a complete set of programming tools.     

Empirical studies on programming language stimuli

Comprehending and debugging computer programs are inherently difficult tasks. The current approach to building program execution and debugging environments is to use exclusively visual stimuli on programming languages whose syntax and semantics has often been designed without empirical guidance. We present an alternative: Sodbeans, an open-source integrated development environment designed to output carefully chosen spoken auditory cues to supplement empirically evaluated visual stimuli. Originally designed for the blind, earlier work suggested that Sodbeans may benefit sighted programmers as well. We evaluate Sodbeans in two experiments. First, we report on a formal debugging experiment comparing (1) a visual debugger, (2) an auditory debugger, and (3) a multimedia debugger, which includes both visual and auditory stimuli. The results from this study indicate that while auditory debuggers on their own are significantly less effective for sighted users when compared with visual and multimedia debuggers, multimedia debuggers might benefit sighted programmers under certain circumstances. Specifically, we found that while multimedia debuggers do not provide instant usability, once programmers have some practice, their performance in answering comprehension questions improves. Second, we created and evaluated a pilot survey analyzing individual elements in a custom programming language (called HOP) to garner empirical metrics on their comprehensibility. Results showed that some of the most widely used syntax and semantics choices in commercial programming languages are extraordinarily unintuitive for novices. For example, at an aggregate level, the word for , as in a for loop, was rated reliably worse than repeat by more than 673% by novices. After completing our studies, we implemented the HOP programming language and integrated it into Sodbeans.     

Assisting comprehension of animation programs through interactive code visualization

Visual languages have been widely used to help people create animation programs. However, current programming environments lack features supporting efficient code exploration and program comprehension, particularly for understanding relationships among parts of animation programs. In this paper, we present novel interactive visualizations aimed at helping people to understand animation programs. We conducted an empirical study to evaluate the impact of these visualizations on programmer comprehension of the code, showing that our approach enabled programmers to comprehend more information with less effort and in less time. This result is potentially significant because it demonstrates an approach for helping users to explore and understand animation code. We anticipate that this approach could be applied in a wide variety of animation programming tools, which could ease common animation programming tasks that require understanding code.     

Cognitive processes in software fault detection: A review and synthesis

The authors discuss empirical research about the cognitive structures and strategies used by programmers during fault location. Empirical evidence indicates the cognitive processes involved in fault detection consist of a comprehension process and a fault location process. The two processes are distinct and separate. The comprehension process is extremely important and was found to be superior in experts due to the semantic encoding they utilize. The semantic representations used by experts consist of abstract hierarchies based on functional meaning. Fault location is less important and usually takes the form of hand simulation or causal reasoning. The fault locating strategies used by experts and novices were similar. The better debugging performance by experts is due to their superior abilities at comprehension. Research indicates that the semantic organizations used by experts can be successfully taught to novices and used by them to improve performance.

The authors also examine two methods, slicing and team reviews, which seek to improve the debugging process. Each was found to affect comprehension and fault location differently. A review of slicing research revealed that it is performed during the fault location process, and does not apply to the comprehension process as some believe. Automating slicing was found to be a technique with potential benefits for debugging. The survey of the team dynamics during inspections and other reviews found them to be effective by enhancing the comprehension process, by improving fault location, and by providing more than one chance to catch each error.     

The role of program structure in software maintenance

A number of claims have been made by the developers of program design methodologies, including the claim that the code produced by following the methodologies will be more understandable and more easily maintained than code produced in other ways. However, there has been little empirical research to test these claims. In this study, student and professional programmers were asked to make either simple or complex modifications to programs that had been generated using each of three different program structures. Data on the programmers' modification performance, cognitive representations formed of the programs and subjective reactions to the programs suggested that problem structure (as created by the different methodologies), problem content, complexity of modification, and programmer experience all play a crucial role in determining performance and the representation formed.     

Debugging hygienic macros

Over the past two decades, Scheme macros have evolved into a powerful API for the compiler front end. Like Lisp macros, their predecessors, Scheme macros expand source programs into a small core language; unlike Lisp systems, Scheme macro expanders preserve lexical scoping, and advanced Scheme macro systems handle other important properties such as source location. Using such macros, Scheme programmers now routinely develop the ultimate abstraction: embedded domain-specific programming languages.Unfortunately, a typical Scheme programming environment provides little support for macro development. This lack makes it difficult for programmers to debug their macros and for novices to study the behavior of macros. In response, we have developed a stepping debugger specialized to the concerns of macro expansion. This debugger presents the macro expansion process as a linear rewriting sequence of annotated terms; it graphically illustrates the binding structure of the program as expansion reveals it; and it adapts to the programmer’s level of abstraction, hiding details of syntactic forms that the programmer considers built-in.     

Debunking the nerd stereotype with pair programming

Our studies show that using pair programming as a structure for incorporating collaboration in the classroom helps increase and broaden participation in computing fields and helps debunk the myth that programmers work alone all the time. It's also a way for students to get a better view of and feel more confident in their preparation for working in the real world. The face of the IT workforce is changing. As the millennial generation makes its way into the working world, the archetype of the nerd as the introverted, obsessed computer programmer must share equal space with the chatty, social software engineer.     

Effects of computer-assisted comprehension training in less skilled comprehenders in second grade: A one-year follow-up study

This study examines the effects of a new CAI program designed to remediate text comprehension difficulties in less skilled comprehenders at the beginning of learning to read. In a randomized control trial design, two groups of second grade children experiencing comprehension difficulties were selected and trained using two CAI programs. One of these was designed to assist comprehension processes (experimental group) whereas the other encouraged word decoding (control group). Four tasks were administered using a classical pre-test/training/post-test design: listening comprehension, reading comprehension, vocabulary and comprehension monitoring. In addition, two post-test sessions were conducted: one just after the training and one 11 months later. The experimental group exhibited a greater level of progress in both listening and reading comprehension, with notable lasting effects. In contrast, the results relating to the vocabulary and comprehension monitoring tasks were more mixed. The results are discussed in the light of the possible use of CAI when training higher-level skills such as text comprehension.     

Does syntax highlighting help programming novices?

Program comprehension is an important skill for programmers – extending and debugging existing source code is part of the daily routine. Syntax highlighting is one of the most common tools used to support developers in understanding algorithms. However, most research in this area originates from a time when programmers used a completely different tool chain. We examined the influence of syntax highlighting on novices’ ability to comprehend source code. Additional analyses cover the influence of task type and programming experience on the code comprehension ability itself and its relation to syntax highlighting. We conducted a controlled experiment with 390 undergraduate students in an introductory Java programming course. We measured the correctness with which they solved small coding tasks. Each test subject received some tasks with syntax highlighting and some without. The data provided no evidence that syntax highlighting improves novices’ ability to comprehend source code. There are very few similar experiments and it is unclear as of yet which factors impact the effectiveness of syntax highlighting. One major limitation may be the types of tasks chosen for this experiment. The results suggest that syntax highlighting squanders a feedback channel from the IDE to the programmer that can be used more effectively.     

Principles of Program Design Induced from Experience with Small Public Programs

The art of programming is taught, learned, and often practiced as if programs are disposable, personal objects owned, solely by the programmer. This paper uses examples to illustrate why real software is neither personal nor disposable; it shows how even simple programs are shared by others. From the examples, the paper extracts four principles for program development. Finally, it draws conclusions about programming practices and the education of programmers.     

Recognizing authors: an examination of the consistent programmer hypothesis

Software developers have individual styles of programming. This paper empirically examines the validity of the consistent programmer hypothesis: that a facet or set of facets exist that can be used to recognize the author of a given program based on programming style. The paper further postulates that the programming style means that different test strategies work better for some programmers (or programming styles) than for others. For example, all‐edges adequate tests may detect faults for programs written by Programmer A better than for those written by Programmer B. This has several useful applications: to help detect plagiarism/copyright violation of source code, to help improve the practical application of software testing, and to help pursue specific rogue programmers of malicious code and source code viruses. This paper investigates this concept by experimentally examining whether particular facets of the program can be used to identify programmers and whether testing strategies can be reasonably associated with specific programmers. Copyright © 2009 John Wiley & Sons, Ltd.     

Ability-training-oriented automated assessment in introductory programming course

Learning to program is a difficult process for novice programmers. AutoLEP, an automated learning and assessment system, was developed by us, to aid novice programmers to obtain programming skills. AutoLEP is ability-training-oriented. It adopts a novel assessment mechanism, which combines static analysis with dynamic testing to analyze student programs. It not only helps students to sufficiently test the programs, but also evaluates whether the programs meet the specification or not. AutoLEP encourages students to find and work through bugs, by providing automatic syntactic and structural checking, and immediate detailed feedback. This can improve students’ learning experience in programming and reduce the workload of the teaching staff. AutoLEP has been used in the C programming course at Harbin Institute of Technology and many other universities since 2004. The feedback on AutoLEP and its incorporation into the introductory programming course has been positive, both by students and teaching staff.