40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study
Marvin Wyrich, Justus Bogner, Stefan Wagner
The relevance of code comprehension in a developer's daily work was
recognized more than 40 years ago. Over the years, several studies have
gathered evidence that developers do indeed invest a considerable amount of
their daily work in code comprehension. Consequently, many studies were
conducted to find out how developers could be supported during code
comprehension and which code characteristics contribute to better
comprehension. Today, such experiments are more common than ever. While this is
great for advancing the field, the number of publications makes it difficult to
keep an overview. Additionally, designing rigorous experiments with human
participants is a challenging task, and the multitude of design decisions and
options can make it difficult for researchers to select a suitable design.
We therefore conducted a systematic mapping study of 95 source code
comprehension experiments published between 1979 and 2019. By systematically
structuring the design characteristics of code comprehension studies, we
provide a basis for subsequent discussion of the huge diversity of design
options in the face of a lack of basic research on their consequences and
comparability. We describe what topics have been studied, as well as how these
studies have been designed, conducted, and reported. Frequently chosen design
options and deficiencies are pointed out. We conclude with five concrete action
items that we as a research community should address moving forward to improve
publications of code comprehension experiments.