Formalização de experimentos controlados em engenharia de software

Abstract

The conduction of empirical studies is very important to gather scientific evidences of new software technologies. Over the last years, a hundred of controlled experiments have been conducted in the software engineering area. A controlled experiment is a technique that allows researchers to test a research hypothesis and the causal effect analysis among the variables involved in the study environment. However, the planning, execution, analysis and packaging of a controlled experiment are considered work intensive, time consuming and error-prone activities. A few existing supporting tools can help the accomplishment of many of these activities but they still have many limitations and improvement needs. In this context, this thesis proposes: (i) to investigate existing approaches and environments to support the formalization and conduction of SE controlled experiments by identifying their limitations and benefits; (ii) to propose a domain-specific language (DSL) to formalize the specification of controlled experiments; and (iii) to develop a model-driven approach that can use the experiment specification in the DSL to generate customized workflows to support the execution of controlled experiments. This work is evaluated through the conduction of: (i) a feasibility study of the modeldriven approach through the modeling of a real experiment and the generation of workflows according to its experimental design; (ii) an empirical study that assesses the expressivity and completeness of the domain-specific language through the modeling of 16 existing experiments; (iii) a controlled experiment that investigates the DSL comprehensibility by the experimenters; and (iv) a controlled experiment that investigates the language usability through the specification of experiments. The studies results bring evidences of the approach feasibility, and the expressiveness and completeness of the DSL. In addition, our controlled experiments results show that: (i) the experimental plan comprehension when written in the proposed DSL is easier to understand and faster to specify when compared to the experiment specification described in scientific papers; and (ii) the experimenters’ perception was positive when using the DSL.