The impact of adopting continuous integration on the delivery time of merged pull requests: an empirical study

Abstract

Continuous Integration (CI) is a software development practice that leads developers to integrate their work more frequently. Software projects have broadly adopted CI to ship new releases more frequently and to improve code integration. The adoption of CI is usually motivated by the allure of delivering new software content more quickly and frequently. However, there is little empirical evidence to support such claims. Over the last years, many available software projects from social coding environments such as GitHub have adopted the CI practice using CI facilities that are integrated in these environments (e.g., Travis-CI). In this dissertation, we empirically investigate the impact of adopting CI on the time-to-delivery of pull requests (PRs), through the analysis of 167,037 PRs of 90 GitHub projects that are implemented in 5 different programming languages. On analyzing the percentage of merged PRs per project that missed at least one release prior being delivered to the end users, the results show that before adopting CI, a median of 13.8% of merged PRs are postponed by at least one release, while after adopting CI, a median of 24% of merged PRs have their delivery postponed to future releases. Contrary to what one might speculate, we find that PRs tend to wait longer to be delivered after the adoption of CI in the majority (53%) of the studied projects. The large increase of PR submissions after CI is a key reason as to why these projects deliver PRs more slowly after adopting CI. 77.8% of the projects increase the rate of PR submissions after adopting CI. To investigate the factors that are related to the time-to-delivery of merged PRs, we train linear and logistic regression models, which obtain sound median R-squares of 0.72-0.74, and good median AUC values of 0.85-0.90. A deeper analysis of our models suggests that, before and after the adoption of CI, the intensity of code contributions to a release may increase the delivery time due to a higher integration-load (in terms of integrated commits) of the development team. Finally, we are able to accurately identify merged pull requests that have a prolonged delivery time. Our regression models obtained median AUC values of 0.92 to 0.97.