A deep dive into continuous integration monitoring practices

Abstract

One of the main activities in software development is monitoring, which plays a vital role in verifying the proper implementation of processes, identifying errors, and discovering opportunities for improvement. Continuous Integration (CI) covers a set of widely adopted practices that enhance software development. However, there are indications that developers may not adequately monitor all CI practices. In this thesis, we delve deep into the ocean of monitoring CI practices. Our goal is to discover how this monitoring is conducted, demonstrate the advantages of monitoring CI practices, and highlight the challenges that must be overcome. In our first study, we analyze the impact of specific CI practices on the volume of Pull Requests and bug-related Issues. Our findings revealed a positive correlation between CI practices and the increase in the number of merged pull requests. We also identified a significant correlation between the number of bug-related issues and the number of bugs. Additionally, our findings suggest that higher values of CI practices can indicate better quality in the development process. In our second study, we investigate the importance attributed to these practices by developers and the support for monitoring CI practices provided by the most popular CI services. We find that developers usually monitor only code coverage and basic build metadata (e.g. build duration and build status). Developers expressed interest in monitoring CI practices if provided the opportunity. Furthermore, we have identified that several leading CI services still have initial support in monitoring CI practices. Finally, we evaluate monitoring in real-world scenarios, conducting a case study on three projects from three different organizations, where we investigate more deeply the developers’ interest in monitoring CI practices, their benefits, challenges, and the evolution of CI practices over a two-month period. The case study revealed that monitoring CI practices provides several benefits to the project and is cost-effective. Participants demonstrated a strong desire to integrate CI monitoring dashboards with popular CI services.