Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions

The slides for the paper "Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions" at FSE24

Transcript

1. Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions FSE 24, Brazil Tao Xiao Hideaki Hata Christoph Treude Kenichi Matsumoto
2. GitHub Copilot 2 Copilot for Pull Requests (PRs) = + Pull Request
3. 3 Copilot for Pull Requests (PRs) copilot:summary copilot:walkthrough copilot:poem copilot:all
4. 4 Impact of Copilot for PRs on code review Merge rate Does it expedite reviews? Expedite Reviews Is there a higher chance of PRs being merged due to its input?
5. 5 Early adoption of Copilot for PRs Developers can modify AI-generated content to meet their needs
6. 6 Goal: Early adoption of Copilot for PRs and its impact on code review (RQ1) To what extent do developers use Copilot for PRs in the code review process? Adoption Trend of Copilot for PRs Proportions of Copilot for PRs Distribution of Copilot commands
7. (RQ2) How are the code reviews affected by the use of Copilot for PRs? 7 Goal: Early adoption of Copilot for PRs and its impact on code review Is there a relationship between the use of Copilot for PRs and review time? Is there a relationship between the use of Copilot for PRs and the likelihood of a PR being merged?
8. 8 Goal: Early adoption of Copilot for PRs and its impact on code review (RQ3) How do developers adapt the content suggested by Copilot for PRs? What kind of supplementary information complements the content suggested by Copilot for PRs? What kind of content suggested by Copilot for PRs undergoes subsequent editing by developers?
9. 9 Goal: Early adoption of Copilot for PRs and its impact on code review (RQ3) How do developers adapt the content suggested by Copilot for PRs? (RQ1) To what extent do developers use Copilot for PRs in the code review process? (RQ2) How are the code reviews affected by the use of Copilot for PRs?
10. 10 (RQ1) To what extent do developers use Copilot for PRs Proportions of Copilot for PRs P R s D e p e n d e n c e Experience Repositories with long experience periods of Copilot for PRs is also dependent on it
11. 11 (RQ1) To what extent do developers use Copilot for PRs Proportions of Copilot for PRs Rely on pull_request_template.md PRs
12. 12 (RQ2) How are the code reviews affected by the use of Copilot for PRs? Expedite Reviews? Merge rate? 17,177 closed/merged 50,695 closed/merged
13. 13 (RQ2) How are the code reviews affected by the use of Copilot for PRs?
14. 14 (RQ2) How are the code reviews affected by the use of Copilot for PRs? Is there a relationship between the use of Copilot for PRs and the likelihood of a PR being merged? PRs (Copilot for PRs) are 1.57 times more likely to be merged than others
15. Is there a relationship between the use of Copilot for PRs and review time? 15 PRs (Copilot for PRs) reduce 19.3 hours of review time (RQ2) How are the code reviews affected by the use of Copilot for PRs?
16. 16 (RQ3) How do developers adapt the content suggested by Copilot for PRs? What kind of supplementary information complements the content suggested by Copilot for PRs? 22.7% 22.8% Associated links Static template Information
17. 17 What kind of supplementary information complements the content suggested by Copilot for PRs? 12.8% Pull Request Intent 9.2% Testing Information 7.3% Visual Representation (RQ3) How do developers adapt the content suggested by Copilot for PRs?
18. 18 What kind of content suggested by Copilot for PRs undergoes subsequent editing by developers? 22.9% Deletion (RQ3) How do developers adapt the content suggested by Copilot for PRs?
19. 19 What kind of content suggested by Copilot for PRs undergoes subsequent editing by developers? 22.9% Deletion 20.8% Refinement (RQ3) How do developers adapt the content suggested by Copilot for PRs?
20. 20 What kind of content suggested by Copilot for PRs undergoes subsequent editing by developers? 22.9% Deletion 20.8% Refinement 17.4% Exclusion (RQ3) How do developers adapt the content suggested by Copilot for PRs?
21. 21 What kind of content suggested by Copilot for PRs undergoes subsequent editing by developers? 22.9% Deletion 20.8% Refinement 17.4% Exclusion 14.7% Replacement 12.3% Exchangement 6% Augmentation (RQ3) How do developers adapt the content suggested by Copilot for PRs?
22. 22 Recommendations Advocate for Copilot for PRs adoption in PRs
23. 23 Incorporate Copilot for PRs tags into PR templates Establish Copilot for PRs tag integration guidelines Recommendations
24. Refinement of template integration 24 Offer developer-specific suggestions Recommendations Learn from Developer Modifications
25. 25 Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions Our Paper! Replication Package! * Most icons and images in the slides were generated by ChatGPT or available at https://www.flaticon.com/.
26. 26 Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions