Work Practices and Challenges in Pull-Based Development: the Integrator’s Perspective

Slides from my talk at ICSE 2015

Transcript

1. Work Practices and Challenges in Pull-Based Development: the Integrator’s Perspective Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, Arie van Deursen
2. How does software grow? http://travelspirit333.com
3. How does OSS grow? http://travelspirit333.com
4. The onion model Core developers Co-developers Community Users
5. The pull-based development model Here are my changes Please fix those issues Here are my updates Looks great, thanks! contributor integrator changes integrated changes examined changes re- examined
6. GitHub: made pull requests popular
7. 45% of collaborative projects Projects with > 1 committers Gousios et al. ICSE14
8. 45% of collaborative projects Projects with > 1 committers 55% use shared repository 45% use pull requests Gousios et al. ICSE14
9. Widely popular and increasing
10. Widely popular and increasing 90k repositories 400k pull requests
11. Widely popular and increasing Per month 90k repositories 400k pull requests
12. Large scale collaboration 2710 committers 309 code reviewers 9120 commenters 657 collaborators 125 code reviewers 3904 commenters 875 committers 92 code reviewers 5200 commenters
13. Too successful?
14. Too successful? “Lack of knowledge of git from contributors; most don’t know how to resolve a merge conflict.” “Sifting through the GitHub information flood to find what, if any, I should address.” “Dealing with loud and trigger-happy developers.”
15. Integrators: guardians of quality
16. WUGRWNNTGSWGUVU! FGEKFGVJGHCVGQHCEQPVTKDWVKQP! GXCNWCVGVJGSWCNKV[QHEQPVTKDWVKQPU! RTKQTKVKUGVJGCRRNKECVKQPQHEQPVTKDWVKQPU! YJCVMG[EJCNNGPIGUFQVJG[HCEG! 4GUGCTEJ3WGUVKQPU *QYFQKPVGITCVQTU
17. Pilot survey GHTorrent 250 integrators (25 responses) Literature Survey Final Survey Analysis
18. How we reached out to integrators Lifelines for 10% slowest pull reqs community participation in commits % comments (red) and commenters (blue) from the community http://ghtorrent.org/pullreq-perf/
19. GHTorrent Final Survey Performance Reports
20. GHTorrent Final Survey Performance Reports 3,200 integrators (749 responses)
21. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses)
22. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses) Results
23. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses) Results 645 integrators
24. Overlay *QYFQKPVGITCVQTUWUGRWNNTGSWGUVU! 
25. How integrators use pull requests? Code reviews Issue fixes Soliciting contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
26. How integrators use pull requests? Code reviews Issue fixes Soliciting contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
27. How integrators use pull requests? Code reviews Issue fixes Soliciting contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
28. How projects do code reviews?
29. How projects do code reviews? 65%: Inline code comments
30. How projects do code reviews? 65%: Inline code comments 60%: Integration pipeline
31. How projects do code reviews? 65%: Inline code comments 60%: Integration pipeline 55%: The community participates
32. Overlay *QYFQKPVGITCVQTUFGEKFGYJKEJ EQPVTKDWVKQPUVQCEEGRV! 
33. Factors for determining acceptance Code quality Code style Project fit Technical fit Testing Documentation Feature importance Code review result 0 5,5 11 16,5 22 % of responses
34. Factors for determining acceptance Code quality Code style Project fit Technical fit Testing Documentation Feature importance Code review result 0 5,5 11 16,5 22 Quality Project Fit % of responses
35. PR characteristics leading to merge PR hotness PR has tests PR track record PR churn PR # comments 0 50 100 150 200 Not or Mildly Important Quite or Very Imporant 100 0 100 % of responses
36. Overlay *QYFQKPVGITCVQTUGXCNWCVGVJG SWCNKV[QHEQPVTKDWVKQPU! 
37. - Following project's code style - Comments in code - Quality of the commit message - Readability of the code - Test coverage and their readability - Included documentation updates “Proper use of C++ language constructs. Knowledge and use of our dependent libraries […]” Views on quality
38. Contribution quality perception Style conformance Test coverage of PR Code quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses
39. Contribution quality perception Style conformance Test coverage of PR Code quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance
40. Contribution quality perception Style conformance Test coverage of PR Code quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance Technical Excellence
41. Contribution quality perception Style conformance Test coverage of PR Code quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance Technical Excellence
42. Overlay *QYFQKPVGITCVQTURTKQTKVKUGVJG CRRNKECVKQPQHEQPVTKDWVKQPU! 
43. Prioritisation factors
44. Prioritisation factors “How important the issue is to the project. How serious the bug is. […]” Criticality
45. Prioritisation factors Urgency Criticality “Whether the functionality is critical (bug fixes or very important feature blocking many users)[…]”
46. Prioritisation factors Urgency Criticality Size/Complexity “If the merge is small, or affects a large issue, it'll be merged fast. If it's a feature request or large change it will take longer.”
47. Prioritisation factors Urgency Mostly FIFO, but critical bug fixes or hot fixes get pushed to the top Age Criticality Size/Complexity
48. Overlay 9JCVCTGVJGMG[EJCNNGPIGUHCEGF D[KPVGITCVQTU! 
49. Technical challenges
50. Technical challenges code base quality “Ensuring the codebase remains consistent”
51. Technical challenges impact assessment “Huge, unwieldy, complected bundles of ‘hey I added a LOT of features and fixes ALL AT ONCE!’ that are hell to review and that I’d like to *partially* reject if only the parts were in any way separable…” code base quality
52. Technical challenges impact assessment contributor experience code base quality “Lack of knowledge of git from contributors; most don’t know how to resolve a merge conflict.”
53. Technical challenges impact assessment “Sifting through the GitHub information flood to find what, if any, I should address” contributor experience code base quality volume
54. Social challenges
55. Social challenges “Time is the biggest challenge -- I would love to have a more hierarchical structure to spread out the work; however, the community's development capacity is somewhat limited.” workload
56. Social challenges workload responsiveness “People become non-responsive, never write tests, and never update style. They make a change once when they need their problem solved and disappear basically.”
57. Social challenges explaining rejection workload responsiveness “Disappointing people when they put in a lot of work but the quality is low.”
58. Social challenges explaining rejection workload responsiveness motivating contributors “On-boarding new contributors: helping them set up the development environment and learn enough Git/ Github to submit a pull request”
59. How does OSS grow? The pull- based development perspective http://travelspirit333.com
60. Integrators use pull requests for code review https://www.digitalassurance.com/sites/default/files/iStock_000017496218_Medium.jpg
61. Integrators use testing as a safety net http://www.tradeandexportme.com/wp-content/uploads/2013/02/shutterstock_91331678.jpg
62. Integrators face technical challenges
63. Integrators face technical challenges Impact assessment Quality evaluation Work prioritisation
64. Integrators do not use track records (much)
65. Not a better online forum experience
66. Recommendations for integrators
67. Recommendations for integrators automation contribution guidelines testing be proactive/reactive
68. New research directions
69. New research directions prioritisation quality analysis impact analysis code review automation
70. New research directions prioritisation quality analysis impact analysis code review automation @gousiosg