Using Data to Inform Decisions in Agile Software Development

Presentation of the paper "Towards using Data to Inform Decisions in Agile Software Development: Views of Available Data" held on July 28th 2019 at the 14th International Conference on Software Technologies (ICSOFT'19) in Prague.

Paper authors: Christoph Matthies, Guenter Hesse. Hasso Plattner Institute, University of Potsdam, Germany

Transcript

1. Hasso Plattner Institute University of Potsdam, Germany christoph.matthies@hpi.de @chrisma0 Using Data to Inform Decisions in Agile Software Development Christoph Matthies, Guenter Hesse Prague, July ’19
2. Motivation 2 for using data in software engineering Software engineering must shed the folkloric advice and anecdotal evidence [...] and replace them with convincing results, established with [...] empirical methods – Bertrand Meyer [Meyer, 2013] “ ” [Meyer, 2013] B. Meyer, H. Gall, M. Harman, and G. Succi, “Empirical Answers to Fundamental Software Engineering Problems (Panel),” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE 2013. New York, USA: ACM, 2013, pp. 14–18. Picture: https://commons.wikimedia.org/wiki/File:Bertrand_Meyer_recent.jpg
3. Importance of Decision-Making 3 in software development projects ▪ Software project success hinges on many factors □ Budget, scope, team organization, technology, architecture □ Customer engagement, requirements analysis [Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254. [Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering
4. Importance of Decision-Making 4 in software development projects ▪ Software project success hinges on many factors □ Budget, scope, team organization, technology, architecture □ Customer engagement, requirements analysis ▪ Correct project decisions critical for success [Drury, 2012] □ Overruns of budget and schedule □ Project cancellation [Taherdoost, 2018] [Drury, 2012] Drury, M., Conboy, K., and Power, K. (2012). Obstacles to decision making in Agile software development teams. Journal of Systems and Software, 85(6):1239–1254. [Taherdoost, 2018] Taherdoost, H. and Keshavarzsaleh, A. (2018). A Theoretical Review on IT Project Success/Failure Factors and Evaluating the Associated Risks. In Mathematical and Computational Methods in Electrical Engineering
5. Motivation 5 for using data in Agile software development processes Scrum is founded on empirical process control theory [...] knowledge comes from experience and making decisions based on what is known. [...] – The Scrum Guide [Schwaber, 2017] [Schwaber, 2017] K. Schwaber, J. Sutherland, “The Scrum Guide - The Definitive Guide to Scrum,” 2017, [online] http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf Picture: https://www.scrum.org/resources/2017-scrum-guide-update-ken-schwaber-and-jeff-sutherland “ ”
6. Decision-Making in Agile 6 Evidence-based decision-making in teams ▪ Agile teams should make use of available data to enable “evidence-based decision making” [Fitzgerald, 2014] ▪ However, few details available on concrete implementations [Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York, New York, USA. ACM Press.
7. Decision-Making in Agile 7 Evidence-based decision-making in teams ▪ Agile teams should make use of available data to enable “evidence-based decision making” [Fitzgerald, 2014] ▪ However, few details available on concrete implementations Main Research Questions ▪ What data is available and relevant to Agile teams? ▪ What methods are transferable from other domains? [Fitzgerald, 2014] Fitzgerald, B., Musiał, M., and Stol, K.-J. (2014). Evidence-based decision making in lean software project management. In Companion Proceedings of the 36th International Conference on Software Engineering - ICSE Companion 2014, pages 93–102, New York, New York, USA. ACM Press.
8. Related Work [Svensson, 2019] 8 Unfulfilled Potential of DDDM in Agile [Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”, 20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948
9. Potential of DDDM 9 Lots of potential, little usage [Svensson, 2019] ▪ Survey of software practitioners ▪ How is data used in the company for making decisions? [Svensson, 2019] Svensson, R.B., Feldt, R., & Torkar, R. “The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development”, 20th International Conference on Agile Software Development (XP), 2019 (preprint), https://arxiv.org/abs/1904.03948
10. Data-Driven Decision-Making 10 In business contexts ▪ DDDM contrasted with “gut feeling” [Provost, 2013] □ Standard in business in the absence of analysis technologies [Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data and Data-Driven Decision Making. Big Data, 1(1):51–59. [Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal. [Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making. Harvard Business Review Digital Articles, pages 2–5
11. Data-Driven Decision-Making 11 In business contexts ▪ DDDM contrasted with “gut feeling” [Provost, 2013] □ Standard in business in the absence of analysis technologies ▪ Technology use in business has fueled DDDM adoption □ Increased company productivity [Brynjolfsson, 2011] □ Transparency & real-time information intensifies tensions, management must adapt [Schrage, 2016] [Provost, 2013] Provost, F. and Fawcett, T. (2013). Data Science and its Relationship to Big Data and Data-Driven Decision Making. Big Data, 1(1):51–59. [Brynjolfsson, 2011] Brynjolfsson, E., Hitt, L. M., and Kim, H. H. (2011). Strength in Numbers: How Does Data-Driven Decisionmaking Affect Firm Performance? SSRN Electronic Journal. [Schrage, 2016] Schrage, M. (2016). How the big data explosion has changed decision making. Harvard Business Review Digital Articles, pages 2–5
12. Agile Software Development 12 Development Processes and their connection to data ▪ Plan-driven, rational decision-making [Moe, 2012] □ Sets of rules that govern behavior □ Assumption: outcome probabilities are known [Zannier, 2007] [Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making: A multiple case study of agile software development. Information and Software Technology, 54(8):853–865. [Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.
13. Agile Software Development 13 Development Processes and their connection to data ▪ Plan-driven, rational decision-making [Moe, 2012] □ Sets of rules that govern behavior □ Assumption: outcome probabilities are known [Zannier, 2007] ▪ Data enables teams to implement Agile feedback cycles ▪ Agile, naturalistic decision-making □ Situational behavior, context-dependent rules [Moe, 2012] Moe, N. B., Aurum, A., and Dybå, T. (2012). Challenges of shared decision-making: A multiple case study of agile software development. Information and Software Technology, 54(8):853–865. [Zannier, 2007] Zannier, C. and Maurer, F. (2006). Foundations of Agile Decision Making from Agile Mentors and Developers. In Abrahamsson, P., Marchesi, M., and Succi, G., editors, Extreme Programming and Agile Processes in Software Engineering, pages 11–20, Berlin, Heidelberg. Springer Berlin Heidelberg.
14. Levels of Decisions 14 The impact of Agile “self-organizing” teams ▪ Strategic plans define long-term goals ▪ Tactical plans involve managing projects to achieve goals ▪ Operational plans define milestones, and the actual work
15. Data Use in Business 15 A wealth of data to use for decisions ▪ Businesses gather detailed statistics □ Users, suppliers, partners & competitors □ ERP systems for collection, BI for analysis [Chen, 2012] [Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS quarterly, 36(4). [Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme Programming, pages 69–85. Springer International Publishing.
16. Data Use in Business 16 A wealth of data to use for decisions ▪ Businesses gather detailed statistics □ Users, suppliers, partners & competitors □ ERP systems for collection, BI for analysis [Chen, 2012] ▪ Continuing digitization allows additional data collection □ “Reality mining”: cars, smart devices, etc. [Svensson, 2019] □ Clickstream data, UI interactions, high time-resolution [Chen, 2012] Chen, H., Chiang, R. H. L., and Storey, V. C. (2012). Business intelligence and analytics: From big data to big impact. MIS quarterly, 36(4). [Svensson, 2019] Svensson, R. B., Feldt, R., and Torkar, R. (2019). The Unfulfilled Potential of Data-Driven Decision Making in Agile Software Development. In Agile Processes in Software Engineering and Extreme Programming, pages 69–85. Springer International Publishing.
17. Data in Software Development 17 Much more than code ▪ Software development process itself: source of valuable data □ Intrinsically relevant to making decisions □ Not only code, but a multitude of supporting artifacts [de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA. ACM, ACM Press. [Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, pages 113–122.
18. Data in Software Development 18 Much more than code ▪ Software development process itself: source of valuable data □ Intrinsically relevant to making decisions □ Not only code, but a multitude of supporting artifacts ▪ Processes are “inscribed” into software artifacts [de Souza, 2005] □ E.g. VCS: who, when, what, why, status □ Also includes evidence of failures [Ziftci & Reardon, 2017] [de Souza, 2005] de Souza, C., Froehlich, J., and Dourish, P. (2005). Seeking the Source: Software Source Code as a Social and Technical Artifact. In Proceedings of the 2005 international ACM SIGGROUP conference on Supporting group work - GROUP ’05, page 197, New York, New York, USA. ACM, ACM Press. [Ziftci & Reardon] Ziftci, C. and Reardon, J. (2017). Who broke the build? Automatically identifying changes that induce test failures in continuous integration at google scale. In Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track, ICSE-SEIP 2017, pages 113–122.
19. Software Project Data 19 [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the 13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016. “a gold-mine of actionable information” [Guo et al., 2016]
20. Software Project Data 20 “a gold-mine of actionable information” [Guo et al., 2016] Primary purpose: communication [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the 13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016.
21. Software Project Data 21 Use this data for reflection [Guo et al., 2016] Guo, J., Rahimi, M., Cleland-Huang, J., Rasin, A., Hayes, J. H., & Vierhauser, M, “Cold-start software analytics”, In Proceedings of the 13th International Workshop on Mining Software Repositories - MSR ’16, pp. 142–153, New York, New York, USA, ACM Press, 2016. “a gold-mine of actionable information” [Guo et al., 2016]
22. Scrum Process 22 Where Agile decision-making happens [Schwaber & Sutherland, 2017] 1 2 3 4 5 6 7 Sprint Planning Sprint Backlog Backlog Sprint Daily Stakeholder Input Product 4 iterations Sprint Retrospective Finished Increment Sprint Review [Schwaber & Sutherland, 2017] Schwaber, K., & Sutherland, J, “The Scrum Guide - The Definitive Guide to Scrum: The Rules of the Game”, 2017, [online] Available: http://scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf
23. Sprint Planning 23 1 2 3 4 5 6 7 Sprint Planning Sprint Backlog Backlog Sprint Daily Stakeholder Input Product 4 iterations Sprint Retrospective Finished Increment Sprint Review Building a backlog of work for the next Sprint
24. Sprint Planning 24 Building a backlog of work for the next Sprint ▪ Selection and prioritization of features for next iteration □ Based on stakeholders’ experiences & opinions [Svensson, 2019] □ Lack explanation and links to evidence
25. Sprint Planning 25 Building a backlog of work for the next Sprint ▪ Selection and prioritization of features for next iteration □ Based on stakeholders’ experiences & opinions [Svensson, 2019] □ Lack explanation and links to evidence ▪ Enable short feedback cycles between devs and real customers □ Metrics from deployed software, real use cases □ Observe actual (mis)usage, engineering relevant metrics □ Base changes on results
26. Daily Scrum 26 Quick status update 1 2 3 4 5 6 7 Sprint Planning Sprint Backlog Backlog Sprint Daily Stakeholder Input Product 4 iterations Sprint Retrospective Finished Increment Sprint Review
27. Daily Scrum 27 Quick status update ▪ Short daily status meeting □ “What did I do?”, “What will I do?”, “Blockers?” □ Strictly time-boxed
28. Daily Scrum 28 Quick status update ▪ Short daily status meeting □ “What did I do?”, “What will I do?”, “Blockers?” □ Strictly time-boxed ▪ Development artifacts hold information on what was done □ Concrete items that were completed □ Rely less on personal recollection □ Optimize for speed
29. Sprint Retrospective 29 Reflecting on past work and improving 1 2 3 4 5 6 7 Sprint Planning Sprint Backlog Backlog Sprint Daily Stakeholder Input Product 4 iterations Sprint Retrospective Finished Increment Sprint Review
30. Retrospectives 30 Reflecting on past work and improving What should be improved? Retrospective Meeting
31. 31 Did we improve what we planned? Retrospective Meeting Retrospectives Reflecting on past work and improving
32. 32 Did we improve what we planned? Retrospective Meeting Project Data Evidence of last iteration’s work code docs stats Retrospectives Reflecting on past work and improving
33. 33 Did we improve what we planned? Retrospective Meeting Project Data Evidence of last iteration’s work code docs stats Retrospectives Reflecting on past work and improving
34. Related Research Fields 34 Mining Software Repositories ▪ Draw from MSR techniques [Prana, 2018] ▪ MSR: Focus on large amounts of repositories □ “most widely used open source license?” [Dyer, 2013] ▪ Little research: Few repositories, intricate knowledge of creators / users [Prana, 2018] Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. “Categorizing the Content of GitHub README Files”. Empirical Software Engineering. 2018. https://doi.org/10.1007/s10664-018-9660-3 [Dyer, 2013] Dyer, R., Nguyen, H. A., Rajan, H., & Nguyen, T. N. “Boa: A language and infrastructure for analyzing ultra-large-scale software repositories”. In Proceedings - International Conference on Software Engineering. pp. 422–431. 2013. IEEE.
35. Conclusion 35 ▪ Agile requires teams to make decisions based on best information available [Schuh, 2004] ▪ Definition of “best information” ongoing research challenge [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10):833–859. [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc., Rockland, MA, USA, 1st edition.
36. Conclusion 36 ▪ Agile requires teams to make decisions based on best information available [Schuh, 2004] ▪ Definition of “best information” ongoing research challenge ▪ Individuals can ignore analyses or own preferences due to rules, traditions or influence of others [Dybå and Dingsøyr, 2008] [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10):833–859. [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc., Rockland, MA, USA, 1st edition.
37. Conclusion 37 ▪ Agile requires teams to make decisions based on best information available [Schuh, 2004] ▪ Definition of “best information” ongoing research challenge ▪ Individuals can ignore analyses or own preferences due to rules, traditions or influence of others [Dybå and Dingsøyr, 2008] Data analysis and interpretation by Agile teams (supported by context changes) can enable better informed decision [Dybå and Dingsøyr, 2008] Dybå, T. and Dingsøyr, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10):833–859. [Schuh, 2004] Schuh, P. (2004). Integrating Agile Development In The Real World. Charles River Media, Inc., Rockland, MA, USA, 1st edition.
38. Summary 38
39. Image Credits 39 In order of appearance ▪ Data directed Decision Making by H Alberto Gongora from the Noun Project (CC BY 3.0 US) ▪ Project by useiconic.com from the Noun Project (CC BY 3.0 US) ▪ Question by unlimicon from the Noun Project (CC BY 3.0 US) ▪ Business by David Glöckler from the Noun Project (CC BY 3.0 US) ▪ Agile Development by Bê Sabino from the Noun Project (CC BY 3.0 US) ▪ Digitization by priyanka from the Noun Project (CC BY 3.0 US) ▪ Developer by becris from the Noun Project (CC BY 3.0 US) ▪ Plan by Gregor Cresnar from the Noun Project (CC BY 3.0 US) ▪ Daily Scrum by Swen-Peter Ekkebus from the Noun Project (CC BY 3.0 US) ▪ Documents by LUFTFI GANI AL ACHMAD from the Noun Project (CC BY 3.0 US) ▪ Idea Generation by priyanka from the Noun Project (CC BY 3.0 US)