Software Analytics with Data Science on Software Data (CraftConf 2024)

Data Science has demonstrated its value in extracting insights from business data, raising the question: Why not apply these principles to our software systems’ data? In this talk, I’ll introduce you to the world of Software Analytics. We’ll explore how to extract valuable insights from software data by using tools and techniques from data science to get rid of big, systemic problems in our software systems.

You’ll learn how to leverage scientific thinking, manage the analysis process, and apply literate statistical programming to analyze software systems in an understandable way. Or to put it in the words of software developers: We automate the analysis of large software systems using open-source tools like Jupyter Notebook, Python, pandas, jQAssistant, and Neo4j. I’ll also demonstrate through live-coding how we can gain new insights from data sources like Git repositories, performance measurements, or source code.

Join me in this session to acquire your starter kit for uncovering deeply hidden issues and change the way of improving systems with data-driven software analysis!

Transcript

1. Software Analytics with Data Science on Software Data CRAFT CONFERENCE 10, MAY 31, 2024 Markus Harrer Software Evolutionist @ INNOQ Social: markusharrer.de
2. The original horror show IMAGE: sethJreid/Pixabay
3. According to https://pkruchten.files.wordpress.com/2013/12/kruchten-colours-yow-sydney.pdf, Icons from https://game-icons.net/, CC BY 3.0 license
4. Adapted from Ray Koopa https://commons.wikimedia.org/wiki/File:Lcars_wallpaper.svg (CC BY-SA 4.0) THE ULTIMATE SOFTWARE QUALITY DASHBOARD DEVELOPERs SANITY PROJECT BUDGET
5. THE
9. ATTACK OF THE DATA
10. Frequency Problems Importance for us Problem-oriented Data ANALYSIS Standard tools for generic problems Software Analytics for very specific problems! !
11. THE SOFTWARE DATA OF OF DEVELOPERS
12. Domain Expertise Data Science Venn diagram by Drew Conway (simplified and slightly changed)
14. 3 Python 1
15. Context Idea Analysis Conclusion Literate Statistical Programming code and data in love
16. Attribution: Tobias ToMar Maier, https://commons.wikimedia.org/wiki/File:VHS_tape_with_time_scale.jpg Demo I Miami Cops Police Academy
17. jQAssistant Neo4j Graph Data Science
18. :Class :Method :Field https://github.com/JavaOnAutobahn/spring-petclinic public class Pet { private LocalDate birthDate; public LocalDate getBirthDate(){ return this.birthDate; } public void setBirthDate(LocalDate birthDate){ this.birthDate = birthDate; } Graph-based Analytics of Source Code
19. :Class :Method :Field :Entity @Entity @Table(name = "pets") public class Pet { ... https://github.com/JavaOnAutobahn/spring-petclinic Graph-based Analytics of Source Code
20. :Class :Method :Field 2 findings 5 changes :Entity 100% usage gain and share Insight from Data Business Subdomain
21. 16 types 17 findings 15 changes 70% usage 5 types 39 findings 51 changes 80% usage A perspective, where also managers can reason about! You can query this! make better Decisions
22. MATCH (t:Type)-[:BELONGS_TO]->(s:Business:Subdomain), (t)-[:HAS_CHANGE]->(ch:Change) RETURN s.name as ASubdomain, COUNT(DISTINCT t) as Types, COUNT(DISTINCT ch) as Changes ORDER BY Types DESC LIMIT 5 What are the top 5 subdomains with the most types and their changes within a subdomain? Changes Types ASubdomain 209 15 "Pet" 119 9 "Visit" 117 8 "Vet" 130 7 "Owner" 102 6 "Clinic" asking the right Questions
23. Attribution: Tobias ToMar Maier, https://commons.wikimedia.org/wiki/File:VHS_tape_with_time_scale.jpg Demo II Terminator Flash Gordon
24. The return of the reason The Two Powers The fellowship of the bling D a t a o r i e n t e d Meta Metric Number of solved problems NO Tool To Rule Them All P r o b l e m d r i v e n become the of the T by analyzing software in a data-driven way
25. slido.com #CraftConf2024 Pink Stage ASK 'EM ALL Social contact links: markusharrer.de Thank you!
26. One more thing…
27. AI = JUDGMENT DAY? THREAT ASSESSMENT ********************** - MAYBE NOT - Absolutely Not! - MAYBE B a c k g r o u n d i m a g e b y F r e e p i k
28. Appendix
29. More on Software Analytics softwareanalytics.de
30. More on Software Analytics Adam Tornhill: Your Code as a Crime Scene & Software Design X-Rays Tim Menzies, Laurie Williams, Thomas Zimmermann: Perspectives on Data Science for Software Engineering Christian Bird, Tim Menzies, Thomas Zimmermann: The Art and Science of Analyzing Software Data
31. More on Data Science Jeff Leek: The Elements of Data Analytic Style Roger D. Peng: Report Writing for Data Science in R Wes McKinney: Python for Data Analysis
32. Jupyter Notebook, python, pandas, matplotlib https://github.com/feststelltaste/software- analytics/tree/master/demos/20240331_CRAFTCONF_2024 jQAssistant & Neo4j https://github.com/JavaOnAutobahn/spring-petclinic Demos
33. More on Graph Data Science Mark Needham & Amy Hodler: Graph Algorithms https://neo4j.com/product/graph-data-science/
34. Paper about jQAssistant/Neo4j https://easychair.org/publications/preprint/893N
35. Thank you very much! innoQ Deutschland GmbH Krischerstr. 100 40789 Monheim am Rhein Germany +49 2173 3366-0 Ohlauer Str. 43 10999 Berlin Germany Ludwigstr. 180E 63067 Offenbach Germany Kreuzstr. 16 80331 Munich Germany Gewerbestr. 11 CH-6330 Cham Switzerland +41 41 743 01 11 Albulastr. 55 8048 Zurich Switzerland innoQ Schweiz GmbH Markus Harrer markus.harrer@innoq.com Slides