Pitfalls in Data Science Projects (and how to avoid them)

Slides for my talk given at the 9th PyData Southampton meetup (September 2024)
https://www.meetup.com/pydata-southampton/events/303138206

Abstract:
This talk discusses the successful delivery of data science projects, by analysing common mistakes that can hamper your plans and hurt your chances of success. It’s commonly acknowledged that a huge proportion of data science projects fail, for a variety of reasons. By sharing some lessons learned the hard way, we’ll look at frequent anti-patterns and how to elude them.

This presentation is mainly aimed at early-career data scientists, but also at anyone who has seen their data science projects going south and would like to improve the outcome of their work.

Transcript

1. Pitfalls in Data Science Projects Marco Bonzanini PyData Southampton Meetup
2. © Bonzanini Consulting Ltd — BonzaniniConsulting.com Nice to meet you • Dr Marco Bonzanini • NLP and Data Science stuff • Consulting, training and coaching on Python + Data Science • Former Chair @ PyData London 2
3. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 3
4. WHY DO PROJECTS FAIL?
5. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 5 Business and Tech Out of sync
6. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 6 Business and Tech Out of sync Data Quality
7. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 7 Business and Tech Out of sync Shiny Object Syndrome Data Quality
8. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 8 Business and Tech Out of sync Shiny Object Syndrome Data Quality No Route to Deployment
9. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 9 Business and Tech Out of sync Shiny Object Syndrome Data Quality No Route to Deployment
10. THE DREADED “POC”
11. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 11 Prototype Idea Product Expectation
12. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 12 Prototype Idea Product Expectation Reality ???
13. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 13 First Mile ≠ Last Mile
14. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 14 R&D Requirements ≠ Production Requirements
15. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 15 Lack of Planning Lack of Communication Pitfalls
16. SHOULD WE WORK ON IT?
17. © Bonzanini Consulting Ltd — BonzaniniConsulting.com Impact vs Effort 17
18. © Bonzanini Consulting Ltd — BonzaniniConsulting.com Impact vs Effort 18 Effort Impact
19. © Bonzanini Consulting Ltd — BonzaniniConsulting.com Impact vs Effort 19 Effort Impact High impact Low effort 👍 👍 Low impact High effort 👎 👎
20. © Bonzanini Consulting Ltd — BonzaniniConsulting.com Impact vs Effort 20 Effort Impact ✅ ✅ ✅ ❌ ❌ ❌ ❌ 🤷 ✅ 🤷 🤷
21. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 21 Align with Business De-risk
22. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 22 Align with Business De-risk • Business case: PoC vs Proof-of-Value
23. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 23 Align with Business De-risk • Business case: PoC vs Proof-of-Value • Stakeholders: buy-in + involvement
24. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 24 Align with Business De-risk • Business case: PoC vs Proof-of-Value • Stakeholders: buy-in + involvement • Data availability: fi rst mile data + additional data
25. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 25 Align with Business De-risk • Business case: PoC vs Proof-of-Value • Stakeholders: buy-in + involvement • Data availability: fi rst mile data + additional data • Data quality, coverage and volume?
26. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 26 Align with Business De-risk • Business case: PoC vs Proof-of-Value • Stakeholders: buy-in + involvement • Data availability: fi rst mile data + additional data • Data quality, coverage and volume? • Deployment, integration, scalability?
27. TOWARDS THE PROTOTYPE
28. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 28 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint
29. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 29 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint “Let’s get the best accuracy”
30. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 30 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint “Let’s get the best accuracy” • Feature engineering • Model building • Hyperparameter tuning • […]
31. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 31 Over Optimising Not Speaking the Language Pitfalls
32. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 32 End-to-end Quickly Iterative Improvement
33. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 33 End-to-end Quickly Iterative Improvement • Promote early feedback
34. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 34 End-to-end Quickly Iterative Improvement • Promote early feedback • Avoid early complexity: - Dif fi cult to diagnose - Delay feedback - Hide bigger risks
35. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 35 End-to-end Quickly Iterative Improvement • Promote early feedback • Avoid early complexity: - Dif fi cult to diagnose - Delay feedback - Hide bigger risks • Optimise for business value, not “accuracy”
36. TOWARDS PRODUCTION
37. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 37 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint
38. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 38 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint
39. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 39 Data Validation Data Processing Model Training Model Validation Deployment Raw Data Model Serving Endpoint What got you here
 won’t get you there
40. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 40 Hidden Technical Debt of Machine Learning Systems, Sculley et al. (2015)
41. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 41 Research ≠ Engineering Not Looking at the Bigger Picture Pitfalls
42. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 42 Testing Packaging
43. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 43 Testing Packaging • Don’t ignore good software engineering principles
44. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 44 Testing Packaging • Don’t ignore good software engineering principles • Testing: unit testing, integration testing
45. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 45 Testing Packaging • Don’t ignore good software engineering principles • Testing: unit testing, integration testing • Code re-usability, DRY (Don’t Repeat Yourself) Single Responsibility Principle
46. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 46 Testing Packaging • Don’t ignore good software engineering principles • Testing: unit testing, integration testing • Code re-usability, DRY (Don’t Repeat Yourself) Single Responsibility Principle • Ditch the notebooks as soon as you: - struggle testing some component - would like to “import from another notebook”
47. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 47 Testing Packaging
48. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 48 Testing Packaging
49. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 49 Code Reviews Integration
50. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 50 Code Reviews Integration • Code reviews done right: foster collaboration - Spot check errors - Clarify the why’s - Knowledge share
51. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 51 Code Reviews Integration • Code reviews done right: foster collaboration - Spot check errors - Clarify the why’s - Knowledge share • Code reviews done wrong: hostile environment - Ask open ended questions - Use professional language - Nitpicks: labelled as such, kept to a minimum
52. AFTER PRODUCTION
53. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 53 Model Degradation Scalability Lack of Explainability Cost of Maintenance 💩
54. SUMMARY
55. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 55 The First Mile Rarely a technology problem Usually a planning + communication problem
56. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 56 The First Mile Rarely a technology problem Usually a planning + communication problem The Last Mile Sometimes a technology problem Still a planning + communication problem
57. © Bonzanini Consulting Ltd — BonzaniniConsulting.com 57 Thank You • Linkedin https://www.linkedin.com/in/marcobonzanini/ • Blog: marcobonzanini.com • Newsletter: marcobonzanini.com/newsletter