Not Invented Here Syndrome and Dark Debt: The PagerDuty Story

A medium length talk I gave at SRECon Europe 2018, about how the development and use of an in house distributed task scheduler/queue within PagerDuty was an example of Dark Debt.

Transcript

1. Not Invented Here Syndrome and Dark Debt: The PagerDuty Story AISH RAJ DAHAL

2. Why are we here ? Section I
3. Not Invented Here Syndrome
4. –The English Wikipedia (2018) “… the tendency towards reinventing the wheel based on the belief that in-house developments are inherently better suited, more secure, more controlled, quicker to develop, and incur lower overall cost than using existing implementations…”
5. Dark Debt
6. – Woods DD, STELLA: Report from the SNAFU catchers Workshop on Coping With Complexity. 2017 “….Dark debt was named to draw a parallel with dark matter. Dark matter has detectable effects on the world but cannot be seen or detected directly….Dark debt is found in complex systems and the anomalies it generates are complex system failures….Dark debt is not recognizable at the time of creation. Its impact is not to foil development but to generate anomalies….”
7. Comparison to technical debt
8. Build vs Buy: The age old question
9. Can building instead of buying lead to dark debt ?
10. Where does Not Invented Here syndrome come in the mix ?
11. WorkQueue Section II
12. October 24, 2011
13. WorkQueue
14. Background
15. High Availability
16. Durability
17. High Scalability
18. Decent Queryability
19. Flexible Schema
20. Backups
21. Synchronous Replication
22. What were the options then?
23. Kafka?
24. Other options were actually considered
25. Result: WorkQueue v1
26. But it did not stop there…
27. Host based partitioning
28. Fair partitioning
29. Time based row bucketing
30. …there were more services using WorkQueue
31. In the meanwhile…
32. Other distributed queues like Kafka gained popularity
33. PaaS based queues made their presence felt
34. Cassandra’s API changed*
35. Thrift RPC fell out of favor within the company
36. WorkQueue’s nature and widespread use was slowly turning into dark debt.
38. Lessons learnt Section III
39. Dark debt is a product of complexity
40. The development and use of in-house software for a task is not an indication dark debt
41. However, the proliferation of such special purpose software to other areas is often an indication of Not Invented Here syndrome
42. Not Invented Here syndrome is an indication of dark debt
43. Proactive evaluation of past custom solutions before proliferation could help alleviate some of the problems
44. Unless there is a really strong case for it, don’t go with build*
45. When in doubt chose simplicity over complexity
46. The past, the present and the future Epilogue
47. Official support for WorkQueue was ended in 2016
48. Cassandra based WorkQueue’s usage in PagerDuty has significantly reduced since 2017
49. –Edsger W. Dijkstra “Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.”
50. Thank You Gracias مكل اركش धन्यवाद Danke Merci 谢谢 ありがとう Asante @aishrajdahal