KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

Evolving software systems often includes data schema changes, and those cause the need for data conversion. This is no different in event sourced systems.
Converting data between schemas while continuing the operation of the system is a challenge.
In this talk, we will look at the possible conversion operations, the techniques to execute them, and strategies for deployment with zero-downtime.
We will look at the possibilities and the trade-offs for designing and implementing upgrades for event sourced systems.

Transcript

1. Copyright Nasa Goddard Michiel Overeem @michielovereem
2. AFAS Software 400+ employees 10.000+ organizations
3. AFAS Profit HRM, CRM, finance, order management, project management
4. Model-Driven Generate a customized application with standardized software
5. Management Zero downtime upgrades
6. daptable odel-based and ser-specific oftware cosystems A M U S E
7. Copyright Nasa Goddard
8. How can an event sourced system be upgraded efficiently when the (implicit) event schema changes?
9. Sources
10. Structure of an Event Store Store A (distributed) collection of streams. An ordered list of events Stream An ordered list of events. Event A name, attributes with values, and metadata.
11. The schema is implicit and hidden! Which streams are known in the system? Which events are allowed in which streams? Which attributes are allowed in which events?
12. Webshop EventStore Customer Stream ShoppingCart Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } ShoppingCartCreated { … } ArticleRemovedFromChart { … } ArticleAddedToChart { … }
13. Change will happen “An software system written to perform some real-world activity must be continually adapted or it becomes progressively less satisfactory” Lehman’s law of continuing change
14. Add attribute to event Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … }
15. Add attribute to event Customer Stream CustomerSignedUp { …, newsletter: <bool> } UpgradedCustomerToPremiumService { … }
16. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Add event to stream
17. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } OptInOnNewsLetter { … } Add event to stream
18. Newsletter Stream CreateNewsletter { … } PublishNewsletter { … } Add stream to store
19. Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> } UpgradedCustomerToPremiumService { … } CustomerSignedUp { …, fullName: <string> } Split attribute
20. Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> } UpgradedCustomerToPremiumService { … } Split attribute
21. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } CustomerSignedUp { …, newsletter: <bool> } Split events
22. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } OptInOnNewsLetter { … } Split events
23. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } OptInOnNewsLetter { … } CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Split streams
24. CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } NewsletterSubscription Stream OptInOnNewsLetter { … } Split streams
25. Store level operations Add / remove streams Merge streams / split / … streams Stream level operations Add / remove events Merge events / split / … events Event level operations Add / update / remove attributes Merge attributes / split / … attributes The operations
26. How can we execute these operations?
27. Always extend Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } CustomerSignedUp_V2 { … }
28. Our codebase would only grow. Blue Cross Blue Shield Tower under construction in Chicago, by Photogonal, wikipedia.org
29. CustomerSignedUp { “customerId”: …, “fullName”: … } Weak schema & serialization CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: … }
30. CustomerSignedUp { “customerId”: …, … } Weak schema & serialization
31. If-then-else explosion? pexels.com
32. Upcasting
33. Upcasters Upcasting
34. CustomerSignedUp { “customerId”: …, “fullName”: … } Upcasting CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: … } CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: …, “newsletter”: … } Upcaster Upcaster
35. Upcasters Upcasting
36. Runtime overhead? pexels.com
37. Think about the order of upgrading components!
43. Do we need immutability? Copyright Stephen Cairns
44. Data Information
45. Upcasters Lazy transformation
46. Upcasters Lazy transformation
47. When will this be done?
48. In place transformation
49. In place transformation
50. Not for the faint of heart?
51. Copy and transform
52. Copy and transform
53. Copy and transform
54. Slow?
55. Store A (distributed) collection of streams. An ordered list of events Store level Stream level Event level Add / remove streams Merge streams / split / … streams Should we perform all operations at runtime?
56. Let’s put all these operations together!
57. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
58. Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transformation Functional suitability +/- - +/- +/- + +
59. Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transformation Maintainability - +/- +/- +/- + +
60. Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transformation Performance efficiency + + + +/- +/- -
61. Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transformation Reliability + + + - - +
62. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
63. Can we execute this upgrade with zero downtime? Copyright SpaceX
64. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform These techniques are executed within the application These techniques need a separate stage
65. Application deployments strategies
66. Big Flip
67. Big Flip
68. Big Flip
69. Big Flip
70. Big Flip
71. Big Flip
72. Big Flip Blue / Green
73. Big Flip Blue / Green
74. Big Flip Blue / Green
75. Big Flip Blue / Green
76. Big Flip Blue / Green
77. Big Flip Blue / Green Rolling Upgrade
78. Big Flip Blue / Green Rolling Upgrade
79. Big Flip Blue / Green Rolling Upgrade
80. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade
81. Database deployments strategies
82. Blue / Green
83. Blue / Green
84. Blue / Green
85. Blue / Green
86. Blue / Green Expand & Contract
87. Blue / Green Expand & Contract
88. Blue / Green Expand & Contract
89. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
90. Event operations Stream operations Store operations simple Store operations complex In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
91. Event operations Stream operations Store operations simple Store operations complex In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
92. Event operations Stream operations Store operations simple Store operations complex Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green Mix & Match!
93. What about the projections?
95. How to keep your clients in sync?
96. It depends. Your context. Your solution.
97. Throw away with a reason It is an option!
98. Time can be of help Expirations?
99. You might not be Google Small enough to replay?
100. Domain design helps Design and test before production
101. Read our paper for more en And more is coming!
102. Recommended!
103. Copyright Nasa Goddard Michiel Overeem @michielovereem