DDDEurope 2018 - Event Sourcing After Launch

Event sourcing can be a solution for certain problems in your application, but it also comes with its own set of challenges. One of those is the evolution of your event schema. How should you handle changes to events?

We found many approaches while doing our research (we’ve published earlier results in our paper “The Dark Side of Event Sourcing”) and we gained a lot of experienced from building a full-size ERP application using CQRS and event sourcing. In this talk we will show what kind of approaches we’ve found and what we use to deal with our specific challenges. We will also share early research results from our interviews with over 20 different companies using event sourcing.

Every request for advice can be answered with ‘it depends’, and this talk will not solve that. But we will tell you what your options are, what we think works, and what many others have done.

Transcript

1. Event Sourcing after launch how to evolve your event store along with your application Michiel Overeem, AFAS, @michielovereem
2. “Versioning in event sourced systems is a big problem”
3. “We dreaded the upgrading. We had some fear in advance.”
4. “There is not much literature about it”
5. How to evolve your event store along with your application?
6. 20 years of ERP 10.000 customers 90% cloud Web Scalable CQRS Event sourcing
7. 20 years of ERP 10.000 customers 90% cloud Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit Put all our knowledge and experience in a model
8. Customized Web Scalable CQRS Event sourcing Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
9. How to evolve your event store along with your application?
11. 24 interviews on event sourcing and upgrades
12. Branches, domains
13. Prevention Techniques Pruning Privacy Read models
15. Which streams are in the store? Which event types are in which stream? Which attributes does an event have?
16. Applications that do not evolve in response to changing requirements or changing technology become less useful -Lehman, 1974
18. Prevention is better than a cure
21. State State’ Event
22. Events = Transactions Or not?
23. Domain-driven design Events = real world events
24. “You align the events with real world events. You are dealing with changes that have a native equivalence. Doing DDD leads to less fragile design.”
25. See what sticks Life before and after release
26. How implicit is your schema?
27. What if prevention is no longer possible?
28. Versioned Events only introduce new events and streams 2/24
29. “I try to keep my domain abstraction pure."
30. Weak Schema aka tolerant reader 11/24
31. Easy, but incomplete
32. Upcasters Transform at runtime 12/24
33. “The approach I took, and which is my impression that everyone takes …”
34. “Then we made our upcasters permanent…”
35. Immutability, not a binaire option
36. “So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.”
37. “Historians rewrite the past too…”
38. “These rewrites aren't destructive first of all.”
39. Data Information
40. “When your events form a contract with the outside, you have to keep those.”
41. In-place transformation Just as in the old days 5/24
42. “Now that would pretty much break the whole premise of having a proper audit trail.”
43. Copy-transform When you just need to start over 14/24
44. Variations 1. Create a new store from the old one. 2. Create a new stream inside of the store, based on old streams. 3. Create new events in an existing stream.
45. “if you have like 10 billion events you just can't do it”
46. Versioned Events Weak Schema Upcasters In-Place Copy- Transform 14/24 5/24 12/24 11/24 2/24
47. Does my event store look big?
48. Year ending Classified ads
49. Privacy… Privacy
50. “I really don’t know”
51. Custom event store structure Encryption Destructive technique
52. What about all those read models?
53. Rebuilt Sync
54. Prevention Techniques Pruning Privacy Read models 24 interviews on event sourcing and upgrades Event sourced systems after launch are not scary
55. It depends. Your context. Your solution.
56. So, what is our solution? 1. Copy-transform of the whole store 1. 100.000 events in 4.5 minute (around 360 per sec) 2. Rebuilt query side 1. All projectors run in parallel 3. We deploy in blue-green
57. What do we (think we) want? 1. Renames are solved on the model level 2. Weak schema: handle new attributes with ease 3. Upcasters: solve complex cases without long delays 4. Copy-transform: 1. Remove upcasters 2. Only those streams that need it 5. Only rebuilt projections when necessary
58. Copyright Nasa Goddard Michiel Overeem @michielovereem Event Sourced Systems after launch: 1. Prevention 2. Techniques for upgrading 3. Prune your store 4. Think about privacy 5. Rebuilt your read models