DDDVienna - 103 years of event sourcing experience

All this talk about event sourcing, and why it is the best. But who uses it? Why do they use it? And what are the challenges that others have run into? We had exactly these questions. In order to find the answers we have interviewed 25 engineers from all over the world. Some with many years of experience, others with just a few. In total they had 103 years of experience. We asked them all about event sourcing: why would you use it, why not, what did you like, what not, and so on. Topics such as schema evolution, optimization of replay, and the steep learning curve will be discussed. This talk presents a summary of 103 years of experience with event sourcing.

Transcript

1. 103 years of event sourcing experience Results of our empirical research on event sourcing
2. Michiel Lead Software Architect @ AFAS Software PhD candidate @ University Utrecht michiel.overeem@afas.nl @michielovereem Overeem
3. Continuous Migration of Mass Customized Applications • Cloud architectures • Model-driven development • Evolution
4. M. Overeem, M. Spoor, and S. Jansen, “The Dark Side of Event Sourcing: Managing Data Conversion,” in IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 193–204.
5. Data upgrade Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with Combined with Weak schema A framework based on literature and a prototype. But do we really understand event sourcing?
6. RQ2 RQ3 RQ1 What types of systems apply event sourcing and why? How can event sourced systems be defined? What are the challenges faced in applying event sourcing?
7. 1. Problem 2. Solution 3. Consequences
8. RQ2 RQ3 RQ1 What types of systems apply event sourcing and why? How can event sourced systems be defined? What are the challenges faced in applying event sourcing?
9. 4 3 18 25 engineers with 103 years of combined experience
10. And 3 engineers who are not doing event sourcing...
13. State State’ Event
14. Events = Transactions Or not?
15. THE PROBLEM
16. Project administration Marketing automation Website building Payment processing Content management Classified advertising …
17. Store (19) S M L Traffic (19) S M L Schema (14) S M L
18. Rationale Flexibility Complexity Trending Audit Immutability Strict Cut-off Mutable Technology .NET JVM PHP Ruby,Scala,Go
19. DDD Yes No MSA Yes No CQRS Yes No
20. No one returned to ‘current state’!
21. THE SOLUTION
22. “An event is a discrete data object specified in domain terms that represents a state change in an ESS.”
23. “The project function takes one or more event streams and creates a projection with the data from the given events.”
24. “The accept function takes a projection P and a command C. The command is validated using the data in the projection, and the accept function either results in an error or in an event “
25. What does a event sourced system with CQRS looks like?
27. THE CONSEQUENCES
28. Steep learning curve Learning curve Finding your way
29. Evolution Small steps, but how to stay stable?
30. Missing tools Related to the learning curve? Or the trend?
31. Duplication?
32. How long do these take?
33. Eventually consistent Things might take a while
34. Evolution
35. Applications that do not evolve in response to changing requirements or changing technology become less useful -Lehman, 1974
36. Which streams are in the store? Which event types are in which stream? Which attributes does an event have?
37. DDD Yes No “You align the events with real world events. You are dealing with changes that have a native equivalence. Doing DDD leads to less fragile design.”
38. Versioned Events (2/24) only introduce new events and streams “I try to keep my domain abstraction pure."
39. Weak Schema (11/24) aka tolerant reader Easy, but incomplete
40. Upcasters (12/24) Transform at runtime “The approach I took, and which is my impression that everyone takes …”
41. Upcasters (12/24) Transform at runtime “Then we made our upcasters permanent…”
42. Immutability, not a binaire option
43. Immutability Strict Cut-off Mutable Rationale Flexibility Complexity Trending Audit
44. “So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.”
45. “So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…”
46. “So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…” “These rewrites aren't destructive first of all.”
47. “So one of the things you should never ever do is change existing events. An event written persisted to datastore is not like a hand wavy thing. It is a fact.” “Historians rewrite the past too…” “These rewrites aren't destructive first of all.” “When your events form a contract with the outside, you have to keep those.”
48. In-place transformation (5/24) Just as in the old days “Now that would pretty much break the whole premise of having a proper audit trail.”
49. Copy-transform (14/24) When you just need to start over “if you have like 10 billion events you just can't do it”
50. Copy-transform (14/24) When you just need to start over “if you have like 10 billion events you just can't do it” 1. Create a new store from the old one. 2. Create a new stream inside of the store, based on old streams. 3. Create new events in an existing stream.
51. Versioned Events Weak Schema Upcasters In-Place Copy- Transform 14/24 5/24 12/24 11/24 2/24
52. Some last topics
53. Does my event store look big?
54. Year ending Classified ads
55. Privacy… Privacy
56. Custom event store structure Encryption Destructive technique
57. What about all those read models?
58. Rebuilt Sync
59. It depends. Your context. Your solution.
60. Hopefully, published soon! M. Overeem, M. Spoor, S. Jansen, and S. Brinkkemper, “An Empirical Characterization of Event Sourced Systems and their Schema Evolution,” to be published.
61. Copyright Nasa Goddard Michiel Overeem @michielovereem All (business) applications can apply event sourcing and benefit from it. There is room for your solution!