DDDVienna - Applying event sourcing and CQRS in a large ERP system

In the last 7 years we have applied event sourcing and CQRS in the development of our new ERP system. During this process we have run into some interesting challenges and we have come up with solutions. In this session I would like to present these patterns and get your feedback on them.

Transcript

1. Applying event sourcing and CQRS in a large ERP system Our work at AFAS Software
2. 400+ employees (4 locations) 10.500 customers (companies) AFAS Software
3. HRM, CRM, finance, order management, project management, workflow, ... AFAS Profit
4. 1996 – AFAS starts 2011 (jan) – First steps in CQRS and event sourcing 2011 (dec) – Started at AFAS 2020 – Live Via JTeam/Trifork/Axon. 2016 – Started on evolution
5. 20 years of ERP 10.000 customers 90% cloud Web Scalable CQRS Event sourcing
6. 20 years of ERP 10.000 customers 90% cloud Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit Put all our knowledge and experience in a model
7. Customized Web Scalable CQRS Event sourcing Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
8. (2.830 tables with 126.705 columns) ERP software is inherently relational
9. Our basic CQRS/Event Sourcing solution
10. Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler
11. Event store Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler Aggregate root Command Projector Event Queue Query store
12. Event store Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler Aggregate root Command Projector Event Queue Query store Data of different projectors can not be joined. Tables are fully inlined, no joins.
13. Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler • Aimed at CRUD • MongoDB, *SQL, DocumentDB • MS Orleans, MS Service Fabric • SPA-frontend
14. How about evolution? Part 1
15. Evolution as a feature of the model? Person Entity Customer Role Order Agreement party Spending limit Address Organisation Entity Delivery BusinessAc tivity party Invoice BusinessAc tivity party Address Address Spending limit Spending limit Own Organisation Workarea workarea Spending limit
16. Data upgrade Deployed with Executed by Executed by Data upgrade Application upgrade Lazy transformation Upcasting In place transfor- mation Multiple versions Copy and transfor- mation Basic & Complex Event Basic & Complex Stream Basic Store Complex Store Big Flip Rolling Upgrade Blue-Green Expand- Contract Blue- Green Deployed with Combined with Weak schema Expensive!
17. Event Publishing part 2
18. Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler
19. Aggregate root Command Event store Event Projector Query store Event Queue Query request Query handler Can give error (needs retry). Command-side in control over publishing.
20. Aggregate root Command Event store Event Projector Query store Query request Query handler
21. Aggregate root Command Event store Event Projector Query store Query request Query handler Projector in control. More load on the event store.
22. When CRUD is not enough business processes
23. Aggregate root Command Event store Event Projector Query store Query request Query handler
24. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager
25. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager But what if these commands fail?
26. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store
27. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Commands that cannot fail become events. We still have some internal commands.
28. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Different instances to scale the work. Event sourced for state.
29. Aggregate root / Process manager Command Event store Event Projector Query store Query request Query handler Is this what we want?
30. Aggregate root / Process manager Command Event store Event Projector Query store Query request Query handler Is this what we want? Technically there is no difference between aggregates and processmanagers
31. The problem of relational data
32. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store
33. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Process manager Process manager Process manager Aggregate root Aggregate root Aggregate root Aggregate root
34. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Process manager Process manager Process manager Aggregate root Aggregate root Aggregate root Aggregate root These ProcessManager all need the same data (lots of data)
35. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Stream Projector Query store Event store
36. Aggregate root Command Event store Event Projector Query store Query request Query handler Process manager Event store Stream Projector Query store Event store StreamProjectors are single-instanced, non- event-sourced processmanagers
37. Eventual consistency communication between client and server
38. Aggregate root Command Event store Event Projector Query store Query request Query handler Client
39. Counter Process Manager Count Command Counted Event Event store Counter Projector Query store Get Counter Count Query handler Getting the next consecutive number in a series.
40. Counter Process Manager Count Command Counted Event Event store Counter Projector Query store Get Counter Count Query handler When is the value ready? Correlationid?
41. Aggregate root Command Event store Event Projector Query store Query request Query handler Client Using websockets to notify the Client
42. Aggregate root Command Event store Event Projector Query store Query request Query handler Client Returning values from the produced event directly
43. Counter Process Manager Count Command Counted Event Event store Counter Projector Query store Get Counter Count Query handler
44. How large is our system anyway?
45. Aggregate root Command Event store Event Projector Query store Query request Query handler Stream Projector Query store Event store 557 464 1927 4849 1075 2718 801 1479
46. Store (19) S M L Traffic (19) S M L Schema (14) S M L
47. Dealing with bounded contexts
49. Aggregate root Command Event store Event Aggregate root Event store Event
50. Aggregate root Command Event store Event Aggregate root Event store Event Team B Team A
51. Aggregate root Command Event store Event Aggregate root Event store Event These events now serve multiple goals. Team B Team A
52. Aggregate root Command Event store Event Aggregate root Event store Event Team B Team A Sync Event store Event
53. Aggregate root Command Event store Event Aggregate root Event store Event Team B Team A Sync Event store Event Introduce synchronization events.
54. How about evolution? Part 2
55. Versioned Events Weak Schema Upcasters In-Place Copy- Transform
56. We will go live in January… No guarantees are given ☺
57. Copyright Nasa Goddard Michiel Overeem @michielovereem • No event bus • Commands do have results • Projectors can publish events • Aggregates can listen to events • You can do what you want ☺ i.e. you do not have to listen to a consultant that does not know your context.