Dealing with change in event sourced applications (JavaZone 2018)

In software development, change is pretty much the only constant factor. In fact, embracing change is one of the twelve principles behind the Agile Manifesto. As time passes, our understanding of the domain we are working in evolves. We develop based on new requirements, (better) insights, opportunities, changes in the market or legislation, or other factors.

These inputs eventually all lead to modifications to our application, which can be very challenging to implement if the application uses event sourcing. Indeed, when applied very strictly, event sourcing can be quite resistant to change. And unfortunately, there’s not a lot of literature on this subject (yet).

In this talk, we’ll explore how to deal with projection updates, event versioning and the GDPR, and the tools offered by a popular framework (Axon).

Transcript

1. DEALING WITH CHANGE
 IN EVENT SOURCED APPLICATIONS Michiel Rook - @michieltcs
3. CHANGE
4. ' Welcome changing requirements, even late in development. -Agile Manifesto
5. MODIFICATIONS
6. EVENT SOURCING
7. CHALLENGE!
8. YOU
9. RAISE YOUR HAND IF YOU HAVE
10. read CQRS / Event Sourcing theory RAISE YOUR HAND IF YOU HAVE
11. read CQRS / Event Sourcing theory followed a tutorial, built a hobby project RAISE YOUR HAND IF YOU HAVE
12. read CQRS / Event Sourcing theory followed a tutorial, built a hobby project used it in production RAISE YOUR HAND IF YOU HAVE
14. QUICK RECAP
15. ' Event Sourcing ensures that all changes to application state are stored as a sequence of events. -Martin Fowler
16. ACTIVE RECORD VS. EVENT SOURCING Account Id Account number Balance 1234 12345678 €50,00 ... ... ... Money Withdrawn Account Id 1234 Amount €50,00 Money Deposited Account Id 1234 Amount €100,00 Account Opened Account Id 1234 Account number 12345678 @michieltcs
17. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00 @michieltcs 1 @Value 2 public class DepositMoney { 3 @TargetAggregateIdentifier 4 String accountId; 5 BigDecimal amount; 6 }
18. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00 command
 handler @michieltcs 1 @CommandHandler 2 public void depositMoney(DepositMoney command) { 3 apply(new MoneyDeposited( 4 command.getAccountId(), 5 command.getAmount(), 6 ZonedDateTime.now())); 7 }
19. COMMANDS TO EVENTS Deposit Money Account Id 1234 Amount €100,00 Money Deposited Account Id 1234 Amount €100,00 command
 handler @michieltcs 1 @Value 2 public class MoneyDeposited { 3 String accountId; 4 BigDecimal amount; 5 ZonedDateTime timestamp; 6 }
20. AGGREGATES @michieltcs an Aggregate handles Commands and generates Events based on the current state
21. AGGREGATES @michieltcs 1 class BankAccount { 2 @AggregateIdentifier 3 private String accountId; 4 private String accountNumber; 5 private BigDecimal balance; 6 7 // ... 8 @EventHandler 9 public void accountOpened(AccountOpened event) { 10 this.accountId = event.getAccountId(); 11 this.accountNumber = event.getAccountNumber(); 12 this.balance = BigDecimal.valueOf(0); 13 } 14 15 @EventHandler 16 public void moneyDeposited(MoneyDeposited event) { 17 this.balance = this.balance.add(event.getAmount()); 18 } 19 }
22. AGGREGATE STATE Account number Balance 12345678 €0,00 Account number Balance 12345678 €100,00 Account number Balance 12345678 €50,00 event
 handler event
 handler event
 handler @michieltcs Money Withdrawn Account Id 1234 Amount €50,00 Money Deposited Account Id 1234 Amount €100,00 Account Opened Account Id 1234 Account number 12345678
23. VALIDATING COMMANDS @michieltcs 1 @CommandHandler 2 public void withdrawMoney(WithdrawMoney command) throws 3 OverdraftDetectedException { 4 if (balance.compareTo(command.getAmount()) >= 0) { 5 apply(new MoneyWithdrawn( 6 command.getAccountId(), 7 command.getAmount(), 8 ZonedDateTime.now())); 9 } else { 10 throw new OverdraftDetectedException(accountNumber, balance, command. 11 getAmount()); 12 } 13 }
24. TESTING AGGREGATES @michieltcs 1 public class BankAccountTest { 2 private FixtureConfiguration<BankAccount> fixture; 3 4 @Before 5 public void createFixture() { 6 fixture = new AggregateTestFixture<>(BankAccount.class); 7 } 8 9 @Test 10 public void noOverdraftsOnEmptyAccount() { 11 fixture.given(new AccountOpened(ACCOUNT_ID, ACCOUNT_NUMBER)) 12 .when(new WithdrawMoney(ACCOUNT_ID, new BigDecimal(20))) 13 .expectException(OverdraftDetectedException.class); 14 } 15 16 private final static String ACCOUNT_ID = "accountId"; 17 private final static String ACCOUNT_NUMBER = "accountNumber"; 18 }
25. REPLAYS AND REBUILDS
26. ANSWERING QUERIES
27. BASED ON EVENTS
28. QUERIES Account Opened Account Opened Account Closed Number of active accounts? @michieltcs
29. QUERIES Money Deposited Money Withdrawn Interest Received Accounts with balance > €100? Money Deposited Money Withdrawn @michieltcs
30. PROJECTION Account Opened Event Handler # of active accounts +1 Account Closed Event Handler # of active accounts -1 @michieltcs
31. PROJECTION Events Event Handler(s) Storage @michieltcs
32. CQRS
33. UI @michieltcs
34. Domain UI Command commands Aggregates @michieltcs
35. Domain UI Command Repository Event Store commands events Aggregates @michieltcs
36. Domain UI Event Bus Event Handlers Command a Database Database Event Store Aggregates @michieltcs commands events events
37. Domain UI Event Bus Event Handlers Command Repository Data Layer Database Database Event Store commands events events queries DTOs Aggregates @michieltcs
38. MULTIPLE AGGREGATES Seller Event Seller Event Seller Event Seller Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Seller Name Listing Date Listing Description ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
39. PROJECTION @michieltcs 1 public class BankAccountProjections { 2 private Map<String, String> activeAccounts = new HashMap<>(); 3 4 @EventHandler 5 public void onAccountOpened(AccountOpened accountOpened) { 6 activeAccounts.put(accountOpened.getAccountId(), accountOpened. 7 getAccountNumber()); 8 } 9 10 @EventHandler 11 public void onAccountClosed(AccountClosed accountClosed) { 12 activeAccounts.remove(accountClosed.getAccountId()); 13 } 14 15 public String getAccountNumberForAccountId(String accountId) { 16 return activeAccounts.get(accountId); 17 } 18 }
40. PROJECTION @michieltcs 1 @GetMapping("accounts/{accountId}") 2 public String getAccountNumber(@PathVariable("accountId") String accountId) { 3 return bankAccountProjections.getAccountNumberForAccountId(accountId); 4 }
41. PROJECTION @michieltcs 1 public class BankAccountProjections { 2 private Map<String, String> activeAccounts = new HashMap<>(); 3 4 @EventHandler 5 public void onAccountOpened(AccountOpened accountOpened) { 6 activeAccounts.put(accountOpened.getAccountId(), accountOpened. 7 getAccountNumber()); 8 } 9 10 @EventHandler 11 public void onAccountClosed(AccountClosed accountClosed) { 12 activeAccounts.remove(accountClosed.getAccountId()); 13 } 14 15 public int getNumberOfActiveAccounts() { 16 return activeAccounts.size(); 17 } 18 }
42. PROJECTION @michieltcs 1 @GetMapping("accounts/active") 2 public int getNumberOfActiveAccounts() { 3 return bankAccountProjections.getNumberOfActiveAccounts(); 4 }
43. NEW PROJECTION
44. NEW STRUCTURE
45. BASED ON EXISTING EVENTS
46. REBUILDING Stop app Cleanup Loop over events Apply to projection Start app @michieltcs
47. ZERO DOWNTIME Loop over existing events Apply to new projection Use projection @michieltcs
48. ZERO DOWNTIME New events Queue Loop over existing events Apply to new projection Apply queued events Use projection @michieltcs
49. LONG RUNNING REBUILDS?
50. IN MEMORY
51. DISTRIBUTED
52. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate Event
53. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate Instance Event Instance
54. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate Instance Event Instance
55. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate Instance Event Event Event Event Instance
56. DIVIDING THE WORK Event Event Event Event Event @michieltcs Aggregate Instance Event Event Event Event Instance Event Event Event
57. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Seller Name Listing Date Listing Description ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
58. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Instance Instance
59. DIVIDING THE WORK Seller Event Seller Event Seller Event Seller Event Seller Event @michieltcs Listing Event Listing Event Listing Event Listing Event Listing Event Seller Listing Instance Instance ?
60. BACKGROUND TASK
61. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
62. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
63. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
64. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
65. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
66. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event Token Store
67. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
68. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
69. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
70. TRACKING EVENTS @michieltcs Event Event Event Event Event Event . . . . . . . . . . . . . . Event Event Event
71. TRACKING EVENT PROCESSOR Get next event Apply to new projection Last event? Use projection yes no @michieltcs
72. TRACKING EVENT PROCESSOR @michieltcs 1 @Target(ElementType.TYPE) 2 @Retention(RetentionPolicy.RUNTIME) 3 public @interface RebuildableProjection { 4 String version() default ""; 5 boolean rebuild() default false; 6 }
73. TRACKING EVENT PROCESSOR @michieltcs 1 @Configuration 2 public class ProjectionsConfiguration { 3 @Autowired 4 private EventHandlingConfiguration eventHandlingConfiguration;
74. TRACKING EVENT PROCESSOR @michieltcs 1 @PostConstruct 2 public void startTrackingProjections() throws ClassNotFoundException { 3 ClassPathScanningCandidateComponentProvider scanner = 4 new ClassPathScanningCandidateComponentProvider(false); 5 scanner.addIncludeFilter(new AnnotationTypeFilter(RebuildableProjection. 6 class)); 7 8 for (BeanDefinition bd : scanner.findCandidateComponents("org.demo")) { 9 Class<?> aClass = Class.forName(bd.getBeanClassName()); 10 RebuildableProjection rebuildableProjection = aClass.getAnnotation( 11 RebuildableProjection. 12 class); 13 14 if (rebuildableProjection.rebuild()) { 15 registerRebuildableProjection(aClass, rebuildableProjection); 16 } 17 } 18 }
75. TRACKING EVENT PROCESSOR @michieltcs 1 private void registerRebuildableProjection(Class<?> aClass, 2 RebuildableProjection 3 rebuildableProjection) { 4 ProcessingGroup processingGroup = aClass.getAnnotation(ProcessingGroup. 5 class); 6 7 String name = Optional.ofNullable(processingGroup).map(ProcessingGroup:: 8 value) 9 .orElse(aClass.getName() + "/" + rebuildableProjection.version()); 10 11 eventHandlingConfiguration.assignHandlersMatching( 12 name, 13 Integer.MAX_VALUE, 14 (eventHandler) -> aClass.isAssignableFrom(eventHandler.getClass())); 15 16 eventHandlingConfiguration.registerTrackingProcessor(name); 17 }
76. TRACKING EVENT PROCESSOR @michieltcs 1 private void registerRebuildableProjection(Class<?> aClass, 2 RebuildableProjection 3 rebuildableProjection) { 4 ProcessingGroup processingGroup = aClass.getAnnotation(ProcessingGroup. 5 class); 6 7 String name = Optional.ofNullable(processingGroup).map(ProcessingGroup:: 8 value) 9 .orElse(aClass.getName() + "/" + rebuildableProjection.version()); 10 11 eventHandlingConfiguration.assignHandlersMatching( 12 name, 13 Integer.MAX_VALUE, 14 (eventHandler) -> aClass.isAssignableFrom(eventHandler.getClass())); 15 16 eventHandlingConfiguration.registerTrackingProcessor(name); 17 }
77. EVENT VERSIONING
78. NEW BUSINESS REQUIREMENTS
79. CHANGING VIEW ON EVENTS
80. IRRELEVANT
81. DIFFERENT FIELDS
82. WRONG NAME
83. TOO COARSE
84. TOO FINE
85. SUPPORT YOUR LEGACY?
86. Commands can be renamed 1 @michieltcs
87. Commands can be renamed 1 Events are immutable 2 @michieltcs
88. Commands can be renamed 1 Events are immutable Correct old events with new events 2 3 @michieltcs
89. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00
90. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00 Ledger Entry Aug 14 Inventory €16500,00 Accounts Payable €16500,00
91. @michieltcs Ledger Entry Aug 14 Inventory €15600,00 Accounts Payable €15600,00 Ledger Entry Aug 14 Inventory €16500,00 Accounts Payable €16500,00 Ledger Correction Entry Aug 14 Inventory €900,00 Accounts Payable €900,00
92. COMPENSATING ACTIONS class MoneyWithdrawn {
 String accountId;
 BigDecimal amount;
 } class WithdrawalRolledBack {
 String accountId;
 BigDecimal amount;
 } Typo: too much withdrawn!
93. COMPENSATING ACTIONS class AccountOpened {
 String accountId;
 String accountNumber;
 } class DuplicateAccountClosed {
 String accountId;
 } Duplicate account number!
94. UPCASTING
95. UPCASTING Event Store @michieltcs
96. UPCASTING @michieltcs 1 @Value 2 @Revision("1.0") 3 public class AccountOpened { 4 String accountId; 5 String accountNumber; 6 }
97. UPCASTING @michieltcs <org.demo.AccountOpened>
 <accountId>80f49161</accountId>
 <accountNumber>12345678</accountNumber> </org.demo.AccountOpened>
98. UPCASTING Event Store AccountOpened (1.0) Event Handler @michieltcs
99. UPCASTING @michieltcs 1 @Value 2 @Revision("2.0") 3 public class AccountOpened { 4 String accountId; 5 String accountNumberIban; 6 }
100. UPCASTING @michieltcs <org.demo.AccountOpened>
 <accountId>80f49161</accountId>
 <accountNumberIban>NL00ABNA012345678</accountNumberIban>
 </org.demo.AccountOpened>
101. UPCASTING Event Store AccountOpened (1.0) Upcaster AccountOpened (2.0) Event Handler @michieltcs
102. UPCASTING @michieltcs 1 @Getter 2 public class AccountOpenedUpcaster implements EventUpcaster { 3 private final SerializedType typeConsumed 4 = new SimpleSerializedType(AccountOpened.class.getTypeName(), "1.0"); 5 private final SerializedType typeProduced 6 = new SimpleSerializedType(AccountOpened.class.getTypeName(), "2.0");
103. UPCASTING @michieltcs 1 @Override 2 public final Stream<IntermediateEventRepresentation> upcast( 3 Stream<IntermediateEventRepresentation> intermediateRepresentations) { 4 return intermediateRepresentations.map(evt -> { 5 if (evt.getType().equals(getTypeConsumed())) { 6 return evt.upcastPayload(getTypeProduced(), Document.class, 7 document -> { 8 Element rootElement = document.getRootElement(); 9 Element accountNumberElement = rootElement.element( 10 "accountNumber"); 11 rootElement.remove(accountNumberElement); 12 rootElement.addElement("accountNumberIban") 13 .setText(toIban(accountNumberElement.getText())); 14 return document; 15 }); 16 } else { 17 return evt; 18 } 19 }); 20 }
104. UPCASTING @michieltcs 1 @Override 2 public final Stream<IntermediateEventRepresentation> upcast( 3 Stream<IntermediateEventRepresentation> intermediateRepresentations) { 4 return intermediateRepresentations.map(evt -> { 5 if (evt.getType().equals(getTypeConsumed())) { 6 return evt.upcastPayload(getTypeProduced(), Document.class, 7 document -> { 8 Element rootElement = document.getRootElement(); 9 Element accountNumberElement = rootElement.element( 10 "accountNumber"); 11 rootElement.remove(accountNumberElement); 12 rootElement.addElement("accountNumberIban") 13 .setText(toIban(accountNumberElement.getText())); 14 return document; 15 }); 16 } else { 17 return evt; 18 } 19 }); 20 }
105. VERSIONED EVENT STORE
106. VERSIONED EVENT STORE events_v1 [
 {
 "id": "12345678",
 "type": "AccountOpened",
 "aggregateType": "Account",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payloadRevision": "1.0",
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
107. COPY & REPLACE
108. VERSIONED EVENT STORE Loop over existing events Apply upcaster Add queued events Use new event store New events Queue @michieltcs
109. VERSIONED EVENT STORE events_v2 [
 {
 "id": "12345678",
 "type": "AccountOpened",
 "aggregateType": "Account",
 "aggregateIdentifier": "1234",
 "sequenceNumber": 0,
 "payloadRevision": "2.0",
 "payload": { ... },
 "timestamp": ...
 ...
 },
 ...
 ] @michieltcs
110. GDPR
111. "RIGHT TO ERASURE"
112. ' ... shall have the right to obtain ... the erasure of personal data concerning him or her without undue delay -GDPR, Article 17
113. PERSONALLY IDENTIFIABLE INFORMATION
114. REMOVED
115. ANONYMIZED?
116. PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event store Remove from read models Notify 3rd parties
117. PROCESSING GDPR ART. 17 REQUESTS @michieltcs RightToErasureInvoked Remove from event store ? Remove from read models Notify 3rd parties
118. IMMUTABLE EVENTS?
119. MODIFY DIRECTLY
120. COPY WITH FILTER
121. STORE PII EXTERNALLY
122. STORE PII EXTERNALLY @michieltcs AccountOpened External Storage 1 @Value 2 public class AccountOpened { 3 String accountId; 4 } Account Id Account number Name 1234 12345678 John Doe ... ... ...
123. CRYPTO ERASURE
124. ENCRYPTING EVENTS @michieltcs <org.demo.AccountOpened>
 <accountId>80f49161</accountId>
 <accountNumberIban>NL00ABNA012345678</accountNumberIban>
 <firstName>Foo</firstName>
 <lastName>Bar</lastName> ...
 </org.demo.AccountOpened>
125. ENCRYPTING EVENTS @michieltcs <org.demo.AccountOpened>
 <accountId>80f49161</accountId>
 <accountNumberIban>2dqjHkY8Mc8+cek4vs/9hzgkob4J3fZJNIJh2sAXlJ0=</accountNumberIban>
 <firstName>N5Y27vd0UbKo6FIu5c7QGQ==</firstName>
 <lastName>OSKrzfuuuayuUNXYS5YUug==</lastName> ...
 </org.demo.AccountOpened>
126. ENCRYPTING EVENTS Generate event Find / create encryption key Encrypt payload values Store
 event @michieltcs
127. DECRYPTING EVENTS Load
 event Find associated encryption key Decrypt payload values Process
 event @michieltcs
128. SHEDDING THE KEY Load
 event Find associated encryption key Decrypt payload values Process
 event @michieltcs X
129. AXON GDPR MODULE @michieltcs 1 @Value 2 public class AccountOpened { 3 @DataSubjectId 4 String accountId; 5 6 @PersonalData 7 String accountNumberIban; 8 9 @PersonalData 10 String firstName; 11 12 @PersonalData 13 String lastName;
 14 }
130. CLOSING WORDS
131. NO SILVER BULLET
132. CHALLENGES
133. (IM)MUTABILITY
134. AUDIT TRAIL
135. SCALABILITY
136. TESTING
137. CQRS + ES = AWESOME
138. THANK YOU! @michieltcs / michiel@michielrook.nl
 
 www.michielrook.nl