Evolving Schemas Without Schema Evolution - Current 2022

Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility.

However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions.
Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today.

The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published.

We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.

Transcript

1. Evolving Schemas Without Schema Evolution
2. Abstract Schema Registry allows us to control how schemas can be evolved through time without breaking compatibility. However, when using high retention, you could end up with a convoluted “dirty” schema depending on the selected compatibility level, as well as many different models in your application logic to support all the schema versions. Upcaster chains allow you to read an old version of a message and bring it to what your logic needs today. The upcasters in the chain describe how to jump from one version to the next. They describe what your logic expects instead of covering all the possible variations that were ever published. We created an upcaster registry to allow you to declare upcasters and their chains in a declarative manner allowing them to be used across all your streaming apps. It is a great combo with Schema Registry; one providing the schemas themselves, the other expressing how to evolve from one schema version to the next.
3. Andreas Evers CTO @ KOR Financial Previously Pivotal / VMware ➢ Staff solution architect consulting for some of the biggest European firms ➢ Focused on application transformation & system design ➢ Worked on Spinnaker, together with the Spring team Who is this guy? @andreasevers
5. Mission Avoid another financial crisis by mapping out risk in financial markets
6. Selena Gomez explains “CDOs”, one of the financial products that were the cause of the 2008 crash
8. KOR Solution Real-time insights based on regulation to make financial products like CDOs transparent to the market and regulators.
11. Most companies use Kafka with a low retention setting, such as 7 days This works great for messaging use cases
12. Kafka can mean so much more for your company
13. The typical use case offloads the streamed events into an external database Those databases typically use commit logs or sequential files internally Such commit logs and sequential files are in fact very similar to Kafka topics and segment files 🤔 Why not use Kafka as the commit log for your data, and become the system of record instead of merely a tunnel
14. KSQL: Turning the database inside-out Source of truth stream Continuous query Derived table
15. High retention offers a lot of benefits, but also comes with its own unique challenges You can’t just wait 7 days for old formats to leave your cluster
16. How do we deal with Evolution?
17. Let’s start with short retention
19. Class People { - Rick - Morty }
20. Class People { - Rick - Morty }
21. - Rick - Morty Class People { - Rick - Morty } 0
22. - Rick - Morty Class People { - Rick - Morty } 0 Class People { - Rick - Morty }
23. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer }
24. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
25. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
26. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1
27. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty } 1
28. Class People { - Rick - Morty } - Rick - Morty 0 1 Class People { - Rick - Morty }
29. Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1 Class People { - Rick - Morty }
30. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1
31. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1 - Rick - Morty - Summer
32. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
33. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
34. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Rick - Morty - Summer 0 1
35. 1 Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty - Summer
36. 1 Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer
37. Isn’t this easier in HTTP APIs?
38. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty }
39. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty }
40. - Rick - Morty - Summer Class People { - Rick - Morty } Class People { - Rick - Morty - Summer }
41. Tolerant Reader Postel’s Law, a.k.a. the Robustness Principle: Be conservative in what you do, be liberal in what you accept from others.
42. Client Functions Client Stub Server Functions Server Stub (skeleton) Network Routines Network Routines kernel client process kernel server process network client server
43. Client Objects Client SOAP Handler Server Objects Server SOAP Handler Network Routines Network Routines network kernel client process kernel server process client server
44. SpecificRecord Avro Serdes SpecificRecord Avro Serdes Network Routines Network Routines network kernel client process kernel server process client server
45. mvn generate-sources Look ma, no programming!
46. - Rick - Morty - Summer Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } Be conservative in what you do, be liberal in what you accept from others. @JsonIgnoreProperties (ignoreUnknown = true)
47. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty - Summer } - Rick - Morty - Summer 1 Be conservative in what you do, be liberal in what you accept from others. Tolerant GenericRecordToDto Deserializer
48. What about some other changes?
49. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick } - Rick 1
50. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick } - Rick 1
51. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
52. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
53. - Rick - Morty 0 Class People { - PickleRick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
54. - Rick - Morty 0 Class People { - PickleRick - Morty } Class People { - PickleRick - Morty } - PickleRick - Morty 1
55. - Rick: String - Morty 0 Class People { - Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1
56. - Rick: String - Morty 0 Class People { - Rick: String - Morty } Class People { - Rick: boolean - Morty } - Rick: boolean - Morty 1
57. How can we prevent breaking consumers?
58. - Rick - Morty 0 Class People { - Rick - Morty } Class People { - Rick - Morty } 1
59. Class People { - Rick - Morty } - Rick - Morty 0 1 Class People { - Rick - Morty }
60. Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1 Class People { - Rick - Morty }
61. Class People { - Rick - Morty - Summer } Class People { - Rick - Morty - Optional<Summer> } - Rick - Morty 0 1
62. Consumer-driven Contract Testing Field-by-field compatibility testing
63. Consumer-driven Contract Testing Field-by-field compatibility testing
64. V2
65. V2 Consumer A is on version 2, only uses PickleRick
66. V2 Consumer A is on version 2, only uses PickleRick
67. V2 Consumer A is on version 2, only uses PickleRick
68. V2 Consumer A is on version 2, only uses PickleRick
69. V2 Consumer-Driven Contract Testing Consumer A is on version 2, only uses PickleRick
70. Consumer-Driven Contract Testing ❯ pact-broker can-i-deploy --pacticipant Foo --version 23 --to production
71. Schema Registry Automatic schema management and validation
72. Schema Evolution through Schema Registry Schema Registry • Versions every schema • Every schema has a unique ID • Uses subjects with Topic, Record, or TopicRecord names • For each subject, each schema change gets a version increase Schema ID: foo Schema ID: bar Subject topic-People - Rick - Morty - Rick - Morty - Summer V1 V2
73. Compatibility Types Compatibility Type Changes allowed Check against which schemas Upgrade first BACKWARD Delete fields Add optional fields Last version Consumers BACKWARD_TRANSITIVE Delete fields Add optional fields All previous versions Consumers FORWARD Add fields Delete optional fields Last version Producers FORWARD_TRANSITIVE Add fields Delete optional fields All previous versions Producers FULL Add optional fields Delete optional fields Last version Any order FULL_TRANSITIVE Add optional fields Delete optional fields All previous versions Any order NONE All changes are accepted Compatibility checking disabled Depends
74. Backward
75. Class People { - Rick - Optional<Summer> } Backward 0 1 - Rick - Morty - Rick - Optional<Summer>
76. Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } Backward - Rick - Morty 0 - Rick - Optional<Summer> 1
77. Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } Backward 0 1 - Rick - Morty - Rick - Optional<Summer>
78. Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } Backward 0 1 - Rick - Morty - Rick - Optional<Summer>
79. Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> Backward
80. Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> Backward
81. Class People { - Rick - Optional<Summer> } Class People { - Rick - Morty } - Rick - Morty Backward
82. Class People { - Rick - Optional<Summer> } Class People { - Rick - Morty } - Rick - Morty Backward
83. - Rick - Morty 0 Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
84. - Rick - Morty 0 Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
85. - Rick - Morty 0 Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
86. - Rick - Morty 0 Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
87. - Rick - Morty 0 Class People { - Rick - Optional<Summer> } Class People { - Rick - Optional<Summer> } - Rick - Optional<Summer> 1 - Rick 2 Backward Transitive
88. Forward
89. Class People { - Rick - Optional<Morty> } Class People { - Rick - Optional<Morty> } Forward - Rick - Optional<Morty>
90. Class People { - Rick - Optional<Morty> } Class People { - Rick - Optional<Morty> } Forward - Rick - Optional<Morty>
91. Class People { - Rick - Optional<Morty> } Class People { - Rick - Summer } Forward - Rick - Optional<Morty> - Rick - Summer
92. Class People { - Rick - Optional<Morty> } Class People { - Rick - Optional<Morty> } - Rick - Optional<Morty> Forward
93. Class People { - Rick - Optional<Morty> } Class People { - Rick - Summer } - Rick - Summer Forward
94. - Rick - Optional<Morty> 0 Class People { - Rick - Optional<Morty> } Class People { - Rick - Optional<Morty> } 1 2 Forward Transitive
95. - Rick - Optional<Morty> 0 Class People { - Rick - Optional<Morty> } Class People { - Rick - Optional<Morty> } 1 2 Forward Transitive
96. - Rick - Optional<Morty> 0 Class People { - Rick - Optional<Morty> } Class People { - Rick } 1 - Rick 2 Forward Transitive
97. - Rick - Optional<Morty> 0 Class People { - Rick - Optional<Morty> } Class People { - Rick - Summer } - Rick - Summer 1 - Rick 2 Forward Transitive
98. Short Retention = Full? Backward when • you want to allow the producer to roll back to a previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers and non-compatible upgrades are controlled, e.g. using CDCT)
99. High Retention = Full Transitive? Backward transitive when • you want to allow new consumers which will start at offset 0, to be added at any time in the future • or you want to allow producers to roll back to any previous version, writing a previous schema, without breaking consumers (provided they are tolerant readers) Forward transitive when • you want to allow consumers to read new versions without needing to be updated first (provided they are tolerant readers) regardless from where they start consuming
100. High Retention = Full Transitive forever? Class People { - Optional<Rick> - Optional<Morty> - Optional<Summer> - Optional<PickleRick> - Optional<Rick:boolean> }
101. Evolution strategies • Tolerant Reader • Versioned Events • Copy-Transform • Upcasting
102. Copy-Transform A.k.a. Copy & Replace A.k.a. The Parallel Universe
103. Copy-Transform
104. Copy-Transform
105. Copy-Transform
106. Copy-Transform
107. Copy-Transform
108. Copy-Transform
109. Copy-Transform
110. Copy-Transform Benefits Drawbacks Relatively simple to implement Complete freedom to do anything you want Data preservation One-time operation Performance
111. Upcasting A.k.a. Message Translators
113. Avro defaults give you wire-compatibility Upcasting gives you business-compatibility
114. - Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick
115. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick
116. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick
117. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3
118. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
119. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3
120. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3
121. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V2 → V3
122. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
123. - Rick - Morty Class People { - PickleRick - Summer } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V3 V1 → V2 V2 → V3 V2 → V3
124. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
125. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
126. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
127. What if some consumers are missing upcasters?
128. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2
129. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2
130. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 → V2
131. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
132. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
133. - Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2
134. - Rick - Morty Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V3 V2 V1 V3 V1 V2 V1
135. Upcaster Registry Centralized repository for up- and downcasters
136. Class People { - Rick - Morty } V1
137. Class People { - Rick - Morty } V1 - Rick - Morty V1
138. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1
139. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1
140. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick - Morty } V1 V1 V1
141. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2
142. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1
143. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
144. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
145. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2
146. - Rick - Morty Class People { - Rick - Morty } Class People { - Rick } V1 V1 V2 V1 → V2 V2 → V1 - Rick V2 V2 → V1
147. - Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2
148. - Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2
149. - Rick - Morty Class People { - Rick } Class People { - Rick } V1 V2 V2 V1 → V2 V2 → V1 - Rick V2 V1 → V2
150. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2
151. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } V1 V2 V3 V1 → V2 V2 → V1 - Rick V2 V3 → V2 V2 → V3
152. - Rick - Morty Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer - Rick V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3
153. https://github.com/KOR-Financial/kopper
154. - Rick - Morty 0 Class People { - Rick } Class People { - PickleRick - Summer } - PickleRick - Summer 1 - Rick 2 V1 V2 V3 V2 V1 → V2 V3 → V2 V1 → V2 V3 → V2 V2 → V1 V2 → V3 V3 Consumer A is on V2 Supports V1 and V3
155. ❯ pact-broker can-i-deploy --pacticipant Foo --version 23 --to production
156. Can we combine these evolution strategies?
157. Combining strategies for short retention • Versioned Events • Tolerant Reader • Consumer-driven Contract Testing
158. Combining strategies for short retention This allows you to: • Avoid transitive • Avoid the optional dance • Chase consumers to upgrade • Prevent producers from mistakenly releasing incompatible version after releasing compatible version • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need
159. Combining strategies for high retention • Versioned Events • Upcasting • Upcaster registry • Tolerant Reader • Copy-Transform
160. Combining strategies for high retention This allows you to: • Avoid Full Transitive Forever • Avoid every field being optional • Avoid having to write to countless deprecated fields • Keep your consumers tolerant to change • Keep your consumers ignorant of information they don’t need
161. Thank you! Accreditations: Graphics by Arthur Shlain, Creative Stal, Siddharth Dasari, Webtechops LLP, Icons Producer, SBTS from Noun Project Andreas Evers @andreasevers aevers@korfinancial.com KOR Financial www.korfinancial.com