Upgrade to Pro — share decks privately, control downloads, hide ads and more …

KanDDDinsky 2017 - The Dark Side of Event Sourc...

KanDDDinsky 2017 - The Dark Side of Event Sourcing: Managing Data Conversion

Evolving software systems often includes data schema changes, and those cause the need for data conversion. This is no different in event sourced systems.
Converting data between schemas while continuing the operation of the system is a challenge.
In this talk, we will look at the possible conversion operations, the techniques to execute them, and strategies for deployment with zero-downtime.
We will look at the possibilities and the trade-offs for designing and implementing upgrades for event sourced systems.

Michiel Overeem

October 20, 2017
Tweet

More Decks by Michiel Overeem

Other Decks in Technology

Transcript

  1. Structure of an Event Store Store A (distributed) collection of

    streams. An ordered list of events Stream An ordered list of events. Event A name, attributes with values, and metadata.
  2. The schema is implicit and hidden! Which streams are known

    in the system? Which events are allowed in which streams? Which attributes are allowed in which events?
  3. Webshop EventStore Customer Stream ShoppingCart Stream CustomerSignedUp { … }

    UpgradedCustomerToPremiumService { … } ShoppingCartCreated { … } ArticleRemovedFromChart { … } ArticleAddedToChart { … }
  4. Change will happen “An software system written to perform some

    real-world activity must be continually adapted or it becomes progressively less satisfactory” Lehman’s law of continuing change
  5. Add attribute to event Customer Stream CustomerSignedUp { … }

    UpgradedCustomerToPremiumService { … }
  6. Add attribute to event Customer Stream CustomerSignedUp { …, newsletter:

    <bool> } UpgradedCustomerToPremiumService { … }
  7. Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> }

    UpgradedCustomerToPremiumService { … } CustomerSignedUp { …, fullName: <string> } Split attribute
  8. Customer Stream CustomerSignedUp { …, firstName: <string>, surName: <string> }

    UpgradedCustomerToPremiumService { … } Split attribute
  9. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … }

    CustomerSignedUp { …, newsletter: <bool> } Split events
  10. Customer Stream CustomerSignedUp { … } UpgradedCustomerToPremiumService { … }

    OptInOnNewsLetter { … } CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Split streams
  11. CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } Customer Stream

    CustomerSignedUp { … } UpgradedCustomerToPremiumService { … } NewsletterSubscription Stream OptInOnNewsLetter { … } Split streams
  12. Store level operations Add / remove streams Merge streams /

    split / … streams Stream level operations Add / remove events Merge events / split / … events Event level operations Add / update / remove attributes Merge attributes / split / … attributes The operations
  13. Our codebase would only grow. Blue Cross Blue Shield Tower

    under construction in Chicago, by Photogonal, wikipedia.org
  14. CustomerSignedUp { “customerId”: …, “fullName”: … } Weak schema &

    serialization CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: … }
  15. CustomerSignedUp { “customerId”: …, “fullName”: … } Upcasting CustomerSignedUp {

    “customerId”: …, “firstName”: …, “surName”: … } CustomerSignedUp { “customerId”: …, “firstName”: …, “surName”: …, “newsletter”: … } Upcaster Upcaster
  16. Store A (distributed) collection of streams. An ordered list of

    events Store level Stream level Event level Add / remove streams Merge streams / split / … streams Should we perform all operations at runtime?
  17. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
  18. Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Functional suitability +/- - +/- +/- + +
  19. Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Maintainability - +/- +/- +/- + +
  20. Always extend Weak schema Upcasting Lazy transformation In place transformation

    Copy and transformation Performance efficiency + + + +/- +/- -
  21. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform
  22. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform These techniques are executed within the application These techniques need a separate stage
  23. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade
  24. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  25. Event operations Stream operations Store operations simple Store operations complex

    In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  26. Event operations Stream operations Store operations simple Store operations complex

    In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green
  27. Event operations Stream operations Store operations simple Store operations complex

    Always extend Weak schema Upcasting Lazy transformation In place transformation Copy and transform Big Flip Blue / Green Rolling Upgrade Expand & Contract Blue / Green Mix & Match!