$30 off During Our Annual Pro Sale. View Details »

Microservices - Where are my Transactions and my Consitency????

Eberhard Wolff
September 27, 2023

Microservices - Where are my Transactions and my Consitency????

Eberhard Wolff

September 27, 2023
Tweet

More Decks by Eberhard Wolff

Other Decks in Technology

Transcript

  1. Microservices: Where Are My
    Transactions And My
    Consistency???
    Eberhard Wolff
    Head of Architecture
    https://swaglab.rocks/
    https://ewolff.com/

    View Slide

  2. Microservices are
    Distributed Systems

    View Slide

  3. Distributed Systems Are Hard:
    Consistency & Reliability

    View Slide

  4. Happened-before
    ordering in
    distributed
    systems earned
    me a reward.
    “Time, Clocks and the Ordering of
    Events in a Distributed System”
    Leslie Lamport

    View Slide

  5. A distributed system is
    one in which the failure
    Leslie Lamport
    of a computer you didn't
    even know existed
    can render your own
    computer unusable.

    View Slide

  6. Databases have a
    hard time
    Kyle Kingsbury
    implementing
    distributed algorithms
    correctly
    https://aphyr.com/tag
    s/jepsen

    View Slide

  7. I am not nearly as smart as
    Leslie Lamport
    or Kyle Kingsbury.

    View Slide

  8. Avoid Consistency Issues!
    •Store each piece of data in one microservice
    •Microservice uses a database
    •Database solves the problem
    …and Kyle Kingsbury helps me to choose the right one.

    View Slide

  9. Entity Service
    Customer
    Service
    Product
    Service
    Order
    Process
    Delivery
    Process
    Invoicing
    Database for scaling
    …and consistency

    View Slide

  10. Entity Service
    • Every call goes through three services.
    • Performance
    • Latency

    View Slide

  11. Entity Service
    • Failure can easily propagate.
    • How can you make the system resilient?
    • Eberhard = default customer, PS5 = default product ?
    Customer
    Service
    Product
    Service
    Order
    Process
    Delivery
    Process
    Invoicing

    View Slide

  12. Transactions
    • Order process changes customer and product
    • Probably need a transaction
    Customer
    Service
    Product
    Service
    Order
    Process
    Delivery
    Process
    Invoicing

    View Slide

  13. View Slide

  14. ACID
    • Properties of classic transactions (tx)
    • Atomicity: Either do full tx full or nothing
    • Change customer and product
    • Consistency: No violated constraints
    • No negative number of products in stock
    • Otherwise: rollback

    View Slide

  15. ACID
    • Isolation: Parallel tx isolated
    • Other orders either processed before or after
    • i.e. no concurrent decrease of stock
    • Durability: Result of tx stored
    • e.g. # in stock stored

    View Slide

  16. ACID
    • Solves parallelism
    • …and increases reliability
    • Very desirable!

    View Slide

  17. Distributed Tx?

    View Slide

  18. Two Phase Commit (2PC): Phase 1
    Coordinator
    Resource
    Resource
    Coordinator identifies
    resources during
    transaction execution
    Can commit?
    Can commit?
    Yes/No
    Yes/No

    View Slide

  19. Two Phase Commit (2PC): Phase 2
    Coordinator
    Resource
    Resource
    Commit/Rollback
    Commit/Rollback
    Done
    Done
    Coordinator

    View Slide

  20. 2PC
    Customer
    Service
    Product
    Service
    Order
    Process
    Delivery
    Process
    Invoicing
    Transfer Tx ID
    via network
    Pass Tx ID to database
    Start commit / rollback
    Execute commit / rollback

    View Slide

  21. 2PC
    •Outdated solutions existed
    •Borland / Inprise ITS
    •CORBA OTS, JTA, C++ interfaces
    •Oracle, IMS, CICS, Tuxedo, MQ Series interfaces
    •🤔 Can I make a fortune implementing that for HTTP,
    gRPC…?

    View Slide

  22. 2PC
    •Problem: slows down the happy path
    •2nd phase
    + additional overhead

    View Slide

  23. 2PC
    •Problem: Blocks resource
    •Resource must be blocked until 2PC is fully done.
    •Potentially long
    •What if client crashes?
    …or if there are network problems?

    View Slide

  24. 2PC
    •Problem: no fully reliable
    Commit phase 2:
    What if resource is now unable to commit?
    Crashes of client / coordinator?
    •Tx should prevent problems
    •I.e. reliability is key

    View Slide

  25. 2PC
    •Not possible with current technologies
    •For good reason!

    View Slide

  26. 2PC = less performance
    + no 100% reliability

    View Slide

  27. 2PC: No option nowadays
    for good reasons

    View Slide

  28. Entity Service: DON’T
    • Bad performance
    • Bad reliability
    • Needs distributed tx – impossible
    Customer
    Service
    Product
    Service
    Order
    Process
    Delivery
    Process
    Invoicing

    View Slide

  29. Entity Service: DON’T

    View Slide

  30. ACID Transactions:
    DON’T and CAN’T

    View Slide

  31. Loose Coupling & Modules
    • Put everything for a set of features in one microservice
    • Including all data
    • Result: bounded context with loose coupling
    Order
    Process
    Including all
    customer & product
    data for orders
    Delivery
    Process
    All data for delivery
    Invoicing
    Process
    All data for invoicing

    View Slide

  32. Saga
    •Saga: Sequence of local transactions
    •Local transaction triggers next local transaction
    •Process done stepwise

    View Slide

  33. Saga
    •Failure: compensation
    •Compensation undoes the original transaction
    •e.g. cancel an order
    •Canceled order are stored
    •i.e. the state change is not undone

    View Slide

  34. Saga
    Order Process Delivery Process
    Invoicing
    Process

    View Slide

  35. Saga
    Order Process Delivery Process
    Invoicing
    Process
    Compensate!
    Reimburse

    View Slide

  36. Saga
    •Closer to what happens in the real world
    •No atomicity
    •No isolation
    •Also: not all steps have received the order
    •i.e. transaction in flight:
    some services have not received it yet
    •WS-Transaction standard (2002)

    View Slide

  37. Saga
    •Requires reliable communication
    •E.g. messaging
    + retransmission
    + idempotency

    View Slide

  38. 2PC instead of Saga
    Order Process Delivery Process
    Invoicing
    Process
    Rollback!
    Atomicity: All information about the order is gone!
    Rollback!

    View Slide

  39. What is the status of my
    order #42?
    WAT?????
    WAT?????????????
    There is no order #42.
    That happens if the order
    was not paid.
    It’s safer and more
    consistent that way.
    But we have no data - can’t
    tell you whether that was
    the problem.
    Transactions

    View Slide

  40. View Slide

  41. View Slide

  42. Saga vs 2PC
    •Saga might catch business logic better
    www.enterpriseintegrationpatterns.com/ramblings/18
    _starbucks.html
    •Sagas cause less problems if distributed
    •Don’t do 2PC!
    •If ACID is needed: Put it in one microservice!

    View Slide

  43. Consistency in DDD
    •Entity: Object distinguished by identity
    •Value object: distinguished by values
    •Aggregate: Cluster of entities and value object
    •Service: Process or transformation outside entity or
    value object
    •Transactions / ACID consistency: where?

    View Slide

  44. Consistency in DDD
    •Use the same aggregate boundaries to govern
    transactions and distribution.
    •Within an aggregate boundary,
    apply consistency rules synchronously.
    Across boundaries, handle updates asynchronously.
    •I.e. aggregates order, delivery, invoice inconsistent

    View Slide

  45. Consistency in DDD
    •Entities / value objects inside order aggregate are
    consistent
    •Aggregate in order process bounded context might be
    inconsistent
    •Bounded contexts might be inconsistent

    View Slide

  46. CAP

    View Slide

  47. CAP Theorem: Chose 2 of C, A, P
    Consistency
    Partition
    Tolerance
    Availability
    Quorum
    Replication
    DNS
    NOT POSSIBLE!*
    * https://codahale.com/you-cant-sacrifice-partition-tolerance

    View Slide

  48. BASE
    Basically
    Available
    Soft state
    Eventually consistent
    AP = Available not consistent

    View Slide

  49. ACID Consistency is not
    CAP Consistency!

    View Slide

  50. ACID vs CAP Consistency
    • ACID Consistency: No violated constraints
    • i.e. invariants, referential integrity etc
    • CAP consistency: All nodes have the same data

    View Slide

  51. What Did I Order?
    Order
    Process
    Invoicing Process
    Delivery
    Process

    View Slide

  52. Has the Invoice been Payed?
    Order
    Process
    Invoicing Process
    Delivery
    Process

    View Slide

  53. Where is my Delivery?
    Order
    Process
    Invoicing Process
    Delivery
    Process

    View Slide

  54. Where is the distribution?

    View Slide

  55. Order Accepted
    •Start delivery & invoicing
    •Possible inconsistency:
    invoice, no delivery yet
    Order
    Process
    Delivery
    Process
    Invoicing Process

    View Slide

  56. Delivery Not Possible
    •Cancel order
    •Reimburse!
    •Possible inconsistence: no reimbursement yet
    Order
    Process
    Delivery
    Process
    Invoicing Process

    View Slide

  57. Conclusion

    View Slide

  58. Conclusion
    •No ACID transactions between microservices
    •Saga: String of transactions + compensation
    •Saga might be the better option to model domain
    logic

    View Slide

  59. Conclusion
    •Aggregates are supposed to be ACID consistent,
    not bounded contexts.
    •CAP shouldn’t be a huge issue
    •All depends on good split of business logic!

    View Slide

  60. https://software-architektur.tv/2021/02/09/folge40.html

    View Slide

  61. View Slide

  62. https://software-architektur.tv/2020/11/13/folge025.html

    View Slide

  63. https://www.youtube.com/watch?v=1ELpfc4JOC0

    View Slide

  64. https://www.socreatory.com/de/trainers/eberhard-wolff
    19.2.2024 Software
    Architektur Kickstart

    View Slide