Distributed Sagas: A Protocol for Coordinating Microservices

Microservices have become the defacto architecture pattern for building services. However separating business logic into small services that operate with a single logical data set has introduced consistency challenges. Previous attempts to solve this problem like two phase commit have not been widely adopted due to availability and liveness issues.

Instead developers implement feral concurrency control mechanism. This technique can be error prone, and often results in “Death Star” architectures which rely on chained calls to enforce application invariants. These architectures become more complicated over time, and are difficult to modify and extend, and often don't correctly handle all failure scenarios.

In this talk I propose a new solution for this problem, Distributed Sagas, a protocol for coordinating requests among multiple micro services, while ensuring application invariants.

Transcript

1. Distributed Sagas A Protocol for Coordinating Microservices
2. Caitie McCaffrey Distributed Systems Engineer caitiem.com @caitie
3. Service Service Service Monoliths
4. Microservices & NoSQL
6. 2015 “Application-level Mechanisms for maintaining database integrity”
7. Reserve a Hotel Front End
8. Reserve a Hotel Front End Feral Concurrency Control Mechanisms
9. Reserve a Hotel Front End Feral Concurrency Control Mechanisms
10. Reserve a Car Front End
11. Reserve a Flight Front End
12. Trips Front End
13. Trips Front End Trips
14. Trips Front End Trips Feral Concurrency Control Mechanisms
15. Trips Front End Trips Feral Concurrency Control Mechanisms
16. Trips Front End Trips Feral Concurrency Control Mechanisms
17. Front End Trips Architecture
18. Death Star Architectures
19. Can We Do Better than Feral Concurrency Control?
21. 2012 “Spanner is Google’s scalable, multi- version, globally distributed, and synchronously-replicated database”
23. 2015 “The biggest barrier…is that consistency mechanisms must integrate across many stateful services”
24. Two Phase Commit
25. 2PC: Prepare Front End Trips Propose Book Hotel Propose Book Car Propose Book Flight
26. 2PC: Prepare Front End Trips Vote Book Hotel Vote Book Car Vote Book Flight
27. 2PC: Commit Front End Trips Commit/Abort Book Hotel Commit/Abort Book Car Commit/Abort Book Flight
28. 2PC: Commit Front End Trips Done Done Done
29. 2PC: Commit Front End Trips Done Done Done Doesn’t Scale •O (N^2) Messages in the worst case •Coordinator is a Single Point of Failure •Reduced Throughput
30. Distributed Sagas A Protocol for Coordinating Microservices
32. 1987 “Sagas are Long Lived Transactions [in a single relational Database]”
33. 1987 “A Saga is a Long Lived Transaction that can be written as a sequence of transactions that can be interleaved. All transactions in the sequence complete successfully or compensating transactions are ran to amend a partial execution.”
34. Distributed Sagas A Protocol for Coordinating Microservices
35. A Distributed Saga is a Collection of Requests Book Hotel Book Car Book Flight Charge Money
36. A Distributed Saga is a Collection of Requests Book Hotel Book Car Book Flight Charge Money and Compensating Requests Cancel Hotel Cancel Car Cancel Flight Refund Money
37. A Distributed Saga is a Collection of Requests Book Hotel Book Car Book Flight Charge Money and Compensating Requests Cancel Hotel Cancel Car Cancel Flight Refund Money that represent a single business level action
38. Distributed Saga Requests
39. Requests Can Abort Book Car
40. Requests Can Abort Book Car
41. Requests Must Be Idempotent Book Car
42. Requests Must Be Idempotent Book Car
43. Requests Must Be Idempotent Book Car
44. Requests Must Be Idempotent Book Car
45. Requests Must Be Idempotent Book Car
46. Requests Must Be Idempotent Book Car Timeout
47. Requests Must Be Idempotent Book Car
48. Requests Must Be Idempotent Book Car
49. Distributed Saga Compensating Requests
50. Compensating Requests Semantically undoes the effect of a request
51. Cancel Car Compensating Requests Cannot Abort
52. Cancel Car Compensating Requests Can Not Abort
53. Cancel Car Compensating Requests Must Be Idempotent
54. Compensating Requests Must Be Commutative with Requests Book Car Cancel Car Cancel Car Book Car is the same as
55. Compensating Requests Book Car Must Be Commutative with Requests
56. Compensating Requests Book Car Timeout Must Be Commutative with Requests
57. Compensating Requests Book Car Book Car Must Be Commutative with Requests
58. Compensating Requests Book Car Book Car Cancel Car Must Be Commutative with Requests
59. Compensating Requests Book Car Book Car Cancel Car Must Be Commutative with Requests
60. Compensating Requests Requests Idempotent Idempotent Commutative Can Not Abort Can Abort
61. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully
62. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
63. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car No Atomicity No Isolation
64. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car No Atomicity No Isolation Book Hotel Book Car Visible before Saga Completes
65. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
66. Distributed Saga defining a
67. Directed Acyclic Graph Distributed Saga Car Flight Hotel Payment Start Saga End Saga
68. Name: Hotel Request: Book Hotel Compensating Request: Cancel Hotel Status: Not Completed Vertex Distributed Saga
69. Start & End Vertices Distributed Saga Start Saga End Saga Car Flight Hotel Payment Start Saga End Saga
70. Distributed Saga Log fault-tolerant & highly available
71. Saga Execution Coordinator SEC Car Flight Hotel Payment Start Saga End Saga Saga Log
72. Distributed Saga Executing a
73. Car Flight Hotel Payment Start Saga End Saga Saga Log Book Trip Request SEC
74. Car Flight Hotel Payment Start Saga End Saga SEC Saga Log Start Saga Book Trip Request
75. Car Flight Hotel Payment Start Saga End Saga SEC Saga Log Done Start Saga Book Trip Request
76. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Start Hotel Start Saga
77. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Done Start Saga Start Hotel
78. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Book Hotel Request Start Saga Start Hotel
79. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Book Hotel Response Start Saga Start Hotel
80. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC End Hotel Start Saga Start Hotel
81. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Done Start Saga Start Hotel End Hotel
82. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Start Car Start Saga Start Hotel End Hotel
83. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Done Book Car Request Start Saga Start Hotel End Hotel Start Car
84. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car Book Car Response End Car
85. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car Done End Car
86. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight End Car
87. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car Done End Car Start Flight Book Flight Request
88. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Book Flight Response End Flight
89. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Done End Flight
90. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Start Payment End Flight
91. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight Done End Flight Start Payment Payment Request
92. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Payment End Flight Start Payment Payment Response
93. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment Done
94. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment End Saga
95. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment Done End Saga
96. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car End Car Start Flight End Flight Start Payment End Payment End Saga
97. Distributed Saga Failure of a
98. Car Flight Hotel Payment Start Saga End Saga Saga Log SEC Book Car Request Start Saga Start Hotel End Hotel Start Car Start Flight
99. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Book Car Response Abort Car Start Car Start Flight
100. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Done Start Car Start Flight Abort Car
101. Car Flight Hotel Payment Start Saga End Saga Saga Log Start Saga SEC Start Hotel End Hotel Done Start Car Start Flight Abort Car Rollback Recovery
102. Car Flight Hotel Payment End Comp Saga Start Comp Saga Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car
103. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
104. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Payment Log Entries? Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
105. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel { } Start Car Start Flight Abort Car Start Comp Saga End Comp Saga
106. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Car Log Entries? Start Comp Saga End Comp Saga
107. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car { Start, Abort } Start Comp Saga End Comp Saga
108. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Hotel Log Entries? Start Comp Saga End Comp Saga
109. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car { Start, End } Start Comp Saga End Comp Saga
110. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car { Start, End } Cancel Hotel Request Start Comp Saga End Comp Saga
111. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Cancel Hotel Response Comp Hotel Start Comp Saga End Comp Saga
112. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Done Comp Hotel Start Comp Saga End Comp Saga
113. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Flight Log Entries? Start Comp Saga End Comp Saga
114. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel { Start } Start Comp Saga End Comp Saga
115. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel { Start } Book Flight Request Start Comp Saga End Comp Saga
116. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel End Flight Book Flight Response Start Comp Saga End Comp Saga
117. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done Cancel Flight Request End Flight Start Comp Saga End Comp Saga
118. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Comp Flight Cancel Flight Response Start Comp Saga End Comp Saga End Flight
119. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done Comp Flight Start Comp Saga End Comp Saga End Flight Comp Flight
120. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel End Saga Start Comp Saga End Comp Saga Comp Flight End Flight Comp Flight
121. Car Flight Hotel Payment Saga Log Start Saga SEC Start Hotel End Hotel Start Car Start Flight Abort Car Comp Hotel Done End Saga Start Comp Saga End Comp Saga Comp Flight End Flight Comp Flight
122. Distributed Saga Guarantee Book Hotel Book Car Book Flight Charge Money All requests were completed successfully Or a subset of requests and the corresponding compensating requests were executed Book Hotel Book Car Cancel Hotel Cancel Car
123. Saga Execution Coordinator Recovering from Failure
124. Front End SEC Start Saga End Saga Saga Log
125. Front End SEC Start Saga End Saga Saga Log
126. Front End SEC Start Saga End Saga Saga Log
127. Front End SEC Saga Log Start Saga End Saga
128. with Distributed Sagas Isolation of Complex Code Fron Tr Complex Code Lives Everywhere Front SEC Saga Log Complex Code Lives Here Complex Code Lives Here
129. Fron Tr with Distributed Sagas Modular Services Front SEC Saga Log Only Flight Booking Flight Booking & Feral Concurrency Control Mechanisms
130. Fron Tr with Distributed Sagas Service Composition Front SEC Saga Log Just Add A New Saga! New Service & Feral Concurrency Control Mechanisms
131. Distributed Sagas Makes Building & Modifying Microservices Easier
132. Thank you @caitie