Maintaining consistency in distributed systems with an effect machine

In the beginning there was nothing. And the programmers said “let there be a monolith”. And a monolith was created, but it was coupled and not scalable, and the programmers saw that it was not good. Then they said “let there be microservices”. And microservices were created. At first it was good, but with microservices came chaos and problems, like: integrity, consistency, testing and concurrency issues. And the programmers thought “What if we could isolate all the side effects leveraging idempotency and functional programming to solve these issues?” So they said “let there be a side effect machine”. And a side effect machine was created. And It was very good. A magic new world was out there.

Transcript

1. Maintaining consistency in distributed systems with an effect machine
2. Carolina Pascale Software Engineer @ Nubank @CarolinaPascale github.com/carolpc instagram.com/carolpc
3. Marcelo Piva Software Engineer @ Nubank @mpivaa github.com/mpivaa instagram.com/m.pivaa
4. In the beginning there was nothing.
5. So the programmers said: “let there be a monolith”
6. And a monolith was created
7. But it was coupled and not scalable, and the programmers saw that it was not good.
8. But it was coupled and not scalable, and the programmers saw that it was not good.
9. Emergency-driven context switching
10. but it was coupled and not scalable, and the programmers saw that it was not good.
11. Hard to implement new architectural designs
12. Long onboarding for the newcomer
13. Biggest source of shared state: database as a global variable
14. Conway's law
15. But it was coupled and not scalable, and the programmers saw that it was not good.
16. So they said: “let there be microservices”.
17. and microservices were created.
18. At first it was good, but with microservices came chaos and problems
20. At first it was good, but with microservices came chaos and problems
23. consistency and testing issues
24. Idempotency
25. At least once
26. Replayable
28. Avoid effects duplication
29. Pure functions
30. Order of effects matter
31. A B C
32. A B C
33. A B C ❌
34. A B C ❌
35. A B C ❌
36. ❌
37. consistency and testing issues eventual
38. A B C
39. ❌ A B C
40. Coordinating
41. db-time db-time db-time A B C sync
42. db-time db-time db-time A B C
43. consistency and testing issues
44. Services need to communicate
46. IO is hard to test
47. Unit Integration E2E
48. Isolate side-effects
49. Pure functions 2
50. What can we do?
51. First attempt: hexagonal architecture
52. Inner layer: domain model
53. Second layer: application
54. Third layer: IO
55. Imperative code and side effects on the second layer
56. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
57. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
58. Why not imperative? (block! card datomic producer) ; =>
59. Why not imperative? (defn block! [card datomic producer] (when (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (datomic.card/update! blocked-card datomic) (producer.card/status-changed! blocked-card producer))))
60. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
61. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
62. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
63. Why not imperative? (defn block-all! [customer cards datomic producer] (doseq [card cards] (block! card datomic producer)) (cards-blocked-notify! customer producer))
64. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
65. Why not imperative? Mocks & Integration Tests
67. Why not imperative? Harder to reason about Harder to compose Harder to test Slower feedback
68. Tests ❌ Consistency ❌
69. Second attempt: Free monads
70. CODE
71. CODE AST
72. CODE AST RUNNER IMPLEMENTATION
74. Tests ✅ Consistency ❌
75. Not simple
76. Not easy
78. Third attempt
79. isolate all side effects leveraging functional programming
81. (defn run-effects! [message] (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t))))
82. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
83. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
84. A deterministic ID to each message
85. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
86. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
87. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
88. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
89. (let [[datoms messages] (handler message) [error db-t] (persist! datoms)] (case error :db-error (abort! error) :idempotency-error (send-messages! messages db-t) nil (send-messages! messages db-t)))
90. so they said: “let there be an effect machine”
91. And an effect machine was created
92. Decouple decisions from effects
93. Decouple intent from execution
95. Data as first class citizens
96. {:effects/datoms [{:db/id [:card/id "some-uuid"] :card/status :status/blocked}] :effects/requests [{:url :some-url :method :post}] :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}]}
97. {:effects/datoms [{:db/id [:card/id "some-uuid"] :card/status :status/blocked}] :effects/requests [{:url :some-url :method :post}] :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}]}
98. Declarative
99. (block card) ; => [{:effects/datoms [...] ; :effects/messages [...]}] Easier to reason about
100. (s/defschema BlockCard {:messages [(s/one StatusChanged)] :datoms [(s/one BlockedCard)]}) (defn block :- (s/either t.effects/BlockCard t.effects/Empty) [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
101. (defn block-all [customer cards] (-> (apply e/merge (map block cards)) (effect.card/card-blocked-notify customer))) Easier to compose
102. Atomic operations
103. (deftest block-card-test (is (= {:effects/datoms [{:db/id [:card/id "id"] :card/status :card/blocked}]} :effects/messages [{:topic :card-status-changed :message {:card/status :status/blocked}}] (block card)))) Easier to test
104. (defspec block-card-always-return-valid-effects-prop (props/for-all [card gen-card] (s/validate t.effects/BlockCard (block card)))) Easier to test
105. Global dependencies
106. Global dependencies Dependency injection
107. Dependency rejection Global dependencies Dependency injection
108. (defn block [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
109. (defn block [card] (if (logic.card/can-block? card) (let [blocked-card (logic.card/block card)] (-> e/empty-effects (effect.card/update blocked-card) (effect.card/status-changed blocked-card)))) e/empty-effects))
110. (s/defn run-effects! [effects :- Effects+Id components] (-> effects add-idempotency-check (run-datoms! components) (run-requests! components) after-run-requests (run-messages! components)))
111. Tests ✅ Consistency ✅
113. https://t.me/clojurebrasil https://www.meetup.com/pt-BR/ClojureSP/
114. Thanks! We're hiring! @mpivaa github.com/mpivaa instagram.com/m.pivaa https://boards.greenhouse.io/nubank @CarolinaPascale github.com/carolpc instagram.com/carolpc