Microservices need Microworkflows w-jax Munich 2023

Microworkflows for Microservices Complex Workflows in Distributed Applications 1

@[email protected] Why me? - Built my first message-driven, asynchronous system
for the Bundesbank starting 2001 - Worked on various event-driven systems, e.g. for Zalando, ING, ista, Maersk - Currently building a bank (!) with Event-driven Architecture, Microservices, DDD 3

@[email protected] Context 1/3 Business Applications (e.g. applications where people can
buy, rent, book something, perform business transactions) (not: IoT, Big Data…) 4

@[email protected] Context 2/3 Communication Between Services 5

@[email protected] 6

@[email protected] Context 3/3 Assuming a Preference for Event-driven (Reminder: all
the advantages of microservices, plus runtime decoupling) 7

@[email protected] So what have we looked at so far? Implementing
a workflow by: - Making one service the coordinator, or - Using a workflow engine for coordination, and - Implementing a saga (can be done with either of the above) What do all approaches have in common? 12

@[email protected] - Introduce a (complex) piece of software - Introduce
a split responsibility - Are not event-driven 13

@[email protected] Technical complexity: - Must be able to resume after
crash - Must be able to handle timeouts - Must be distributed to avoid single point of failure - Needs distributed coordination (stateful) - (Some of) your business logic is now in a YAML or XML file 14

@[email protected] Organizational complexity: - Business logic is split - Error
handling is split, leading to reduced clarity of responsibility if something goes wrong 15

@[email protected] 16 ❌ ❌

@[email protected] Reminder: Events vs. Commands 17 Is.. Command An intention
to perform an operation or change a state Event A fact, something that undisputedly happened in the past

@[email protected] Reminder: Events vs. Commands 18 Is.. Expected Response Communication
Pattern Command An intention to perform an operation or change a state A confirmation that the command has been executed, or an error message Request-Response Event A fact, something that undisputedly happened in the past None Fire-And-Forget

@[email protected] Q: How to implement complex workflows over multiple microservices?
A: Don’t. Instead, break up the workflow. To match actual event-driven architecture. Monolith -> Microservices Workflow -> Microworkflows 22

@[email protected] Q: But what about visibility of the end-to-end process?
A: Don’t mix control and observability! 25

@[email protected] Digression: What the “end” is, is very arbitrary. Does
it “end” when the parcel is sent? When it has arrived? When the goods have not been returned in 2 weeks? Where does one workflow end, and a new one start? It’s designed. 27

@[email protected] Q: But what about timeouts? A: - flaky connections,
retries - within service - processing times - SLA - service needs to scale, e.g. based on consumer lag - overall process - observer 28

@[email protected] Q: But what about error handling? A: Errors must
be handled within a service. The service gives a promise. This is a fundamental decoupling in event-driven architecture. 29

@[email protected] Events are “fire-and-forget”... … but based on previous agreements,
promises 30

@[email protected] Q: Surely, this only works for unrealistic, simplistic examples?
A: No, it’s just much harder to fit it on slides in a non-confusing way 󰤇. 31

@[email protected] Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 32

@[email protected] Image: https://blog.codecentric.de/wer-microservices-richtig-macht-braucht-keine-workflow-engine-und-kein-bpmn 33

@[email protected] Challenge accepted! Send your workflow to: [email protected] 34

@[email protected] Technology is not the problem, it’s a mindset issue.
Think Promises. 35

@[email protected] Some ideas seem outlandish, but they’re nothing new. Prior
art: - Unix pipes - Actors - Stream Processing 36

@[email protected] Think Unix Philosophy 1. Make each program do one
thing well. To do a new job, build afresh rather than complicate old programs by adding new "features". 2. Expect the output of every program to become the input to another, as yet unknown, program. ... 37 $ Cat file3.txt | grep “dwx” | tee file4.txt | wc –l

@[email protected] Think Actors 38 Supervisor Actor Client

@[email protected] Think Stream Processing 39

@[email protected] Enjoy the organizational clarity 40 ❌ ❌

@[email protected] Enjoy the (load) testing 41 While you’re freed from
having e.g. the workflow engine in your end-to-end test, there are other challenges. Also: Testing a microworkflow is still an integration test. But it’s nice to be able to test e.g. throughput in a very isolated way.

@[email protected] Orchestration vs. Choreography While these terms might be fitting
to describe the concepts, most comparisons are not useful, but based on some folkloristic concerns. Hence I tried to avoid them. 43

@[email protected] Example Heuristic “Complex workflows..” Problem: Taking a workflow as
given, not as something that is designed and can be changed. 44

@[email protected] Think reducing complexity, not managing it. https://twitter.com/bitfield/status/1411731604335214592?ref_src=twsrc%5Etfw 45

@[email protected] Think Agile. And then putting SAFE on top of
it. 46

@[email protected] Workflows are useful tools. Workflow engines too, probably. But
not for implementing complex workflows on top of Microservices. That use case shouldn’t exist. Consider going event-driven instead. Break the workflow down into microworkflows. 47

@[email protected] People are quick to jump to sagas and workflow
engines! - Is it someone who’s familiar with it, but not with EDA, and is scared to leave their comfort zone? - Is it someone who wants to sell you something? 48

@[email protected] Summary: Workflows and Sagas are at odds with event
driven architecture. If you really want to get the benefits of event-driven architecture, go all-in, don’t impose an orchestrator on top of it. Keep your event-driven architecture sane. 49

Let’s Discuss! Follow me on Mastodon @[email protected] Follow me on
LinkedIn https://linkedin.com/in/lutzh Read my blog https://www.reactivesystems.eu/ 50

Microservices need Microworkflows w-jax Munich ...

Microservices need Microworkflows w-jax Munich 2023

More Decks by Lutz Hühnken

Other Decks in Programming

Featured

Transcript