Microservice Pitfalls

The Microservice pattern has become very popular in recent years. It was a very good fit for the cultural shift in the industry towards a more agile, horizontal, autonomous way of building software. A few years in, microservices have since permeated even the enterprise world and some disillusion settled in, as long-term costs of the pattern emerge. To successfully leverage the potential of Microservices it is key to take the disadvantages into careful consideration and avoid common pitfalls.
Lothar has used the Microservice approach since 2015. In this he shares microservice drawbacks and approaches to work around those.

Topics:

- Contracts - Lawyer up! Ambiguities and unmet expectations
- Domains - None of your concern! Slicing microservices properly
- Distributed Systems - Your consensus is a house of cards
- Observability - How to X-Ray a hairball
- Micro or Macro Services - Empty vessels make the most noise
- API flavors - Which style would you prefer

Transcript

1. Addressing the most frequent pitfalls when transitioning to Microservices Microservices pitfalls
2. Lothar Schulz Head of Engineering lotharschulz.info lnkd.in/lotharschulz Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz Software Engineer github.com/mkulke lnkd.in/magnuskulke Magnus Kulke
3. Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz • Micro vs. Macro • Domains • Contracts • API Flavors • Distributed Systems • Observability
4. Micro or Macro Empty vessels make the most noise
5. Micro Services loosely coupled service oriented architecture with bounded contexts microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies. Adrian Cockcroft: State of the Art in Microservices James Lewis and Martin Fowler: Microservices Guide Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
6. Micro Services Modularisation - Modular programming focus independent, interchangeable modules Decomposition • Interfaces • Contracts • Packages • Data Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
7. Big Ball of Mud http://www.laputan.org/mud/ Forces - Time - Cost - Experience - Skill - Complexity - Change - Scale Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
8. Big Ball of Mud http://www.laputan.org/mud/ Throw away code Debug logs in Production Code HotFix Git Branch Selection/Cherry Picking Fixes NOT deployed Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
9. How small is micro ? Monoliths vs Microservices is Missing the Point—Start with Team Cognitive Load - Team Topologies https://speakerdeck.com/tastapod/microservices-software-that-fits-in-your-head?slide=62 Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
10. Domains None of your concern! Slicing microservices properly
11. Database as Microservice Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
12. Database as Microservice Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
13. Monolith first My Shop Find goods Buy goods Pay the goods Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
14. Domains Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2021 12 01 - Magnus Kulke/Lothar Schulz
15. Domains Scaling Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
16. Domains Scaling - Vertical Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
17. Domains Scaling - Vertical - Horizontal Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
18. Domains Scaling - Vertical - Horizontal - Partitioning Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
19. Domains Scaling - Vertical - Horizontal - Partitioning - Sharding Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
20. Domains - Bounded Contexts Book Search Pay Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
21. Domains - Bounded Contexts Book Search Pay Recommendation Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
22. Domains - Bounded Contexts Book Search Pay Recommendation Voucher Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
23. Domains - Bounded Contexts Book Search Pay Recommendation Voucher Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
24. Contracts Lawyer up! Ambiguities and Unmet Expectations
25. Microservices are (also/primarily?) a social tool • There is a relation between architecture and team setup • “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.” Conway’s Law • Enables teams to make autonomous decisions Remove placeholder Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
26. • Codify expectations towards an API from the consumer’s perspective ◦ Behaviour: does not change unexpectedly ◦ Availability: when can we retire an API? • How to express such a contract? ◦ Machine readable: Swagger/OpenAPI, JSON Schema, GraphQL ◦ API Versions • Abstain from breaking changes ◦ Additional properties? ◦ Extending enums? • Make everything optional: Protobuf3 Service Boundaries are Defined by Contracts Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
27. Problem: A Schema might not be expressive enough • Documents can be formally correct • But semantics have changed ◦ References in a document ◦ Content: New ID for entity • Pragmatic solution: Contract tests Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
28. 28 Performance Characteristics • Service level objectives • Rate limits • Request budgets Remove placeholder Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
29. • Unforeseen (ab)use patterns • How to attribute incoming traffic? ◦ Correlation Ids ◦ Callers need to tag their requests • Manage access ◦ Service Accounts ◦ Declarative: Service Mesh ◦ Rate Limiting The Other Side: Protection from Harmful Workloads Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
30. API flavors Which style would you prefer?
31. Remote Procedure Calls https://de.wikipedia.org/wiki/Remote_Method_Invocation Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
32. Remote Procedure Calls - GRPC Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz Consumer (Client) Book Info (Server) gRPC Stub gRPC Skel Protocol Buffer over HTTP/2 IDL Compiler Generate client stub Generate server Skeleton BookInfo.proto lib lib
33. HTTP APIs GET /books/ POST /books/ Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
34. REST Maturity Models - Level 0 - remote interactions - Level 1 - Resources - /books/123 - /books/123/author - Level 2 - HTTP Verbs - GET, POST, PUT, HEAD, PATCH, DELETE - Level 4 - Hypermedia Controls - <link rel=”slots/book” uri=”/slots/123” https://martinfowler.com/articles/richardsonMaturityModel.html Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
35. GraphQL Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
36. GraphQL https://www.lotharschulz.info/2020/03/18/how-to-break-github-graphql-pagination/ https://www.lotharschulz.info/2021/02/13/github-graphql-pagination-orderby-issue-fixed/ Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
37. Distributed Systems Your Consensus is a House of Cards
38. • Rate limits ◦ Urgent rollback, 3am ◦ Node cannot pull redis:latest 🙀 • DNS Load Balancing • DNS transport is UDP • UDP Packages are limited in size • Per Spec DNS allows <= 512 bytes Scenario 1: DockerHub Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
39. Scenario 1: DockerHub, cont. • DNS responses > 512 bytes fall back to TCP ◦ Your sysadmin might not know this ◦ Security Group blocks tcp/53 • Not all resolvers are alike / agree on the spec ◦ Glibc “salvages” truncated DNS messages ◦ Golang DNS resolver (Docker) does not ◦ Quick fix: CGO_ENABLED=1 Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
40. Scenario 2: DNS, again (it’s always DNS) • Our J2EE service is stuck in an exception loop ◦ Logs a lot of large stack traces (lots of lines) • Engineers integrate cool .io SaaS for tailing logs in Logstash ◦ Every line a request to cool .io data sink ◦ Every line a hostname is resolved • Cloud Providers disapproves, starts rate-limiting DNS for the service’s node • K8S api-server/node comm. is affected. ◦ Node is marked as broken ◦ Scheduler moved ever-crashing service to fresh, healthy node • Repeat Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
41. Scenario 3: Seemingly unlimited resources Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
42. Scenario 3: Seemingly unlimited resources, cont. ISPs & Facebook Outage, Oct 2021 • Facebook is removed from DNS servers worldwide • Gazillions of IoT devices, SmartTVs, etc are deployed and want to talk to facebook’s servers • Resolving fb fails, sloppy programs retry host resolution in a hot loop (no exponential backoff, no jitter) • ISPs DNS get overwhelmed by requests • Internet is 💥 not just fb Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
43. Observability How to X-Ray a hairball
46. 46 Tailor towards audience Example: - 24x7 - the engineering teams - Management - End customers Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
47. Intuition, experience, and an understanding of what engineers know about the services they serve is used to define - service level indicators (SLIs), - objectives (SLOs), - and agreements (SLAs). Service Level Objectives SRE Book - Service Level Objectives Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
48. - request latency - request response time and/or timeout rate - traffic / system throughput - demand placed on the system - http requests, static & dynamic - error rate - proportion of service errors - saturation - measures the system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). - availability - what’s the uptime of a service Guidance - The Four Golden SRE Signals SRE Book - The Four Golden Signals Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
49. Google's DevOps Research and Assessment (DORA) team DevOps Reports Guidance - DORA Key Metrics to measure DevOps Performance Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
50. key metrics that indicate the performance of a software development team: - Deployment Frequency - How often an organization successfully releases to production - Lead Time for Changes - The amount of time it takes a commit to get into production - Change Failure Rate - The percentage of deployments causing a failure in production - Time to Restore Service - How long it takes an organization to recover from a failure in production - Reliability - How long it takes an organization to recover from a failure in production Guidance - DORA Key Metrics to measure DevOps Performance DORA 2021 state of DevOps Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
51. Results Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
52. Results Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz
53. 53 Your Questions Please • Micro or Macro • Domains • Contracts • API Flavors • Distributed Systems • Observability Addressing the most frequent pitfalls when transitioning to Microservices - 2023 04 13 - Lothar Schulz