Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Underestimated costs of microservice architectures

Underestimated costs of microservice architectures

With many business success stories, our beautiful software systems can degrade into monolithic Big Balls of Mud. And to fix these kinds of monstrosities, we as developers and architects have begun to reach for microservices as our solution. Beautifully-designed architecture diagrams and org charts clearly show the benefits in terms of coordination, batch size, and codebase understandability, but is it really all unicorns and rainbows?

As folks who have been down this road can tell you, microservice architectures don’t solve all our problems. Tradeoffs abound, and in this talk we’ll see the costs we need to be prepared to pay when we introduce microservices, including team dynamics as well as technical tradeoffs around consistency, failure handling, and observability.

Colin Jones

May 02, 2018
Tweet

More Decks by Colin Jones

Other Decks in Technology

Transcript

  1. On other hand But to gain any benefit from microservice

    thinking, you have to understand what it is, how to do it, and why you should usually do something else. - Martin Fowler
  2. eliminate legacy code independent deployability independent scalability fault tolerance avoid

    dependency hell architectural boundaries small team ownership
  3. eliminate legacy code independent deployability independent scalability fault tolerance avoid

    dependency hell architectural boundaries small team ownership microservices!
  4. Additional infrastructure Mitigations • Containers (e.g. Docker) • Infrastructure automation

    & configuration management • Virtual machines / cloud • Auto-scaling (metered cost) • Serverless
  5. Data consistency Mitigations • Design for eventual consistency • Canonical

    source for data (aka “system of record” / “source of truth”) and derived data • Backend sync processes • Service teams co-own ETL for analytics/ business intelligence / data warehouse
  6. Failure modes Mitigations • Use retries (with backoff; cap the

    max time) • Read the remote end to see if it succeeded • Use fallbacks for read timeouts • Use circuit breaker to limit cascading failures • Use bulkheads to protect independent modules
  7. Mitigations • Expand testing mindset to staging/production observability efforts •

    Integrate only a few services / API checking and rely more on unit tests • Unify on lightweight runtimes and single database technology [only delays the problem; conflicts with team ownership] • Test against external environments with services set up [risks test pollution] • Orchestrate new isolated infrastructure for each test run Development & testing
  8. Observability Mitigations • Log aggregation with correlation IDs • Error

    reporting / alerting [generally on symptoms, not causes] • Distributed tracing tools • Monitoring tools / dashboards • Fancier 3rd-party observability tools
  9. Mitigations • Measure business metrics, not team velocity • Make

    sure team / service incentives are aligned with the company’s • Rotations / team exchanges / dynamic re- teaming • Cross-org communication Tunnel vision
  10. Implicit connection data Failure modes Development & testing Observability Tunnel

    vision Data consistency Additional infrastructure Latency
  11. Mitigations • Well known API contracts / specifications (e.g. JSON

    Schema, Swagger, Protocol Buffers, Thrift) • Centralized/standardized repository to track service metadata for service discovery • Put all services in one codebase (monorepo) for easier searchability • Custom tooling based on log aggregation / monitoring Implicit connection data
  12. Inter-team priority conflicts Failure modes Development & testing Observability Tunnel

    vision Implicit connection data Data consistency Additional infrastructure Latency
  13. Mitigations • Make our case really well to the service

    team, or management • Add staff on heavily-used microservices • Split heavily-used services further • Contribute to their project (aka “internal open- source”) • Rebuild a similar service with the changes we need Inter-team priority conflicts
  14. Hard to change across boundaries Failure modes Development & testing

    Observability Tunnel vision Inter-team priority conflicts Implicit connection data Data consistency Additional infrastructure Latency
  15. Mitigations • Be deliberate about the choice to Extract Microservice

    • Version your API? [controversial] • If versioning / breaking: Have a well-defined way to communicate breaking changes / deadlines • Sticking with the same runtime (e.g. JVM) makes Inline Microservice possible • Cross-org communication Hard to change across boundaries
  16. Mitigations • Skill and culture of backwards compatibility (SemVer, Postel’s

    Law) • Don’t make breaking changes • Well known API contracts / specifications (e.g. JSON Schema, Swagger, Protocol Buffers, Thrift) • Consumer-driven contract tests in CI • See also “Connection data is implicit” mitigations Hard to change across boundaries
  17. Failure modes Development & testing Observability Tunnel vision Inter-team priority

    conflicts Implicit connection data Data consistency Additional infrastructure Latency Hard to change across boundaries microservices!
  18. Learn more •Ben Christensen. “Don’t Build a Distributed Monolith”: https://

    www.microservices.com/talks/dont-build-a-distributed-monolith/ •Michael Feathers. “Microservices and the Failure of Encapsulation”: https:// michaelfeathers.silvrback.com/microservices-and-the-failure-of-encapsulaton •Michael Feathers. Working Effectively with Legacy Code: https:// www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/ 0131177052 •Martin Fowler. Enterprise Application Architecture: https:// www.martinfowler.com/books/eaa.html •Martin Fowler. “MicroservicePremium”: https://www.martinfowler.com/ bliki/MicroservicePremium.html •Martin Fowler. “Microservice Prerequisites”: https://www.martinfowler.com/ bliki/MicroservicePrerequisites.html •Martin Fowler. “Microservices Resource Guide”: https:// www.martinfowler.com/microservices/
  19. Learn more •Susan Fowler. Production-Ready Microservices: http://shop.oreilly.com/product/ 0636920053675.do •John Gall.

    The Systems Bible: https://www.amazon.com/Systems-Bible-Beginners- Guide-Large/dp/0961825170 •David Heinemeier Hansson. “The Majestic Monolith”: https://m.signalvnoise.com/ the-majestic-monolith-29166d022228 •Rich Hickey. “Hammock Driven Development”: https://www.youtube.com/watch? v=f84n5oFoZBc •Gregor Hohpe and Bobby Woolf. Enterprise Integration Patterns: http:// www.enterpriseintegrationpatterns.com/ •Mike Knepper. “The Hidden Costs of Leaving a Monolith”: https://8thlight.com/ blog/mike-knepper/2016/01/20/hidden-costs-of-leaving-a-monolith.html •Dan Manges. “The Modular Monolith: Rails Architecture”: https://medium.com/ @dan_manges/the-modular-monolith-rails-architecture-fb1023826fc4 •Sam Newman. Building Microservices: https://samnewman.io/books/ building_microservices/
  20. Learn more •Michael Nygard. “The Entity Microservice Antipattern”: http:// www.michaelnygard.com/blog/2017/12/the-entity-service-antipattern/

    •Michael Nygard. Release It!, 2nd edition: https://pragprog.com/book/mnee2/ release-it-second-edition •Ozan Onay. “You are not Google”: https://blog.bradfieldcs.com/you-are-not- google-84912cf44afb •Arnon Rotem-Gal-Oz. “Fallacies of Distributed Computing Explained”: http:// www.rgoarchitects.com/Files/fallacies.pdf •Cindy Sridharan. “Testing Microservices, the Sane Way”: https://medium.com/ @copyconstruct/testing-microservices-the-sane-way-9bb31d158c16 •Cindy Sridharan. “Testing in Production, the safe way”: https://medium.com/ @copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1 •Jim Waldo, Geoff Wyant, Ann Wolrath, and Sam Kendall. “A Note on Distributed Computing”: http://web.cs.wpi.edu/~cs3013/a11/Papers/ Waldo_NoteOnDistributedComputing.pdf