Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A journey from complexity to observability

A journey from complexity to observability

AWS Community Day, Dublin, May 25th, 2023

Managing the inherent complexity of software from development to production presents a significant challenge. Drawing insights from complexity theory, we explore how dependencies can lead to unexpected effects in software applications. By employing well-bounded components and smart design patterns, we can mitigate these dependencies and promote loose coupling. Given the vast array of interacting components in modern applications, we have to make sure applications are not just working, but also easy to understand and track. Join us to learn more about how to turn complexity into clarity.

Danilo Poccia

May 25, 2023
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. A journey from complexity to observability Danilo Poccia (he/him) Chief Evangelist (EMEA) AWS
  2. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Photo by Kalen Emsley on Unsplash
  3. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. https://github.com/SebLague/Ant-Simulation https://www.youtube.com/@SebastianLague
  4. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. “Complexity arises when the dependencies among the elements become important.” Scott E. Page, John H. Miller Complex Adaptive Systems
  5. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Monolithic Application Services Microservices
  6. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. How Amazon SQS works Front End Back End Metadata Amazon DynamoDB Load Manager
  7. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. “A complex system that works is invariably found to have evolved from a simple system that worked.” Gall’s Law
  8. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. “A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
  9. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. “Amazon S3 is intentionally built with a minimal feature set. The focus is on simplicity and robustness.” – Amazon S3 Press Release, March 14, 2006
  10. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Amazon S3 8 → more than 200 microservices Mai-Lan Tomsen Bukovec AWS Vice President (Storage, Streaming, Messaging, and Monitoring/Observability)
  11. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Teams building microservice architectures often find that integration with applications and external services can result in more monolithic and tightly coupled workloads
  12. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Coupling: Integration’s magic word Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and run-time Coupling isn’t binary Coupling isn’t one-dimensional A B Source: EnterpriseIntegrationPatterns.com
  13. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries Source: EnterpriseIntegrationPatterns.com
  14. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints. Gregor Hohpe Enterprise Integration Patterns
  15. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Receiver Request Response
  16. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Receiver Disadvantages • Receiver failure Request
  17. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Sender Sender Sender Sender Request Request Request Request Request Disadvantages • Receiver failure • Receiver throttled Receiver
  18. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • Decreases temporal coupling • Resilient to receiver failure • Receiver controls consumption rate Queue Ack Ack
  19. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • … • Dead-letter queue (DLQ) for errors Dead-letter queue Queue Ack Ack
  20. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (queue) Sender Receiver Receiver M1 M2 Ack Ack Advantages • … • More than one receiver consuming messages M1 M2 Ack
  21. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (queue) Sender Receiver Receiver M1 M2 Ack Ack Advantages • … • More than one receiver consuming messages Disadvantages • Response correlation • Backlog recovery time • Fairness in multi-tenant systems M1 M2 Ack
  22. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Receiver M2 M2 Purple channel Blue Blue Purple Purple
  23. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Disadvantages • Increases location coupling • Sender maintains routing logic Receiver M2 M2 Purple channel Blue Blue Simple color logic Purple Purple
  24. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous point-to-point model (router) Sender Receiver Disadvantages • Increases location coupling • Sender maintains routing logic • Sender complexity increases with time Complex color logic Receiver Receiver Receiver Receiver M1 M2 M1 M3 M1 M2 M3 M2 M3 M1
  25. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Asynchronous message-router (bus) Sender Receiver M1 Advantages • Reduces location coupling • Efficient for senders and receivers Receiver M2 Blue Purple Bus M1 M2 Ack Blue Purple Blue? Purple?
  26. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Event-driven architecture
  27. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. What is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2023-05-25 09:07:06 CustomerCreated 2023-05-25 09:07:09 OrderCreated 2023-05-25 09:07:13 PaymentSuccessful 2023-05-25 09:07:17 CustomerUpdated . . . . . .
  28. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Should you focus on the current status, or what is happening? Current status Domain model Commands Control ”CreateAccount” “AddProduct” What happens Domain events Event-driven Autonomy ”AccountCreated” “ProductAdded”
  29. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Commands Vs Events Command Has an intent Directed to a target Personal communication ”CreateAccount” “AddProduct” Event It’s a fact For others to observe Broadcast one to many ”AccountCreated” “ProductAdded”
  30. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment
  31. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment Notify me when an order is created Subscription
  32. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment Notify me when an order is created Subscription OrderCreated Notification
  33. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Orchestrate a business process within a domain, resulting in a published event Retail 1 Order up! Workflow Orchestrator InventoryManager InvoiceManager 2 Create order In stock?
  34. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Serverless Serverful
  35. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Don’t build a network of connected “black boxes” Building observable applications is a developer responsibility
  36. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Observability & Instrumentation • “Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings.” • “In order to be able to ask those questions of a system, the application must be properly instrumented. That is, the application code must emit signals such as traces, metrics, and logs.” • “An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need.” https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability
  37. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. OpenTelemetry An open-source framework for instrumenting, generating, collecting, and exporting telemetry data
  38. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. What is OpenTelemetry (OTel)? • CNCF popular project • Instrument once • Collect and aggregate • Send to multiple destinations • Cloud-centered framework • Supports hundreds of destinations and 11 programing languages
  39. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. OpenTelemetry data sources and protocol • Traces & Spans – What happens when a request is made • Metrics – A measurement about a service, captured at runtime • Logs – A timestamped text record with metadata • Baggage – Contextual information that’s passed between spans • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol
  40. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. OpenTelemetry reference architecture Code OTel Instrumentation Destination 1 Destination 2 Destination N … Receiver Exporter Processor OTel collector
  41. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Working with the open-source community
  42. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. AWS Distro for OpenTelemetry (ADOT) A secure, production-ready open-source distribution supported by AWS Certified by AWS for security and predictability Upstream-first distro of the popular CNCF project Backed by AWS support One-click deploy and configure from AWS container and AWS Lambda consoles Exporters for AWS monitoring solutions including – Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch Service, Prometheus, SigV4Auth, and partner solutions
  43. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. ADOT Public Roadmap
  44. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Tracing with AWS auto instrumentation agent ADD https://github.com/aws-observability/ aws-otel-java-instrumentation/releases/download/ version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar" ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp" ENV OTEL_IMR_EXPORT_INTERVAL="10000” ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" S A M P L E D O C K E R F I L E
  45. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Custom tracing attributes using annotations public class MyObservedClass { public void myObservedMethod(String param1, long param2) { <...> } } S A M P L E J A V A C O D E
  46. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Custom tracing attributes using annotations import io.opentelemetry.extension.annotations.WithSpan; import io.opentelemetry.extension.annotations.SpanAttribute; public class MyObservedClass { @WithSpan public void myObservedMethod(@SpanAttribute("param1") String param1, @SpanAttribute("param2") long param2) { <...> } } S A M P L E J A V A C O D E
  47. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Instrument once, send to multiple destinations P A R T N E R S
  48. © 2023, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Takeaways Complexity Microservices Integration Patterns → Loose coupling Observability OpenTelemetry Instrument once → Collect & analyze everywhere
  49. Thank you! © 2023, Amazon Web Services, Inc. or its

    affiliates. All rights reserved. Danilo Poccia @danilop