A journey from complexity to observability

AWS Community Day, Dublin, May 25th, 2023

Managing the inherent complexity of software from development to production presents a significant challenge. Drawing insights from complexity theory, we explore how dependencies can lead to unexpected effects in software applications. By employing well-bounded components and smart design patterns, we can mitigate these dependencies and promote loose coupling. Given the vast array of interacting components in modern applications, we have to make sure applications are not just working, but also easy to understand and track. Join us to learn more about how to turn complexity into clarity.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. A journey from complexity to observability Danilo Poccia (he/him) Chief Evangelist (EMEA) AWS
2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Photo by Kalen Emsley on Unsplash
3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://github.com/SebLague/Ant-Simulation https://www.youtube.com/@SebastianLague
4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Complexity arises when the dependencies among the elements become important.” Scott E. Page, John H. Miller Complex Adaptive Systems
5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monolithic Application Services Microservices
6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Amazon SQS works Front End Back End Metadata Amazon DynamoDB Load Manager
7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “A complex system that works is invariably found to have evolved from a simple system that worked.” Gall’s Law
8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Amazon S3 is intentionally built with a minimal feature set. The focus is on simplicity and robustness.” – Amazon S3 Press Release, March 14, 2006
10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 8 → more than 200 microservices Mai-Lan Tomsen Bukovec AWS Vice President (Storage, Streaming, Messaging, and Monitoring/Observability)
11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Teams building microservice architectures often find that integration with applications and external services can result in more monolithic and tightly coupled workloads
12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Coupling: Integration’s magic word Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and run-time Coupling isn’t binary Coupling isn’t one-dimensional A B Source: EnterpriseIntegrationPatterns.com
13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The many facets of coupling • Technology dependency: Java vs. C++ • Location dependency: IP addresses, DNS • Data format dependency: Binary, XML, JSON, ProtoBuf, Avro • Data type dependency: int16, int32, string, UTF-8, null, empty • Semantic dependency: Name, middle name, ZIP • Temporal dependency: sync, async • Interaction style dependency: messaging, RPC, query-style (GraphQL) • Conversation dependency: pagination, caching, retries Source: EnterpriseIntegrationPatterns.com
14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints. Gregor Hohpe Enterprise Integration Patterns
15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Receiver Request Response
16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Receiver Disadvantages • Receiver failure Request
17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Synchronous request-response model Advantages • Low latency • Simple • Fail fast Sender Sender Sender Sender Sender Request Request Request Request Request Disadvantages • Receiver failure • Receiver throttled Receiver
18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • Decreases temporal coupling • Resilient to receiver failure • Receiver controls consumption rate Queue Ack Ack
19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver M1 M2 M1 M2 Advantages • … • Dead-letter queue (DLQ) for errors Dead-letter queue Queue Ack Ack
20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver Receiver M1 M2 Ack Ack Advantages • … • More than one receiver consuming messages M1 M2 Ack
21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (queue) Sender Receiver Receiver M1 M2 Ack Ack Advantages • … • More than one receiver consuming messages Disadvantages • Response correlation • Backlog recovery time • Fairness in multi-tenant systems M1 M2 Ack
22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Receiver M2 M2 Purple channel Blue Blue Purple Purple
23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver M1 M1 Blue channel Disadvantages • Increases location coupling • Sender maintains routing logic Receiver M2 M2 Purple channel Blue Blue Simple color logic Purple Purple
24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous point-to-point model (router) Sender Receiver Disadvantages • Increases location coupling • Sender maintains routing logic • Sender complexity increases with time Complex color logic Receiver Receiver Receiver Receiver M1 M2 M1 M3 M1 M2 M3 M2 M3 M1
25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Asynchronous message-router (bus) Sender Receiver M1 Advantages • Reduces location coupling • Efficient for senders and receivers Receiver M2 Blue Purple Bus M1 M2 Ack Blue Purple Blue? Purple?
26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Event-driven architecture
27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is an “event” ? “something that happens” Events tell us a fact Immutable time series Time What 2023-05-25 09:07:06 CustomerCreated 2023-05-25 09:07:09 OrderCreated 2023-05-25 09:07:13 PaymentSuccessful 2023-05-25 09:07:17 CustomerUpdated . . . . . .
28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Should you focus on the current status, or what is happening? Current status Domain model Commands Control ”CreateAccount” “AddProduct” What happens Domain events Event-driven Autonomy ”AccountCreated” “ProductAdded”
29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Commands Vs Events Command Has an intent Directed to a target Personal communication ”CreateAccount” “AddProduct” Event It’s a fact For others to observe Broadcast one to many ”AccountCreated” “ProductAdded”
30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment
31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment Notify me when an order is created Subscription
32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Choreograph events between domains using subscriptions Retail Fulfillment Notify me when an order is created Subscription OrderCreated Notification
33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Orchestrate a business process within a domain, resulting in a published event Retail 1 Order up! Workflow Orchestrator InventoryManager InvoiceManager 2 Create order In stock?
34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless Serverful
36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Don’t build a network of connected “black boxes” Building observable applications is a developer responsibility
37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Observability & Instrumentation • “Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings.” • “In order to be able to ask those questions of a system, the application must be properly instrumented. That is, the application code must emit signals such as traces, metrics, and logs.” • “An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need.” https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability
38. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry An open-source framework for instrumenting, generating, collecting, and exporting telemetry data
39. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is OpenTelemetry (OTel)? • CNCF popular project • Instrument once • Collect and aggregate • Send to multiple destinations • Cloud-centered framework • Supports hundreds of destinations and 11 programing languages
40. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry data sources and protocol • Traces & Spans – What happens when a request is made • Metrics – A measurement about a service, captured at runtime • Logs – A timestamped text record with metadata • Baggage – Contextual information that’s passed between spans • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol
41. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry reference architecture Code OTel Instrumentation Destination 1 Destination 2 Destination N … Receiver Exporter Processor OTel collector
42. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working with the open-source community
43. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Distro for OpenTelemetry (ADOT) A secure, production-ready open-source distribution supported by AWS Certified by AWS for security and predictability Upstream-first distro of the popular CNCF project Backed by AWS support One-click deploy and configure from AWS container and AWS Lambda consoles Exporters for AWS monitoring solutions including – Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch Service, Prometheus, SigV4Auth, and partner solutions
44. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ADOT Public Roadmap
45. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing with AWS auto instrumentation agent ADD https://github.com/aws-observability/ aws-otel-java-instrumentation/releases/download/ version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar" ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp" ENV OTEL_IMR_EXPORT_INTERVAL="10000” ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" S A M P L E D O C K E R F I L E
46. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Custom tracing attributes using annotations public class MyObservedClass { public void myObservedMethod(String param1, long param2) { <...> } } S A M P L E J A V A C O D E
47. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Custom tracing attributes using annotations import io.opentelemetry.extension.annotations.WithSpan; import io.opentelemetry.extension.annotations.SpanAttribute; public class MyObservedClass { @WithSpan public void myObservedMethod(@SpanAttribute("param1") String param1, @SpanAttribute("param2") long param2) { <...> } } S A M P L E J A V A C O D E
48. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instrument once, send to multiple destinations P A R T N E R S
49. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Takeaways Complexity Microservices Integration Patterns → Loose coupling Observability OpenTelemetry Instrument once → Collect & analyze everywhere
50. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Danilo Poccia @danilop