From complexity to observability using OpenTelemetry

CloudConf, Turin, May 16th, 2023

Understanding the behavior of modern applications can be challenging because they are built using many components that interact with each other. Starting with an application deployed using containers and serverless functions, we use OpenTelemetry to instrument and collect telemetry data such as traces and metrics. Then, we visualize telemetry data to gain insight into the running system, detect anomalies, troubleshoot issues, and perform root-cause analysis. Through this end-to-end example, you will learn how to build observable applications.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. From complexity to observability using OpenTelemetry Chief Evangelist (EMEA), AWS Danilo Poccia (he/him) @danilop
2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudConf 2013
3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudConf 2013
4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Monolithic Application Services Microservices
5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Complexity arises when the dependencies among the elements become important.” Scott E. Page, John H. Miller Complex Adaptive Systems
6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. How Amazon SQS works Front End Back End Metadata Amazon DynamoDB Load Manager
7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “A complex system that works is invariably found to have evolved from a simple system that worked.” Gall’s Law
8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.”
9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Amazon S3 is intentionally built with a minimal feature set. The focus is on simplicity and robustness.” – Amazon S3 Press Release, March 14, 2006
10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon S3 8 → more than 200 microservices Mai-Lan Tomsen Bukovec AWS Vice President (Storage, Streaming, Messaging, and Monitoring/Observability)
11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Coupling: Integration’s magic word Coupling is a measure of independent variability between connected systems Decoupling has a cost, both at design and run-time Coupling isn’t binary Coupling isn’t one-dimensional A B Source: EnterpriseIntegrationPatterns.com
12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. The appropriate level of coupling depends on the level of control you have over the endpoints. Gregor Hohpe Enterprise Integration Patterns
13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Serverless Serverful
15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Observability & Instrumentation • “Observability lets us understand a system from the outside, by letting us ask questions about that system without knowing its inner workings.” • “In order to be able to ask those questions of a system, the application must be properly instrumented. That is, the application code must emit signals such as traces, metrics, and logs.” • “An application is properly instrumented when developers don’t need to add more instrumentation to troubleshoot an issue, because they have all of the information they need.” https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability
16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Don’t build a network of connected “black boxes” Building observable applications is a developer responsibility
17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry An open-source framework for instrumenting, generating, collecting, and exporting telemetry data
18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is OpenTelemetry (OTel)? • CNCF popular project • Instrument once • Collect and aggregate • Send to multiple destinations • Cloud-centered framework • Supports hundreds of destinations and 11 programing languages
19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. OpenTelemetry data sources and protocol • Traces – What happens when a request is made by user or an application • Metrics – A measurement about a service, captured at runtime • Logs – A timestamped text record with metadata • Baggage – Contextual information that’s passed between spans • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol
20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Reference architecture Code OTel Instrumentation Destination 1 Destination 2 Destination N … Receiver Exporter Processor OTel collector
21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Working with the open-source community
22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS Distro for OpenTelemetry (ADOT) A secure, production-ready open-source distribution supported by AWS Certified by AWS for security and predictability Upstream-first distro of the popular CNCF project Backed by AWS support One-click deploy and configure from AWS container and AWS Lambda consoles Exporters for AWS monitoring solutions including – Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch Service, Prometheus, SigV4Auth, and partner solutions
23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. ADOT Public Roadmap
24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tracing with AWS auto instrumentation agent ADD https://github.com/aws-observability/ aws-otel-java-instrumentation/releases/latest/download/ aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar" ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp" ENV OTEL_IMR_EXPORT_INTERVAL="10000” ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317" S A M P L E D O C K E R F I L E
25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Custom tracing attributes using annotations import io.opentelemetry.extension.annotations.WithSpan; import io.opentelemetry.extension.annotations.SpanAttribute; public class MyObservedClass { @WithSpan public void myObservedMethod(@SpanAttribute("firstParam") String firstParam, @SpanAttribute("secondParam") long secondParam) { <...> } } S A M P L E J A V A C O D E
26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Instrument once, send to multiple destinations P A R T N E R S
27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Demo
28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” app • Nice way to test a new programming language or new environments • Random number between 1 and 100 • Try your guess • Too high, too low, correct • What happens if I want to run a scalable, multitenant version?
29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” app architecture Internet Browser Database Compute Winner Message Message Queue User
30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” app architecture Internet Browser User ??? Amazon DynamoDB Winner Message Amazon SQS
31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” app architecture Internet Browser User ??? Amazon DynamoDB Winner Message AWS App Runner Amazon Elastic Container Service (Amazon ECS) Amazon Elastic Kubernetes Service (Amazon EKS) AWS Lambda Amazon SQS
33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” app architecture Internet Browser User ??? Amazon DynamoDB Winner Message AWS App Runner Amazon Elastic Container Service (Amazon ECS) Amazon Elastic Kubernetes Service (Amazon EKS) AWS Lambda Amazon SQS
34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Guess the Number” compute AWS App Runner Amazon Elastic Container Service (Amazon ECS) Amazon Elastic Kubernetes Service (Amazon EKS) AWS Lambda Single web container Managed OTel agent Tracing (X-Ray) Any container OTel sidecar Simplified console experience Tracing Metrics Any container OTel sidecar EKS add-on support for ADOT Operator Tracing Metrics Serverless function X-Ray active tracing Lambda layer with OTel agent Tracing Metrics
35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Code & Workshop Code used in the demo https://github.com/danilop/CloudConf2023-observability-demo AWS Observability Workshop https://catalog.workshops.aws/observability
36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Takeaways Complexity Observability OpenTelemetry AWS Distro for OpenTelemetry (ADOT) Instrument once → Collect & analyze everywhere
37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved. @danilop Give me your feedback!