Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From complexity to observability using OpenTelemetry

From complexity to observability using OpenTelemetry

CloudConf, Turin, May 16th, 2023

Understanding the behavior of modern applications can be challenging because they are built using many components that interact with each other. Starting with an application deployed using containers and serverless functions, we use OpenTelemetry to instrument and collect telemetry data such as traces and metrics. Then, we visualize telemetry data to gain insight into the running system, detect anomalies, troubleshoot issues, and perform root-cause analysis. Through this end-to-end example, you will learn how to build observable applications.

Danilo Poccia

May 16, 2023
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    From complexity to observability
    using OpenTelemetry
    Chief Evangelist (EMEA), AWS
    Danilo Poccia (he/him)
    @danilop

    View Slide

  2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    CloudConf 2013

    View Slide

  3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    CloudConf 2013

    View Slide

  4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Monolithic
    Application
    Services Microservices

    View Slide

  5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Complexity arises when
    the dependencies among
    the elements become
    important.”
    Scott E. Page, John H. Miller
    Complex Adaptive Systems

    View Slide

  6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    How Amazon SQS works
    Front End
    Back End
    Metadata
    Amazon
    DynamoDB
    Load
    Manager

    View Slide

  7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “A complex system that
    works is invariably found
    to have evolved from a
    simple system that
    worked.”
    Gall’s Law

    View Slide

  8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “A complex system
    designed from scratch
    never works and cannot
    be patched up to make it
    work. You have to start
    over with a working
    simple system.”

    View Slide

  9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Amazon S3 is intentionally
    built with a minimal feature set.
    The focus is on simplicity and
    robustness.”
    – Amazon S3 Press Release,
    March 14, 2006

    View Slide

  10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Amazon S3
    8 → more than 200
    microservices
    Mai-Lan Tomsen Bukovec
    AWS Vice President (Storage, Streaming, Messaging,
    and Monitoring/Observability)

    View Slide

  11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Coupling: Integration’s magic word
    Coupling is a measure of independent
    variability between connected systems
    Decoupling has a cost, both at design and run-time
    Coupling isn’t binary
    Coupling isn’t one-dimensional
    A B
    Source: EnterpriseIntegrationPatterns.com

    View Slide

  12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    The appropriate level of
    coupling depends on the
    level of control you have
    over the endpoints.
    Gregor Hohpe
    Enterprise Integration Patterns

    View Slide

  13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    View Slide

  14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Serverless Serverful

    View Slide

  15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Observability & Instrumentation
    • “Observability lets us understand a system from the outside, by letting us ask
    questions about that system without knowing its inner workings.”
    • “In order to be able to ask those questions of a system, the application must be
    properly instrumented. That is, the application code must emit signals such as
    traces, metrics, and logs.”
    • “An application is properly instrumented when developers don’t need to add
    more instrumentation to troubleshoot an issue, because they have all of the
    information they need.”
    https://opentelemetry.io/docs/concepts/observability-primer/#what-is-observability

    View Slide

  16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Don’t build a network of connected “black boxes”
    Building observable applications
    is a developer responsibility

    View Slide

  17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry
    An open-source framework for instrumenting, generating,
    collecting, and exporting telemetry data

    View Slide

  18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What is OpenTelemetry (OTel)?
    • CNCF popular project
    • Instrument once
    • Collect and aggregate
    • Send to multiple destinations
    • Cloud-centered framework
    • Supports hundreds of destinations
    and 11 programing languages

    View Slide

  19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OpenTelemetry data sources and protocol
    • Traces – What happens when a request is made by user or an application
    • Metrics – A measurement about a service, captured at runtime
    • Logs – A timestamped text record with metadata
    • Baggage – Contextual information that’s passed between spans
    • OpenTelemetry Protocol (OTLP) – Telemetry data delivery protocol

    View Slide

  20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Reference architecture
    Code
    OTel
    Instrumentation
    Destination
    1
    Destination
    2
    Destination
    N

    Receiver Exporter
    Processor
    OTel collector

    View Slide

  21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Working with
    the open-source
    community

    View Slide

  22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    AWS Distro for OpenTelemetry (ADOT)
    A secure, production-ready open-source distribution supported by AWS
    Certified by AWS for security and predictability
    Upstream-first distro of the popular CNCF project
    Backed by AWS support
    One-click deploy and configure from AWS container and
    AWS Lambda consoles
    Exporters for AWS monitoring solutions including –
    Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch
    Service, Prometheus, SigV4Auth, and partner solutions

    View Slide

  23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    ADOT Public Roadmap

    View Slide

  24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Tracing with AWS auto instrumentation agent
    ADD https://github.com/aws-observability/
    aws-otel-java-instrumentation/releases/latest/download/
    aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
    ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar"
    ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp"
    ENV OTEL_IMR_EXPORT_INTERVAL="10000”
    ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
    S A M P L E D O C K E R F I L E

    View Slide

  25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Custom tracing attributes using annotations
    import io.opentelemetry.extension.annotations.WithSpan;
    import io.opentelemetry.extension.annotations.SpanAttribute;
    public class MyObservedClass {
    @WithSpan
    public void myObservedMethod(@SpanAttribute("firstParam") String firstParam,
    @SpanAttribute("secondParam") long secondParam) {
    <...>
    }
    }
    S A M P L E J A V A C O D E

    View Slide

  26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Instrument once, send to multiple destinations
    P A R T N E R S

    View Slide

  27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Demo

    View Slide

  28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” app
    • Nice way to test a new programming language or new environments
    • Random number between 1 and 100
    • Try your guess
    • Too high, too low, correct
    • What happens if I want to run a scalable, multitenant version?

    View Slide

  29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” app architecture
    Internet
    Browser
    Database
    Compute
    Winner
    Message
    Message
    Queue
    User

    View Slide

  30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” app architecture
    Internet
    Browser
    User ??? Amazon DynamoDB
    Winner
    Message
    Amazon SQS

    View Slide

  31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” app architecture
    Internet
    Browser
    User ??? Amazon DynamoDB
    Winner
    Message
    AWS App Runner Amazon Elastic Container
    Service (Amazon ECS)
    Amazon Elastic Kubernetes
    Service (Amazon EKS)
    AWS Lambda
    Amazon SQS

    View Slide

  32. View Slide

  33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” app architecture
    Internet
    Browser
    User ??? Amazon DynamoDB
    Winner
    Message
    AWS App Runner Amazon Elastic Container
    Service (Amazon ECS)
    Amazon Elastic Kubernetes
    Service (Amazon EKS)
    AWS Lambda
    Amazon SQS

    View Slide

  34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    “Guess the Number” compute
    AWS App Runner Amazon Elastic Container
    Service (Amazon ECS)
    Amazon Elastic Kubernetes
    Service (Amazon EKS)
    AWS Lambda
    Single web container
    Managed OTel agent
    Tracing (X-Ray)
    Any container
    OTel sidecar
    Simplified console
    experience
    Tracing
    Metrics
    Any container
    OTel sidecar
    EKS add-on support for
    ADOT Operator
    Tracing
    Metrics
    Serverless function
    X-Ray active tracing
    Lambda layer
    with OTel agent
    Tracing
    Metrics

    View Slide

  35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Code & Workshop
    Code used in the demo
    https://github.com/danilop/CloudConf2023-observability-demo
    AWS Observability Workshop
    https://catalog.workshops.aws/observability

    View Slide

  36. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Takeaways
    Complexity
    Observability
    OpenTelemetry
    AWS Distro for OpenTelemetry (ADOT)
    Instrument once → Collect & analyze everywhere

    View Slide

  37. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Thank you!
    © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    @danilop Give me your feedback!

    View Slide