Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Of the Order of Billions: Building Observability at Twitter

Caitie McCaffrey
October 21, 2015
600

Of the Order of Billions: Building Observability at Twitter

Every minute Twitter’s Observability stack processes 1.5+ billion metrics in order to provide Visibility into Twitter’s distributed microservices architecture. In this talk will focus on some of the challenges associated with building and running this large scale distributed system. We will also focus on lessons learned and how to build services that scale that are applicable for services of any size.

Caitie McCaffrey

October 21, 2015
Tweet

Transcript

  1. Caitie McCaffrey
    Tech Lead Observability | @caitie
    Of the Order of Billions: Building
    Observability at Twitter

    View full-size slide

  2. Of the Order of Billions
    BUILDING OBSERVABILITY AT TWITTER

    View full-size slide

  3. Observability
    PROVIDES VISIBILITY INTO TWITTER SERVICES

    View full-size slide

  4. Dashboards & Charts

    View full-size slide

  5. Alerts
    KOALABIRD
    ALERTING

    View full-size slide

  6. 2 Billion
    METRICS PER MINUTE
    1 MINUTE 10 MILLION METRICS
    OCTOBER, 2015

    View full-size slide

  7. 14 Days
    x
    24 Hours
    120 Billion Metrics
    x
    =
    1 Hour
    40 Trillion
    Metrics
    2 Billion Metrics
    OCTOBER, 2015

    View full-size slide

  8. ABSORBER
    SERVICE
    Serves requests, looks
    to Manhattan
    Life Of A Metric
    TWITTER
    SERVICE
    CUCKOO-
    WRITE
    CUCKOO-
    READ
    KOALABIRD
    VIZ Dashboards
    Manhattan
    Gathers Metrics
    Every
    Analyzes data & decides
    where it needs to be
    Alerts

    View full-size slide

  9. Building Highly
    Available, Scalable &
    Fault Tolerant Systems

    View full-size slide

  10. Idempotent Operations
    F(F(X) = F(X))

    View full-size slide

  11. Idempotent Operations
    F(F(X) = F(X))
    ABSORBER
    CUCKOO-WRITE

    View full-size slide

  12. Idempotent Operations
    F(F(X) = F(X))
    ABSORBER
    CUCKOO-WRITE

    View full-size slide

  13. Idempotent Operations
    F(F(X) = F(X))
    ABSORBER
    CUCKOO-WRITE

    View full-size slide

  14. Avoid Coordination
    Minimizing coordination is key to
    maximizing scalability, availability,
    and high performance
    Value of n? n?
    N=2
    N=2
    N=?

    View full-size slide

  15. Avoid Coordination
    Minimizing coordination is key to
    maximizing scalability, availability,
    and high performance
    Value of n?
    N=2
    N=2
    N=?
    Value of n?

    View full-size slide

  16. Make Assumptions Explicit
    PRODUCER
    CONSUMER
    .

    View full-size slide

  17. Make Assumptions Explicit
    PRODUCER
    .
    CONSUMER

    View full-size slide

  18. Make Assumptions Explicit
    .
    CONSUMER
    PRODUCER

    View full-size slide

  19. Make Assumptions Explicit
    .
    CONSUMER
    PRODUCER

    View full-size slide

  20. Make Assumptions Explicit
    .
    CONSUMER
    PRODUCER

    View full-size slide

  21. Make Assumptions Explicit
    .
    CONSUMER
    PRODUCER

    View full-size slide

  22. Make Assumptions Explicit
    .
    CONSUMER
    PRODUCER

    View full-size slide

  23. Make Assumptions Explicit
    • Queue Lengths
    • Input Data Length
    • Requests Per Second
    • In Memory Data Structure Sizes

    View full-size slide

  24. SOME
    LESSONS
    LEARNED
    Idempotent Operations
    Avoid Coordination
    Make Assumptions Explicit

    View full-size slide

  25. Thank You
    @caitie

    View full-size slide