Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling big with Apache Kafka

Scaling big with Apache Kafka

Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.

Nikolay Stoitsev

November 16, 2018
Tweet

More Decks by Nikolay Stoitsev

Other Decks in Technology

Transcript

  1. Partition - ordered, immutable sequence Message Message Message Message Message

    Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
  2. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2

    P2 P2 Partitioned and Replicated
  3. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2

    P2 P2 Producers Producer Producer
  4. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2

    P2 P2 Producer Producer Consumer Consumer Consumer Group 1 Consumer Consumer Consumer Consumer Group 2 Consumers
  5. Broker Broker P0 P0 P1 P1 P2 P2 Producer Consumer

    Consumer Consumer Group 1 ZooKeeper Get Broker ID Update Offset
  6. Tune for lower latency • Acknowledgement after persisted on the

    leader • Can lost message on leadership changes • At-most-once semantic
  7. Tune for durability • Acknowledgement after persisted on all ISR

    (after committed) • No data loss • At-least-once sematic
  8. Kafka Upstream Downstrea m Schema Registry Publish Schema Fetch Schema

    https://docs.confluent.io/current/schema-registry/docs/index.html
  9. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue

    Payment Retry 0 Payment Retry 1 Dead Letter Queue
  10. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue

    Payment Retry 0 Payment Retry 1 Dead Letter Queue Payment Consumer Payment Consumer
  11. Chaperone Collector Regional Kafka Service Kafka REST Proxy Audit Library

    Audit Library Aggregate Kafka Chaperone Service Chaperone Service Chaperone Collector DB
  12. Summary Tune for durability Define Avro Schemas Use Kafka REST

    Proxy Add idempotency checks Use Dead Letter Queue
  13. Summary Tune for durability Define Avro Schemas Use Kafka REST

    Proxy Add idempotency checks Use Dead Letter Queue Monitor everything