Event streaming fundamentals with Apache Kafka

Transcript

1. Event Streaming Fundamentals with Apache Kafka Keith Resar Sr. Kafka Developer @KeithResar
2. Data-Driven Operations
3. Data-Driven Operations
4. Data-Driven Operations
6. @KeithResar
7. @KeithResar
8. The Rise of Event Streaming 2010 Apache Kafka created at LinkedIn 2022 Most fortune 100 companies trust and use Kafka
9. A company is built on _DATA FLOWS_ but all we have are _DATA STORES_
10. Example Application Architecture Serving Layer (Microservices, Elastic, etc.) Java Apps with Kafka Streams or ksqlDB Continuous Computation High-Throughput Event Streaming Platform API-Based Clustering @KeithResar
11. Apache Kafka is an Event Streaming Platform 1. Storage 2. Pub / Sub 3. Processing @KeithResar
12. Storage 12 @KeithResar
13. Core Abstractions @KeithResar • DB → table • Hadoop → file • Kafka - ?
14. LOG
15. Immutable Event Log New Messages are added at the end of the log Old @KeithResar
16. Messages are KV Bytes key: byte[] value: byte[] Headers => [Header] @KeithResar
17. Messages Inside Topics Clicks Orders Customers Topics are similar to database tables @KeithResar
18. Topics divide into Partitions Messages are guaranteed to be strictly ordered within a partition @KeithResar P 0 Clicks P 1 P 2
20. Pub / Sub 20 @KeithResar
21. Producing Data New Messages are added at the end of the log Old @KeithResar
22. Consuming Data New Consume via sequential data access starting from a specific offset. Old @KeithResar Read to offset & scan
23. Distinct Consumer Positions New Old @KeithResar Sally offset 12 Fred offset 3 Rick offset 9
25. Messages are KV Bytes key: byte[] value: byte[] Headers => [Header] @KeithResar
26. Producing to Kafka - No Key @KeithResar P 0 P 1 P 2 P 3 Messages will be produced in a round robin fashion
27. Producing to Kafka - No Key @KeithResar P 0 P 1 P 2 P 3 Messages will be produced in a round robin fashion
28. Producing to Kafka - With Key @KeithResar P 0 P 1 P 2 P 3 hash(key) % numPartitions = N
29. Producing to Kafka - With Key @KeithResar P 0 P 1 P 2 P 3 hash(key) % numPartitions = N
30. Consumer from Kafka - Single @KeithResar P 0 P 1 P 2 P 3 Single consumer reads from all partitions
31. Consumer from Kafka - Multiple @KeithResar P 0 P 1 P 2 P 3 Consumers can be split into multiple groups each of which operate in isolation
32. CONSUMER GROUP COORDINATOR CONSUMERS CONSUMER GROUP
33. Consumer from Kafka - Multiple @KeithResar P 0 P 1 P 2 P 3 Consumers can be split into multiple groups each of which operate in isolation
34. Consumer from Kafka - Multiple @KeithResar P 0 P 1 P 2 P 3 Consumers can be split into multiple groups each of which operate in isolation
35. Grouped Consumers @KeithResar P 0 P 1 P 2 P 3 Consumers can be split into multiple groups each of which operate in isolation
36. Grouped Consumers @KeithResar P 0 P 1 P 2 P 3 Consumers can be split into multiple groups each of which operate in isolation X
38. Linearly Scalable Architecture @KeithResar Producers • Many producers machines • Many consumer machines • Many Broker machines Consumers Single topic, No Bottleneck!
39. Replicate for Fault Tolerance @KeithResar Broker A Broker B Message ✓ Leader Replicate
40. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Partition 1 Partition 2 Partition 3 Follower Leader
41. Replication Provides Resiliency @KeithResar Producers Consumers Replica followers become leaders on machine failure X X X X X
42. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Partition 1 Partition 2 Partition 3 Follower Leader
43. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Partition 1 Partition 2 Partition 3 Follower Leader
44. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Partition 1 Partition 2 Partition 3 Follower Leader
45. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Partition 1 Partition 2 Partition 3 Follower Leader Partition 2 Partition 1 Partition 3
46. Partition Leadership / Replication @KeithResar Broker 1 Broker 2 Broker 3 Broker 4 P 0 P 1 P 2 P 3 Partition 0 Partition 2 Partition 3 Partition 0 Partition 1 Partition 3 Partition 0 Partition 1 Partition 2 Follower Leader Partition 2 Partition 1 Partition 3
48. The log is a type of durable messaging system @KeithResar Similar to a traditional messaging system (ActiveMQ, Rabbit, etc.) but with: • Far better scalability • Built-in fault tolerance/HA • Storage
50. Origins in Stream Processing Serving Layer (Microservices, Elastic, etc.) Java Apps with Kafka Streams or ksqlDB Continuous Computation High-Throughput Event Streaming Platform API-Based Clustering
51. Processing 51 @KeithResar
52. Streaming is the toolset for working with events as they move! @KeithResar
53. What is stream processing? @KeithResar auth attempts possible fraud
54. What is stream processing? @KeithResar User Population Coding Sophistication Core developers who use Java/Scala Core developers who don’t use Java/Scala Data engineers, architects, DevOps/SRE BI analysts streams
55. Standing on the Shoulders of Streaming Giants Producer, Consumer APIs Kafka Streams ksqlDB Ease of use Flexibility ksqlDB UDFs Powered by Powered by
56. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
57. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
58. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
59. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
60. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
61. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
62. What is stream processing? @KeithResar CREATE STREAM possible_fraud AS SELECT card_number, count(*) FROM authorization_attempts WINDOW TUMBLING (SIZE 5 MINUTE) GROUP BY card_number HAVING count(*) > 3;
64. Wrap Up 64 @KeithResar
65. developer.confluent.io Learn Kafka. Start building with Apache Kafka at Confluent Developer.
66. Free eBooks Designing Event-Driven Systems Ben Stopford Kafka: The Definitive Guide Neha Narkhede, Gwen Shapira, Todd Palino Making Sense of Stream Processing Martin Kleppmann I ❤ Logs Jay Kreps http://cnfl.io/book-bundle
68. Thank You @KeithResar Kafka Developer confluent.io