Kafka Will Get The Message Across, Guaranteed.

Presentation given at SymfonyCon 2016 in Berlin, Germany.

Transcript

1. KAFKA WILL GET THE MESSAGE ACROSS. GUARANTEED. SymfonyCon 2016 Berlin, Germany
2. David Zuelke
4. dz@heroku.com
5. @dzuelke
6. KAFKA
7. LinkedIn
8. APACHE KAFKA
9. "uh oh, another Apache project?!"
11. KEEP CALM AND LOOK AT THE WEBSITE
13. "Basically it is a massively scalable pub/sub message queue. architected as a distributed transaction log."
14. "so it's a queue?"
15. it's not a queue
16. queues are not multi-subscriber :(
17. "so it's a pubsub thing?"
18. it's not a pubsub thing
19. pubsub broadcasts to all subscribers :(
20. it's a log
22. not that kind of log
23. WAL
24. Write-Ahead Log
25. WRITE-AHEAD LOG
27. 1 foo 2 bar 3 baz 4 hi
28. 1 create document: "foo", data: "…" 2 update document: "foo", data: "…" 3 create document: "bar", data: "…" 4 remove document: "foo"
30. never corrupts
31. sequential I/O
33. sequential I/O
34. every message will be read at least once, no random access
35. FileChannel.transferTo (shovels data straight from e.g. disk cache to network interface, no copying via RAM)
36. "HI, I AM KAFKA" "Buckle up while we process (m|b|tr)illions of messages/s."
37. TOPICS
38. streams of records
39. 1 2 3 4 5 6 7 …
40. 1 2 3 4 5 6 7 8 … producer writes consumer reads
41. can have many subscribers
42. 1 2 3 4 5 6 7 8 … producer writes consumerB reads consumerA reads
43. can be partitioned
44. P0 1 2 3 4 5 6 7 … P1 1 2 3 4 … P2 1 2 3 4 5 6 7 8 … P3 1 2 3 4 5 6 …
45. partitions let you scale storage!
46. partitions let you scale consuming!
48. all records are retained, whether consumed or not, up to a configurable limit
49. PRODUCERS
50. byte[]
51. (typically JSON, XML, Avro, Thrift, Protobufs)
52. (typically not funny GIFs)
53. can choose explicit partition, or a key (which is used for auto-partitioning)
54. https://github.com/edenhill/librdkafka & https://arnaud-lb.github.io/php-rdkafka/
55. BASIC PRODUCER $rk = new RdKafka\Producer(); $rk->addBrokers("127.0.0.1"); $topic = $rk->newTopic("test"); $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Unassigned partition, let Kafka choose"); $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Yay consistent hashing", $user->getId()); $topic->produce(1, 0, "This will always be sent to partition 1");
56. CONSUMERS
57. cheap
58. only metadata stored per consumer: offset
59. guaranteed to always have messages in right order (within a partition)
60. can themselves produce new messages!
62. BASIC CONSUMER $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); }
63. AT-MOST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); $topic->offsetStore($msg->partition, $msg->offset); do_something($msg); }
64. AT-LEAST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); $topic->offsetStore($msg->partition, $msg->offset); }
65. EXACTLY-ONCE DELIVERY
66. you cannot have exactly-once delivery
67. THE BYZANTINE GENERALS "together we can beat the monsters. let's both attack at 07:00?" "confirm, we attack at 07:00" ☠
68. USE CASES
69. • LinkedIn • Yahoo • Twitter • Netflix • Square • Spotify • Pinterest • Uber • Goldman Sachs • Tumblr • PayPal • Airbnb • Mozilla • Cisco • Etsy • Foursquare • Shopify • CloudFlare
70. ingest Twitter firehose and turn it into a pointless demo ;)
72. messaging, of course
73. track user activity
74. record runtime metrics
75. IoT
76. replicate information between data centers
77. billing!
78. "shock absorber" between systems to avoid overload of DBs, APIs, etc.
79. in PHP: mostly producing messages; better languages exist for consuming
80. The End
81. THANK YOU FOR LISTENING! Questions? Ask me: @dzuelke & dz@heroku.com