Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kafka Will Get The Message Across, Guaranteed.

Kafka Will Get The Message Across, Guaranteed.

Presentation given at PHP Benelux 2017 near Antwerp, Belgium.

David Zuelke

January 28, 2017
Tweet

More Decks by David Zuelke

Other Decks in Programming

Transcript

  1. WAL

  2. 1 create document: "foo", data: "…" 2 update document: "foo",

    data: "…" 3 create document: "bar", data: "…" 4 remove document: "foo"
  3. 1 2 3 4 5 6 7 8 … producer

    writes consumer reads
  4. 1 2 3 4 5 6 7 8 … producer

    writes consumerB reads consumerA reads
  5. P0 1 2 3 4 5 6 7 … P1

    1 2 3 4 … P2 1 2 3 4 5 6 7 8 … P3 1 2 3 4 5 6 …
  6. BASIC PRODUCER $rk = new RdKafka\Producer(); $rk->addBrokers("127.0.0.1"); $topic = $rk->newTopic("test");

    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Unassigned partition, let Kafka choose"); $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Yay consistent hashing", $user->getId()); $topic->produce(1, 0, "This will always be sent to partition 1");
  7. can themselves produce new messages!
 (but there is also a

    Streams API for pure transformations)
  8. BASIC CONSUMER $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk =

    new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.interval.ms', 100); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); }
  9. AT-MOST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk

    = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); $topic->offsetStore($msg->partition, $msg->offset); do_something($msg); }
  10. AT-LEAST ONCE DELIVERY $conf = new RdKafka\Conf(); $conf->set('group.id', 'myConsumerGroup'); $rk

    = new RdKafka\Consumer($conf); $rk->addBrokers("127.0.0.1"); $topicConf = new RdKafka\TopicConf(); $topicConf->set('auto.commit.enable', false); $topic = $rk->newTopic("test", $topicConf); $topic->consumeStart(0, RD_KAFKA_OFFSET_STORED); while (true) { $msg = $topic->consume(0, 120*10000); do_something($msg); $topic->offsetStore($msg->partition, $msg->offset); }
  11. THE BYZANTINE GENERALS "together we can beat the monsters. let's

    both attack at 07:00?" "confirm, we attack at 07:00" ☠
  12. • LinkedIn • Yahoo • Twitter • Netflix • Square

    • Spotify • Pinterest • Uber • Goldman Sachs • Tumblr • PayPal • Airbnb • Mozilla • Cisco • Etsy • Foursquare • Shopify • CloudFlare