[DevNexus-2018] Apache Kafka A Streaming Data Platform

@ Apache Kafka A Streaming Data Platform

@ @gamussa @confluentinc Who am I?

@ @gamussa @confluentinc Solutions Architect Who am I?

@ @gamussa @confluentinc Solutions Architect Developer Advocate Who am I?

@ @gamussa @confluentinc Solutions Architect Developer Advocate @gamussa in internetz
Who am I?

@ @gamussa @confluentinc Solutions Architect Developer Advocate @gamussa in internetz
Hey you, yes, you, go follow me in twitter © Who am I?

@ @gamussa @confluentinc

@ @gamussa @confluentinc A company is build on

@ @gamussa @confluentinc A company is build on DATA FLOWS
but All we have is DATA STORES

@ @gamussa @confluentinc Streaming Platform 1. Pub/Sub 2. Store 3.
Process

@ @gamussa @confluentinc Core abstraction

@ @gamussa @confluentinc Core abstraction DB - table

@ @gamussa @confluentinc Core abstraction DB - table Hadoop -
file

@ @gamussa @confluentinc Core abstraction DB - table Hadoop -
file Messaging -?

@ @gamussa @confluentinc LOGS

@ @gamussa @confluentinc Producing to Kafka Time

@ @gamussa @confluentinc Producing to Kafka Time C C C

@ @gamussa @confluentinc Producing to Kafka - With Key Time
A B C D hash(key) % numPartitions = N

@ @gamussa @confluentinc Producing to Kafka - No Key Time
Messages will be produced in a round robin fashion

@ @gamussa @confluentinc Consuming From Kafka - Single Consumer C

@ @gamussa @confluentinc Consuming From Kafka - Grouped Consumers C
C C1 C C C2

@ @gamussa @confluentinc Consuming From Kafka - Grouped Consumers C
C C C

@ @gamussa @confluentinc Consuming From Kafka - Grouped Consumers 0
1 2 3

@ @gamussa @confluentinc Consuming From Kafka - Grouped Consumers 0,
3 1 2 3

@ @gamussa @confluentinc Producers Consumers

@ @gamussa @confluentinc Kafka Connect does hard work so you
don’t

@ @gamussa @confluentinc Kafka Connect does hard work so you
don’t 1. Scale out

Process

@ @gamussa @confluentinc Why Store?

@ @gamussa @confluentinc Scalability of a filesystem

@ @gamussa @confluentinc Scalability of a filesystem Throughput 100s mb/s

TBs per server

TBs per server Commodity Hardware

TBs per server Commodity Hardware O(1) writes

@ @gamussa @confluentinc Guarantees of a database

@ @gamussa @confluentinc Guarantees of a database Persistence

@ @gamussa @confluentinc Guarantees of a database Persistence Strict ordering

@ @gamussa @confluentinc Distributed by Design

@ @gamussa @confluentinc Replication Distributed by Design

@ @gamussa @confluentinc Replication Fault Tolerance Distributed by Design

@ @gamussa @confluentinc Replication Fault Tolerance Partitioning Distributed by Design

@ @gamussa @confluentinc Replication Fault Tolerance Partitioning Scale Distributed by
Design

@ @gamussa @confluentinc Partition Leadership and Replication Broker 1 Topic1
partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4

@ @gamussa @confluentinc Partition Leadership and Replication - node failure
Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4

Process

@ @gamussa @confluentinc What is Stream Processing? A machine for
combining streams of events

@ @gamussa @confluentinc https://www.confluent.io/download/

@ @gamussa @confluentinc We are hiring! https://www.confluent.io/careers/

@ @gamussa @confluentinc One more thing…

@ @gamussa @confluentinc A Major New Paradigm

@ @gamussa @confluentinc Thanks! questions? @gamussa [email protected] We are hiring!
https://www.confluent.io/careers/

[DevNexus-2018] Apache Kafka A Streaming Data P...

[DevNexus-2018] Apache Kafka A Streaming Data Platform

More Decks by Viktor Gamov

Other Decks in Technology

Featured

Transcript