Everything you Wanted to Know about Apache Kafka But you Were Too Afraid to Ask!

Transcript

1. Join the Confluent Community Slack Subscribe to the Confluent blog cnfl.io/slack cnfl.io/read Welcome to the DURHAM, NC Apache Kafka® Meetup! 6:00pm Doors open 6:00pm - 6:30pm Pizza, Drinks and Networking 6:30pm - 7:00pm John turner, mediamath 7:00pm - 8:00pm Ricardo ferreira, confluent Apache, Apache Kafka, Kafka and the Kafka logo are trademarks of the Apache Software Foundation. The Apache Software Foundation has no affiliation with and does not endorse the materials provided at this event.
2. @riferrei | #kafkameetup | @CONFLUENTINC 2 Before we begin
3. NOMINATE YOURSELF OR A PEER AT CONFLUENT.IO/NOMINATE
4. KS19Meetup. CONFLUENT COMMUNITY DISCOUNT CODE 25% OFF* *Standard Priced Conference pass
5. @riferrei | #kafkameetup | @CONFLUENTINC Everything you wanted to know about kafka But you were too afraid to ask! @riferrei | #kafkameetup | @CONFLUENTINC
6. @riferrei | #kafkameetup | @CONFLUENTINC Wakanda? Forever! Hulk? Smash! Apache Kafka? Is it Like Messaging?
7. About Us: • Ricardo Ferreira ❑ Developer Advocate @ Confluent ❑ Ex-Oracle, Red Hat, IONA Tech ❑ Ricardo@confluent.io ❑ https://riferrei.net • Alexa (amazon echo) ❑ The voice behind Amazon ❑ Ex-Raspberry Pi, Arduino ❑ She is a female in character! @riferrei @alexa99
8. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed streaming platform?
9. @riferrei | #kafkameetup | @CONFLUENTINC ? ? ?
10. @riferrei | #kafkameetup | @CONFLUENTINC Let’s go back in time. Shaw we?
11. @riferrei | #kafkameetup | @CONFLUENTINC databases 30 years ago… Dude, you're embarrassing me in front of the wizards… Databases these days…
12. @riferrei | #kafkameetup | @CONFLUENTINC 12 Database problems
13. @riferrei | #kafkameetup | @CONFLUENTINC It is so limited Makes mess
14. @riferrei | #kafkameetup | @CONFLUENTINC Etl is the result of databases being limited!
15. @riferrei | #kafkameetup | @CONFLUENTINC What does it cost to extract data from the transactional?
16. @riferrei | #kafkameetup | @CONFLUENTINC Silos = mess
17. @riferrei | #kafkameetup | @CONFLUENTINC How easy is to change a etl code because the data model changed?
18. @riferrei | #kafkameetup | @CONFLUENTINC 18 workarounds
19. @riferrei | #kafkameetup | @CONFLUENTINC Another day... Another doug
20. @riferrei | #kafkameetup | @CONFLUENTINC Databases to eliminate the need to process data! Database for large amounts of data!
21. @riferrei | #kafkameetup | @CONFLUENTINC And lots of messaging!
22. @riferrei | #kafkameetup | @CONFLUENTINC
23. @riferrei | #kafkameetup | @CONFLUENTINC Let’s go back in time. Shaw we?
24. @riferrei | #kafkameetup | @CONFLUENTINC
25. @riferrei | #kafkameetup | @CONFLUENTINC
26. 26 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
27. 27 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Thinking
28. @riferrei | #kafkameetup | @CONFLUENTINC Question: What is a distributed streaming platform?
29. @riferrei | #kafkameetup | @CONFLUENTINC “The truth is the log. The database is a cache of a subset of the log.” — pat helland Immutability changes everything http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf
30. @riferrei | #kafkameetup | @CONFLUENTINC 01 Messaging done right 02 Scalable data pipelines 03 Stream processing @riferrei | #kafkameetup | @CONFLUENTINC
31. @riferrei | #kafkameetup | @CONFLUENTINC Messaging done right @riferrei | #kafkameetup | @CONFLUENTINC
32. @riferrei | #kafkameetup | @CONFLUENTINC @riferrei | #kafkameetup | @CONFLUENTINC Scalable data pipelines
33. @riferrei | #kafkameetup | @CONFLUENTINC @riferrei | #kafkameetup | @CONFLUENTINC Stream processing
34. @riferrei | #kafkameetup | @CONFLUENTINC 34 demo
35. @riferrei | #kafkameetup | @CONFLUENTINC
36. Join The fun
37. @riferrei | #kafkameetup | @CONFLUENTINC Question: is apache kafka a database?
38. @riferrei | #kafkameetup | @CONFLUENTINC It can be!
39. @riferrei | #kafkameetup | @CONFLUENTINC Have you EVER HEARD ABOUT THE #ASKCONFLUENT INITIATIVE?
40. @riferrei | #kafkameetup | @CONFLUENTINC There was an idea…
41. @riferrei | #kafkameetup | @CONFLUENTINC of Bringing TOGETHER A GROUP OF REMARKABLE PEOPLE
42. @riferrei | #kafkameetup | @CONFLUENTINC THAT COULD ANSWER THE QUESTIONS…
43. @riferrei | #kafkameetup | @CONFLUENTINC THAT we NEVER COULD...
44. @riferrei | #kafkameetup | @CONFLUENTINC @tlberglund @gwenshap
45. @riferrei | #kafkameetup | @CONFLUENTINC https://www.youtube.com/playlist?list= PLa7VYi0yPIH0snucuYWkuUXwasMr-HR7Y Kafka is so Cool!
46. @riferrei | #kafkameetup | @CONFLUENTINC Question: IS THERE SUCH A THING OF OVERSUBSCRIPING KAFKA?
47. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS MOST COSTLY... UP CONVERT OR DOWN CONVERT?
48. @riferrei | #kafkameetup | @CONFLUENTINC Question: HOW MANY PARTITIONS A TOPIC CAN HAVE?
49. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT ARE THE PROS AND CONS OF THE EXACTLY-ONCE FEATURE?
50. @riferrei | #kafkameetup | @CONFLUENTINC Question: WHAT IS THE IMPACT OF CONSUMER APPS BEING DOWN FOR 4 DAYS?
51. @riferrei | #kafkameetup | @CONFLUENTINC Question: NOT USING KEYS WHILE PRODUCING RECORDS IS A BAD PRACTICE?
52. @riferrei | #kafkameetup | @CONFLUENTINC