Kafka - A Little Introduction

This is a presentation I gave on Kafka (http://incubator.apache.org/kafka/)- a persistent, distributed messaging system developed at LinkedIn.

I gave it during one of our monthly tech presentation evenings in our office.

Transcript

1. Kafka A little introduction
3. Pub-Sub Messaging System
8. Distributed
10. Performance
11. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100 1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
12. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100 1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
13. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100 1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second
14. Disk/Memory Performance Source: http://queue.acm.org/detail.cfm?id=1563874 Disk SSD Memory 1 10 100 1,000 10,000 100,000 1M 10M 100M 1000M Random access Sequential Access Read values/second Sequential disk read faster than random memory read
15. Persistent
19. Length Magic Value Checksum Payload 4 bytes 1 byte 4 bytes n bytes
23. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
24. Token Offset: 0 Broker: kafka.local Topic: Testing Input MR Job Output Offset: 130098 Broker: kafka.local Topic: Testing Output Sequence File
26. Useful Things • http://incubator.apache.org/kafka/ • https://github.com/pingles/clj-kafka