Systems that enable data agility

Talk given at Strata + Hadoop World London, 6 May 2015. http://strataconf.com/big-data-conference-uk-2015/public/schedule/detail/39689

Abstract:

Congratulations, you’ve got a lot of data! Now what? How do you enable your organisation to create value from that data? What tools do your data scientists need in order to create data-driven products? How do you empower your teams to experiment, to innovate, and to be agile in response to customer needs?

In this session we will discuss LinkedIn’s approach to solving these problems, and the open source tools that were created at LinkedIn to support data agility in a large organisation. The approach boils down to a few simple ideas:

1. Make all data available centrally, in real time. If it’s too difficult to access data across silos, you can’t derive value from it. For this purpose, LinkedIn created Apache Kafka, which can be the data exchange backbone of your organisation.

2. Make it easy to analyse and process that data. You’ve hired smart people, now empower them to easily try out new ideas for data-driven products, and rapidly get them into production if they are good. To support this, LinkedIn created Apache Samza, a stream processing framework that provides powerful, reliable tools for working with data in Kafka.

Since Kafka and Samza are open source, you can apply these lessons and start implementing your own agile data pipeline today.

In this talk you’ll learn about:

- How Kafka and Samza reliably scale to millions of messages per second
- What kinds of real-time data problems you can solve with Samza
- How Samza compares to other stream processing frameworks
- How data streams support collaboration between different data science, product and engineering teams within an organisation
- Lessons learnt on how to move fast without breaking things

Transcript

21. 216.58.210.78 - - [27/Feb/2015:17:55:11 +0000] "GET /css/typography.css HTTP/1.1” 200 3377 "http://martin. kleppmann.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36"
64. References 1.  Jay Kreps: “Putting Apache Kafka to use: A practical guide to building a stream data platform (part 1).” 25 February 2015. http://blog.confluent.io/2015/02/25/stream-data-platform-1/ 2.  Jay Kreps: “I ♥︎ Logs.” O’Reilly Media, September 2014. http://shop.oreilly.com/product/ 0636920034339.do 3.  Martin Kleppmann: “Designing data-intensive applications.” O’Reilly Media, to appear in 2015. http://dataintensive.net 4.  Martin Kleppmann: “Bottled Water: Real-time integration of PostgreSQL and Kafka.” 23 April 2015. http://blog.confluent.io/2015/04/23/bottled-water-real-time-integration-of- postgresql-and-kafka/ 5.  Shirshanka Das, Chavdar Botev, Kapil Surlaker, et al.: “All Aboard the Databus!,” at ACM Symposium on Cloud Computing (SoCC), October 2012. http://www.socc2012.org/s18- das.pdf