Lessons from Building Distributed RocksDB

Presented as part of Geeknight Chennai - November 2016

Video - https://www.youtube.com/watch?v=PSCa9_Avne0
References - https://github.com/ashwanthkumar/distributed-rocksdb-talk

1. ashwanth kumar @_ashwanthkumar Building Distributed RocksDB
2. overview - What’s RocksDB - Why Distributed RocksDB - Lessons from running it in production
3. rocksdb
4. From facebook Fast persistent KV store Server Workloads Embeddable Optimized for SSDs rocksdb Fork of LevelDB Modelled after BigTable LSM Tree based SST files Written in C++
5. Simple C++ API Has bindings in C Java Go Python rocksdb
6. but why build a distributed database?
7. data system paradigms
8. sum of numbers
9. sum of numbers select col from table;
10. sum of numbers select col from table; <<results>>
11. sum of numbers select col from table; <<results>>
12. sum of numbers (attempt 2)
13. sum of numbers select sum(col) from table;
14. sum of numbers select sum(col) from table;
15. sum of numbers <<result>> select sum(col) from table;
16. data shipping function shipping select sum(col) from table; select col from table;
17. but why ship functions?
18. data locality for low latency / data intensive applications
19. recursive reduction aggregations in distributed systems
20. recursive reduction select sum(col) from table;
21. recursive reduction select sum(col) from table;
22. recursive reduction select sum(col) from table;
23. recursive reduction select sum(col) from table; <<result>>
24. - sum / multiplication - (sorted) top-K elements - operations on a graph - eg. link reach on twitter graph - function should be associative and optionally commutative recursive reduction
25. rocksdb @indix
26. - Serving our API in production for 2+ years - Search on hierarchical documents - Dynamic fields didn’t scale well on Solr - Brand / Store / Category Counts for a filter - Price History Service - More than a billion prices and serve online to REST queries rocksdb @indix
27. - Stats (as Monoids) Storage System - All we want was approximate aggregates real-time - HTML Archive System - Stores ~120TB of url and timestamp indexed HTML pages - Real-time scheduler for our crawlers - Finds out which of the 20 urls to crawl now out of 3+ billion urls - Helps crawler crawl 20+ million urls everyday rocksdb @indix
28. suuchi toolkit for building distributed function shipping applications github.com/ashwanthkumar/suuchi
29. ops lessons running rocksdb in production
30. leveled compaction
31. https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide Tuning RocksDB for Read amplification Write amplification Space amplification compaction + other tuning
32. backup & restore - Incremental backups - Store backups in S3 - Sometimes high CPU during backups - Restore happens outside the app lifecycle https://github.com/indix/rocks
33. questions? https://github.com/ashwanthkumar/distributed-rocksdb-talk