Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Luck Driven Development: Building for Serendipi...

Ronnie Chen
October 20, 2017

Luck Driven Development: Building for Serendipity in Slack's Data Platform

Ronnie Chen

October 20, 2017
Tweet

More Decks by Ronnie Chen

Other Decks in Technology

Transcript

  1. Luck Driven Development Building for Serendipity in Slack's Data Platform

    Ronnie Chen @rondo!w Data Engineering Slack CrunchConf 2017 | Ronnie Chen @rondo!w 1
  2. Mar 2016 5 data engs / 350+ employees / 2M+

    DAU Oct 2017 10 data engs / 800+ employees / 6M+ DAU 1 in 9 employees access the data warehouse / day 1 in 3 employees access the data warehouse / week CrunchConf 2017 | Ronnie Chen @rondo!w 3
  3. Your Lucky Day — Recognizing and fostering luck — Taking

    advantage of luck — Building resiliency to bad luck — Saving your luck for when you need it — Slack data's lucky break CrunchConf 2017 | Ronnie Chen @rondo!w 5
  4. Luck is what happens when preparation meets opportunity — [Not]

    Seneca CrunchConf 2017 | Ronnie Chen @rondo!w 7
  5. If one does not know to which port one is

    sailing, no wind is favorable. — Seneca, Letter LXXI CrunchConf 2017 | Ronnie Chen @rondo!w 8
  6. ❌ Build a new service for running large queries using

    [XYZ technology] CrunchConf 2017 | Ronnie Chen @rondo!w 12
  7. ✅ Make querying the data warehouse reliable & performant even

    for large data sets ✅ Supports queries with billions of rows ✅ 95% of typical queries run in under 20 seconds ✅ Data shows up within 5 min of being written CrunchConf 2017 | Ronnie Chen @rondo!w 13
  8. If I had asked people what they wanted, they would

    have said faster horses — [Not] Henry Ford CrunchConf 2017 | Ronnie Chen @rondo!w 18
  9. ! – earliest testable ! – earliest usable ! –

    earliest lovable CrunchConf 2017 | Ronnie Chen @rondo!w 20
  10. You get love for it. You get hate for it.

    You get nothing if you wait for it, wait for it. — Alexander Hamilton (via Lin Manuel Miranda) CrunchConf 2017 | Ronnie Chen @rondo!w 23
  11. 28

  12. – make a query – create a graph – build

    a dashboard with selectors CrunchConf 2017 | Ronnie Chen @rondo!w 33
  13. Me and My Shadow function presto_query($query, $opts=[]){ if (is_select($query) &&

    !$opts['is_shadow']){ presto_query($query, $opts + ['is_shadow' => true]); } ... } CrunchConf 2017 | Ronnie Chen @rondo!w 36
  14. Data for the Data God data engineers queries success rate

    is shadow 3623 0.99558 false 3083 0.96497 true CrunchConf 2017 | Ronnie Chen @rondo!w 37
  15. ! Luck is real. The effects of luck are undeniable.

    CrunchConf 2017 | Ronnie Chen @rondo!w 44