Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
情報漏洩させないための設計
kubotak
5
1.3k
watsonx.ai Dojo #6 継続的なAIアプリ開発と展開
oniak3ibm
PRO
0
170
Flatt Security XSS Challenge 解答・解説
flatt_security
0
740
Beyond ORM
77web
11
1.6k
Scaling your build logic
antalmonori
1
100
カンファレンス動画鑑賞会のススメ / Osaka.swift #1
hironytic
0
170
DevinとCursorから学ぶAIエージェントメモリーの設計とMoatの考え方
itarutomy
0
150
php-conference-japan-2024
tasuku43
0
430
いりゃあせ、PHPカンファレンス名古屋2025 / Welcome to PHP Conference Nagoya 2025
ttskch
1
180
『改訂新版 良いコード／悪いコードで学ぶ設計入門』活用方法−爆速でスキルアップする！効果的な学習アプローチ / effective-learning-of-good-code
minodriven
28
4.2k
ゼロからの、レトロゲームエンジンの作り方
tokujiros
3
1.1k
テストコード書いてみませんか？
onopon
2
340

Featured

See All Featured
Writing Fast Ruby
sferik
628
61k
Agile that works and the tools we love
rasmusluckow
328
21k
Making the Leap to Tech Lead
cromwellryan
133
9k
Statistics for Hackers
jakevdp
797
220k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
Practical Orchestrator
shlominoach
186
10k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
3
240
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.2k
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]