Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Building a macOS screen saver with Kotlin (Android Makers 2025)
zsmb
1
140
AI Coding Agent Enablement - エージェントを自走させよう
yukukotani
13
5.8k
AHC 044 混合整数計画ソルバー解法
kiri8128
0
330
タイムゾーンの奥地は思ったよりも闇深いかもしれない
suguruooki
1
570
エンジニア未経験が最短で戦力になるためのTips
gokana
0
260
サービスレベルを管理してアジャイルを加速しよう！！ / slm-accelerate-agility
tomoyakitaura
1
170
API for docs
soutaro
1
590
サービスクラスのありがたみを発見したときの思い出 #phpcon_odawara
77web
4
630
地域ITコミュニティの活性化とAWSに移行してみた話
yuukis
0
230
Chrome Extension Techniques from Hell
moznion
1
160
フロントエンドテストの育て方
quramy
11
2.9k
Making TCPSocket.new "Happy"!
coe401_
1
130

Featured

See All Featured
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Designing for Performance
lara
607
69k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.4k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.6k
Building Applications with DynamoDB
mza
94
6.3k
The Pragmatic Product Professional
lauravandoore
33
6.5k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
9
740
Testing 201, or: Great Expectations
jmmastey
42
7.4k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]