Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
My daily life on Ruby
Avatar for Akira Matsuda a_matsuda
3
390
いつか誰かが、と思っていた フロントエンド刷新5年間の実践知
Avatar for Kiichi Sugihara kiichisugihara
1
280
なぜあなたのコードには「コシ」がないのか？〜AI時代に問う、最後まで美味しい設計と戦略〜 #phpconkagawa / phpconkagawa2026
Avatar for shogogg shogogg
0
210
過去のレビュー知見をSkillsで資産化した話
Avatar for PKSHA Technology（パークシャテクノロジー） pkshadeck
PRO
1
2k
Hive Metastoreを通して学ぶIceberg REST Catalog ― 仕様から実装まで
Avatar for okumin okumin
0
130
t *testing.T は どこからやってくるの？
Avatar for Kotaro Otaka otakakot
1
940
AgentCore Optimizationを始めよう！
Avatar for matsukada licux
3
240
AIを導入する前にやるべきこと
Avatar for Negima negima
2
360
エラー処理の温故知新 / history of error handling technic
Avatar for nakaryo ryotanakaya
7
1.9k
Kingdom of the Machine
Avatar for yui-knk yui_knk
2
1.5k
Back to the roots of date
Avatar for jinroq jinroq
0
850
PicoRuby for IoT: Connecting to the Cloud with MQTT
Avatar for Y_uuu yuuu
2
770

Featured

See All Featured
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
Avatar for UX Y'all uxyall
0
1.4k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
1.1k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
528
40k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
3.2k
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
7
640
Ethics towards AI in product and experience design
Avatar for Skipper Chong Warson skipperchong
2
270
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.6k
Exploring the relationship between traditional SERPs and Gen AI search
Avatar for Ray Grieselhuber raygrieselhuber
PRO
2
4k
Code Review Best Practice
Avatar for Trisha Gee trishagee
74
20k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
140
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.1k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]