Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
[PyCon KR 2026] More Variants, More Diversity for AI Accelerators
Avatar for Joongi Kim achimnol
0
120
「人を評価する AI」の 設計と実装 
Avatar for 奈良亮耶 ryoyanara
0
260
バグを直したら useEffect が消えた
Avatar for taki colorful12
3
770
夏だ！祭りだ！祭りとはドメインモデリングでは？
Avatar for ryugen04 ryugen04
0
440
承認済みなのに差戻しできてしまうバグ、型で潰せます
Avatar for SHINCHI Takahiro shinchit
0
120
Jindong: Introducing Declarative Haptics in Compose Multiplatform
Avatar for HyunWoo Lee l2hyunwoo
0
130
コンパウンドプロダクト開発のためのローカルプロセスマネージャー再発明 #layerxgo
Avatar for izumin5210 izumin5210
0
470
Japan Community Day at Kubecon + CloudNativeCon Japan 2026: Learning Container Privilege Control by Building My Own Low-Level Container Runtime
Avatar for ternbusty ternbusty
1
170
AI時代に学ぶ 好きなルール 嫌いなルール Linter編
Avatar for Shota Nemoto shorty5121
0
200
TSX の <Hoge<Fuga>> という構文に驚いた話 / tsx-type-argument-syntax
Avatar for Kanaru kanaru0928
0
240
書籍「プロフェッショナルAI駆動開発」紹介スライド
Avatar for Juntaro Matsumoto juntaromatsumoto
0
780
進化を続けるGo toolsの現在地 / The Current State of Ever-Evolving Go Tools
Avatar for hond hond0413
0
270

Featured

See All Featured
30 Presentation Tips
Avatar for Ian Lurie portentint
PRO
1
380
So, you think you're a good person
Avatar for Per Axbom axbom
PRO
2
2.1k
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
780
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
42
3.1k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.8k
Navigating the Design Leadership Dip - Product Design Week Design Leaders+ Conference 2024
Avatar for Andy Polaine apolaine
2
410
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
220
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
Avatar for Eileen M. Uchitelle eileencodes
141
35k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
201
75k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
41
2.7k
Agile Leadership in an Agile Organization
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
210
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
1
10k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]