Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
re:Invent 2025 トレンドからみる製品開発への AI Agent 活用
Avatar for Kohei Yoshikawa yoskoh
0
720
0→1 フロントエンド開発 Tips🚀 #レバテックMeetup
Avatar for 弁護士ドットコム bengo4com
0
540
Honoを使ったリモートMCPサーバでAIツールとの連携を加速させる！
Avatar for とすり tosuri13
1
170
AIで開発はどれくらい加速したのか？AIエージェントによるコード生成を、現場の評価と研究開発の評価の両面からdeep diveしてみる
Avatar for どすこい daisuketakeda
1
960
ZJIT: The Ruby 4 JIT Compiler / Ruby Release 30th Anniversary Party
Avatar for Takashi Kokubun k0kubun
1
390
QAフローを最適化し、品質水準を満たしながらリリースまでの期間を最短化する #RSGT2026
Avatar for shibayu36 shibayu36
2
4.2k
余白を設計しフロントエンド開発を 加速させる
Avatar for karacoro / からころ tsukuha
7
2.1k
AI時代のキャリアプラン「技術の引力」からの脱出と「問い」へのいざない / tech-gravity
Avatar for MinoDriven minodriven
17
5.6k
OCaml 5でモダンな並列プログラミングを Enjoyしよう!
Avatar for Haochen Kotoi-Xie haochenx
0
110
KIKI_MBSD Cybersecurity Challenges 2025
Avatar for ikema ikema
0
1.3k
Basic Architectures
Avatar for Denys Poltorak denyspoltorak
0
650
dchart: charts from deck markup
Avatar for Anthony Starks ajstarks
3
990

Featured

See All Featured
How to Get Subject Matter Experts Bought In and Actively Contributing to SEO & PR Initiatives.
Avatar for Liv Day livdayseo
0
53
Lessons Learnt from Crawling 1000+ Websites
Avatar for Charles Meaden charlesmeaden
PRO
1
1.1k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
Documentation Writing (for coders)
Avatar for Carmen Chung carmenintech
77
5.2k
Color Theory Basics | Prateek | Gurzu
Avatar for Gurzu gurzu
0
190
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
Avatar for Aleyda Solis aleyda
1
1.1k
BBQ
Avatar for Matthew Crist matthewcrist
89
10k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
Avatar for Michelle Schulp Hunt marktimemedia
31
2.7k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
527
40k
16th Malabo Montpellier Forum Presentation
Avatar for AKADEMIYA2063 akademiya2063
PRO
0
46

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]