Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
さぁV100、メモリをお食べ・・・
Avatar for nilpe nilpe
0
150
フロントエンドとバックエンドで「1文字」を揃えよう
Avatar for てきめん tekimen youkidearitai
PRO
0
740
AIだと陥りがちなJakarta EE最新技術への移行時の落とし穴と解決策
Avatar for tnagao7 tnagao7
0
120
Strategic Design in the Frontend: Moduliths & Micro Frontends @DDDEurope
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
Performance Engineering for Everyone
Avatar for Elena Tanasoiu elenatanasoiu
0
210
トークンをケチるな、設計しろ：GitHub Copilotを賢く使うコンテキスト戦略
Avatar for Ochtum ochtum
0
160
The ROI of Quarkus for Spring Boot Applications
Avatar for Holly Cummins hollycummins
0
140
Signal Forms: Details & Live Coding @enterJS 2026 in Mannheim
Avatar for Manfred Steyer manfredsteyer
PRO
0
190
JavaDoc 再入門
Avatar for nagise nagise
1
410
はてなアカウント基盤 State of the Union
Avatar for cockscomb cockscomb
0
640
コンテキストの使い捨てをやめる — ビジネスルール駆動開発と miko —
Avatar for ioki ioki
0
230
キャリア迷子上等 ─ "ない道"は自分で作ればいい
Avatar for 16bit_idol 16bitidol
3
2.3k

Featured

See All Featured
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
62
44k
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
57
14k
Building a A Zero-Code AI SEO Workflow
Avatar for Ian Lurie portentint
PRO
0
610
A Soul's Torment
Avatar for Nat Ma seathinner
6
3k
Have SEOs Ruined the Internet? - User Awareness of SEO in 2025
Avatar for Akash Hashmi akashhashmi
0
370
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
440
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
200
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
615
220k
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
2
1.6k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.2k
Highjacked: Video Game Concept Design
Avatar for Ryan Kendrick rkendrick25
PRO
1
400
How to Grow Your eCommerce with AI & Automation
Avatar for Katarina Dahlin katarinadahlin
PRO
1
210

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]