Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
AsyncSequenceとAsyncStreamのプロポーザルを全部読む！！
Avatar for Shigure Shimotori s_shimotori
1
230
Blazing Fast UI Development with Compose Hot Reload (Bangladesh KUG, October 2025)
Avatar for Márton Braun zsmb
2
460
One Enishi After Another
Avatar for Koji SHIMADA snoozer05
PRO
0
180
Reactive Thinking with Signals and the Resource API
Avatar for Manfred Steyer manfredsteyer
PRO
0
120
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
530
Making Angular Apps Smarter with Generative AI: Local and Offline-capable
Avatar for Christian Liebel christianliebel
PRO
0
110
Amazon ECS Managed Instances が リリースされた！キャッチアップしよう！！ / Let's catch up Amazon ECS Managed Instances
Avatar for 02 cocoeyes02
0
130
Webサーバーサイド言語としてのRustについて
Avatar for kaza kouyuume
1
5.1k
ネストしたdata classの面倒な更新にさようなら！Lensを作って理解するArrowのOpticsの世界
Avatar for shiita0903 shiita0903
1
260
NIKKEI Tech Talk#38
Avatar for cipepser cipepser
0
380
Designing Repeatable Edits: The Architecture of . in Vim
Avatar for Satoru Kitaguchi satorunooshie
0
240
問題の見方を変える「システム思考」超入門
Avatar for panda_program panda_program
0
110

Featured

See All Featured
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
8
340
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
46
2.5k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
470k
Navigating Team Friction
Avatar for Lara Hogan lara
190
15k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.7k
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.4k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
127
17k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
55
3.1k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
190
11k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
186
22k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
31
2.9k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]