Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
アセットのコンパイルについて
Avatar for ojun ojun9
0
130
AIと私たちの学習の変化を考える - Claude Codeの学習モードを例に
Avatar for azukiazusa azukiazusa1
11
4.5k
How Android Uses Data Structures Behind The Scenes
Avatar for HyunWoo Lee l2hyunwoo
0
480
ぬるぬる動かせ！ Riveでアニメーション実装🐾
Avatar for Kuno Ayana kno3a87
1
230
Updates on MLS on Ruby (and maybe more)
Avatar for sylph01 sylph01
1
180
Introducing ReActionView: A new ActionView-compatible ERB Engine @ Rails World 2025, Amsterdam
Avatar for Marco Roth marcoroth
0
710
Azure SRE Agentで運用は楽になるのか？
Avatar for KAMEGAWA Kazushi kkamegawa
0
2.6k
The Past, Present, and Future of Enterprise Java with ASF in the Middle
Avatar for ivargrimstad ivargrimstad
0
190
AI Coding Agentのセキュリティリスク：PRの自己承認とメルカリの対策
Avatar for さうす s3h
0
240
print("Hello, World")
Avatar for 高見龍 eddie
2
530
ファインディ株式会社におけるMCP活用とサービス開発
Avatar for starfish719 starfish719
0
2.1k
AI Agents: How Do They Work and How to Build Them @ Shift 2025
Avatar for Slobodan Stojanović slobodan
0
110

Featured

See All Featured
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.4k
StorybookのUI Testing Handbookを読んだ
Avatar for Shingo Yamazaki zakiyama
31
6.1k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
3
63
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
8
930
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
303
21k
Side Projects
Avatar for Sacha Greif sachag
455
43k
Imperfection Machines: The Place of Print at Facebook
Avatar for Scott Boms scottboms
268
13k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.2k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
188
55k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.8k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
431
66k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]