Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
Avatar for panda_program panda_program
6
2.2k
AIと”コードの評価関数”を共有する / Share the "code evaluation function" with AI
Avatar for Shintani Teppei euglena1215
1
170
Agentic Coding: The Future of Software Development with Agents
Avatar for Armin Ronacher mitsuhiko
0
110
The Niche of CDK Grant オブジェクトって何者？/the-niche-of-cdk-what-isgrant-object
Avatar for hassaku63 hassaku63
1
230
The Evolution of Enterprise Java with Jakarta EE 11 and Beyond
Avatar for ivargrimstad ivargrimstad
0
120
What's new in AppKit on macOS 26
Avatar for 1024jp 1024jp
0
120
技術同人誌をMCP Serverにしてみた
Avatar for 74th(Atsushi Morimoto) 74th
1
670
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
Avatar for Takuya Katsusa takuyakatsusa
2
570
なぜ適用するか、移行して理解するClean Architecture 〜構造を超えて設計を継承する〜 / Why Apply, Migrate and Understand Clean Architecture - Inherit Design Beyond Structure
Avatar for shiro seike seike460
PRO
3
780
ISUCON研修おかわり会 講義スライド
Avatar for Yuki Yata arfes0e2b3c
1
450
RailsGirls IZUMO スポンサーLT
Avatar for 16bit_idol 16bitidol
0
190
なんとなくわかった気になるブロックテーマ入門/contents.nagoya 2025 6.28
Avatar for Chiaki Okamoto chiilog
1
280

Featured

See All Featured
KATA
Avatar for Megan Lloyd mclloyd
30
14k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.7k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
25
1.7k
Git: the NoSQL Database
Avatar for Brandon Keepers bkeepers
PRO
430
65k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
3.9k
Statistics for Hackers
Avatar for Jake VanderPlas jakevdp
799
220k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
Avatar for Joe Casabona jcasabona
32
2.4k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
23
3.5k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
278
23k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.3k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]