Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Gemini CLIの"強み"を知る！ Gemini CLIとClaude Codeを比較してみた！
Avatar for Kota Hisafuru kotahisafuru
3
920
新しいモバイルアプリ勉強会（仮）について
Avatar for 上ちょ / uetyo uetyo
1
250
AIコーディングエージェント全社導入とセキュリティ対策
Avatar for hikae hikaruegashira
16
9.4k
Google I/O Extended Incheon 2025 ~ What's new in Android development tools
Avatar for pluulove (노현석) pluu
1
220
Terraform やるなら公式スタイルガイドを読もう 〜重要項目 10選〜
Avatar for hiyanger hiyanger
11
2.9k
Understanding Kotlin Multiplatform
Avatar for HyunWoo Lee l2hyunwoo
0
250
知って得する@cloudflare_vite-pluginのあれこれ
Avatar for chimame chimame
1
140
React は次の10年を生き残れるか：3つのトレンドから考える
Avatar for Yuka O’oka oukayuka
41
16k
AWS Summit Japan 2024と2025の比較／はじめてのKiro、今あなたは岐路に立つ
Avatar for Satoshi Kaneyasu satoshi256kbyte
1
260
Go製CLIツールをnpmで配布するには
Avatar for syumai syumai
2
1.1k
[DevinMeetupTokyo2025] コード書かせないDevinの使い方
Avatar for yoshi takumiyoshikawa
2
260
新世界の理解
Avatar for Akihito Koriyama koriym
0
130

Featured

See All Featured
Code Reviewing Like a Champion
Avatar for maltzj maltzj
524
40k
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
43
7.4k
Embracing the Ebb and Flow
Avatar for Simon Collison colly
86
4.8k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
7
790
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1031
460k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
379
70k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
130
19k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
126
17k
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
207
24k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
182
54k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]