Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Evolving Sustainable Data Pipelines
Search
Hakka Labs
February 13, 2015
Programming
0
3.5k
Evolving Sustainable Data Pipelines
Full post here:
Hakka Labs
February 13, 2015
Tweet
Share
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
12k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
Rubyとクリエイティブコーディングの輪の広がり / The Growing Circle of Ruby and Creative Coding
chobishiba
1
270
o1モデルのプロンプトエンジニアリングって?
ktc_wada
0
260
Android開発以外のAndroid開発経験の活かしどころ
konifar
2
1k
Amazon BedrockでサーバレスなAIお料理ボットを作成する!!
tosuri13
0
230
What you can do with Ruby on WebAssembly
kateinoigakukun
0
170
2024 컴포즈 정원사
jisungbin
0
150
大公開!iOS開発の悩みトップ5 〜iOSDC Japan 2024〜
ryunakayama
0
190
GraphQL あるいは React における自律的なデータ取得について
quramy
11
3k
Google Sign-inの移行から始めるCredential Manager活用
clockvoid
0
310
仮想ファイルシステムを導入して開発環境のストレージ課題を解消する
segadevtech
2
550
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
490
A New Era of Testing
mannodermaus
2
510
Featured
See All Featured
Documentation Writing (for coders)
carmenintech
65
4.3k
Designing for Performance
lara
604
68k
Being A Developer After 40
akosma
84
590k
Side Projects
sachag
451
42k
Agile that works and the tools we love
rasmusluckow
327
20k
Navigating Team Friction
lara
183
13k
Scaling GitHub
holman
458
140k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
326
21k
Code Reviewing Like a Champion
maltzj
517
39k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
225
22k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
89
16k
Mobile First: as difficult as doing things right
swwweet
221
8.8k
Transcript
None
ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29
January 2015
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
None
1. 2. 3.
WHERE THE MAGIC HAPPENS
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION
reservation calendar
None
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 0
None
YUP
KEEP IT TOGETHER
PHASE 1 primordial soup
STABILITY
PHASE 2 process
orderwarehouse.job
$ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job
--only 2
runjob.py
ADAPTING ensuring data quality
PHASE 3 exposing weaknesses
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
dependency manager
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 4 ownership
RELIABILITY
COMMUNICATION
THE FUTURE @OMGannaks
[email protected]