Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Evolving Sustainable Data Pipelines
Search
Hakka Labs
February 13, 2015
Programming
3.5k
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Evolving Sustainable Data Pipelines
Full post here:
Hakka Labs
February 13, 2015
Transcript
35 slides
None
ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29
January 2015
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
None
1. 2. 3.
WHERE THE MAGIC HAPPENS
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation
DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION
reservation calendar
None
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 0
None
YUP
KEEP IT TOGETHER
PHASE 1 primordial soup
STABILITY
PHASE 2 process
orderwarehouse.job
$ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job
--only 2
runjob.py
ADAPTING ensuring data quality
PHASE 3 exposing weaknesses
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
dependency manager
None
None
A bit about RTR A bit about our analytics Some
evolution stuff A bit about Luigi The future bits
PHASE 4 ownership
RELIABILITY
COMMUNICATION
THE FUTURE @OMGannaks
[email protected]
More Decks by Hakka Labs
See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.9k
Groupcache with Evan Owen
hakka_labs
2
5.4k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2.1k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5.1k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k
Other Decks in Programming
See All in Programming
TS 7.0 で baseUrl が使えなくなった今、パスエイリアスをどうするか
bicstone
2
760
Spec-Driven Development - Why Vaadin Is the Perfect Fit
simas
PRO
0
260
祈るテストから、探索するテストへ 〜Lincheckではじめる並行処理テスト〜
yyh_gl
0
350
조합이폭발하는디자인시스템이걸눈으로도테스트하라고요
giljongyu
0
780
作れるエンジニアから、価値を届けるエンジニアへ
ryunakayama
1
310
Kotlin JVM用に作られたライブラリのKMPを前提とした再設計
jsoizo
0
220
Proxmox x Talos Linux x Terraform で実現するなんかいい感じの Kubernetes クラスタ / A Sleek Kubernetes Cluster Powered by Proxmox, Talos Linux, and Terraform
nnaka2992
0
270
Python Lambdaにおけるテストしやすい設計とAI支援の活用 / Designing Testable Python Lambda Functions with AI
kitano_yuichi
1
340
オーケストレーション記述言語 Katari
yukikurage
0
270
複数ドメインを一つのアプリに統合するFlutterの設計と運用_FlutterKaigi2026登壇資料(須永高弘)
takahirosunaga
0
200
小さなアプリを作って公開するまで
onozaty
0
210
Claude Code ハーネスDeep Dive – 君はどこまで制御する? -
takateeen_knj
6
6.2k
Featured
See All Featured
16th Malabo Montpellier Forum Presentation
akademiya2063
PRO
0
470
Speed Design
sergeychernyshev
33
2.2k
The Illustrated Guide to Node.js - THAT Conference 2024
reverentgeek
1
610
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
28
3.8k
New Earth Scene 8
popppiees
5
2.9k
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
1.7k
Principles of Awesome APIs and How to Build Them.
keavy
128
18k
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
Fantastic passwords and where to find them - at NoRuKo
philnash
53
4k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
tammyeverts
2
580
Groundhog Day: Seeking Process in Gaming for Health
codingconduct
0
490