Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
hakka_labs
0
2.8k
Groupcache with Evan Owen
hakka_labs
2
5.3k
Testing Android at Spotify
hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
hakka_labs
0
5k
Deep Learning and NLP Applications
hakka_labs
3
13k
Go and the Gophers
hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Keeping it Ruby: Why Your Product Needs a Ruby SDK - RubyWorld 2024
envek
0
190
ChatGPT とつくる PHP で OS 実装
memory1994
PRO
2
110
Haze - Real time background blurring
chrisbanes
1
510
rails statsで大解剖 🔍 “B/43流” のRailsの育て方を歴史とともに振り返ります
shoheimitani
2
940
テストコード文化を0から作り、変化し続けた組織
kazatohiei
2
1.5k
PSR-15 はあなたのための ものではない？ - phpcon2024
myamagishi
0
140
SymfonyCon Vienna 2025: Twig, still relevant in 2025?
fabpot
3
1.2k
なまけものオバケたち -PHP 8.4 に入った新機能の紹介-
tanakahisateru
1
120
Amazon S3 NYJavaSIG 2024-12-12
sullis
0
100
これが俺の”自分戦略” プロセスを楽しんでいこう！ - Developers CAREER Boost 2024
niftycorp
PRO
0
190
CQRS+ES の力を使って効果を感じる / Feel the effects of using the power of CQRS+ES
seike460
PRO
0
140
testcontainers のススメ
sgash708
1
120

Featured

See All Featured
Build your cross-platform service in a week with App Engine
jlugia
229
18k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
48
2.2k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
26
1.9k
Measuring & Analyzing Core Web Vitals
bluesmoon
4
170
Rails Girls Zürich Keynote
gr2m
94
13k
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
StorybookのUI Testing Handbookを読んだ
zakiyama
27
5.3k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
Music & Morning Musume
bryan
46
6.2k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.6k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]