Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
モックわからないマン卒業記 ~振る舞いを起点に見直した、フロントエンドテストにおけるモックの使いどころ~
Avatar for Tasuku Watanabe tasukuwatanabe
3
440
Coding at the Speed of Thought: The New Era of Symfony Docker
Avatar for Kévin Dunglas dunglas
0
4.1k
Migration to Signals, Signal Forms, Resource API, and NgRx Signal Store @Angular Days 03/2026 Munich
Avatar for Manfred Steyer manfredsteyer
PRO
0
210
20260320登壇資料
Avatar for 松井遼(まついはるか) pharct
0
140
Rethinking API Platform Filters
Avatar for Vincent Amstoutz vinceamstoutz
0
5k
「速くなった気がする」をデータで疑う
Avatar for Leaf senleaf24
0
130
AWS×クラウドネイティブソフトウェア設計 / AWS x Cloud-Native Software Design
Avatar for nrs nrslib
16
3.5k
AI時代のシステム設計：ドメインモデルで変更しやすさを守る設計戦略
Avatar for 増田 亨 masuda220
PRO
6
1.2k
Claude Codeログ基盤の構築
Avatar for giginet giginet
PRO
7
3.8k
forteeの改修から振り返るPHPerKaigi 2026
Avatar for muno92 muno92
PRO
3
110
RSAが破られる前に知っておきたい 耐量子計算機暗号(PQC)入門 / Intro to PQC: Preparing for the Post-RSA Era
Avatar for mackey0225 mackey0225
3
110
脱 雰囲気実装！ AgentCoreを良い感じに WEBアプリケーションに組み込むために
Avatar for Takuya Yonezawa takuyay0ne
3
420

Featured

See All Featured
My Coaching Mixtape
Avatar for mlcsv mlcsv
0
91
Primal Persuasion: How to Engage the Brain for Learning That Lasts
Avatar for Mike Taylor tmiket
0
310
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
Avatar for Helen Beal helenjbeal
1
160
Code Reviewing Like a Champion
Avatar for maltzj maltzj
528
40k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
100
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
75
12k
The untapped power of vector embeddings
Avatar for Frank van Dijk frankvandijk
2
1.7k
Large-scale JavaScript Application Architecture
Avatar for Addy Osmani addyosmani
515
110k
The B2B funnel & how to create a winning content strategy
Avatar for Katarina Dahlin katarinadahlin
PRO
1
320
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
12
1.6k
Dominate Local Search Results - an insider guide to GBP, reviews, and Local SEO
Avatar for Greg Gifford greggifford
PRO
0
130
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
320

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]