Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
そこに3びきプロダクトがいるじゃろう——生成AI時代における“価値が届かない理由”の構造
Avatar for kosuke T kosuket
0
440
Claude CodeとAgentCore Gatewayを繋ぐ際の認証認可 / Authentication and authorization when connecting Claude Code with AgentCore Gateway
Avatar for Har1101 har1101
2
260
作るコストが小さくなった時代 幸せに働くために改めて考えたいこと 〜エンジニアとして価値を出し続けるために注視している二分野〜
Avatar for YuppeEng yuppeeng
0
190
霧の中の代数的エフェクト
Avatar for funnycat funnyycat
1
490
React本体のコードリーディング
Avatar for high-g high_g_engineer
1
140
ドリフトを絶対に許さない（？）CDK運用 / CDK Ops with Zero Tolerance for Drifts (?)
Avatar for アキキー | Akihisa Ikeda akihisaikeda
1
180
これって Effect でできたのでは? / TSKaigi Mashup Kansai #2
Avatar for Susisu susisu
0
150
源内ハンズオン概要編
Avatar for Hideyuki Nagata hideg
0
140
AI時代に設計が 最大の生産性レバーになる 意図駆動開発とデータを消さない設計｜Don't Delete Your Data or Your Intent — Design as the Deepest Lever in the AI Era
Avatar for Tomohisa Takaoka tomohisa
1
740
Welcome to the "Parametricity" 🏙️ − Generic だけど Specific な世界 −
Avatar for TAKASE Kazuyuki guvalif
PRO
1
210
複数の Claude Code が"放置"されてしまう問題をCLI ダッシュボードを自作して解決した話
Avatar for sumihiro3 sumihiro3
1
670
jsmini JavaScript Engine を作ってみた話
Avatar for Yosuke Furukawa yosuke_furukawa
PRO
0
300

Featured

See All Featured
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2.2k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
63
45k
Pawsitive SEO: Lessons from My Dog (and Many Mistakes) on Thriving as a Consultant in the Age of AI
Avatar for David Carrasco davidcarrasco
0
210
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Visual Storytelling: How to be a Superhuman Communicator
Avatar for David Neal reverentgeek
2
610
Deep Space Network (abreviated)
Avatar for Tony Rice tonyrice
0
250
What does AI have to do with Human Rights?
Avatar for Per Axbom axbom
PRO
1
2.3k
Ten Tips & Tricks for a 🌱 transition
Avatar for Manu Carrasco Molina stuffmc
0
160
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
201
75k
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
2
450
Scaling GitHub
Avatar for Zach Holman holman
464
140k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]