Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Node-RED を（HTTP で）つなげる MCP サーバーを作ってみた
Avatar for High U highu
0
120
VS Code Update for GitHub Copilot
Avatar for 74th(Atsushi Morimoto) 74th
2
670
Hack Claude Code with Claude Code
Avatar for Akihiro Okuno choplin
6
2.4k
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ～継続的な取り組みを添えて～
Avatar for Takuya TAKAHASHI takutakahashi
1
310
はじめてのWeb API体験 ー 飲食店検索アプリを作ろうー
Avatar for Aki Nakahara akinko_0915
0
130
『自分のデータだけ見せたい！』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
Avatar for AkitoTsukahara akitotsukahara
2
650
CDK引数設計道場100本ノック
Avatar for kazuho cryer-shinozuka badmintoncryer
2
360
PHPでWebSocketサーバーを実装しよう2025
Avatar for kubotak kubotak
0
310
Result型で“失敗”を型にするPHPコードの書き方
Avatar for Takuma Kajikawa kajitack
5
980
dbt民主化とLLMによる開発ブースト ~ AI Readyな分析サイクルを目指して ~
Avatar for yosh_yumyum yoshyum
3
1.1k
PicoRuby on Rails
Avatar for makicamel makicamel
2
140
LT 2025-06-30: プロダクトエンジニアの役割
Avatar for Keisuke Yamamoto yamamotok
0
810

Featured

See All Featured
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
Automating Front-end Workflow
Avatar for Addy Osmani addyosmani
1370
200k
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
31
1.3k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
299
21k
[RailsConf 2023] Rails as a piece of cake
Avatar for Vladimir Dementyev palkan
55
5.7k
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
10
700
Building Applications with DynamoDB
Avatar for Matt Wood mza
95
6.5k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
35
6.7k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
29
5.4k
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.3k
Designing for Performance
Avatar for Lara Hogan lara
610
69k
Fireside Chat
Avatar for paigeccino paigeccino
37
3.5k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]