Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
一人でAIプロダクトを作るための工夫 〜技術選定・開発プロセス編〜 / I want AI to work harder
Avatar for r-kagaya rkaga
13
2.8k
MCPで実現するAIエージェント駆動のNext.jsアプリデバッグ手法
Avatar for NakamuraTakumi nyatinte
6
840
令和最新版手のひらコンピュータ
Avatar for KOBA789 koba789
14
8k
レガシープロジェクトで最大限AIの恩恵を受けられるようClaude Codeを利用する
Avatar for tk1351 tk1351
2
1.1k
decksh - a little language for decks
Avatar for Anthony Starks ajstarks
4
21k
Terraform やるなら公式スタイルガイドを読もう 〜重要項目 10選〜
Avatar for hiyanger hiyanger
13
3.2k
Scale out your Claude Code ~⁨⁩自社専用Agentで10xする開発プロセス~
Avatar for Yuku Kotani yukukotani
9
2.6k
TDD 実践ミニトーク
Avatar for Daisuke Garaike contour_gara
0
140
画像コンペでのベースラインモデルの育て方
Avatar for tattaka tattaka
3
1.8k
私の後悔をAWS DMSで解決した話
Avatar for 平目 hiramax
4
130
CEDEC2025 長期運営ゲームをあと10年続けるための0から始める自動テスト ~4000項目を50%自動化し、月1→毎日実行にした3年間~
Avatar for アカツキゲームス_Tech akatsukigames_tech
0
150
CSC305 Summer Lecture 04
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
110

Featured

See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
283
13k
GraphQLとの向き合い方2022年版
Avatar for Yosuke Kurami quramy
49
14k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
83
9.1k
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
PRO
23
1.4k
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
53k
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.4k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.7k
4 Signs Your Business is Dying
Avatar for Josh Pigford shpigford
184
22k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
Avatar for Chris Coyier chriscoyier
507
140k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
279
23k
Build your cross-platform service in a week with App Engine
Avatar for Jose L jlugia
231
18k
How GitHub (no longer) Works
Avatar for Zach Holman holman
315
140k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]