Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
意外と簡単！？フロントエンドでパスキー認証を実現する WebAuthn
Avatar for teamLab teamlab
PRO
2
780
アセットのコンパイルについて
Avatar for ojun ojun9
0
130
FindyにおけるTakumi活用と脆弱性管理のこれから
Avatar for adachi.ryo rvirus0817
0
540
アルテニア コンサル／ＩＴエンジニア向け 採用ピッチ資料
Avatar for ALTENIR altenir
0
110
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
Avatar for Taito Tanaka taitotnk
0
870
Processing Gem ベースの、2D レトロゲームエンジンの開発
Avatar for tokujiros tokujiros
2
130
もうちょっといいRubyプロファイラを作りたい (2025)
Avatar for osyoyu osyoyu
1
460
HTMLの品質ってなんだっけ？ “HTMLクライテリア”の設計と実践
Avatar for una unachang113
4
2.9k
奥深くて厄介な「改行」と仲良くなる20分
Avatar for おぐえもん oguemon
1
570
パッケージ設計の黒魔術/Kyoto.go#63
Avatar for kadota kyohei lufia
3
440
Android 16 × Jetpack Composeで縦書きテキストエディタを作ろう / Vertical Text Editor with Compose on Android 16
Avatar for cc4966 cc4966
2
270
複雑なドメインに挑む.pdf
Avatar for Yuki Sakai yukisakai1225
5
1.2k

Featured

See All Featured
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.6k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
271
21k
We Have a Design System, Now What?
Avatar for Morgane Peng morganepeng
53
7.8k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
15
1.7k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
113
20k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
667
120k
Helping Users Find Their Own Way: Creating Modern Search Experiences
Avatar for Dan Newman danielanewman
29
2.9k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
26
1.9k
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
127
53k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]