Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.3k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Reactive Thinking with Signals, Resource API, and httpResource @Devm.io Angular 20 Launch Party
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
ユーザーにサブドメインの ECサイトを提供したい (あるいは) 2026年函館で一番熱くなるかもしれない言語の話
Avatar for uvb_76 uvb_76
0
170
AI時代のリアーキテクチャ戦略 / Re-architecture Strategy in the AI Era
Avatar for dachi023 dachi023
0
190
What Spring Developers Should Know About Jakarta EE
Avatar for ivargrimstad ivargrimstad
1
580
Agent Rules as Domain Parser
Avatar for Yoda Keisuke yodakeisuke
1
330
Language Server と喋ろう – TSKaigi 2025
Avatar for pizzacat83 pizzacat83
2
660
TypeScriptのmoduleオプションを改めて整理する
Avatar for おおいし bicstone
4
420
當開發遇上包裝： AI 如何讓產品從想法變成商品
Avatar for Caesar Chi clonn
0
2.5k
Efficiency and Rock 'n’ Roll (Really!)
Avatar for Holly Cummins hollycummins
0
590
Investigating Multithreaded PostgreSQL
Avatar for Thomas Munro macdice
0
150
型付け力を強化するための Hoogle のすゝめ / Boosting Your Type Mastery with Hoogle
Avatar for TAKASE Kazuyuki guvalif
1
230
TypeScript Language Service Plugin で CSS Modules の開発体験を改善する
Avatar for mizdra mizdra
PRO
3
2.4k

Featured

See All Featured
Sharpening the Axe: The Primacy of Toolmaking
Avatar for Bryan Cantrill bcantrill
42
2.3k
Writing Fast Ruby
Avatar for Erik Berlin sferik
628
61k
Making Projects Easy
Avatar for Brett Harned brettharned
116
6.2k
It's Worth the Effort
Avatar for 3n 3n
184
28k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Fontdeck: Realign not Redesign
Avatar for Paul Robert Lloyd paulrobertlloyd
84
5.5k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
69
4.7k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
78
6.4k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
92
6k
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
462
33k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
840
Bootstrapping a Software Product
Avatar for Garrett Dimon garrettdimon
PRO
307
110k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks asmith@renttherunway.com