Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
モダンJSフレームワークのビルドプロセス 〜なぜReactは503行、Svelteは12行なのか〜
Avatar for Maple fuuki12
0
180
connect-python: convenient protobuf RPC for Python
Avatar for Anuraag Agrawal anuraaga
0
350
CSC305 Lecture 17
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
270
テストやOSS開発に役立つSetup PHP Action
Avatar for Atsushi Matsuo matsuo_atsushi
0
140
新卒エンジニアのプルリクエスト with AI駆動
Avatar for Yuta Fukunaga fukunaga2025
0
100
AI時代もSEOを頑張っている話
Avatar for しらはま shirahama_x
0
220
Rediscover the Console - SymfonyCon Amsterdam 2025
Avatar for Robin Chalas chalasr
2
130
AWS CDKの推しポイントN選
Avatar for アキキー akihisaikeda
1
240
Level up your Gemini CLI - D&D Style!
Avatar for Riccardo Carlesso palladius
1
170
スタートアップを支える技術戦略と組織づくり
Avatar for pospome pospome
8
15k
全員アーキテクトで挑む、 巨大で高密度なドメインの紐解き方
Avatar for Agata Naomichi agatan
8
18k
著者と進める！『AIと個人開発したくなったらまずCursorで要件定義だ！』
Avatar for yasuna yasunacoffee
0
110

Featured

See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.6k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
370
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.3k
Art, The Web, and Tiny UX
Avatar for Lynn Fisher lynnandtonic
303
21k
ReactJS: Keep Simple. Everything can be a component!
Avatar for Pedro Nauck pedronauck
666
130k
Mobile First: as difficult as doing things right
Avatar for Swwweet swwweet
225
10k
Keith and Marios Guide to Fast Websites
Avatar for Keith Pitt keithpitt
413
23k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
8
1.2k
Building an army of robots
Avatar for Kyle Neath kneath
306
46k
Building Adaptive Systems
Avatar for Chris Keathley keathley
44
2.9k
Build The Right Thing And Hit Your Dates
Avatar for Maggie C. maggiecrowley
38
3k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
329
39k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]