Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
ドメイン駆動設計のエッセンス
Avatar for 増田 亨 masuda220
PRO
15
7.2k
スマホから Youtube Shortsを見られないようにする
Avatar for lemolatoon lemolatoon
27
34k
When Dependencies Fail: Building Antifragile Applications in a Fragile World
Avatar for Selçuk Usta selcukusta
0
120
Temporal Knowledge Graphで作る！ 時間変化するナレッジを扱うAI Agentの世界
Avatar for po3rin po3rin
5
1.1k
Module Proxyのマニアックな話 / Niche Topics in Module Proxy
Avatar for kuro kuro_kurorrr
0
970
三者三様 宣言的UI
Avatar for Keita Kagurazaka kkagurazaka
0
330
Kotlin 2.2が切り拓く: コンテキストパラメータで書く関数型DSLと新しい依存管理のかたち
Avatar for Kenichi SUZUKI knih
0
210
ビルドプロセスをデバッグしよう！
Avatar for Yuta Tomiyama yt8492
0
200
The Past, Present, and Future of Enterprise Java
Avatar for ivargrimstad ivargrimstad
0
660
CSC305 Lecture 11
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
0
320
Kotlinで実装するCPU/GPU 「協調的」パフォーマンス管理
Avatar for matuyuhi matuyuhi
0
230
Vueのバリデーション、結局どれを選べばいい？ ― 自作バリデーションの限界と、脱却までの道のり ― / Which Vue Validation Library Should We Really Use? The Limits of Self-Made Validation and How I Finally Moved On
Avatar for ねぎなす neginasu
3
1.8k

Featured

See All Featured
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
34
8.9k
Context Engineering - Making Every Token Count
Avatar for Addy Osmani addyosmani
8
330
No one is an island. Learnings from fostering a developers community.
Avatar for Antonio thoeni
21
3.5k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
CSS Pre-Processors: Stylus, Less & Sass
Avatar for Bermon Painter bermonpainter
359
30k
[RailsConf 2023 Opening Keynote] The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
31
9.7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
34
2.3k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.7k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
7
700
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
190
11k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]