Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
0
3.5k

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015
Tweet

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
go directiveを最新にしすぎないで欲しい話──あるいは、Go 1.26からgo mod initで作られるgo directiveの値が変わる話 / Go 1.26 リリースパーティ
Avatar for Arthur arthur1
2
180
AIコーディングの理想と現実 2026 | AI Coding: Expectations vs. Reality 2026
Avatar for Tomohisa Takaoka tomohisa
0
560
CSC307 Lecture 09
Avatar for Javier Gonzalez-Sanchez javiergs
PRO
1
850
FOSDEM 2026: STUNMESH-go: Building P2P WireGuard Mesh Without Self-Hosted Infrastructure
Avatar for Date Huang tjjh89017
0
190
Head of Engineeringが現場で回した生産性向上施策 2025→2026
Avatar for gessy0129 gessy0129
PRO
0
190
Oxlint JS plugins
Avatar for kazupon kazupon
1
1.1k
個人開発は儲からない - それでも開発開始1ヶ月で300万円売り上げた方法
Avatar for Taishi Yamasaki taishiyade
0
110
Metaprogramming isn't real, it can't hurt you
Avatar for Masafumi Okura okuramasafumi
0
130
朝日新聞のデジタル版を支えるGoバックエンド ー価値ある情報をいち早く確実にお届けするために
Avatar for JunkiIshida junkiishida
1
250
Raku Raku Notion 20260128
Avatar for hareyaka hareyakayuruyaka
0
420
Go 1.26でのsliceのメモリアロケーション最適化 / Go 1.26 リリースパーティ #go126party
Avatar for mazrean mazrean
1
110
浮動小数の比較について
Avatar for Kishikawa Katsumi kishikawakatsumi
0
340

Featured

See All Featured
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
37
7.2k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.4k
Public Speaking Without Barfing On Your Shoes - THAT 2023
Avatar for David Neal reverentgeek
1
320
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
670
State of Search Keynote: SEO is Dead Long Live SEO
Avatar for Ryan Jones ryanjones
0
140
Optimising Largest Contentful Paint
Avatar for Harry Roberts csswizardry
37
3.6k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
28
2.5k
Sam Torres - BigQuery for SEOs
Avatar for Tech SEO Connect techseoconnect
PRO
0
200
svc-hook: hooking system calls on ARM64 by binary rewriting
Avatar for Akira Moroo retrage
1
130
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8k
A better future with KSS
Avatar for Kyle Neath kneath
240
18k
The Language of Interfaces
Avatar for Des Traynor destraynor
162
26k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]