Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Evolving Sustainable Data Pipelines

Avatar for Hakka Labs Hakka Labs
February 13, 2015
Programming
3.5k
0

Evolving Sustainable Data Pipelines

Full post here:

Avatar for Hakka Labs

Hakka Labs

February 13, 2015

More Decks by Hakka Labs

See All by Hakka Labs
New Workflows for Building Data Pipelines
Avatar for Hakka Labs hakka_labs
0
2.9k
Collaborative Topic Models for Users and Texts
Avatar for Hakka Labs hakka_labs
0
2.8k
Groupcache with Evan Owen
Avatar for Hakka Labs hakka_labs
2
5.4k
Testing Android at Spotify
Avatar for Hakka Labs hakka_labs
1
4.5k
It's Not a Bug, It's a Feature!
Avatar for Hakka Labs hakka_labs
0
3.2k
K-means Clustering to Understand Your Users
Avatar for Hakka Labs hakka_labs
0
2k
Building Amy: The Email-based Virtual Assistant by x.ai
Avatar for Hakka Labs hakka_labs
0
5k
Deep Learning and NLP Applications
Avatar for Hakka Labs hakka_labs
3
13k
Go and the Gophers
Avatar for Hakka Labs hakka_labs
2
11k

Other Decks in Programming

See All in Programming
Symfony + NelmioApiDocBundle を使った スキーマ駆動開発 / Schema Driven Development with NelmioApiDocBundle
Avatar for Shohei Okada okashoi
0
260
Java 21/25 Virtual Threads 소개
Avatar for Sunghyouk Bae (Debop) debop
0
320
AWS×クラウドネイティブソフトウェア設計 / AWS x Cloud-Native Software Design
Avatar for nrs nrslib
16
3.5k
へんな働き方
Avatar for Yusuke Wada yusukebe
6
2.9k
おれのAgentic Coding 2026/03
Avatar for TsukasaSekiguchi tsukasagr
1
130
AIコードレビューの導入・運用と AI駆動開発における「AI4QA」の取り組みについて
Avatar for ハゲワシ hagevvashi
0
590
GoのDB アクセスにおける 「型安全」と「柔軟性」の両立 - Bob という選択肢
Avatar for tak tak848
0
300
最初からAWS CDKで技術検証してもいいんじゃない？
Avatar for アキキー | Akihisa Ikeda akihisaikeda
4
180
Linux Kernelの1文字のミスで 権限昇格ができた話
Avatar for rona rqda
0
2.2k
Tamach-sre-3_ANDPAD-shimaison93
Avatar for shimaison93 mane12yurks38
0
240
PHP でエミュレータを自作して Ubuntu を動かそう
Avatar for memory m3m0r7
PRO
2
160
Coding at the Speed of Thought: The New Era of Symfony Docker
Avatar for Kévin Dunglas dunglas
0
4.3k

Featured

See All Featured
Site-Speed That Sticks
Avatar for Harry Roberts csswizardry
13
1.1k
Heart Work Chapter 1 - Part 1
Avatar for Lake Fama lfama
PRO
5
35k
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.2k
Kristin Tynski - Automating Marketing Tasks With AI
Avatar for Tech SEO Connect techseoconnect
PRO
0
210
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
A brief & incomplete history of 
UX Design for the World Wide Web: 1989–2019
Avatar for Jason CranfordTeague jct
1
340
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
300
Organizational Design Perspectives: An Ontology of Organizational Design Elements
Avatar for Dr. Kim W Petersen kimpetersen
PRO
1
660
AI: The stuff that nobody shows you
Avatar for John Nunemaker jnunemaker
PRO
4
510
ラッコキーワード サービス紹介資料
Avatar for ラッコ株式会社 rakko
1
2.9M
A designer walks into a library…
Avatar for Paul Jervis Heath pauljervisheath
211
24k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
4.1k

Transcript

  1. None

  2. ESP Evolving Sustainable {data} Pipelines Anna Smith - @OMGannaks 29

    January 2015

  3. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  4. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits
  5. None

  6. 1. 2. 3.

  7. WHERE THE MAGIC HAPPENS

  8. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  9. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation

  10. DEMAND website user interaction transactions SUPPLY warehouse inventory allocation INTERACTION

    reservation calendar
  11. None
  12. None
  13. None

  14. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  15. PHASE 0

  16. None

  17. YUP

  18. KEEP IT TOGETHER

  19. PHASE 1 primordial soup

  20. STABILITY

  21. PHASE 2 process

  22. orderwarehouse.job

  23. $ ./runjob.py orderwarehouse.job $ ./runjob.py orderwarehouse.job --show $ ./runjob.py orderwarehouse.job

    --only 2

  24. runjob.py

  25. ADAPTING ensuring data quality

  26. PHASE 3 exposing weaknesses

  27. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  28. dependency manager

  29. None
  30. None

  31. A bit about RTR A bit about our analytics Some

    evolution stuff A bit about Luigi The future bits

  32. PHASE 4 ownership

  33. RELIABILITY

  34. COMMUNICATION

  35. THE FUTURE @OMGannaks [email protected]