Upgrade to Pro — share decks privately, control downloads, hide ads and more …
Sign up for free

Szymon Sobczak - Hadoop + Storm

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Base Lab Base Lab
May 07, 2015
Technology
140
0

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
170
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
150
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
330
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
270
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
170
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
500
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
270

Other Decks in Technology

See All in Technology
Agent 活用環境の構築・運用のための AWS サービスのご紹介/agentcore-for-building-and-operating-agents
Avatar for Renya Kujirada ren8k
1
310
視野はAIに、視座は人に / ai-for-breadth-humans-for-perspective
Avatar for Yukiya Nakagawa nkzn
6
2.8k
re:Invent の去年のスケジュールを公開します！
Avatar for Kazuki Miura miu_crescent
PRO
0
160
「作れる」と「リリースできる」の間にあったもの── PdMを外して見えた、PdMがやるべき仕事/Buildable ≠ Shippable: What Removing the PM Taught Me About the PM’s Job
Avatar for Yappli Developers yappli_developers
0
420
Agent Loopを組み込んだ アンビエントなAI機能を実現する / Ambient AI fetures with Agent Loop
Avatar for Mitsuhiro Komuro (nekorush14) nekorush14
0
130
書き手が人でもAIでも正しくなる道、 SREを埋め込んだDeveloper Platform / tech drivers day8
Avatar for mekka chmikata
4
890
FinOpsを導入してマルチプロダクトのコスト最適化を実現する
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
0
110
AIエージェント時代、Kaggleにどう取り組むか
Avatar for Naoya Takashima takanao
2
470
Don't write the architecture only in the prompt.
Avatar for Neo Hsu neohsu
0
300
AIコーディングエージェント開発_全体像
Avatar for コロッケそば happysamurai294
0
150
AI時代の開発はウォーターフォールに回帰する？ / development-with-ai-back-to-wf
Avatar for 岡本卓也 okamototakuyasr2
0
170
"AWSはちょっと苦手"でも大丈夫。AWS Blocks で手軽に作る Flutter のバックエンド #FlutterKaigi
Avatar for Kihara, Takuya tacck
PRO
0
130

Featured

See All Featured
How To Speak Unicorn (iThemes Webinar)
Avatar for Michelle Schulp Hunt marktimemedia
1
620
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
123
22k
Improving Core Web Vitals using Speculation Rules API
Avatar for Sergey Chernyshev sergeychernyshev
21
1.7k
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
6k
WENDY [Excerpt]
Avatar for Tessa Abrams tessaabrams
14
40k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
11
1.1k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
143
7.8k
Designing for humans not robots
Avatar for Tammie Lister tammielis
254
26k
HDC tutorial
Avatar for Michiel Stock michielstock
2
990
Ethics towards AI in product and experience design
Avatar for Skipper Chong Warson skipperchong
2
450
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
390
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
41
2.8k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com