Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Base Lab
May 07, 2015
Technology
0
94

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
baselab
0
110
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
baselab
0
110
Tomasz Nowak - Web Application Testing made easy
baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
baselab
3
470
Jerzy Chałupski - Data model on Android
baselab
4
220

Other Decks in Technology

See All in Technology
Classmethod AI Talks(CATs) #21 司会進行スライド(2025.04.17) / classmethod-ai-talks-aka-cats_moderator-slides_vol21_2025-04-17
shinyaa31
0
420
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming
tomzoh
0
200
はてなの開発20年史と DevOpsの歩み / DevOpsDays Tokyo 2025 Keynote
daiksy
5
1.4k
YOLOv10~v12
tenten0727
3
850
Tokyo dbt Meetup #13 dbtと連携するBI製品＆機能ざっくり紹介
sagara
0
420
AI Agentを「期待通り」に動かすために：設計アプローチの模索と現在地
kworkdev
PRO
2
380
はじめてのSDET / My first challenge as a SDET
bun913
1
190
ブラウザのレガシー・独自機能を愛でる-Firefoxの脆弱性4選- / Browser Crash Club #1
masatokinugawa
1
390
Ops-JAWS_Organizations小ネタ3選.pdf
chunkof
2
110
MCP Documentation Server @AI Coding Meetup #1
yyoshiki41
2
2.6k
SRE NEXT CfP チームが語る 聞きたくなるプロポーザルとは / Proposals by the SRE NEXT CfP Team that are sure to be accepted
chaspy
1
560
LangChainとLangGiraphによるRAG・AIエージェント実践入門「10章 要件定義書生成Alエージェントの開発」輪読会スライド
takaakiinada
0
120

Featured

See All Featured
How to Think Like a Performance Engineer
csswizardry
23
1.5k
Testing 201, or: Great Expectations
jmmastey
42
7.4k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
380
Building Adaptive Systems
keathley
41
2.5k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.2k
Thoughts on Productivity
jonyablonski
69
4.6k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Designing Experiences People Love
moore
141
24k
Designing for Performance
lara
607
69k
GraphQLの誤解/rethinking-graphql
sonatard
71
10k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.8k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
13
660

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com