Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
94

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems
Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
120
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
110
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
470
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
220

Other Decks in Technology

See All in Technology
AI Engineering Summit Pre Event LT #10
Avatar for Kaoru okaru
2
560
AWS と定理証明 〜ポリシー言語 Cedar 開発の舞台裏〜 #fp_matsuri / FP Matsuri 2025
Avatar for y_taka_23 ytaka23
9
2.3k
Securing your Lambda 101
Avatar for Ilia chillzprezi
0
240
「規約、知識、オペレーション」から考える中規模以上の開発組織のCursorルールの 考え方・育て方 / Cursor Rules for Coding Styles, Domain Knowledges and Operations
Avatar for YuitoSato yuitosato
5
1.4k
vLLM meetup Tokyo
Avatar for jpishikawa jpishikawa
1
190
Workflows から Agents へ ~ 生成 AI アプリの成長過程とアプローチ~
Avatar for Belong inc. belongadmin
2
130
Autonomous Database サービス・アップデート (FY25)
Avatar for oracle4engineer oracle4engineer
PRO
2
760
Kotlinで学ぶ 代数的データ型
Avatar for Kanon ysknsid25
5
1k
Grafana MCP serverでなんかし隊 / Try Grafana MCP server
Avatar for kohbis kohbis
0
320
ゆるSRE #11 LT
Avatar for Kaoru okaru
1
570
Snowflake Intelligenceで 実現できるノーコードAI活用
Avatar for tak takumimukaiyama
1
180
開発効率と信頼性を両立する Ubieのプラットフォームエンジニアリング
Avatar for teru0x1 teru0x1
0
130

Featured

See All Featured
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Performance Is Good for Brains [We Love Speed 2024]
Avatar for Tammy Everts tammyeverts
10
900
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
Avatar for Tammy Everts tammyeverts
4
130
Docker and Python
Avatar for Tania Allard trallard
44
3.4k
Music & Morning Musume
Avatar for Bryan Veloso bryan
46
6.6k
Building Applications with DynamoDB
Avatar for Matt Wood mza
95
6.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
Avatar for Eileen M. Uchitelle eileencodes
26
2.8k
KATA
Avatar for Megan Lloyd mclloyd
29
14k
Writing Fast Ruby
Avatar for Erik Berlin sferik
628
61k
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
790
250k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
90
590k
Done Done
Avatar for chrislema chrislema
184
16k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you szymon.sobczak@getbase.com bigdata.getbase.com