Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Avatar for Base Lab Base Lab
May 07, 2015
Technology
0
100

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Avatar for Base Lab

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
Avatar for Base Lab baselab
0
140
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
Avatar for Base Lab baselab
0
120
Tomasz Nowak - Web Application Testing made easy
Avatar for Base Lab baselab
0
300
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
Avatar for Base Lab baselab
0
250
Mateusz Herych - LIKE '%smth%' is not the way
Avatar for Base Lab baselab
0
150
Jerzy Chałupski - Offline mode in Android apps
Avatar for Base Lab baselab
3
490
Jerzy Chałupski - Data model on Android
Avatar for Base Lab baselab
4
240

Other Decks in Technology

See All in Technology
器用貧乏が強みになるまで ～「なんでもやる」が導いたエンジニアとしての現在地～
Avatar for KAKEHASHI kakehashi
PRO
5
630
インシデント対応入門
Avatar for gr1m0h grimoh
7
5.3k
LINEヤフーにおけるAI駆動開発組織のプロデュース施策
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
180
サンタコンペ2025完全攻略 ～お前らの焼きなましは遅すぎる～
Avatar for terry-u16 terryu16
1
530
Windows ネットワークを再確認する
Avatar for Murachi Akira murachiakira
PRO
0
160
バクラクにおける Document Understanding の挑戦：書類の「読取」から「意思決定」へ / document-understanding-in-bakuraku-2026
Avatar for Yuya Matsumura yuya4
0
140
AIに視覚を与えモバイルアプリケーション開発をより円滑に行う
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
570
【PyCon mini Shizuoka 2026】生成AI時代に画像処理やオーディオ処理のノードエディターを作る理由
Avatar for 高橋かずひと kazuhitotakahashi
0
180
なぜAIは組織を速くしないのか 令和の腑分け
Avatar for su8 / denchu sugino
80
49k
AWS Bedrock Guardrails / 機密情報の入力・出力をブロックする — Blocking Sensitive Information Input/Output
Avatar for Kazuhito Nakayama kazuhitonakayama
2
180
【SLO】"多様な期待値" と向き合ってみた
Avatar for kaita z63d
2
230
「データとの対話」の現在地と未来
Avatar for kobakou kobakou
0
910

Featured

See All Featured
KATA
Avatar for Megan Lloyd mclloyd
PRO
35
15k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
38
2.8k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
73
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
275
41k
Designing Experiences People Love
Avatar for Jonathan Moore moore
144
24k
Marketing Yourself as an Engineer | Alaka | Gurzu
Avatar for Gurzu gurzu
0
140
Applied NLP in the Age of Generative AI
Avatar for Ines Montani inesmontani
PRO
4
2.1k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
Avatar for r-kagaya rkaga
60
42k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
128
17k
How to Think Like a Performance Engineer
Avatar for Harry Roberts csswizardry
28
2.5k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
110
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon szymonslowik
1
940

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com