Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Szymon Sobczak - Hadoop + Storm

Base Lab
May 07, 2015
Technology
0
94

Szymon Sobczak - Hadoop + Storm

Combo for realtime big data systems

Base Lab

May 07, 2015
Tweet

More Decks by Base Lab

See All by Base Lab
Slawek Skowron - Monitoring @ Scale
baselab
0
110
Karol Nowak - Monitoring clock drift in Amazon EC2 environment
baselab
0
100
Tomasz Nowak - Web Application Testing made easy
baselab
0
290
Szymon Pawlik - UX i Automatyzacja czyli jak testerzy mogą poprawić produkt.
baselab
0
240
Mateusz Herych - LIKE '%smth%' is not the way
baselab
0
140
Jerzy Chałupski - Offline mode in Android apps
baselab
3
470
Jerzy Chałupski - Data model on Android
baselab
4
220

Other Decks in Technology

See All in Technology
PHPからGoへのマイグレーション for DMMアフィリエイト
yabakokobayashi
1
170
開発生産性向上！ 育成を「改善」と捉えるエンジニア育成戦略
shoota
2
400
非機能品質を作り込むための実践アーキテクチャ
knih
5
1.5k
ハイテク休憩
sat
PRO
2
170
LINE Developersプロダクト(LIFF/LINE Login)におけるフロントエンド開発
lycorptech_jp
PRO
0
120
10個のフィルタをAXI4-Streamでつなげてみた
marsee101
0
170
ガバメントクラウドのセキュリティ対策事例について
fujisawaryohei
0
560
pg_bigmをRustで実装する（第50回PostgreSQLアンカンファレンス@オンライン 発表資料）
shinyakato_
0
100
サーバーなしでWordPress運用、できますよ。
sogaoh
PRO
0
110
オプトインカメラ：UWB測位を応用したオプトイン型のカメラ計測
matthewlujp
0
180
GitHub Copilot のテクニック集/GitHub Copilot Techniques
rayuron
37
15k
第3回Snowflake女子会_LT登壇資料（合成データ）_Taro_CCCMK
tarotaro0129
0
200

Featured

See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
810
Keith and Marios Guide to Fast Websites
keithpitt
410
22k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Building Applications with DynamoDB
mza
91
6.1k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
48
2.2k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
The World Runs on Bad Software
bkeepers
PRO
65
11k
What's in a price? How to price your products and services
michaelherold
243
12k
Optimizing for Happiness
mojombo
376
70k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.3k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Building a Scalable Design System with Sketch
lauravandoore
460
33k

Transcript

  1. Szymon Sobczak

  2. Hadoop + Storm Combo for realtime big data systems

  3. Plan • Hadoop & Storm • Our setup • What

    projects are we running • Decisions we had to make

  4. Hadoop

  5. None

  6. Storm “Ala ma kota Artura" “ala”, “ma”, “kota”, “artura" “ala”,

    “ma”, “kota”, “artura" a: 2 k: 1 m: 1 “ala”

  7. Storm “Ala ma kota Artura" “ma” “ala”, “ma”, “kota”, “artura"

    “ala” a: 2 m: 1 k: 1 “ala”, “artura" “kota”

  8. Common traits

  9. None

  10. Base infrastructure services

  11. Understand how the entire Base system works services

  12. Big Data S3 uploader

  13. Four example projects • Debugging • Reporting • Email intelligence

    • Forecasting

  14. Debugging S3 uploader

  15. Reporting

  16. Email analysis S3 uploader

  17. Forecasting S3 uploader

  18. Decisions we made ☑ Collect *all* data ☑ Put them

    in one place ☐ Build platform for engineers ☐ Same code on Hadoop and Storm

  19. Summary S3 uploader

  20. Questions?

  21. Thank you [email protected] bigdata.getbase.com