Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon Kinesisでストリーム解析 - Node学園12時限目
Search
nashibao
April 24, 2014
Technology
4
3.9k
Amazon Kinesisでストリーム解析 - Node学園12時限目
Node学園12時限目で話します.
Amazon Kinesisをストリーム解析に使えるか検証した時の話です.
http://plaid.co.jp
nashibao
April 24, 2014
Tweet
Share
More Decks by nashibao
See All by nashibao
ちゃんと使えるBotについて考える
nashibao
0
2.2k
PySparkを使った レコメンドアルゴリズムの改良
nashibao
1
5.8k
カジュアルにリアクティブウェブを実現している話
nashibao
1
5.4k
fishシェルからのエトセトラ - nanapi勉強会 vol2
nashibao
1
490
Other Decks in Technology
See All in Technology
時間がないなら、つくればいい 〜数十人規模のチームが自律性を発揮するために試しているいくつかのこと〜
kakehashi
PRO
15
1.9k
MCPを理解する
yudai00
13
9.5k
Winning at PHP in Production in 2025
beberlei
1
270
AI-in-the-Enterprise|OpenAIが公開した「AI導入7つの教訓」——ChatGPTで変わる企業の未来とは?
customercloud
PRO
0
130
Aspire をカスタマイズしよう & Aspire 9.2
nenonaninu
0
360
Microsoft Fabric vs Databricks vs (Snowflake) -若手エンジニアがそれぞれの強みと違いを比較してみた- "A Young Engineer's Comparison of Their Strengths and Differences"
reireireijinjin6
1
130
AIとSREで「今」できること
honmarkhunt
3
690
2025-04-14 Data & Analytics 井戸端会議 Multi tenant log platform with Iceberg
kamijin_fanta
1
180
Part2 GitHub Copilotってなんだろう
tomokusaba
0
250
生成AIによるCloud Native基盤構築の可能性と実践的ガードレールの敷設について
nwiizo
7
1.4k
Асинхронная коммуникация в Go: от понятного к душному. Дима Некрасов, Otello, 2ГИС
lamodatech
0
1.8k
ビジネスとデザインとエンジニアリングを繋ぐために 一人のエンジニアは何ができるか / What can a single engineer do to connect business, design, and engineering?
kaminashi
2
870
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
462
33k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Facilitating Awesome Meetings
lara
54
6.3k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
5
590
Code Review Best Practice
trishagee
67
18k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Become a Pro
speakerdeck
PRO
28
5.3k
Product Roadmaps are Hard
iamctodd
PRO
53
11k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Being A Developer After 40
akosma
91
590k
Adopting Sorbet at Scale
ufuk
76
9.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
47
2.7k
Transcript
Amazon KinesisͰ ετϦʔϜղੳ (ג)ϓϨΠυ @nashibao
0. Who are you?
@nashibaoͱ͍͍·͢ ࣲࢁथ తࢄܕػցֶशతΤϯδχΞ(31) ! ~201208: େֶͱ͔Ͱʑͱ͢ΔɽMapReduceͱߦྻղ͕ɽ 201209: Node.jsʹ֮ΊΔ 201304: ݩָఱ(31)ͱىۀ
σʔλղੳ&CRMαʔϏε 201309: ത࢜υϩοϓΞτ
(ג)ϓϨΠυͱ͍͍·͢ http://plaid.co.jp EC͚ͷղੳ&CRMαʔϏεΛεςϧε։ൃதɽ ! ݱࡏ։ൃͨͬͨ2ਓ(t-nakamura)ͳͷͰɺ ԠืΛຊʹ͓͓ͪͯ͠Γ·͢ʂ ෲյ͢·Ͱ͕৯͑ΔձࣾʂͦΕ͕ϓϨΠυʂ ܴձγϡϥείͰ͢ʂ
Amazon KinesisΛετϦʔϜ ղੳʹ͑Δ͔ݕূ͓ͨ͠
1. Kinesisͱʁ 2. Node.js͔ΒKinesisΛ͏ 3. ύϑΥʔϚϯεݕূ 4. ײ
1. Kinesisͱʁ
Amazon Kinesis
“a fully managed service for real-time processing of streaming data
at massive scale.” ͲͰ͔͍σʔλ͔Β ϦΞϧλΠϜॲཧ͕ग़དྷΔ ϑϧϚωʔδυͳαʔϏε
5JCTF 5JCTF 5JCTF -# -# -# 2TQFWEGT -KPGUKU#RRNKECVKQPU 5VTGCO 2TQFWEGT
2TQFWEGT 2TQFWEGT 2TQFWEGT 哋┮䠉㏰ا揉ⓧ
5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT
WEGT WEGT EGT ϑϧϚωʔδυͳ෦ ! • 24࣌ؒอ࣋ • ࠶औಘՄʢex: Τϥʔ࣌ʣ • 1000tps/shard put • 5tps/shard get
5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT
WEGT WEGT EGT ϑϧϚωʔδυͳ෦ ! • 24࣌ؒอ࣋ • ࠶औಘՄʢex: Τϥʔ࣌ʣ • 1000tps/shard put • 5tps/shard get • hashKey(ࣗ༝)ͰৼΓ͚
5JCTF 5JCTF 5JCTF -# -# -# WEGT -KPGUKU#RRNKECVKQPU 5VTGCO WEGT
WEGT WEGT EGT େྔʹॻ͖ࠐΜͰ ·ͱΊͯ औಘ͢ΔͨΊͷΩϡʔ? ॻ͖ࠐΈΛΩϟογϡ͢Δ ड͚ࡼͷΑ͏ͳΠϝʔδ
ෛՙूதͷड͚ࡼʹ Ͱ͖Δʁ
5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT
Ý ݱঢ় YGDUQEMGV
5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT
Ý ݱঢ় ෛՙ͕ूத͢Δͱ͘͞ͷେมʂ YGDUQEMGV
5KVG# 5KVG$ &$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG ! .QIIGT
Ý Kinesis? YGDUQEMGV
5KVG# 5KVG$ &$ ! 5VTGCO #PCN[VKEU 5GOKDCVEJ/CR4GFWEG Kinesis? -KPGUKU !
Ý YGDUQEMGV
2.Node.js͔Βѻ͏
plaidev/kinesis-wrapper.js # install npm install kinesis-wrapper.js # stream ϕʔε mhart/kinesis
! # ΫϥγΧϧͳͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ plaidev/kinesis-wrapper.js
plaidev/kinesis-wrapper.js # install npm install kinesis-wrapper.js # stream ϕʔε mhart/kinesis
! # bacon.js ϕʔε andyhorng/kinesis ! # ΫϥγΧϧͷ͕ཉ͍͠ͷɽɽ͔ͩΒॻ͍ͨͷɽɽ plaidev/kinesis-wrapper.js
1. putRecordͰ͛ࠐΉ var kinesis = require('..'); ! # ετϦʔϜͱͭͳ͛Δ var
stream = kinesis.stream('hoge'); ! # ‘key’MD5ͰϋογϡԽ͞ΕͯγϟʔυʹৼΓ͚ΒΕΔ stream.putRecord('key', {date: new Date()}, function(err, res){ console.log(err, res); });
2. getRecordsͰϙʔϦϯά var kinesis = require('..'); ! # ετϦʔϜͱͭͳ͛Δ var
stream = kinesis.stream('hoge'); ! # ϙʔϦϯάͯ͠औಘɽγϟʔυ͝ͱग़དྷ·͢ stream.getRecords(function(err, records){ console.log(new Date(), records[0].val.date); });
͍ͭͰʹ ಉظతʹฒྻԽ͢Δ • async.jsʁ • ͜Ε͔Βyieldʂ • node-thunkify • nashibao/co-gateѱ͘ͳ͍Αʂ
co-gateϞδϡʔϧ ैདྷͷίʔϧόοΫܗࣜͷؔΛͦͷ··ฒ ྻԽɾಉظతʹॻ͚Δ ICVG ECNNDCEM ECNNDCEM ECNNDCEM ECNNDCEM KP QWV
TGUWNVU
var co = require('co'); var Gate = require('co-gate'); ! co(function
*(){ ! var gate = new Gate(); ! async_function(gate.in()); ! var results = yield gate.out(); ! })(); co-gateϞδϡʔϧ
3. େྔʹ͛ࠐΉʢ࠶ʣ co(function *(){ var gate = new Gate(); !
for (var i = 0; i < 100; i++){ stream.putRecord('key', {date: new Date()}, gate.in()); } ! var res = yield gate.out(); }
4. ϙʔϦϯάʢ࠶ʣ co(function *(){ var gate = new Gate(); !
stream.getRecords(gate.in()); ! var records = yield gate.out(); })();
3. KinesisͷύϑΥʔϚϯε
ݕূڥɾ݅ • EC2(c3-4xlarge)1͔Β͛ͯݕূɽ • ΫϥΠΞϯτଆͷλΠϜελϯϓͰtpsΛܭଌ • 1~3shard • ৄ͘͠ɺ͏গ͠ݕূඞཁ
putRecord / shard UQTͱ͔ೖͬͨΓ͢Δɽɽ
put & getRecords UQTఔͰݻ·ͬͯऔಘͰ͖Δ
ࢥͬͨΑΓॻ͖ࠐΊΔɽͰ ͪΐͬͱ͍? • max3000~ tps/shard put • ॻ͖ࠐΈ͗͢Δͱ݁ہऔΓग़͢·Ͱʹ͔͔࣌ؒΔɽ • ݁ہ400~700tpsఔ·Ͱ͍͚Δ͜ͱ֬ೝ
• avg1sec get • ϙʔϦϯάࠐΈͰ͜ͷఔ • 200msҎʹฦͬͯ͘ΔΘ͚͡Όͳ͍ • ॻ͖ࠐΜͰऔΓग़͢·Ͱʹͳ͔ͥϥά͕͋ͬͨΓͳ͔ͬͨΓ
ײ • ϑϧϚωʔδυʁ • KAࣗͰཧɽγϟʔυʹϫʔΧʔͤΒΕͳ͍ͷ͔ɽɽ ! • ϦΞϧλΠϜʁ • ϙʔϦϯά
• औΓग़͢ͷʹϥά͕͚ͬ͜͏͋Δ ! • ύϑΥʔϚϯε • writeߴ͍ɽγϟʔυ૿͔ͤͳΓ͍͚ͦ͏
͋Γ͕ͱ͏͍͟͝·ͨ͠ɽ
http://plaid.co.jp