Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amazon EMR でのバッチ処理 Spark アプリケーション運用で 気をつけたいこと

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for soonraah soonraah
March 29, 2015
Technology
650
1

Amazon EMR でのバッチ処理 Spark アプリケーション運用で 気をつけたいこと

2015-03-29 第2回 “Learning Spark” 読書会 LT

Avatar for soonraah

soonraah

March 29, 2015

More Decks by soonraah

See All by soonraah
MLCT #5 Pipeline Jungles とたたかう話
Avatar for soonraah soonraah
0
2k
Apache Spark による推薦システム案件例
Avatar for soonraah soonraah
11
8.1k
Spark MLlib でやってみる協調フィルタリング
Avatar for soonraah soonraah
8
7.8k

Other Decks in Technology

See All in Technology
Oracle MCP Servers Explained
Avatar for thatjeffsmith thatjeffsmith
1
520
平文パスワードはログに“残り” ── 肝心の侵入は“痕跡すら残らない”
Avatar for kuroneko kuroneko13
0
130
Kiro５兄弟のいまどきのセキュリティ基礎知識
Avatar for Hiroo Katoh kentapapa
0
290
dbt と Snowflake と tag
Avatar for Kosaku Ono kevinrobot34
1
250
巨大気象データと戦う ― サロゲートモデル学習を高速化する圧縮技術
Avatar for GPU UNITE gpuunite_official
0
280
Microsoft MVP プログラムを紹介するから目指す人増えてくれ
Avatar for Yuta Matsumura tsubakimoto_s
0
110
サイボウズ 開発本部採用ピッチ / Cybozu Engineer Recruit
Avatar for Cybozu cybozuinsideout
PRO
12
86k
[ホンマでっか SRE] あなたはなぜ SRE に?
Avatar for _awache _awache
0
260
Kiro Crew で始める マルチエージェント開発
Avatar for Kazuki Miura miu_crescent
PRO
0
190
暗号化？某ファイルストレージはどうなるの！？ 3rd Partyとうまく付き合う秘密度ラベル設計
Avatar for Kaz Asada | しがない情シス kasada
0
200
AIレビュー時代に必要なのは、SLOで引く撤退ライン
Avatar for nobuoooo nobuoooo
0
100
AI for Science時代を切り開く、政府の次世代HPC戦略の展望
Avatar for GPU UNITE gpuunite_official
0
260

Featured

See All Featured
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
440
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
10k
Are puppies a ranking factor?
Avatar for Jono Alderson jonoalderson
2
3.8k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
Avatar for Kenny Baas-Schwegler baasie
0
480
Game over? The fight for quality and originality in the time of robots
Avatar for Wayne Barker wayneb77
1
250
The innovator’s Mindset - 
Leading Through an Era of Exponential Change - McGill University 2025
Avatar for Jean-Marc De Jonghe jdejongh
PRO
1
290
The Spectacular Lies of Maps
Avatar for Per Axbom axbom
PRO
1
940
Amusing Abliteration
Avatar for ianozsvald ianozsvald
1
260
Test your architecture with Archunit
Avatar for Yoan thirion
2
2.4k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
56
3.4k
The AI Search Optimization Roadmap by Aleyda Solis
Avatar for Aleyda Solis aleyda
1
6.1k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
85
9.6k

Transcript

  1. "NB[PO&.3Ͱͷόονॲཧ 4QBSLΞϓϦέʔγϣϯӡ༻Ͱ ؾΛ͚͍ͭͨ͜ͱ  ୈճl-FBSOJOH4QBSLzಡॻձ-5 !TPPOSBBI

  2. "(&/%" •  "NB[PO&.3 •  ΠϯελϯεੑೳWTίετ •  +PCqPXΛ։࢝Ͱ͖ͳ͍৔߹ͷอݥ •  XFC6*ͷݟํ • 

    ·ͱΊ 2

  3. "NB[PO&.3 &MBTUJD.BQ3FEVDF  •  )BEPPQϑϨʔϜϫʔΫΛΫϥ΢υఏڙ •  )%'4Λར༻Մೳ •  4QBSL΍1SFTUP౳Λ࣮ߦՄೳ • 

    ॳظඅ༻Λ཈͑ͯ෼ࢄॲཧΛࢼͤΔ •  όονॲཧ޲͖ʁ  3

  4. ΠϯελϯεੑೳWTίετ •  ҎԼͷཁૉΛߟྀܾͯ͠ఆ •  ඞཁͳϝϞϦྔ͸ʁ •  ϩʔΧϧͰͷ࣮ݧ΍4QBSLXFC6*͔Βݟੵ •  ཁٻ͞ΕΔॲཧ࣌ؒ͸ʁ • 

    ϩʔΧϧͰͷ࣮ݧͰݟੵ •  ༧ࢉʹऩ·Δ͔ʁ •  εϙοτΠϯελϯεͷ৔߹͸࣌Ձ •  ೖࡳՁ֨ͷࢦఆ 4

  5. ΠϯελϯεՁ֨มಈ͸େ͖͍ʜ 5

  6. +PCqPXΛ։࢝Ͱ͖ͳ͍৔߹ͷอݥ •  +PCΛ։࢝Ͱ͖ͳ͍ •  #PPUTUSBQ"DUJPOͰͷࣦഊ •  ೖࡳՁ͕֨࣌ՁΑΓ௿͍ •  ΘΓͱى͜Δ • 

    &.3KPCqPXϦΫΤετଆʹอݥͷॲཧΛ •  +PCqPXΛ؂ࢹ •  ࣦഊͨ͠ΒΠϯελϯελΠϓ΍ೖࡳՁ֨Λ ม͑ͯϦτϥΠ͢Δ࢓૊Έ 6

  7. 8FC6* •  ࣮ߦதͷϝτϦΫε͕ݟΕΔ •  εςʔδ͝ͱͷλεΫ࣮ߦঢ়گ •  ֤3%%ͷ࢖༻ϝϞϦྔ •  ϊʔυ৘ใ • 

    Ϧιʔεௐ੔΍νϡʔχϯάͷͨΊʹ 7

  8. 8FC6* 8

  9. 8FC6*ͷݟํʢʣ 9

  10. 8FC6*ͷݟํʢʣ 10

  11. 8FC6*ͷݟํʢʣ 11

  12. 8FC6*ͷݟํʢʣ 12

  13. 8FC6*ͷݟํʢʣ 13

  14. ·ͱΊ •  ඞཁͳΠϯελϯεੑೳΛݟੵ΋ͬͯίετΛ ܭࢉ͠Α͏ •  +PCqPXͷ։࢝ʹࣦഊͨ͠ͱ͖ͷอݥ͕͋Δͱ ӡ༻Λָʹ͠Α͏ •  8FC6*ΛݟΑ͏ 14