Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw awa...
Search
Hiroyuki Nakazato / 中里 浩之
March 14, 2019
Technology
0
4.6k
1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day
Hiroyuki Nakazato / 中里 浩之
March 14, 2019
Tweet
Share
More Decks by Hiroyuki Nakazato / 中里 浩之
See All by Hiroyuki Nakazato / 中里 浩之
Data Engineering Study 21 - Microsoft Fabric
nakazax
0
770
Azure Databricks Learning Series #3 - Network Design
nakazax
1
1.1k
Azure Databricks Learning Series #2 - Databricks SQL
nakazax
0
600
Microsoft Build 2022 Recap Party!! Azure のデータ & 分析サービス 注目アップデート / microsoft-build-2022-recap-azure-data-and-analytics
nakazax
0
690
Microsoft Build 2022 - Azure のデータ & 分析サービス 最新アップデート / Microsoft Build 2022 Updates on Azure Data and Analytics Services
nakazax
1
910
Architecture patterns of Azure Cosmos DB & Azure Synapse Analytics
nakazax
1
480
CY2021 Updates for ADF & Synapse & Cosmos
nakazax
0
98
Ignite 2020 Update Azure Synapse Analytics
nakazax
0
1.1k
Ignite 2020 Update - Azure DB for MySQL and Postgres - Azure Arc - and more
nakazax
2
420
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructure:2025年6月度サービス・アップデート
oracle4engineer
PRO
2
140
Observability в PHP без боли. Олег Мифле, тимлид Altenar
lamodatech
0
290
BigQuery Remote FunctionでLooker Studioをインタラクティブ化
cuebic9bic
2
230
ローカルLLMでファインチューニング
knishioka
0
130
Amazon ECS & AWS Fargate 運用アーキテクチャ2025 / Amazon ECS and AWS Fargate Ops Architecture 2025
iselegant
16
4.6k
エンジニア向け技術スタック情報
kauche
0
110
Azure AI Foundryでマルチエージェントワークフロー
seosoft
0
150
ハノーバーメッセ2025座談会.pdf
iotcomjpadmin
0
150
Snowflake Summit 2025 データエンジニアリング関連新機能紹介 / Snowflake Summit 2025 What's New about Data Engineering
tiltmax3
0
240
登壇ネタの見つけ方 / How to find talk topics
pinkumohikan
3
300
rubygem開発で鍛える設計力
joker1007
1
130
Amazon S3標準/ S3 Tables/S3 Express One Zoneを使ったログ分析
shigeruoda
2
390
Featured
See All Featured
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
920
What's in a price? How to price your products and services
michaelherold
245
12k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
Product Roadmaps are Hard
iamctodd
PRO
53
11k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Site-Speed That Sticks
csswizardry
10
650
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
Rails Girls Zürich Keynote
gr2m
94
14k
Transcript
݄ ιϑτόϯΫגࣜձࣾதཬߒ೭ ݸҎ্ͷ )BEPPQΫϥελʔΛ ͍ࣺͯΔํ๏ )BEPPQ4QBSL$POGFSFODF+BQBO
ࣗݾհ தཬߒ೭ φΧβτώϩϢΩ ॴଐ ιϑτόϯΫגࣜձࣾ גࣜձࣾ"HPPQ ϙδγϣϯ όοΫΤϯυνʔϜϦʔμʔ
"845FDIOJDBM.FJTUFS
͜ΜͳهࣄΛॻ͍͍ͯ·͢ IUUQTRJJUBDPNOBLB[BYJUFNTFEEGBB
Ґஔใऩू ղੳ ਓͷྲྀΕ
.VTUͷϩάΛؒόον։࢝·Ͱʹ%8)ʹೖ 8BOU%8)ͷೖૣ͚Εૣ͍΄Ͳϕλʔ ػೳ໘ ඇػೳ໘ ϩάͷϘϦϡʔϜԯ݅5#Φʔμʔ ࠓޙϩάྔ͕૿͑ଓ͚ΔͨΊεέʔϥϏϦςΟ͕ඞཁ ίετޮٴͼ҆ఆੑ͕ߴ͍͜ͱ &5-ॲཧͷཁ݅
ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT
&-# &$ ऩूɾੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ
ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT
&-# &$ ऩूɾੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ εέʔϥϏϦςΟ ͳͲʹ՝
࣌ؒͷϩάΛೖྗɺ1ZUIPONVMUJQSPDFTTJOHͰ $16Λͬͯ%8)ʹϩʔυՄೳͳܗࣜʹม ͋Δͷରͷϩά͕एׯҟͳΔͨΊ ૿͑ଓ͚Δϩάʹରͯ͠ &$ͷεέʔϧΞοϓͰରԠ ࣌ͷ&5-ॲཧͷ࣮
DYMBSHF ϦϦʔε W$16(J#.FN ֹ݄ສԁ DYMBSHF Լ०ϦϦʔε W$16(J#.FN ֹ݄ສԁ
εέʔϧΞοϓ ಄ଧͪϦεΫ ίϯϐϡʔςΟϯά࠷దԽΠϯελϯελΠϓ
࣌&5-ॲཧ ࣌&5-ॲཧ ͋Δ࣌ؒͷॲཧ͕Ҿ͘ͱޙଓ͕٧·Δ
&$ 4 ࣌&5-ॲཧ &5-ॲཧΠϝʔδ
ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT
&-# &$ ऩूɾੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ ՝ͷղܾʹணख
৽ΞʔΩςΫνϟʔߏ εέʔϧΞτϞσϧ 4QBSLPO&.3Λ࠾༻ ॲཧͷಠཱੑ ࣌ؒͷ&5-ॲཧΛ Ϋϥελʔ͕୲ ͍ࣺͯՄೳͳ &.3όονॲཧج൫
ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5-
ղੳ ऩूɾੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ৽ΞʔΩςΫνϟʔ (MVF
৽ΞʔΩςΫνϟʔͷϙΠϯτ &.3ͷεςοϓػೳͷ׆༻ -BNCEBʹΑΔ&.3Ϋϥελʔͷ੍ޚ ϑϧϚωʔδυ)JWFϝλετΞͷ׆༻
ΞυϗοΫੳ༻&.3Ϋϥελʔͷఏڙ
&.3ͷεςοϓػೳͷ׆༻ 1 2 3 4 εςοϓҙͷॲཧΛॱ൪ʹ࣮ߦ͢ΔΈ શεςοϓྃޙʹΫϥελʔΛऴྃ͢Δͷ੍ޚ͕ ؆୯ʹߦ͑Δ
݅ذ܁Γฦ͕͠ͳ͍γϯϓϧͳόονॲཧϑϩʔ Ͱ͋ΕεςοϓػೳͰेΧόʔՄೳ
&5-ॲཧͷεςοϓ &.3 ຊ൪ࢿࡐ +BS 4IFMM ςετࢿࡐ 4IFMM +BS
ੜϩά 1BSRVFU &5-ॲཧ༻ͷࢿࡐΛ4͔Βμϯϩʔυ TEJTUDQͰॲཧରͷϩάΛू TQBSLTVCNJUͰσʔλՃͱ1BSRVFUมΛ࣮ࢪ 1 2 3
-BNCEBʹΑΔΫϥελʔ੍ޚ 1 2 3 4 &.3Ϋϥελʔىಈ༻ͷ-BNCEBؔΛ࣮ 1ZUIPOͰ3VO+PC'MPXΛίʔϧ 3VO+PC'MPXύϥϝʔλʔ͕ඇৗʹଟ͍
ઃఆϑΝΠϧϥΠϒϥϦ)0$0/Ͱཧ͠࠶ར༻ੑΛ্ 1MBZ'SBNFXPSLͳͲͰ༗໊ͳ-JHIUCFOE͕ࣾ։ൃ ϩάऩूγεςϜͳͲͰ1MBZ'SBNFXPSLΛ׆༻ͯ͠ ͓Γ)0$0/ʹ׳Ε͍ͯͨͷ͕࠾༻ཧ༝
&5-༻Ϋϥελʔىಈϑϩʔ 1 2 3 4 Name = hourly-etl
ReleaseLabel = emr-5.20.0 Steps = [ { spark-submit hourly-etl.jar ${YYYYMMDD}/${HH} } ... IPVSMZDPOG IPVSMZDPOG )0$0/ύʔε 3VO+PC'MPX࣮ߦ
&5-ॲཧͷϦΧόϦʔ ޭ ޭ ࣦഊ ͕ҟৗऴྃ 1 2 3
4 ϦΧόϦ༻ ىಈ༻ Ϋϥελʔͷঢ়ଶΛ ఆظతʹνΣοΫ ىಈ༻-BNCEBΛ ݺͼग़͠ ࣦഊͨ࣌ؒ͠ͷ ΫϥελʔΛىಈ
ϑϧϚωʔδυ)JWFϝλετΞ 1 2 3 4 "84(MVF%BUB$BUBMPH <&.3 ରԠ>
Ҏલ.Z42-PO3%4Λ)JWFϝλετΞͱͯ͠ར༻ 3%4͕410'ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋ͬͨ ΫϥελʔΛେྔʹىಈ͢ΔͱଓͰ͖ͳ͍FUD (MVF%BUB$BUBMPHϑϧϚωʔδυ͔ͭಉ࣌ଓ੍ݶͳ͠
ΞυϗοΫੳ༻Ϋϥελʔ ΤϯδχΞΞφϦετ͕ ؆୯ʹ&.3ΫϥελʔΛىಈՄೳʹ 1 2 3 4
Ϋϥελʔͷམͱ͠Εࢭ 1 2 3 4 I I I
ࢹ༻ Ϋϥελʔͷ Քಇ࣌ؒΛνΣοΫ 4MBDL௨ BEIPDDMVTUFS
ίετͷ ຖ࣌ͷ&5-ॲཧҎલΑΓ҆Ձʹ εϙοτΠϯελϯεΛੵۃతʹ׆༻͍ͯ͠ΔͨΊ 4શମͷίετ૿Ճ ΞυϗοΫੳ͕ؾܰʹͰ͖ΔͨΊੳ݁Ռͷσʔλ͕ 4ʹੵ͍͍ͯͬͯ͠Δ ෆཁͳσʔλΛఆظతʹআ͢ΔΑ͏ͳΈ͕ඞཁ
ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5-
ղੳ ऩूɾੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ࠶ܝ৽ΞʔΩςΫνϟʔ (MVF
THANK YOU! ANY QUESTIONS?