Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop b...
Search
Hiroyuki Nakazato / 中里 浩之
April 05, 2018
Technology
0
270
Agoop における AWS をフル活用したビッグデータ収集・分析の仕組み / Agoop big data collection and analysis system fully leveraging AWS
Hiroyuki Nakazato / 中里 浩之
April 05, 2018
Tweet
Share
More Decks by Hiroyuki Nakazato / 中里 浩之
See All by Hiroyuki Nakazato / 中里 浩之
Data Engineering Study 21 - Microsoft Fabric
nakazax
0
660
Azure Databricks Learning Series #3 - Network Design
nakazax
0
670
Azure Databricks Learning Series #2 - Databricks SQL
nakazax
0
490
Microsoft Build 2022 Recap Party!! Azure のデータ & 分析サービス 注目アップデート / microsoft-build-2022-recap-azure-data-and-analytics
nakazax
0
650
Microsoft Build 2022 - Azure のデータ & 分析サービス 最新アップデート / Microsoft Build 2022 Updates on Azure Data and Analytics Services
nakazax
1
850
Architecture patterns of Azure Cosmos DB & Azure Synapse Analytics
nakazax
1
400
CY2021 Updates for ADF & Synapse & Cosmos
nakazax
0
80
Ignite 2020 Update Azure Synapse Analytics
nakazax
0
1.1k
Ignite 2020 Update - Azure DB for MySQL and Postgres - Azure Arc - and more
nakazax
2
390
Other Decks in Technology
See All in Technology
Shopifyアプリ開発における Shopifyの機能活用
sonatard
4
260
マルチモーダル / AI Agent / LLMOps 3つの技術トレンドで理解するLLMの今後の展望
hirosatogamo
37
12k
TanStack Routerに移行するのかい しないのかい、どっちなんだい! / Are you going to migrate to TanStack Router or not? Which one is it?
kaminashi
0
600
個人でもIAM Identity Centerを使おう!(アクセス管理編)
ryder472
4
230
New Relicを活用したSREの最初のステップ / NRUG OKINAWA VOL.3
isaoshimizu
3
630
障害対応指揮の意思決定と情報共有における価値観 / Waroom Meetup #2
arthur1
5
490
Making your applications cross-environment - OSCG 2024 NA
salaboy
0
200
BLADE: An Attempt to Automate Penetration Testing Using Autonomous AI Agents
bbrbbq
0
320
アジャイルでの品質の進化 Agile in Motion vol.1/20241118 Hiroyuki Sato
shift_evolve
0
180
マルチプロダクトな開発組織で 「開発生産性」に向き合うために試みたこと / Improving Multi-Product Dev Productivity
sugamasao
1
310
VideoMamba: State Space Model for Efficient Video Understanding
chou500
0
190
強いチームと開発生産性
onk
PRO
35
11k
Featured
See All Featured
Fireside Chat
paigeccino
34
3k
Thoughts on Productivity
jonyablonski
67
4.3k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
93
16k
Designing the Hi-DPI Web
ddemaree
280
34k
Why Our Code Smells
bkeepers
PRO
334
57k
Practical Orchestrator
shlominoach
186
10k
How to Ace a Technical Interview
jacobian
276
23k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Six Lessons from altMBA
skipperchong
27
3.5k
GitHub's CSS Performance
jonrohan
1030
460k
The Invisible Side of Design
smashingmag
298
50k
Code Review Best Practice
trishagee
64
17k
Transcript
"84Λϑϧ׆༻ͨ͠ Ϗοάσʔλऩूɾੳ "HPPQ$PSQ
Ґஔใऩू ղੳ ਓͷྲྀΕ
5FDI1BSU
4FMGJOUSPEVDUJPO தཬߒ೭ /BLB[BUP)JSPZVLJ ϩάऩूγεςϜͷҡ࣋ɾվળ ΞϓϦͷόοΫΤϯυ։ൃ ΤϯδχΞνʔϜͷ։ൃɾੳͷͨΊͷ ֤छΈͷඋ גࣜձࣾ"HPPQ ٕज़ຊ෦"*։ൃ෦ όοΫΤϯυνʔϜϦʔμʔ
ϩάऩूɾੳγεςϜͷΞʔΩςΫνϟ "84ΞφϦςΟΫεαʔϏεͷϢʔεέʔε "HFOEB
ϩάऩूɾੳγεςϜ ΞʔΩςΫνϟ
͔Βӡ༻։࢝ ͯࣗ͢લͰ։ൃ ݱࡏ໊ͷόοΫΤϯυνʔϜͰ։ൃ ͯ͢"84্ͰՔಇ ॳ&$ɺ4ɺ3%4ͳͲϕʔγοΫͳαʔϏεͰߏங ৽ن"84αʔϏεΛܧଓతɾੵۃతʹಋೖ ϩάྔ૿͑ଓ͚Δ ίετޮͱ҆ఆੑΛ্ͤ͞ΔͨΊ γεςϜͷجຊใ
ϩάσʔλͷ֓ཁ εϚʔτϑΥϯΞϓϦ͔Βऩू த௨৴ঢ়گͳͲʹؔ͢Δϩά +40/ܗࣜ Ͱऩू͢ΔσʔλϘϦϡʔϜ ݅ԯ݅ σʔλαΠζ5# γεςϜ͕ऩू͢Δϩά
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ऩूɾੵ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF &5-
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ੳ
ऩूɾੵ +BWB൛ͷ,JOFTJT1SPEVDFS-JCSBSZͱ,JOFTJT$POTVNFS-JCSBSZΛར༻ ,JOFTJTϓϩσϡʔαʔड৴ͨ͠+40/Λ΄΅ͦͷ··,JOFTJTʹೖ ,JOFTJTʹೖΕ͓͚ͯ࣌ؒ ࠷ؒ σʔλϩετͷ৺͕ແ͍ͨΊ ,JOFTJTίϯγϡʔϚʔҎԼͷॲཧΛ࣮ࢪ +40/Λύʔεͯ͠ޙଓͷੳͰ༻͢ΔใΛՃ &$ͰՔಇ͢ΔqVFOUEʹόοϑΝͯ͠ҰఆִؒͰ4ʹ([JQܗࣜͰग़ྗ <1PJOU>,JOFTJTಋೖͰ҆ఆੑ͕͔ͳΓ্ˠσʔλऩूܥʹ͓͢͢Ί
"VUP4DBMF&$ ,JOFTJT "VUP4DBMF&$ 4
&5- &.3Ϋϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- &.3Ϋϥελʔىಈ <2">ͳͥ$MPVE8BUDI&WFOUTͰͳ͘+FOLJOTʁ +FOLJOTΛδϣϒεέδϡʔϥʔΘΓʹ͏จԽ ӡ༻νʔϜ͕+FOLJOTͷऔΓѻ͍ʹ׳Ε͓ͯΓϦΧόϦʔͷ͢͠͞Λ༏ઌ <༨ஊ>ෳνʔϜɾෳ༻్Ͱ+FOLJOTΛ͏ͱΧΦεԽ͍͢͠ ᷖᮣʹ04ύοέʔδϓϥάΠϯΛΞοϓσʔτ͢ΔͱΒͳ͍δϣϒ͕ࣦഊ ͍ͭԿ͔ͷδϣϒ͕ಈ͍͍ͯͯϓϥάΠϯͷΠϯετʔϧ͕Ͱ͖ͳ͍FUDʜ +FOLJOT͔Βͷ٫Λࡧதˠ$MPVE8BUDI&WFOUT "84#BUDIΛݕ౼
4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- &.3Ϋϥελʔىಈ <2">ͳͥ-BNCEBؔʁ ༷ʑͳνʔϜͰ&.3ΫϥελʔΛؾܰʹར༻ͯ͠Β͏ͨΊ &5-ॲཧҎ֎ʹ༷ʑͳνʔϜɾ༻్Ͱ&.3Λར༻͍ͯ͠Δ &.3Ϋϥελʔͷىಈύϥϝʔλʔ͕ଟ͘Ұ͔Βىಈ͢Δͷେม Ϋϥελʔىಈͷڞ௨తͳॲཧΛ1ZUIPO CPUP Ͱ࣮ ֤νʔϜ͔ΒͷཁΛݩʹΫϥελʔͷεϖοΫΛઃఆϑΝΠϧʹهड़͠1ZUIPO
ϞδϡʔϧʹؚΊͯ-BNCEBؔͱͯ͠Ξοϓϩʔυ ֤νʔϜઃఆϑΝΠϧΛࢦఆ͠-BNCEBؔΛݺͼग़͢͜ͱͰΫϥελʔىಈ 4QBSLPO&.3 +FOLJOTPO&$ -BNCEB ࣌ؒʹճ -BNCEBؔΛίʔϧ &.3ΫϥελʔΛىಈ
&5- 3FETIJGUɾ)JWFϩʔυ ݱࡏ.Z42-3%4Λ)JWFϝλετΞ ςʔϒϧఆٛͷཧઌ ͱͯ͠ར༻ (MVFσʔλΧλϩάͷҠߦɾସΛ࣮ࢪத 3%4͔Β(MVFͷҠߦ(JU)VCͷBXTMBCTͷπʔϧΛར༻ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFS VUJMJUJFT)JWF@NFUBTUPSF@NJHSBUJPO ॳͷέʔε͕͍͔ͭ͋͘Γগ͠ϋϚ͕ͬͨ"84αϙʔτͷ͓ӄͰղܾʂ
4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
&5- 3FETIJGUɾ)JWFϩʔυ <1PJOU>)JWFϝλετΞ3%4ΑΓ(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͕410' ୯Ұো ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋Δ 3%4ಉ࣌ଓͷ੍ݶ͕͋ΔͨΊ&.3Ϋϥελʔͷಉ࣌ىಈʹ੍ݶ͕ੜ͡Δ ࠷େಉ࣌ଓϝϞϦґଘͷઃఆ ࠷େಉ࣌ଓΛ্͛ΔʹΑΓϝϞϦ͕ଟ͍ΠϯελϯελΠϓͷมߋ͕ඞཁ ϝλσʔλ͔͠ཧ͠ͳ͍ͷʹϦονͳεϖοΫମແ͍
(MVF%BUB$BUBMPHϑϧϚωʔδυ͔ͭಉ࣌ଓͷ੍ݶͳ͠ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
&5- 3FETIJGUɾ)JWFϩʔυ 4QBSLPO&.3Ͱ࣌ؒͷϩάΛೖྗͱͯ͠ҎԼεςοϓΛ࣮ߦ 4%JTU$QͰ ݸ ,#ڃ ͷϑΝΠϧΛݸ .#d(#ڃ ͷϑΝΠϧʹू <1PJOU>4QBSL
)BEPPQ େྔͷখ͞ͳϑΝΠϧΛѻ͏ͷෆಘҙ 4QBSLͰ([JQQFE+40/Λग़ྗ)JWFςʔϒϧʹ*/4&3507&383*5& εςοϓऴྃޙɺ&.3Ϋϥελʔࣗಈআ ࣌ؒͷॲཧΛಠཱͨ͠ΫϥελʔͰ࣮ߦ͢ΔͷͰϦΧόϦʔͳͲ͕Γ͍͢ 4্ͷ([JQϑΝΠϧΛ3FETIJGUͷ$01:ίϚϯυͰϩʔυ 4QBSL PO&.3 4 ([JQ 4 1BSRVFU 3FETIJGU 3%4 )JWF.FUBTUPSF ऩूɾੵ 4
σʔλऩूܥʹ,JOFTJT4USFBN͕͓͢͢Ί )JWFϝλετΞ(MVF%BUB$BUBMPH͕͓͢͢Ί 3%4͔Β(MVFͷҠߦʹ(JU)VCͷBXTMBCTͷπʔϧ͕͑Δ IUUQTHJUIVCDPNBXTMBCTBXTHMVFTBNQMFTUSFFNBTUFSVUJMJUJFT )JWF@NFUBTUPSF@NJHSBUJPO 4QBSL )BEPPQ େྔͷখ͍͞ϑΝΠϧΛѻ͏ͷෆಘҙͳͨΊɺ 4%JTU$QͳͲΛ༻͍ͯ͋Δఔͷେ͖͞ʹ·ͱΊΔͷ͕͓͢͢Ί &.3ϕετϓϥΫςΟεͰ([JQѹॖͷ߹ɺʙ(#ϑΝΠϧΛਪ
IUUQTNFEJBBNB[POXFCTFSWJDFTDPNKQXQ "84@"NB[PO@&.3@#FTU@1SBDUJDFTQEG લͷ·ͱΊ
"84ΞφϦςΟΫεαʔϏε Ϣʔεέʔε
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF ੳ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF 3FETIJGU +FOLJOT ఆظఆܕੳ
γεςϜΞʔΩςΫνϟ "VUP4DBMJOH &$ ,JOFTJT "VUP4DBMJOH &$ 4 4 3FETIJGU "UIFOB
1SFTUP PO&.3 4 4QBSL PO&.3 (MVF +FOLJOT PO&$ -BNCEB %FWFMPQFST %BUB"OBMZTUT σʔλϑϩʔ ࢀরͳͲ 3%4 )JWF.FUBTUPSF "UIFOB1SFTUPPO&.3 3FEBTI ΞυϗοΫੳ
3FETIJGU ఆظఆܕੳ͕த৺ +FOLJOTΛδϣϒεέδϡʔϥʔͱͯ͠ར༻ σʔλαΠζͱϫʔΫϩʔυʹ߹ͬͨεϖοΫͷ3*Λߪೖ 3*ͷظݶΕ͕͍ۙͨΊ)JWFPO&.3ͷҠߦΛਪਐத ΞφϦςΟΫεαʔϏεͷ͍͚
"UIFOB 1SFTUPPO&.3 ΞυϗοΫੳ͕த৺ 3FEBTI 044ͷμογϡϘʔυπʔϧ Λ(6*ͱͯ͠ར༻ ॳ಄͔Β3FEBTI "UIFOBಋೖ ݩʑ3FETIJGUͰΞυϗοΫੳߦͳ͍ͬͯͨ σʔλྔ૿Ճɺूܭύλʔϯ૿Ճ
ఆظఆܕੳͱΞυϗοΫੳͰͷϦιʔεͷୣ͍߹͍ ΞφϦςΟΫεαʔϏεͷ͍͚
ͳͥ"UIFOBͱ1SFTUPPO&.3ͷ྆ํΛར༻ʁ "UIFOBͷར༻ྉ͕ۚఆΑΓߴ͘ͳͬͯ͠·ͬͨ ྉۚ ݄ εΩϟϯαΠζ1# ݄ ࣍ળͷࡦͱͯ͠1SFTUPPO&.3Λಋೖ ΞφϦςΟΫεαʔϏεͷ͍͚
ΞυϗοΫੳͷද 3FETIJGU ར༻։࢝ ݄ w 3FEBTIར༻։࢝ w σʔλιʔεʹ
3FETIJGUͱ "UIFOBΛՃ ݄ 3FEBTIͷ σʔλιʔε͔Β 3FETIJGUΛআ ݄ ✖3FETIJGUෛՙ૿େ ✔ࣾʹ͘ීٴ ✖"UIFOBͷ ར༻ྉ͕ۚ ݄ 1# εΩϟϯ w 1SFTUPPO&.3 ར༻։࢝ w "UIFOBͷΫΤϦ Λ1SFTUPʹҠߦ ✖ੑೳ໘Ͱෆຬ ✔ར༻ྉ͕ۚ ݄Ͱ҆ఆ ݱࡏ ΞυϗοΫੳͷ සग़ͷूܭॲཧΛ )JWFPO&.3Ͱ όονॲཧԽ ݱࡏਐߦத ݄
ϋʔυΣΞϦιʔεΛҙࣝͤͣʹ5#ΦʔμʔͷεΩϟϯΛݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ ͋ΔఔʮແʯͳΫΤϦ"UIFOBͳΒݱ࣮తͳ࣌ؒͰॲཧͰ͖ͯ͠·͏ "UIFOBͷ͍উखͷྑ͞ ԯ݅ͷϩάΛ࡞ผʹूܭ "UIFOBඵͰྃ εΩϟϯαΠζ(# 1SFTUPPO&.3Ͱྃ NYMBSHF .BTUFS $PSF
5BTL ඵ "UIFOB 1SFTUPPO&.3
"UIFOBͷ͍উखͷྑ͞ ϋʔυΣΞϦιʔεͷ੍ "UIFOB σϑΥϧτͷಉ࣮࣌ߦΫΤϦͷ্ݶ ಉ࣮࣌ߦͨ͠ΫΤϦؒͰϦιʔεͷୣ͍߹͍ൃੜ͠ͳ͍ 1SFTUPPO&.3 ΫϥελʔͷϝϞϦͳͲͷϦιʔεΛෳΫΤϦͰγΣΞ͢Δ ԼهͷϝϞϦઃఆΛ͑ΔΫΤϦ࣮ߦͰ͖ͳ͍ ઃఆ߲ આ໌
RVFSZNBYNFNPSZ ΫΤϦͰ༻͢ΔϝϞϦͷ࠷େ RVFSZNBYNFNPSZQFSOPEF ΫΤϦϊʔυͰ༻͢ΔϝϞϦͷ࠷େ
"UIFOBͷར༻ྉۚߴಅͷݪҼ 3FEBTIͷΫΤϦεέδϡʔϧػೳ 3FEBTIʹอଘͨ͠ΫΤϦΛࢦఆͨ͠සͰ࣮ߦͰ͖Δ "UIFOBͷ͍উखͷྑ͞ ͋ΔఔʮແʯͳΫΤϦݱ࣮తͳ࣌ؒͰॲཧͰ͖Δ 5#ΛεΩϟϯ͢ΔΫΤϦ܈͕සൟʹ࣮ߦ ߹ܭεΩϟϯαΠζ͕1#ΛΦʔόʔ
ݱࡏ1SFTUPPO&.3ΛϝΠϯͰར༻ نͰ"UIFOBʹશવఢΘͳ͍ʜ εΩϟϯαΠζΛখ͘͢͞ΔͨΊɺසग़ͷूܭॲཧΛ )JWFPO&.3ͰόονॲཧԽ͢ΔऔΛਪਐத ্هऔ͕ྃͨ͠Β"UIFOBͷํ͕҆͘ͳΔՄೳੑ͕ ͋ΔͨΊɺ"UIFOBΛϝΠϯʹ͢Δ͜ͱݕ౼ ΞυϗοΫੳͷݱࡏͱࠓޙ
ϏοάσʔλͷΞυϗοΫੳͷจ຺ʹ͓͍ͯ "UIFOB࠷ڧͷબࢶͷҰͭ ಛʹ੍ݶΛઃ͚ͣʹ"UIFOBΛಋೖ͢ΔͱఆΑΓར༻ྉ͕ۚߴ͘ ͳΔέʔε͕͋Δ 3FEBTIͱͷ߹ͤͰ"UIFOBීٴͷϋʔυϧάϯͱԼ͕Δ͕ɺ ΫΤϦεέδϡʔϧػೳͳͲҙ͕ඞཁ "84ར༻ྉۚΛීஈ͋·Γҙࣝ͠ͳ͍Ϣʔβʔ͕"UIFOBΛར༻ ͢Δ߹ɺίετҙࣝͳͲΛఆணͤ͞Δ׆ಈ͕ඞཁ ޙͷ·ͱΊɾڭ܇
ޙͷ·ͱΊɾڭ܇ ίετҙࣝ ΫΤϦ࣮ߦʹΑΓԿ(#εΩϟϯ͢Δͷ͔ɺͦͷ݁Ռྉ͍ۚ͘Βֻ͔Δ͔Λ Ϣʔβʔ͕ҙࣝ͢Δ͜ͱ͕ඇৗʹॏཁ "84ཧऀ͚ Ϣʔβʔ͕ར༻ྉۚΛ֬ೝͰ͖ΔΈΛ༻ҙ͢Δ %3:ݪଇ %PO`U3FQFBU:PVSTFMG εΩϟϯαΠζ͕େ͖͍ΫΤϦΛԿճ࣮ߦ͠ͳ͍ʗͤ͞ͳ͍
සग़ͷΫΤϦճ͚࣮ͩߦͯ݁͠ՌΛอଘ͢Δ ͓࡞๏ͷపఈ εΩϟϯαΠζݮͷͨΊύʔςΟγϣϯΛඞͣࢦఆ͢Δ ඞཁ࠷ݶͷΧϥϜΛ4&-&$5͢Δ ҆қʹ4&-&$5 ͠ͳ͍ FUD "UIFOBΛίετޮྑ͘׆༻͢ΔͨΊͷϚΠϯυηοτ
&0'