AI時代の大規模データ活用とセキュリティ戦略

データ管理とApacheIcebergについてセキュリティ監視の文脈から、ハンズオンしながら話すスライド。

近年、AIの発展とともに扱われるデータ量は飛躍的に増大しています。膨大なデータを効率的に活用するには、ストレージやOpen Table Format（データの構造を扱いやすくするための形式）、DataOps（データの開発・運用プロセスを管理する手法）などの技術を正しく理解し、運用する必要があります。これらの仕組みを把握しておくことで、高精度なAIモデル開発だけでなく、ビジネスのあらゆる場面でデータ利活用をスムーズに進められるようになります。

また、AIが業務全般に浸透し始めた今だからこそ、サイバーセキュリティ対策の重要性が増しています。本講義では、大規模データとAIを活用する際の注意点を体系的に学ぶとともに、組織的なセキュリティマネジメントにおいて留意すべきポイントを解説します。また、AI時代に求められる新人像やキャリア観にも触れ、これからのエンジニアが持つべき知識・姿勢をディスカッションしながら身につけていきます。

Transcript

1. @ken5scal, 2025/08/13 AI࣌୅ͷେن໛σʔλ׆༻ͱ ηΩϡϦςΟઓུ
2. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
3. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
4. Πϯτϩ
5. - ݸਓͷݖརͱͯ͠ͷϓϥΠόγʔɺՁ஋૑ग़ʹ͋ͨͬͯϦεΫΛ཈͑Δͷ ͨΊͷηΩϡϦςΟɺͲͪΒ΋σʔλ͕த৺ɻ - ࣾձతʹීٴͭͭ͋͠ΔAIʹͱͬͯ΋σʔλ͸ॏཁ - ͕ͨͬͯ͠ɺσʔλ؅ཧ͸ॏཁ - σʔλ؅ཧͱͦΕʹඞཁͳཁૉ΁ͷηΩϡϦςΟͱ͍͏؍఺ͰຊߨٛΛਐ Ί͍ͨ ࠓ೔ͷ͓࿩
6. 1. AIʹ΋ඞཁͳσʔλΛ؅ཧ͢Δͱ͸Ͳ͏͍͏ঢ়ଶ͔ཧղ 2. ͦͷதͰApache IcebergΛ࢖ͬͨγεςϜͷಛੑͷཧղ 3. ͦ͏͍ͬͨγεςϜʹ͓͚ΔηΩϡϦςΟͷઓུతͳରԠͷཧղ ࠓ೔ͷΰʔϧ
7. - ॴଐ - LayerX Fintechࣄۀ෦ - ࡾҪ෺࢈σδλϧɾΞηοτϚωδϝϯτ CISOʢˢ͔Βग़޲ʣ - Dev Ops, Corp Eng, ηΩϡϦςΟ౳Λ·Δͬͱ - Φϧλφ৴ୗɹγεςϜϦεΫ౷ׅʢˢͷࢠձࣾʣ - σδλϧிɹηΩϡϦςΟϢχοτ - ݸਓ׆ಈ - िؒχϡʔεϨʔλʔɺPodCastɺಉਓࢽ - ηΩϡϦςΟຊͷ຋༁ - ηΩϡΩϟϯߨࢣʢBίʔεʣ: 2019, 2023 ɹ※ຊߨٛʹશৼΓͷͨΊɺ্ه͸͢΂ͯҰ࣌ఀࢭத… @ken5scal ʢླ໦ݚޗʣ χϡʔεϨλʔ: https://ken5scal.notion.site/54bda4932da14add9e9911ab3e9a6e5c podcast: https://open.spotify.com/show/73sFeKzUIkSYfCZWVBNO70
8. - 2011~2014: NRIηΩϡΞςΫϊϩδʔζ (NRI͔Βͷग़޲ʣ - 2014~2018: Money Forward - 2018~2020: FOLIO - 2020~: ݱ৬ (ॴଐઌx4ʣ - ʢ݁ՌతʹʣελʔτΞοϓ+ ۚ༥ܥ͹ͬ ͔͠ɻ @ken5scal - དྷྺ χϡʔεϨλʔ: https://ken5scal.notion.site/54bda4932da14add9e9911ab3e9a6e5c podcast: https://open.spotify.com/show/73sFeKzUIkSYfCZWVBNO70
9. - Discordνϟωϧʮd3-ai࣌୅ͷ(ҎԼུʯͷઆ໌Օॴʹهࡌ - ϑΝΠϧஔ͖৔: https://drive.google.com/drive/folders/ 1COhIcFklpMndRdE3Xd6G9t9jxnjfa2wq - ΫΤϦ࣮ߦ: https://docs.google.com/document/d/ 10nYY7Z3aOW7Mn5gkZzF0MQJWENXt9oYf_QZo_F0cuco ڭࡐ
10. - ࢀՃऀͷ໊લʢॱෆಉʣ - দඌ͞Μɺࡈ౻͞Μɺେ௬͞Μɺยؠ͞Μɺ੨໦͞ɺ௽ా ͞Μɺࡾాଜ͞Μɺډྛ͞Μɺদౡ͞Μɺ౻઒͞Μ - ࣄલ४උͷঢ়ଶ֬ೝ - AWS IAM Userൃߦલͷঢ়ଶ֬ೝऴΘͬͯΔਓʢڍखʣ - AWS IAM Userൃߦޙͷঢ়ଶ֬ೝऴΘͬͯΔਓʢڍखʣ - ऴΘͬͯͳ͚Ε͹࣍ͷٳΈ࣌ؒʹҰॹʹ΍Γ·͠ΐ͏ ࢀՃऀ֬ೝ
11. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
12. ຊߨٛͷ಺༰
13. ຊΫϥεͰ͸ɺAI։ൃʹඞཁͳߟ͑ํΛʮ߈ܸʯͱʮ๷ޚʯͷ྆໘͔Β ࣮ફతʹֶͼ·͢ɻಛʹੜ੒AI΍ը૾ࣝผAIʹର͢Δ۩ମతͳ߈ܸख๏ ͱͦͷରࡦɺ৴པͰ͖ΔAIΛ࡞ΔͨΊͷֶशσʔλͷѻ͍ɺ͞Βʹ͸AI ΛηΩϡϦςΟλεΫʹ׆༻͢Δํ๏·Ͱɺ࠲ֶͱϋϯζΦϯܗࣜͰ෯ ޿͘औΓѻ͍·͢ɻՃ͑ͯɺੜ੒AI࣌୅ʹ͓͚ΔΩϟϦΞܗ੒΍ηΩϡ ϦςΟϚωδϝϯτʹ͍ͭͯ΋ɺઐ໳Ոͱͷٞ࿦Λ௨ͯ͡ߟ࡯ΛਂΊͯ ͍͖·͢ɻAI͕ΑΓ਎ۙͳଘࡏͱͳΔࠓɺٕज़Λਖ਼͘͠ཧղ͠ɺ҆શʹ ׆༻͢ΔͨΊͷʮ໨ʯͱʮࢹ఺ʯΛڞʹֶͼ·͠ΐ͏ɻ͝Ԡื͓଴ͪ͠ ͓ͯΓ·͢ AIηΩϡϦςΟΫϥεʢDίʔεʣझࢫ
14. ଞͷߨٛ ߨ໊ٛ ϥϕϧ ಺༰ ʢݸਓͷղऍʣ ࣮ફLLMΞϓϦέʔγϣϯɾηΩϡϦςΟ D1/D6 શମతͳ߈ܸɾ๷ޚ؍఺ Ϋϥ΢υιʔγϯάʹΑΔֶशσʔλ࡞੒ೖ໳ D2 σʔλ࡞੒ ෺ཧAIηΩϡϦςΟ D4 ը૾ೝࣝΛத৺ͱͨ͠ ෺ཧαʔϏεಛԽηΩϡϦςΟ AIΤʔδΣϯτͰ࡞ΔηΩϡϦςΟࢧԉπʔϧ D5 γεςϜ։ൃ
15. AIγεςϜͱDίʔεߨٛͷҐஔ෇͚ https://www.microsoft.com/en-us/securityengineering/sdl/practices D1/D6 D4 D5 D2
16. AIγεςϜͱDίʔεߨٛͷҐஔ෇͚ σʔλ؅ཧ΍Γ͍ͨ
17. Ͳ͜Β΁Μͷ࿩ʁ D1/D6 D4 D5 D2 D3
18. - A: Կ͔͠Βͷ໨తΛୡ੒͢Δͷʹඞཁ͔ͩΒ - A: ·ͨ͸ͦͷޙͷৼฦΓʹඞཁ͔ͩΒ ͳͥσʔλ؅ཧ͕ඞཁͳͷ͔ʁ
19. • DίʔεͰ͸AI࣌୅ॏཁʹͳΔσʔλ؅ཧͷ͏ͪʮPreparationʯͷΈ͕Χ όʔ͞Ε͍ͯΔঢ়گ • ͦ΋ͦ΋աڈ̑೥ؒͷηΩϡΩϟϯߨٛΈ͚ͨͲɺσʔλ؅ཧͬͯΧόʔ ͞Εͯͳͦ͞͏ʁ • ෺ཧηΩϡϦςΟͰ΋OTͰ΋WebͰ΋σʔλͷॏཁੑͬͯ·͍ͯ͠Δͷ ʹɺͦΕͰ͍͍ͷ͔ʁ • ͍͍΍ɺྑ͘͸ͳ͍ʢ൓ޠʣ ຊߨٛΛ͢ΔϞνϕ
20. - ηΩϡϦςΟʹઐ໳ੑΛ࣋ͭ໨ઢ͔Βσʔλ؅ཧͱ͸Կ͔ཧղ͢Δ - σʔλ؅ཧʹ͓͚Δ༗ྗͳٕज़ʹ৮ΕΔ - ηΩϡϦςΟ؂ࢹʹσʔλ؅ཧΛԠ༻͢ΔͱͲ͏ͳΔ͔Λ஌Δ ૂ͍
21. ຊߨٛͷείʔϓ σʔλ؅ཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓ৘ใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲໨Ͱ͸ͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance
22. - “࠲ֶͱϋϯζΦϯܗࣜͰ෯޿͘औΓѻ͍·͢” - ਐΊํ: ৽نੑͷ͋Δཁૉͱීวੑͷ͋ΔཁૉΛ࠲ֶͱϋϯζΦϯΛ௨ֶͯ͠श - ৽نੑ: Apache IcebergΛ࢖ͬͨσʔλ؅ཧ - ීวੑ: σʔλ؅ཧͷγεςϜʹର͢Δอޢ - σʔλར༻ͷϢʔεέʔε: ηΩϡϦςΟ؂ࢹ - ࢖͏σʔλ: AWSͷ؂ࠪϩά - https://bit.ly/40Wiw1M <- ͪ͜ΒͷϦϯΫ͔ΒదٓϑΝΠϧDLΛ͓ئ͍͢ΔͷͰɺURLΛ ͓ؾʹೖΓʹ͍ΕͨΓ͢ΔͳͲ͓ئ͍͠·͢ - Discordνϟωϧͷઆ໌ཝʹ΋͋Γ·͢ɻ ຊߨٛΛͲ͏ਐΊΔ͔
23. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
24. σʔλ؅ཧ
25. ͍͖ͳΓϋϯζΦϯʂ
26. - γνϡΤʔγϣϯ: - ౤ࢿՈͷࢿۚΛ༬͔Δূ݊αʔϏε - ҙਤ͠ͳ͍ग़ۚɾചങ͕͋ΔͱΫϦςΟΧϧ - Ұํɺਖ਼౰ͳۀ຿ͱͯ͠ޱ࠲͔Βग़ۚ͢Δ͜ͱ͕͋Δɻ͜ͷ৔߹͸ࣄલʹ᜛ٞͰঝೝɾܾࡋΛಘΔ - ૝ఆ͢ΔϦεΫ͸ɺͦ͏͍ͬͨਖ਼౰ͳۀ຿Λߦ͏ࣾһͷۜߦΞΧ΢ϯτ͕ෆਖ਼ར༻͞Εͨɹ΋͘͠͸಺෦ෆਖ਼ΛߦΘΕͨ - ଘࡏ͢Δϩά - ۜߦAͷग़ۚཤྺϩά - ۜߦBͷೖग़ۚཤྺϩά - ग़ۚͷঝೝɾܾࡋΛͱΔϫʔΫϑϩʔϩά - ※ঝೝɾܾࡋ೔ = ग़ۚ೔ͱ͸ݶΓ·ͤΜ - ͜ΕΛ݄࣍Ͱʢ؂ࠪʣ͢Δϩά෼ੳͷϓϩηεΛߟ͍͑ͯͩ͘͞ - ※ͲͷϩάΛͲͷΑ͏ʹಥ߹͠ɺूܭͯ͠ɺԿΛ࣋ͬͯ໰୊ͳ͍ͱ͢Δ͔ɺɺɺͱ͍͏ϓϥϯΛͨͯΔ - ϋϯζΦϯ: 10min - ൃද: 7min? ϋϯζΦϯ01: ηΩϡϦςΟ෼ੳΛ΍ͬͯΈΑ͏
27. - ҎԼͷཧ༝Ͱ࢖͍ʹ͍͘ - จࣈίʔυ - ΧϥϜ໊ͷݴޠ - ֤γεςϜͰҟͳΔεΩʔϚ - ࣌ؒϑΥʔϚοτ - ݸਓ৘ใͷࠞೖ ϋϯζΦϯ: Ͳ͏͍͏લఏ͔ͩͬͨ
28. σʔλ؅ཧͷ֓ཁɾҙٛ
29. - σʔλͱ৘ใࢿ࢈Λܭըɺ؅ཧɺ഑৴͢ΔϏδωεػೳ by DMBOK - σʔλ؅ཧͰ͸ɺҙࢥܾఆͷͨΊʹσʔλΛऩूͯ͠࢖༻͢ΔͨΊͷϙϦ γʔͱखॱΛ੍ఆ by Tableau - ʮσʔλͱ৘ใࢿ࢈Λऩूɾอ؅ɾར༻Ͱ͖ΔΑ͏ʹఏڙ͢Δػೳ ʯ σʔλ؅ཧʢData Managementʣͱ͸Կ͔ʁ DAMA DMBOK
30. - A: Կ͔͠Βͷ໨తΛୡ੒͢Δͷʹඞཁ͔ͩΒ - A: ·ͨ͸ͦͷޙͷৼฦΓʹඞཁ͔ͩΒ ͳͥσʔλ؅ཧ͕ඞཁͳͷ͔ʁʢ࠶ܝʣ
31. - A: Կ͔͠Βͷ໨తΛୡ੒͢Δͷʹඞཁ͔ͩΒ - A: ·ͨ͸ͦͷޙͷৼฦΓʹඞཁ͔ͩΒ ͳͥσʔλ؅ཧ͕ඞཁͳͷ͔ʁʢ࠶ܝʣ ҙࢥܾఆ Ψόφϯε
32. - ੜσʔλ͸ଟछଟ༷ - ѹॖํࣜ( `json.gz` , `zip`, ·ͨ͸ѹॖͳ͠) - ܗࣜ( `jpg`, `csv`, `json`, `txt`, `mp3`, `md`) - εΩʔϚɺΧϥϜ໊ - ஋(ྫ: גࣜձࣾͱ(ג)) - ߋ৽͞ΕΔεΩʔϚ - ੜσʔλ͸ɺͦͷ··Ͱ͸࢖͑ͳ͍ʮૉࡐʯͷࢁ σʔλ͸ͦͷ··࢖͑Δ͔
33. - ৘ใΛར༻ऀͷϢʔεέʔεʹ͋Θͤͯ࠷దԽ͍ͯ͠Δ͜ͱ - ࠷దԽ͸ʮσʔλΛ੔උ͠ɺՃ޻͢Δʯ͜ͱ - ໨తʹԊͬͨσʔλͷఏڙ - ద੾ͳൣғͷσʔλͷఏڙ - ద੾ͳܗࣜͷσʔλͷఏڙ - ൃݟɾཧղɾॲཧͰ͖Δ؀ڥΛ੔͑Δ͜ͱ - ʮσʔλͱ৘ใࢿ࢈Λऩूɾอ؅ɾར༻Ͱ͖ΔΑ͏ʹఏڙ͢Δػೳ ʯ - ར༻ऀ͕AIͰ΋ਓؒͰ΋มΘΒͳ͍ - ͔͠͠ɺAI࣌୅͸͜ͷϓϩηεͷ࣭ͱεϐʔυ͕ɺڝ૪ྗΛ௚઀ࠨӈ͢Δ σʔλΛ؅ཧɾར༻Ͱ͖Δɺͱ͸
34. σʔλ؅ཧͷཁૉ01 ཁૉ໊ આ໌ Data preparation ੜͷσʔλΛΫϦʔϯΞοϓ͠ɺม׵͢Δϓϩηε Data pipelines σʔλΛγεςϜ͔Βผͷஈ֊ͷγεςϜ΁ྲྀ͢Ұ࿈ͷγεςϜίϯϙʔωϯτ Data ETL σʔλΛσʔλ΢ΣΞϋ΢εʹϩʔυ͢ΔͨΊɺநग़ɾม׵͢ΔϓϩηεɻData preparationʹ΋࢖ΘΕΔ͠ɺύΠϓϥΠϯ্ʹ΋ࡌΔ
35. σʔλ؅ཧͷཁૉ02 ཁૉ໊ આ໌ Data catalog σʔλͷશମ૾ͷ೺Ѳ͠ɺ໨తͷσʔλΛൃݟ͠΍͘͢͢ΔΑ͏ɺϝλσʔλΛ؅ཧ͢Δ ίϯϙʔωϯτ Data warehouses ར༻͢ΔσʔλιʔεΛ౷߹͍ͯ͠Δίϯϙʔωϯτ Data governance σʔληΩϡϦςΟͱίϯϓϥΠΞϯεΛҡ࣋͢ΔͨΊͷϙϦγʔͱखॱɺͦͯͦ͠ΕΛ ؅ཧ͢Δଶ੎ Data architecture σʔλϑϩʔΛ؅ཧ͢ΔεΩʔϚ Data security σʔλΛอޢ͢ΔͨΊͷηΩϡϦςΟ
36. σʔλ؅ཧͷཁૉʢ࠶ܝʣ σʔλ؅ཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓ৘ใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲໨Ͱ͸ͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance
37. ηΩϡϦςΟ؂ࢹ΁ σʔλ؅ཧΛԠ༻
38. - ෆਖ਼ͷૣظݕ஌ - ίϯϓϥΠΞϯεཁ݅ͷୡ੒ঢ়گ ηΩϡϦςΟ؂ࢹͷ໨త
39. - ෆਖ਼ͷૣظݕ஌ - ίϯϓϥΠΞϯεཁ݅ͷୡ੒ ηΩϡϦςΟ؂ࢹͷ໨తʢਅʣ - γεςϜʹ͓͚ΔϦεΫ͕ݦࡏԽͯ͠ͳ͍͔൑அɺͦͯ͠ଓ͘ΞΫγϣϯΛ ܾΊΔ͜ͱɹʢҙࢥܾఆʣ - ·ͨɺϦεΫରࡦ͕ػೳ͍ͯ͠Δ͔ϞχλϦϯά͢Δ͜ͱʢΨόφϯεʣ
40. - ͱΓ͋͑ͣϩάΛू໿ɾҰݩԽ - Webϒϥ΢β্ͰΫΤϦ - ΫΤϦΛ΋ͱʹͨ͠Ξϥʔτ - ΫΤϦΛ΋ͱʹͨ͠μογϡϘʔυ؂ࢹ - ͔͠͠… ηΩϡϦςΟ؂ࢹͷʮΑ͋͘Δ෩ܠʯ
41. - ͦ΋ͦ΋औಘ͕େม໰୊ - σʔλར༻͠ʹ͍͘໰୊ - ϑΥʔϚοτ͕όϥόϥ໰୊ - อ؅Ͱ͖Δσʔλྔ͕ݶఆత ηΩϡϦςΟ؂ࢹʹ·ͭΘΔʮσʔλʯͷ՝୊ ಛʹϦεΫϕʔεͰ؂ࢹ͍߲ͨ͠໨Ͱ͸ݦஶ
42. - ໰୊1: ίωΫλͷน - ໰୊2: औಘํ๏ͷଟ༷ੑɾࠔ೉͞ ͦ΋ͦ΋औಘ͕େม໰୊
43. - ؂ࢹ؀ڥͱσʔλΛܨ͙طଘͷίωΫλʹ͸੍ݶ͕͋Δɻ - ੍ݶ1: ίωΫλ͕ଘࡏ͠ͳ͍ - ࣗࣾͰ࢖͏શͯͷαʔϏεʹରԠ͢ΔίωΫλ͕͋Δͱ͸ ݶΒͳ͍ - ಛʹɺ֤ۜߦͷೖग़ۚ໌ࡉͷΑ͏ͳɺ೔ຊࠃ಺ͷಛఆۀ຿ ʹಛԽͨ͠ίωΫλ͸·ͣఏڙ͞Ε͍ͯͳ͍ - ੍ݶ2: ࣗ࡞ͷίεύ͕ྑ͍ͱ͸ݴ͑ͳ͍ - ࣗ࡞ίωΫλͷ։ൃɾอकίετ͸ɺσʔλΛࣗલͷετ Ϩʔδʹอ؅͢Δ࢓૊ΈΛ૊Ήͷͱಉ౳ఔ౓ ίωΫλͷน ಛʹࣗࣾͷϦεΫ ϕʔεͰ؂ࢹͨ͠ ͍߲໨Ͱ͸ݦஶ ࡞ۀ
44. - શͯͷσʔλ͕ɺ៉ྷͳAPI͔ΒऔಘͰ͖ΔΘ͚Ͱ͸ͳ͍ɻݱ৔Ͱ͸ɺటष͍ํ๏ Ͱ͔͠खʹೖΒͳ͍σʔλ΋ଟ͍ɻ - API͕ͳ͍৔߹ͷऔಘํ๏ - ؂ࢹ͞ΕΔଆͷγεςϜ͕Webhookࢦఆʢྫ: GitHub Enterprise - Webը໘͔Βͷखಈμ΢ϯϩʔυ (ྫ: ۜߦͷCSV) - WebαΠτΛ८ճͯ͠৘ใΛूΊΔ (ΫϩʔϦϯά) - ࠷ѱͷ৔߹ɺը໘ͷεΫϦʔϯγϣοτ औಘํ๏ͷଟ༷ੑɾࠔ೉͞ - ᶃ
45. - ϑΟϧλϦϯάͷ໰୊ - ͜͏ͨ͠ੜσʔλʹ͸ɺ෼ੳʹෆཁͳ৘ ใ΍ɺѻ͏ͷ͕ةݥͳݸਓ৘ใؚ͕·Ε ͍ͯΔ͜ͱ͕ଟʑ͋Δɻ - ʮ͏͔ͬΓݸਓ৘ใΛւ֎ͷαʔόʔʹ ૹͬͯ͠·ͬͨʯͰ͸ࡁ·ͳ͍ɻ - σʔλΛ෼ੳπʔϧʹೖΕΔલͷʮࣄલ ϑΟϧλϦϯάʯΛ͍ͨ͠ɻ औಘํ๏ͷଟ༷ੑɾࠔ೉͞ - ᶄ
46. - ֎෦σʔλ΋ؚΊͨෳ਺ͷσʔλͱͷಥ͖߹Θ͕ͤࠔ೉ - γεςϜɾ࡞ۀ؀ڥ͝ͱʹΫΤϦํ๏͕ҟͳΔ - ಛʹϩʔΧϧͰΫΤϦΛͬ͘͞ͱݕূ͠ʹ͍͘ σʔλར༻Λ͠ʹ͍͘໰୊
47. - ϑΝΠϧܗࣜͷҧ͍ - จࣈίʔυͷҧ͍ - ݴޠͷҧ͍ - λΠϜκʔϯͷҧ͍ - ࣌ࠁϑΥʔϚοτͷҧ͍ - ΦϨΦϨσʔλ஋ - ಛఆ࣌ظͷϑΥʔϚοτҧ͍ ϑΥʔϚοτ͕όϥόϥ໰୊ ಛʹࣗࣾͷϦ ϕʔεͰ؂ࢹ͠
48. ྫ: ๭passwordϚωʔδϟͷϩά - Time͕ΞϝϦΧ౦෦ඪ४࣌EST - ্ʹ͍ͭͯԿ΋ϩά্ʹهࡌ͕ͳ͍ - ԿͳΒAPI࢓༷ʹ΋ͳ͔ͬͨ
49. ྫ: ۜߦͷೖग़ۚϩά - ೔ຊޠจࣈίʔυલఏ - CSV - ΧϥϜ͕೔ຊޠ - ۜߦ͝ͱͷṖ৘ใ - `240002184 008 0000008` ͬͯͳʹ... - Time͕JSPલఏ - ஋΋೥݄೔࣌෼ - ଞͷγεςϜͱҧ͍ඵ͕ͳ͍ - ݸਓ৘ใ
50. ྫ: ۜߦͷೖग़ۚϩά - ϦχϡʔΞϧʹ൐͏ഁյతͳεΩʔϚม ߋ - ͜͏͍͏ͷ͕͋ΔͱɺաڈͷσʔλΛ ৽εΩʔϚʹม׵͢ΔͳͲͷ൑அ͕ඞ ཁ - ͦͯ͠ɺ͍͔ͭΒ͍ͭ·Ͱ͕چόʔδϣ ϯͰ͔͋ͬͨΛϝλతʹೝ஌͢Δඞཁ ͕Ͱͯ͘Δ...
51. - શ෦͕શ෦·ͬͨ͘Ͱ͖ͳ͍ιϦϡʔγϣϯ͸গͳ͍ - ͔͠͠ɺଳʹ୹͠᧪ʹ௕͠ - ͲͷΑ͏ͳιϦϡʔγϣϯͰ΋ɺ݁ہɺෆ଍෼͸ࣗલͰΧόʔ͢Δ͔͠ͳ͍ - Χόʔൣғ͕ڱ͚Ε͹Α͍͕ɺͦ͏Ͱͳ͍έʔεͷ΄͏͕ଟ͍ - ιϦϡʔγϣϯϕʔεͩͱɺ݁ہίϯτϩʔϧՄೳͳൣғ͕ڱ͘ɺ׌ͭίετ ΋ߴ͍ ιϦϡʔγϣϯ͝ͱʹҰ௕Ұ୹
52. ηΩϡϦςΟ؂ࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ σʔλར༻΋؅ཧ΋ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠ΋ιϦϡʔγϣϯಛ ༗… औΓࠐΊͳ͍… औΓࠐΊͳ͍… ᶃϑΥʔϚοτ͕ ᶄ઀ଓܗ͕ࣜ ᶅݸਓ৘ใͷআڈ͕ αϙʔτ͞Εͯͳ͍͔Β औΓࠐΊͳ͍…
53. ࣗલ ίωΫλ ηΩϡϦςΟ؂ࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ ࣗ લ தܧ DB σʔλར༻΋؅ཧ΋ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠ΋ιϦϡʔγϣϯಛ༗… ʢղܾ͞Εͳ͍…ʣ ࣗલͰதܧ͢Ε͹ͳΜͱ͔ͳ ΔɻͰ΋͜͜·Ͱ΍ΔͳΒ….
54. ࣗલ ίωΫλ ηΩϡϦςΟ؂ࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ ࣗ લ தܧ DB σʔλར༻΋؅ཧ΋ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠ΋ιϦϡʔγϣϯಛ༗… ʢղܾ͞Εͳ͍…ʣ ࣗલͰதܧ͢Ε͹ͳΜͱ͔ͳ ΔɻͰ΋͜͜·Ͱ΍ΔͳΒ…. ηΩϡϦςΟ؂ࢹͷ෇ՃՁ஋Λ্͛ΔͨΊ σʔλΛʰίϯτϩʔϧʱͰ͖ΔൣғΛ޿͍͛ͨ ≒ σʔλ؅ཧΛऔΓࠐΊΔܗʢιϦϡʔγϣϯɺ࣮૷ʣʹ͍ͨ͠
55. લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ
56. - ✅ ऩूɾอ؅ɾ෼ੳΛҰݩԽ͢Δ͚ͩͰͳ͘ɺίϯτϩʔϥϏϦςΟΛߴΊΒΕΔɻ - ✅ λΠϜκʔϯ΍จࣈίʔυΛɺ౷ҰతͳϧʔϧͰ੔ܗɻ - ✅ ෼ੳͨ͘͠ͳ͍ݸਓ৘ใͳͲΛɺࣄલʹআ֎ (ϑΟϧλ)ɻ - ✅ ඞཁͳ৘ใʢྫɿࣾһϚελʣΛ෇༩ͯ͠ɺσʔλΛϦονʹ (ΤϯϦον)ɻ - ✅ ໨తʹԠͨ͡৽͍͠σʔληοτΛɺࣗ༝ʹ࡞੒ɻ - ✅ ݁Ռతʹɺ෼ੳίετ΋࠷దԽͰ͖Δ σʔλ؅ཧ͕΋ͨΒ͢ϝϦοτ
57. ౰ࣾͷࣄྫ
58. ౰ࣾͷࣄྫ ϑΥʔϚοτΛ ඪ४Խ ϑΥʔϚοτͷඪ४Խ
59. ౰ࣾͷࣄྫ ෳ਺औಘํࣜͷαϙʔτ - APIΫϩʔϦϯάʢෳ਺αʔϏεɾ πʔϧɺࣗલ࣮૷ʣ - Webhookड͚ - ετϨʔδ΁ͷ௚઀อ؅ - खಈ
60. ౰ࣾͷࣄྫ ϑΟϧλϦϯά - ݸਓ৘ใͷϑΟϧλϦϯά - ݕ஌
61. ౰ࣾͷࣄྫ ҆શͰ࢖͍΍͘͢ͳͬͨσʔλ΍ɺ֎ ෦σʔλΛ૬ؔ෼ੳ ҆શͰ࢖͍΍͘͢ͳͬͨσʔλ΍ɺ֎ ෦σʔλΛ૬ؔ෼ੳ
62. ౰ࣾͷࣄྫ ৭ʑ΍͚ͬͨͲ… ݁ہɺʢ͖Ε͍ʹͳͬͨʣ σʔλ͸ͲΜͳܗͰͲ͜ʹͲ Μͳ෩ʹ͋ͬͯɺ ͦΕΛͲ͜ʹ໰͍߹ΘͤΕ͹ Θ͔Δͷʁ
63. ౰ࣾͷࣄྫ ͦͷͨΊͷData LakeHouse΍ʂ
64. B-1Ͱ΋ࣅͨΞϓϩʔνΛͱͬͯΔ https://github.com/m-mizutani/seccamp-2025-b1
65. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
66. ٳܜ (+ηοτΞοϓϔϧϓ࣌ؒʣ
67. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
68. σʔλ΢ΣΞϋ΢ε
69. σʔλ؅ཧͷཁૉʢ࠶ܝʣ ཁૉ໊ આ໌ Data catalog σʔλͷશମ૾ͷ೺Ѳ͠ɺ໨తͷσʔλΛൃݟ͠΍͘͢͢ΔΑ͏ɺϝλσʔλΛ؅ཧ͢Δ ίϯϙʔωϯτ Data warehouses ར༻͢ΔσʔλιʔεΛ౷߹͍ͯ͠Δίϯϙʔωϯτ Data governance σʔληΩϡϦςΟͱίϯϓϥΠΞϯεΛҡ࣋͢ΔͨΊͷϙϦγʔͱखॱɺͦͯͦ͠ΕΛ ؅ཧ͢Δଶ੎ Data architecture σʔλϑϩʔΛ؅ཧ͢ΔεΩʔϚ Data security σʔλΛอޢ͢ΔͨΊͷηΩϡϦςΟ
70. લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ
71. ͍͖ͳΓϋϯζΦϯʂ
72. - AWSΞΧ΢ϯτ297323088823, 085773780922 ͷ౦ژϦʔδϣϯʹ͓͍ͯ2025/07/14 18:00 ~ 2025/07/15 09:00ʹ࣮ࢪ͞ΕͨAWS্ͷActionɺ࣮ߦͨ͠ΞΫλʔΛΧ΢ϯτ͢Δ ϓϩηεͱϩδοΫΛ࡞͍ͬͯͩ͘͞ɻ - ΋ͱ΋ͱͲ͜ʹ͋Δ͔Λಛఆ͢Δͷ͸೉͍͠ͱ͓΋͏ͷͰɺࣄલʹఏڙͨ͠ɹhttps:// bit.ly/40Wiw1M > ʮੜawsϩάʯΛετϨʔδͱΈͨͯͯΛ࢖͍ͬͯͩ͘͞ - औಘ͔Β෼ੳ·Ͱͷखॱ΋໌͍ࣔͯͩ͘͠͞ - πʔϧूܭϩδοΫ͸ΧϥϜ໊Λ࢖ͬͯઆ໌Ͱ͖Ε͹͓̺Ͱ͢ɻਖ਼֬Ͱͳͯ͘΋͍͍Ͱ ͢ɻ ɹ※AIʹ౤͛ࠐΉ͸͋Γɻͦͷ৔߹ɺpython/sqlͷΫΤϦΛఏڙͯ͘͠Δͱ͓ΉͷͰɺͦͷΫ ΤϦͷʮूܭϩδοΫʯΛ͓࿩͍ͩ͘͞ɻ - ࣮ߦ: 10min, ൃද: 1minx7 ϋϯζΦϯ02: ηΩϡϦςΟ؂ࢹΛ΍ͬͯΈΑ͏
74. ϋϯζΦϯ02-01: σʔλ؅ཧԼʹ͋Δσʔλ෼ੳΛͯ͠ΈΑ͏ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯࢀߟ > ʮϋϯζΦϯ02-01: σ ʔλ؅ཧԼʹ͋Δσʔλ෼ੳΛͯ͠ΈΑ͏ʯʹίϐϖͰ͖ΔΫΤϦ͕͋Γ ·͢ɻ ΫϥΠΞϯτ͸duckDBͱ͍͏πʔϧ͕ͩɺඪ४ʹै͍ͬͯΕ͹ଞ ͷπʔϧ΍ϥΠϒϥϦͰ΋Մೳ㲈૬ޓޓ׵ੑ
77. ൺֱ ඇσʔλ؅ཧԼͷରԠ σʔλ؅ཧԼͷରԠ ෆཁʹͳͬͨʁ ετϨʔδ୳ࡧ aws s3όέοτΛ୳͢ ʢWeb, cliʣ ςʔϒϧ໊Λ୳͢ ✗ ετϨʔδཧղ όέοτ഑ԼͷϑΥϧμߏ ੒Λ೺Ѳ - ◦ ϑΝΠϧൃݟ json.gz` Λ औಘ - ◦ σʔλཧղ `jq`ͳͲΛ࢖ͬͯߏ଄೺Ѳ describeҰൃɻ ࢓༷Λຬ͍ͨͯ͠Ε͹πʔ ϧ͸໰Θͳ͍ ˚ɻΫΤϦͳͷͰΑΓ೺Ѳ ͠΍͘͢ͳͬͨ σʔλར༻ `jq`ͳͲΛ࢖ͬͯ෼ੳ ΫΤϦͰ෼ੳɻ ࢓༷Λຬ͍ͨͯ͠Ε͹πʔ ϧ͸໰Θͳ͍ ˚ɻ࢓૊ΈʹΑΓσʔλऔ ಘ͕ޮ཰Խ͞Ε͍ͯΔ
78. ηΩϡϦςΟ؂ࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ औΓࠐΊͳ͍… औΓࠐΊͳ͍… ᶃϑΥʔϚοτ͕ ᶄ઀ଓܗ͕ࣜ ᶅݸਓ৘ใͷআڈ͕ αϙʔτ͞Εͯͳ͍͔Β औΓࠐΊͳ͍… σʔλར༻΋؅ཧ΋ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠ΋ιϦϡʔγϣϯಛ ༗…
79. લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ ΫΤϦ ΫΤϦ σʔλ؅ཧଆʹґଘ͠ͳ͍ ׳ΕͯΔΫΤϦݴޠΛ ࢖͑ΔΑ͏ʹͳͬͨʂ σʔλߏ଄΋؆୯ʹΘ͔Δʂ
80. લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ ΫΤϦ ΫΤϦ σʔλ؅ཧଆʹґଘ͠ͳ͍ ׳ΕͯΔΫΤϦݴޠΛ ࢖͑ΔΑ͏ʹͳͬͨʂ σʔλߏ଄΋؆୯ʹΘ͔Δʂ πʔϧʹޓ׵ੑ͕͋Δʂ
81. - ݁࿦: Open Table Formatͱ͍͏ඪ४ʹै͍ͬͯΔ͔Β - ͍͖ͳΓOTFʹೖΔલʹɺͦΕҎલͷ֓೦ʹ͍ͭͯ৮Ε ·͢ - OLAP - σʔλ΢ΣΞϋ΢ε - σʔλϨΠΫ ͳͥ૬ޓޓ׵ੑ͕͋Δ͔
82. - Q: σʔλϕʔεͱ͸ҧ͏ͷʁ - A: DB͸OLTPʮOnLine Transaction Processingʯͱ͍͏τϥϯβΫγϣϯ୯ҐͰͷ҆ શͳॲཧ͕ಘҙɻେྔͷσʔλΛ͔ͭͬͨ෼ੳͱ ͸·ͨҟͳΔ෼໺ OLAP: Online Analytical Processing
83. - Q: ͳΜͰOLTPͱOLAPΛΘ͚Δͷʁ - A: - OLAPͱϦιʔεΛୣ͍߹Θͳ͍Α͏ʹ - OLAP͸େྔͷσʔλΛର৅ʹ͍ͨ͠ͷͰɺΧϥϜϕʔεͰ͕ͬ͞ͱ෼ੳ͍ͨ͠ - ྫʮ͓ళͷϨδʢOLTPʣͱɺࡏݿ؅ཧ૔ݿʢOLAPʣ͸໾ׂ͕ҧ͏͔Βɻ - Ϩδ͸ߴ଎ͳձܭ͕ٻΊΒΕɺ૔ݿ͸େྔͷࡏݿ͔Β໨తͷ΋ͷΛ୳͢ೳྗ͕ٻΊΒΕΔ - OLAP͸ෳ਺σʔλΛͬ͞ͱ૬ؔͤ͞ΔηΩϡϦςΟ؂ࢹɺಛʹ௕ظؒͷ෼ੳʹ૬ੑ͕͍͍ OLAP: Online Analytical Processing
84. OLAPͷίϯϙʔωϯτ ίϯϙʔωϯτ໊ ֓ཁ ίϯϐϡʔτΤϯδϯ σʔλॲཧɺ·ͨ͸ͦͷϫʔΫϩʔυͷ࣮ߦ Χλϩά ར༻͍ͨ͠σʔλͷൃݟɾಛఆΛޮ཰Խ͢ΔͨΊʹɺσʔλͷϝλσʔ λʢςʔϒϧ໊ɺεΩʔϚɺ෺ཧతͳ഑ஔՕॴʣΛίϯϐϡʔτΤϯδ ϯʹఏڙ ςʔϒϧϑΥʔϚοτ σʔλͷ࣮ମͰ͋Δଟ਺ͷϑΝΠϧ܈͕ͲͷΑ͏ʹετϨʔδ্ʹ഑ஔ ͞Ε͍ͯΔ͔Λࣔ͢ϝλσʔλɺ·ͨ͸࿦ཧతͳදͷΑ͏ͳ΋ͷɻ ετϨʔδΤϯδϯ ςʔϒϧϑΥʔϚοτʹैͬͯ෺ཧσʔλΛอଘɾߋ৽ɾ࡟আɻ ഑ஔ࠷ దԽɺظؒΛ௒͑ͨσʔλͷ࡟আ౳ͷ࠷దԽ΋࣮ࢪ ετϨʔδ σʔλͷอଘɻΦϒδΣΫτετϨʔδͱ͍͏ͷ͕͍͍͔΋ɻσʔλʹ ΑͬͯɺϑΝΠϧϑΥʔϚοτͱςʔϒϧϑΥʔϚοτ͸ҟͳΔ ϑΝΠϧܗࣜ ෺ཧతʹσʔλΛ֨ೲ͢ΔϑΝΠϧܗࣜɻߏ଄ vs ඇߏ଄,͔Β͞Βʹ ߦ ࢥߟ vs ྻࢥߟͰ͞Βʹ෼ྨ͞ΕΔ
85. - OLAPΛγεςϜʹམͱ͠ࠐΜͩઃܭɺ·ͨ͸ϓϩμΫτ - 1988೥ॳग़ͷ֓೦ - ࣍Λ࣮ݱ - σʔλͷू໿ϦϙδτϦ - ߏ଄Խ͞Εͨߴ଎ͳΫΤϦΛఏڙ σʔλ΢ΣΞϋ΢ε
86. - σʔλͷ੔߹ੑΛอͪͳ͕Βߋ৽΍݁߹ͱ͍ͬͨૢ࡞͕ෳࡶɾࠔ೉ - εΩʔϚมߋͷ؅ཧ - ίϯϐϡʔτͱετϨʔδ͕ີ݁߹ͳͷͰίετ΋྆ํʹҾͬுΒΕΔ - ີ݁߹ͳͷͰσʔλ΁ͷΞΫηε͸ɺσʔλ΢ΣΞϋ΢εಠࣗͷίϯ ϐϡʔτΤϯδϯʹݶఆ͞ΕΔʢϩοΫΠϯʣ - ϚϧνϞʔμϧରԠ͕Ͱ͖ͳ͍ʢॊೈੑʹ͔͚Δ - MLରԠ͕ࠔ೉ σʔλ΢ΣΞϋ΢εͰ͸ෆ଍͍ͯͨ͠ͱ͜Ζ
87. - ࡉ͔͍͜ͱΛল͘ͱɺੜσʔλࣄʹҟͳΔෳ਺ͷϑΝΠϧϑΥʔϚοτΛϦʔζφ ϒϧʹอଘɾ෼ੳͰ͖ΔॊೈੑΛɺϫʔΫϩʔυ͝ͱʹҟͳΔίϯϐϡʔτΤϯδ ϯΛ࠾༻Ͱ͖ΔΑ͏ͳ࢓૊Έʹ͢Δ͜ͱͰ࣮ݱ - ίϯϐϡʔτͱετϨʔδͷ෼཭ʂ - ϑΝΠϧͷΦʔϓϯԽ! - …ͱ͍͏ίϯηϓτͰFacebook͕Hadoop্ͰSQLΛهड़͠΍͘͢͢ΔͨΊʹHive ϑϨʔϜϫʔΫΛ։ൃ (2008) - σʔλ΢ΣΞϋ΢εͷ໰୊͕ղܾ͢Δͱͱ΋ʹɺҰ෦σάϨ·ͨ͸ΑΓਂࠁԽ σʔλϨΠΫొ৔ https://www.borthakur.com/ftp/hadoopworld.pdf
88. - ߋ৽΍࡟আͷࠔ೉͞ - ্هΛΧόʔ͢ΔͨΊͷෳࡶͳσʔλύΠϓϥΠϯ - ্هʹΑΔεΩʔϚ؅ཧͷࠔ೉͞ σʔλϨΠΫͰ͸ෆ଍͍ͯͨ͜͠ͱ
89. - ߋ৽΍࡟আͷࠔ೉͞ - ্هΛΧόʔ͢ΔͨΊͷෳࡶͳσʔλύΠϓϥΠ ϯ - ্هʹΑΔεΩʔϚ؅ཧͷࠔ೉͞ σʔλϨΠΫͰ͸ෆ଍͍ͯͨ͜͠ͱ ݪҼ͸ςʔϒϧϑΥʔϚοτͷઃܭʂ
90. OLAPͷίϯϙʔωϯτʢ࠶ܝʣ ίϯϙʔωϯτ໊ ֓ཁ ίϯϐϡʔτΤϯδϯ σʔλॲཧɺ·ͨ͸ͦͷϫʔΫϩʔυͷ࣮ߦ Χλϩά ར༻͍ͨ͠σʔλͷൃݟɾಛఆΛޮ཰Խ͢ΔͨΊʹɺσʔλͷϝλσʔ λʢςʔϒϧ໊ɺεΩʔϚɺ෺ཧతͳ഑ஔՕॴʣΛίϯϐϡʔτΤϯδ ϯʹఏڙ ετϨʔδΤϯδϯ ςʔϒϧϑΥʔϚοτʹैͬͯ෺ཧσʔλΛอଘɾߋ৽ɾ࡟আɻ ഑ஔ࠷ దԽɺظؒΛ௒͑ͨσʔλͷ࡟আ౳ͷ࠷దԽ΋࣮ࢪ ϑΝΠϧܗࣜ ෺ཧతʹσʔλΛ֨ೲ͢ΔϑΝΠϧܗࣜɻߏ଄ vs ඇߏ଄,͔Β͞Βʹ ߦ ࢥߟ vs ྻࢥߟͰ͞Βʹ෼ྨ͞ΕΔ ςʔϒϧϑΥʔϚοτ σʔλͷ࣮ମͰ͋Δଟ਺ͷϑΝΠϧ܈͕ͲͷΑ͏ʹετϨʔδ্ʹ഑ஔ ͞Ε͍ͯΔ͔Λࣔ͢ϝλσʔλɺ·ͨ͸࿦ཧతͳදͷΑ͏ͳ΋ͷɻ ετϨʔδ σʔλͷอଘɻΦϒδΣΫτετϨʔδͱ͍͏ͷ͕͍͍͔΋ɻσʔλʹ ΑͬͯɺϑΝΠϧϑΥʔϚοτͱςʔϒϧϑΥʔϚοτ͸ҟͳΔ O'Reilly-ApacheIceberg-ਤ1-5-දܗࣜΛ࢖ͬͯσʔλϑΝΠϧΛදʹ·ͱΊΔ
91. - ޮ཰తͳσʔλΞΫηε: - Ϣʔβʔ΍෼ੳπʔϧ͸ɺσʔλ͕࣮ࡍʹͲͷϑΥϧμʹɺ͍ͭ͘ͷϑΝΠϧ ͱͯ͠อଘ͞Ε͍ͯΔ͔ͱ͍ͬͨ෺ཧతͳৄࡉΛҙࣝ͢Δඞཁ͕ͳ͘ͳΔɻ - ࿦ཧతͳҰͭͷදΛૢ࡞Ͱ͖Δɻ - Ұ؏ੑͷ୲อ: - ༷ʑͳπʔϧ΍ෳ਺ͷνʔϜ͕σʔλΛར༻͢Δࡍʹɺৗʹ࠷৽ͷσʔλ͕ࢀ র͞Ε͍ͯΔ͜ͱΛอূ - ٯʹݴ͏ͱɺίϨ͕อূ͞Εͳ͍ͱਓʹΑͬͯ෼ੳ݁Ռ͕ҟͳΔͱ͍͏ࣄଶʹ ςʔϒϧϑΥʔϚοτͷ໾ׂ O'Reilly-ApacheIceberg-ਤ1-5-දܗࣜΛ࢖ͬͯσʔλϑΝΠϧΛදʹ·ͱΊΔ
92. - σʔλϨΠΫ·Ͱ͸ Apache Hiveςʔϒϧ ϑΥʔϚοτ - ςʔϒϧͷఆ͕ٛϑΝ ΠϧγεςϜͷ෺ཧత ͳσΟϨΫτϦߏ଄ʹ ڧ͘ґଘ σʔλϨΠΫͷϑΥʔϚοτ https://estuary.dev/blog/explaining-data-lakes-lakehouses-catalogs/
93. - σʔλϨΠΫ·Ͱ͸Apache HiveςʔϒϧϑΥʔ Ϛοτ - ςʔϒϧͷఆ͕ٛϑΝΠϧγεςϜͷ෺ཧతͳ σΟϨΫτϦߏ଄ʹڧ͘ґଘ σʔλϨΠΫͷϑΥʔϚοτ ίϨͷԿ͕໰୊ͳͷ͔
94. - ॻ͖ࠐΈதʹϑΝΠϧΛಡΈऔΒΕΔͷͰɺ෼ੳ݁ՌʹҰ؏ੑ ͕ͳ͘ͳΔ - Ұ෦ͷϑΝΠϧ͕ॻ͖ࠐΈͳͲͰഁଛ͠ɺಡΈࠐΊͳ͘ͳΔ - εΩʔϚมߋ࣌ʹ৽چͰޓ׵ੑ͕୲อ͞Εͣɺݹ͍ϑΝΠϧ͕ಡ ΈࠐΊͳ͘ͳΔ - ϑΝΠϧͷϦωʔϜ͕Ͱ͖ͳ͍ - ෺ཧతͳॻ͖׵͑ʹͳΔͷͰɺσόοά΍࠶ݱ͕ࠔ೉ʹͳΔ - ্هΛΧόʔ͢Δʹ͸૬Ԡͷίετ͕ඞཁ σʔλϨΠΫʹ͓͚ΔςʔϒϧϑΥʔϚοτͷ໰୊
95. Open Table FormatʢOTFʣͱσʔλϨΠΫϋ΢εͷొ৔ - σΟϨΫτϦߏ଄Ͱ͸ͳ͘ݸʑͷσʔλϑΝΠϧ ϑΝΠϧ୯Ґʹɺෳ਺ͷ֊૚ʹ෼͔Εͨϝλσʔ λΛςʔϒϧͱͯ͠؅ཧ - ϑΝΠϧॻࠐΈͷͨͼʹɺϝλσʔλ͕ߋ৽͞Ε Δ - OTFΛ֩ͱͨ͠ΞʔΩςΫνϟ͕ʮσʔλϨΠΫϋ ΢εʯ - ΦϒδΣΫτετϨʔδΛετϨʔδ૚ͱͯ͠ ར༻ͭͭ͠ɺDBMSͷΑ͏ͳػೳɺಛʹACIDಛ ੑΛσʔλϨΠΫʹ΋ͨΒͨ͠ - ۩ମతͳOTFͷ࣮ྫͷ͕̍ͭʮApache Icebergʯ https://estuary.dev/blog/explaining-data-lakes-lakehouses-catalogs/
96. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
97. Apache Iceberg
98. - 2015೥ʹNet fl ixͷΤϯδχΞͰ͋ΔDaniel WeeksࢯͱRyan Blueࢯ͕։ൃ ͨ͠ΦʔϓϯςʔϒϧϑΥʔϚοτ - Net fl ix͸ϖλόΠτϨϕϧͷσʔλΛѻ͏ͷͰɺͦ͜·ͰεέʔϧͰ͖ Δඞཁ͕͋Δ - ͭ·Γඪ४Ͱ͋Γɺ੡඼΍ιϑτ΢ΣΞͦͷ΋ͷͰ͸ͳ͍ - https://iceberg.apache.org/spec/ - 2018೥ʹΦʔϓϯιʔεԽ & Apache Software FoundationʢASFʣʹد ଃ - ݱ࣌఺Ͱv4Λ։ൃத Apache Icebergͱ͸
99. - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓΛฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
100. - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ - Partition Evolution, Hidden Partitioning, Time travel, Version Rollback, Schema Evolutionͷαϙʔτ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓ͸ฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
101. - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ֦େੑʢScaleʣ - ϝλσʔλΛ׆༻ͨ͠ΫΤϦϓϥχϯάΛΫϥΠΞϯτଆͰ࣮ ݱ͠ɺϘτϧωοΫΛςʔϒϧଆʢαʔόʔଆʣʹ͠ͳ͍ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓ͸ฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
102. - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ΫΤϦϓϥχϯάΛO1ʹ͢Δ - σʔλϨΠΫͷ৔߹ɺ෺ཧతͳσΟϨΫτϦߏ଄Λςʔϒϧͷఆٛͱ͍ͯ͠Δͨ ΊɺΫΤϦΛ࣮ࢪ͢ΔલʹύʔςΟγϣϯ౳ͷ෺ཧߏ੒Λߟྀ͢Δඞཁ͕͋ͬͨɻ ͜Ε͸ɺσʔλར༻ऀ͸΋ͱΑΓΫΤϦΛ࣮ࢪ͢ΔϫʔΫϩʔυʹ΋༏͘͠ͳ͍ - ಡΈऔΓ͸ฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
103. - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓΛฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ACIDಛੑͷ֬อ, ߦ୯ҐͰͷςʔϒϧૢ࡞ - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
104. - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓ͸ฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ݎ࿚ͳܕʢDependable typesʣ - ໌֬ʹఆ͚ٛͮΒΕͨςʔϒϧܕ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ
105. - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓ͸ฒߦ͢Δॻ͖ࠐΈ͔Βͷ෼཭ʢSerializable Isolationʣ - ෺ཧతετϨʔδ͔Βͷ࿦ཧతͳ෼཭ʢετϨʔδͷ෼཭ʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτ - ಡΈऔΓ࠷దԽͱॻ͖ࠐΈ࠷దԽͷ྆ํͷܗ͕ࣜར༻Մೳ - ·ͨϑΝΠϧܗࣜʹ͔͔ΘΒͣɺఆٛ͞Εͨσʔλܕ΍εΩʔϚ؅ཧ ϓϩηεΛద༻Ͱ͖Δ Apache Iceberg͕࣮ݱͨ͜͠ͱ
106. ຊߨٛͰσϞΛ·͑͡Δػೳ ػೳ໊ ಺༰ ର৅ ACIDಛੑ- ָ؍తฒߦ੍ޚʹΑΔACIDಛੑ ॻ͖ࠐΈ׬ྃޙʹɺ࠷৽ͷϝλσʔλϑΝΠϧΛ࡞੒͢Δ͕ɺͦͷ࠷৽ͷϑΝΠ ϧʹϙΠϯλΛม͑Α͏ͱ͢ΔλΠϛϯάͰʢଞͷϥΠλʔ͕ଞͷมߋΛίϛο τ͍ͯ͠ͳ͍͔ʣڝ߹Λݕग़͢Δɻͳ͚Ε͹ίϛοτ͞Εɺ͋Ε͹ίϛοτ͞Ε ͳ͍ ˚ HiddenύʔςΟγϣχϯά σʔλར༻ऀͷΫΤϦΛղੳ͢ΔࡍʹɺύʔςΟγϣχϯάʹ൓ө͞ΕΔΑ͏ಛ ఆͷॲཧΛ಺෦తʹద༻͠ΫΤϦΛมߋ͢Δɻ ◦ ύʔςΟγϣϯߋ৽ʢEvolutio ύʔςΟγϣϯ؅ཧΛ෺ཧతͳσΟϨΫτϦߏ଄͔ΒϝλσʔλϑΝΠϧ্Ͱ؅ ཧ͢ΔܗࣜͱͳͬͨͨΊɺύʔςΟγϣϯΛมߋͯ͠΋ςʔϒϧͷ࠶࡞੒͕ෆཁ ͱͳΔɻ·ͨɺύʔςΟγϣϯόʔδϣϯΛҙࣝͤͣΫΤϦ͕Մೳ ˚ εΩʔϚߋ৽ʢEvolution ↑ͷςʔϒϧεΩʔϚόʔδϣϯ ◦ λΠϜτϥϕϧ ߋ৽࣌ʹϝλσʔλϑΝΠϧ͕࡞੒͞ΕΔͱͱ΋ʹɺΠϛϡʔλϒϧͳεφοϓ γϣοτ΋ੜ੒͞ΕΔͨΊɺಛఆͷλΠϛϯάͰͷσʔλΛΫΤϦͰ͖ΔΑ͏ʹ ◦ όʔδϣϯϩʔϧόοΫ εφοϓγϣοτ͕͋ΔͷͰʢࡶɺϝλσʔλϑΝΠϧͷ࠷৽ϙΠϯλΛߋ৽͢ ΔͳͲͯ͠ϩʔϧόοΫՄೳ ◦
107. - Net fl ix͕Ͳ͏ηΩϡϦςΟϦεΫΛఆྔԽ͍ͯ͠Δ͔ - https://www.youtube.com/watch?v=wgQkqPmDIAQ&t=1184s - Α͋͘ΔAE͕ͩɺͦ͜͸Net fl ixɺ͜ͷϏσΦͰHowΛ͕ͬͭΓઆ໌ͯ͠Δ - ϞϯςΧϧϩγϛϡϨʔγϣϯͳͲ౷ܭ΍ϞσϦϯάख๏Λ͕ͬͭΓ࢖͍ͬͯΔ - ͜ΕʹݶΒͣNet fl ix͸਺೥લ·Ͱߴ౓ͳηΩϡϦςΟϒϩάΛൃ৴͍ͯͨ͠ͷͰ͓͢͢Ί - https://net fl ixtechblog.com/all?topic=security - ͦͷޙɺͲ͜ʹ͍ͬͨͷ͔.... ༨ஊ(ඈ͹͔͢΋ʣ
108. Apache IcebergͷΞʔΩςΫνϟ https://iceberg.apache.org/spec/#overview - ࣮ࡍͷσʔλʹΞΫηε͢Δ·Ͱɺͬ͘͟Γ3ͭͷϨΠϠʔͱ ̐ͭͷϑΝΠϧΛܦ༝ - σʔλར༻ऀͱͯ͠ମݧΛఏڙ͠ͳ͕Βղઆ
109. πʔϧͷղઆ https://iceberg.apache.org/spec/#overview - ΫΤϦΤϯδϯ - DuckDB: OLAPϫʔΫϩʔυ޲͚ʹઃܭ͞ΕͨΠϯϝϞϦͳ෼ੳDBɻ ༷ʑͳ֦ுπʔϧ͕͋Δ - Athena: AWSͷαʔόʔϨεͳϑϧϚωʔδυͳΫΤϦαʔϏε - Iceberg SDK: IcebergͷSDKɻݴޠ͝ͱʹ͋Δ - Lake Formation: Amazon S3 ςʔϒϧͱ౷߹͞ΕɺS3 ʹอଘ͞Εͨσʔ λʹର͢ΔूதతͳΞΫηε੍ޚͱΨόφϯεΛՄೳʹ͠·͢ɻ - S3 Tables: Apache Iceberg ܗࣜͷςʔϒϧϑΥʔϚοτσʔλͱͯ͠࠷ద Խ͞ΕͨS3 BucketετϨʔδΛఏڙ - σʔλϨΠϠͷS3͸҉໧తʹ࡞ΒΕΔ
110. σʔλར༻ऀͱͯ͠σʔλΛ ಡΈͳ͕ΒɺΞʔΩςΫνϟΛֶΜͰ ΈΑ͏
111. ΧλϩάϨΠϠʔ https://iceberg.apache.org/spec/#overview - σʔλར༻ऀ͕Icebergςʔϒϧͱର࿩͢Δ࠷ॳͷϙΠϯτ - ݱࡏͷϝλσʔλ΁ͷϙΠϯλΛอ࣋ - ACIDಛੑͷAtomicityઈର֬อ͢Δ͘Μ - ͜ͷςʔϒϧʹͲͷΑ͏ͳσʔλؚ͕·Ε͍ͯΔ͔Λ࠷ॳʹఏڙ - ಛఆͷλΠϛϯάͰͷςʔϒϧͷεΩʔϚɺύʔςΟγϣχϯάɺεφοϓγϣ οτཤྺͳͲͱ͍ͬͨςʔϒϧͷঢ়ଶʹؔ͢ΔΛϝλσʔλΛอ࣋ɻ - ࣮ࡍʹςʔϒϧ઀ଓΛ͢Δࡍ͸ɺ͜ͷΧλϩάΛࢦఆ͍ͯ͠Δ
112. Χλϩάͱͷ઀ଓʢDuckDB) $ cat duckdb_init.sql INSTALL httpfs; INSTALL iceberg; INSTALL aws; LOAD iceberg; LOAD httpfs; LOAD aws; FORCE INSTALL iceberg FROM core_nightly; FORCE INSTALL avro FROM core_nightly; CREATE SECRET secret ( TYPE s3, PROVIDER credential_chain, REGION 'ap-northeast-1' ); ATTACH 'arn:aws:s3tables:ap-northeast-1:297323088823:bucket/security- camp-table-bucket' AS s3_tables ( TYPE iceberg, secret secret, ENDPOINT_TYPE s3_tables ); SET unsafe_enable_version_guessing = true; % export AWS_ACCESS_KEY_ID=hogehoge % export AWS_SECRET_ACCESS_KEY=fugafuga % duckdb --init duckdb_init.sql https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡΈࠐΈ > Χλϩάͱͷ઀ଓ
113. ϝλσʔλϨΠϠʔ https://iceberg.apache.org/spec/#overview - Apache Iceberg͕ఏڙ͢Δػೳͷ؊ - ΤϯδϯͱετϨʔδͷؒͰϝλσʔλ/ந৅ԽϨΠϠʔͱͯ͠ػೳ - ۩ମతʹ͸σʔλͱͦͷϝλσʔλɺ͓ΑͼͦΕΒ΁ͷૢ࡞Λ௥੻͢Δߏ଄ - Metadata fi leɺManifest listɺManifest fi leͱ͍ͬͨ̏छྨͷϑΝΠϧ͔Βߏ੒ - ΫΤϦΤϯδϯ͕Χλϩάܦ༝ͰΞΫηε͢Δ৔߹ɺσϑΥϧτͰ͸࠷৽ͷ metadata fi le͕ಡΈࠐ·ͤΒΕΔ - ͜ΕΒ͸࣮ࡍͷϑΝΠϧͱಉ͡ετϨʔδʹอ؅͞ΕΔ
114. ϝλσʔλϑΝΠϧΛಡΜͰΈΑ͏ - ΫΤϦ͸ɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ > ʮϝλσʔλϑΝΠϧΛಡΜͰΈΑ͏ʯ͔Βऔಘɻ - Ͱ͖ͳ͚Ε͹ɺhttps://bit.ly/40Wiw1M > metadata-sample.json ͔Βऔಘ
115. - ಛఆͷλΠϛϯάͰͷςʔϒϧͷεΩʔ ϚɺύʔςΟγϣχϯάɺεφοϓγϣ οτཤྺͳͲͱ͍ͬͨςʔϒϧͷঢ়ଶʹ ؔ͢ΔΛϝλσʔλΛอ࣋ɻ - ಛʹॏཁͳͷ͕ ςʔϒϧͷݱঢ়Λද͢ `current-snapshot-id` ϝλσʔλϑΝΠϧ
116. ϝλσʔλͷεΩʔϚ(ࢀߟ·Ͱʣ
117. ϝλσʔλϑΝΠϧ͔ΒϚχϑΣετϦετ΁ https://iceberg.apache.org/spec/#overview - `current-snapshot-id` ͷ஋Λಛఆ - `snapshots` Ϧετ͔Βɺcurrent-snapshot-idͱಉ͡idΛ࣋ͭ `snapshot-id` Λ snapshots͔Β୳ࡧ - ↑ͷ `manifest-list` ͷ஋ = ϑΝΠϧͷύε ʢ `*.avro`ʣΛಛఆ - DuckDBͰ΍Δ৔߹ - ΫΤϦ͸ɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ >ɹʮϝλσʔλϑΝΠϧ͔ΒϚχϑΣετϦετ΁ʯɻ - Ͱ͖ͳ͚Ε͹ɺhttps://bit.ly/40Wiw1M > snap-manifest-list-sample.avro Λऔಘ –ϚχϑΣετϦετϑΝΠϧͷύεͷऔಘ D SELECT s."manifest-list" FROM read_json('s3://a33d7730-dc61-4b7d-1rerceqph8j3ubw9fj6snbyfxtqyhapn1b--table-s3/metadata/00868- b975b279-8eac-4cfd-96db-0bb85bbdc1ea.metadata.json') t CROSS JOIN UNNEST(t.snapshots) AS u(s) WHERE s."snapshot-id" = t."current-snapshot-id"; –ϚχϑΣετϦετϑΝΠϧಡΈࠐΈ D SELECT * FROM read_avro('s3://a33d7730-dc61-4b7d-1rerceqph8j3ubw9fj6snbyfxtqyhapn1b--table-s3/metadata/ snap-8236464468384760848-0-f0f21f1d-d821-4b42-b969-7cde2eef7d59.avro');
118. - ಛఆλΠϛϯάʹ͓͚Δςʔϒϧͷ εφοϓγϣοτͱ1:1Ͱඥͮ͘Ϧ ετ - ໊લͷ௨ΓϚχϑΣετϑΝΠϧ ʢޙड़ʣͷϦετ - ଞʹϚχϑΣετϑΝΠϧͷύεɺ ύʔςΟγϣϯͷൣғ͕Θ͔Δ - `where`۟ͳͲͰύʔςΟγϣϯ Λࢦఆ͢Ε͹ɺ͜ͷ஋Λ΋ͱʹෆ ཁͳσʔλΛಡΈࠐ·ͳ͍ - ·ͨɺલճͷεφοϓγϣοτͱ ͷࠩ෼͕Θ͔Δ ϚχϑΣετϦετ
119. - Ϧετ - Ϧετ - Ϧετ - Ϧετͷڧௐจࣈ - Ϧετ ϚχϑΣετϦετͷεΩʔϚ
120. ϚχϑΣετϦετ͔ΒϚχϑΣετϑΝΠϧ΁ https://iceberg.apache.org/spec/#overview - ֤ཁૉͷ `manifest_path` ͷ஋( `*-m0.avro`)ΛಡΈࠐΉ - ͱΓ͋͑ͣ࠷ॳͷཁૉΛಡΈࠐΜͰΈΑ͏ - duckdbͷ৔߹ - ΫΤϦ͸ɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡΈࠐΈ >ɹʮϚχϑΣετϦετ͔ΒϚχϑΣετϑΝΠϧ΁ʯɻ - Ͱ͖ͳ͚Ε͹ɺhttps://bit.ly/40Wiw1M > manifests-samle-m0.avroΛऔಘ
121. - ↑ϚχϑΣετϑΝΠϧ - ↓ϚχϑΣετϑΝΠϧͷ data_ fi leͷ஋ - σʔλ૚ͷσʔλϑΝΠϧͷα ϒηοτΛ௥੻͠ɺ֤ϑΝΠϧ ͷৄࡉ΍౷ܭ৘ใΛؚΉ - ͜͜Ͱ͍͏౷ܭ৘ใ͸ΧϥϜͷ ࠷େ஋ɾ࠷খ஋ɺnull஋ͷ༗ແɺ Ϩίʔυ਺ͳͲ - ΍͸Γ`where` ۟ͰΑΈ͜Ήϑ ΝΠϧΛ੍ݶͰ͖Δ ϚχϑΣετϑΝΠϧ
122. ϚχϑΣετϑΝΠϧͷεΩʔϚ
123. ϚχϑΣετϑΝΠϧ͔ΒσʔλϑΝΠϧ΁ https://iceberg.apache.org/spec/#overview - ϚχϑΣετϑΝΠϧ಺ͷ `data_ fi le`.` fi le_path`ͷ஋ΛಡΉ - data_ fi leΧϥϜࣗମ͸ϑΝΠϧͱ͍͏ΑΓɺ࣮ࡍͷσʔλͷύεͱͦͷϝλσ ʔλΛຒΊͨ΋ͷ - ࣮ࡍͷσʔλϑΝΠϧͷऔಘ͸࣍ͷΑ͏ʹ࣮ࢪʢDuckdb) - ΫΤϦ͸ɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ >ɹʮσʔλϑΝΠϧ΁ʯɻ - Ͱ͖ͳ͚Ε͹ɺhttps://bit.ly/40Wiw1M > data- fi le-sample.parquetΛऔಘ
124. σʔλϨΠϠʔ https://iceberg.apache.org/spec/#overview - ࣮ࡍͷσʔλ͕֨ೲ͞ΕΔ෺ཧ૚ - ෼ࢄϑΝΠϧγεςϜɺΦϒδΣΫτετϨʔδʹͳΓ͕ͪ - ࿦ཧతʹσʔλར༻ऀ͔Β͸Ӆṭ͞ΕͯΔͷͰҙࣝ͠ͳͯ͘ ྑ͍ - …͕S3Table͸σʔλ؅ཧऀͰ΋௚઀ΞΫηεͰ͖ͳ͍ͷͰ ϝϯςతʹ͸ࠔΔʢཪ࿩ʣ
125. - ͍ͭʹੜσʔλʹ౸ୡʂ - ࣮ࡍͷσʔλͷϑΝΠϧ - AWS؂ࠪϩά - ෳ਺ͷϑΝΠϧܗࣜΛαϙʔτ: parquet, avro, orc - ͳͷͰϚϧνϞʔμϧ - Parquet͔͠Έͨ͜ͱͳ͍͔΋ - εΩʔϚ͸ϩά͝ͱʹҟͳΔ σʔλϑΝΠϧ
126. ͓·͚: - ಛఆͷλΠϛϯάͷεφοϓγϣοτΛݟΔ(Time Travel) - Hidden Partitioning - ύʔςΟγϣϯΛઃఆ͍ͯ͠Δલఏ - ࠓճͷ৔߹ͩͱɺregion, accountId, time_dt_dayΛఆٛͯ͠ ͍Δɻtime_dt_day͸timestamp͔Β೔෇Λࣗಈܭࢉ͢Δύʔ ςΟγϣϯ஋ʹ͍ͯ͠Δɻ - σʔλར༻ऀ͸ͦΕΛҙࣝͤͣλΠϜελϯϓʢྫ: current_date() - Interval 3 day)ͱ͔͢Ε͹͍͍
127. σʔλ؅ཧऀͱͯ͠σʔλΛ ؅ཧʢॻ͖ࠐΈɺεΩʔϚߋ৽ʣ ͠ͳ͕Βɺ ΞʔΩςΫνϟΛֶΜͰΈΑ͏
128. ॻ͖ࠐΈͷ࢓૊Έ ਤ3-2-Apache Icebergʹ͓͚Δॻ͖ࠐΈॲཧ - جຊ͸Χλϩά͕ϙΠϯτ͢ΔϝλσʔλΛ ࢖ͬͯޮ཰త׌ͭ҆શʹσʔλΛॻ͖ࠐΉ - ಡΈࠐΈͱҧ͏ͷ͸ɺຖॲཧ͝ͱʹϝλσʔ λϑΝΠϧɺεφοϓγϣοτ౳ͷϑΝΠϧ ͕ੜ੒͞ΕΔ͜ͱ - ͞Βʹָ؍తฒߦੑʹΑͬͯΧλϩά͕ࣔ͢ ϝλσʔλϑΝΠϧͷϙΠϯλઌ͕࠷৽ͷ΋ ͷʹͳΔ - ͜ͷࡍʹɺॳΊͯڝ߹ʢίϯϑϦΫτʣν ΣοΫΛ͢ΔɻͷͰɺAtomicੑ͕୲อ͞Ε Δ
129. - https://bit.ly/40Wiw1M > ʮtable-create.zip Λղౚ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ > Apache Iceberg > ॻࠐΈ > ςʔϒϧ࡞੒ - ͦͷޙɺϝλσʔλΛಡΜͰΈΑ͏ ςʔϒϧ࡞੒
130. σʔλॻࠐΈ - https://bit.ly/40Wiw1M > ʮdata-ingest.zipΛ ղౚ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ॻࠐΈ > σʔλ஫ೖ - ͦͷޙɺϝλσʔλΛಡΜͰΈΑ͏ - snapshots, snapshot-logͳͲͷมԽʹண໨
131. - duckdb͸·ͩॻࠐΈૢ࡞Λαϙʔτͯ͠ͳ͍ͷͰɺ͜͜Ͱ͸αϯϓϧͷΈ - AthenaͰࡶʹ࡟আ - https://bit.ly/40Wiw1M > delete.metadata.json Λམͱ͠ಡΜͰΈΔ σʔλ࡟আ DELETE FROM "system_logs"."tojitsu" WHERE class_uid = 3002;
132. σʔλ࡟আ - ϝλσʔλͷsnapshotsͰҎԼΛ஫໨ - added-position-delete- fi les: 7 - added-delete- fi les: 7 - added-position-deletes: 36 - total-delete- fi les: 7 - total-position-deletes: 36
133. σʔλ࡟আ - manifest list - https://bit.ly/40Wiw1M > delete-snap.avro Λམͱ͠ಡΜͰΈΔ - ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete-snap.avro'); - Manifest listͷσʔλ࢓༷͔Β `content` ͸ `The type of fi les tracked by the manifest, either data or delete fi les`Λࣔ͢ɻ - `int` with meaning: `0: data`, `1: deletes` - ௥Ճ͞Εͨdelete fi les ͕̓ͭͱ͍͏͜ͱ - Metadata ͱໃ६ͳ͠ https://iceberg.apache.org/spec/#manifest-lists
134. σʔλ࡟আ - manifest fi le - https://bit.ly/40Wiw1M > delete-m0.avro Λམͱ͠ಡΜͰΈΔ - ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete-m0.avro'); - `content` ͸ `Type of content stored by the data fi le: data, equality deletes, or position deletes`Λࢦ͠ɺ`1` ͷ৔߹͕ `position delete` ʹ֘౰͢Δ - `status` ͷ 1͸ added Λҙຯ͢Δ͕ɺdelete fi le ͱͯ͠௥Ճ͞Εͨɺͱ͍ ͏͜ͱ - Manifest-listͷηΫγϣϯͰॻ͍ͨ௨ΓɺίϨ͸૝ఆͲ͓Γɻ
135. σʔλ࡟আ - delete fi le - https://bit.ly/40Wiw1M > delete- fi le.parquetΛऔಘ - ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete- fi le.parquet'); - ࣮ࡍͷεΩʔϚΛΈΔͱɺී௨ͷcloudtrailϩάͱҟͳΔ delete fi leͷεΩʔϚʢ্ਤʣ - fi le_pathʹ͸ʢ࿦ཧ࡟আͨ͠ʣσʔλ - posʹ͸ɹ࡟আͨ͠ΠϯσοΫε
136. σʔλ࡟আ - ࣮ࡍͷσʔλ - https://bit.ly/40Wiw1M > deleted-logically-but-physical-data- fi le.parquetɹΛऔಘ - ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./deleted-logically-but-physical-data- fi le.parquet'); - ࣮ࡍͷϑΝΠϧʢӈਤʣ - ΫΤϦͰ࡟আͨ͠஋͕ొ৔͢Δidx͕ɺdelete fi leʢԼਤʣͷposʹ߹க͍ͯ͠Δ - ↓࡟আΫΤϦ DELETE FROM "system_logs"."ex04_with_partition" WHERE class_uid = 3002;
137. σʔλ࡟আ - ࣮ࡍͷσʔλ - ࣮ࡍͷϑΝΠϧʢӈਤʣ - ΫΤϦͰ࡟আͨ͠஋͕ొ৔͢Δidx͕ɺdelete fi le ʢԼਤʣͷposʹ߹க͍ͯ͠Δ - ↓࡟আΫΤϦ DELETE FROM "system_logs"."ex04_with_partition" WHERE class_uid = 3002; ϝλσʔλϨΠϠʔ͕࡟আΛ࿦ཧతʹ؅ཧͨ͜͠ͱ͕Θ͔Δ
138. - ͜͜Ͱ͸αϯϓϧͷΈ - AthenaͰࡶʹcommonΧϥϜ௥Ճɻ - ϝλσʔλϑΝΠϧʹεΩʔϚ (id =1ʣ͕௥Ճ ͞Ε͍ͯΔ - εφοϓγϣοτࣗମ͸૿͑ͯແ͍ - εΩʔϚͷid = 0 εΩʔϚߋ৽ ALTER TABLE "system_logs"."ex04_with_partition" ADD COLUMNS (comment string)
139. - AthenaͰࡶʹcommonΧϥϜ௥Ճɻ - ϝλσʔλϑΝΠϧʹεΩʔϚ (id =1ʣ͕ ௥Ճ͞Ε͍ͯΔ - εφοϓγϣοτࣗମ͸૿͑ͯແ͍ - εΩʔϚͷid = 0 εΩʔϚߋ৽ ALTER TABLE "system_logs"."ex04_with_partition" ADD COLUMNS (comment string) ϝλσʔλϨΠϠʔ͕εΩʔϚߋ৽Λ࿦ཧతʹ؅ཧͨ͜͠ͱ͕ Θ͔ΔʢSchema Evolutionʣ
140. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
141. ٳܜ
142. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
143. γεςϜϦεΫΞηεϝϯτ
144. ຊߨٛͷείʔϓ σʔλ؅ཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓ৘ใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲໨Ͱ͸ͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance
145. - σʔλΛ؅ཧɾ׆༻͢Δ໨త͸༷ʑͰ͋Δ - ͔͠͠ɺ໨తୡ੒ͱͦͷӡ༻ʹ͸ਖ਼͍͠ܕͱखஈ͕ඞཁʹͳΔ - ਖ਼͘͠ͳ͍ܕɾखஈ͸ɺϚΠφεޮՌ㲈 **ϦεΫ** ʹͳΓ͏Δ - ໨తୡ੒·ͰͷܭըʹϦεΫΛ৫ΓࠐΈɺରࡦ͠ɺৼΓฦ͑Γɺ࠶ݱੑΛ ࣋ͨͤͳ͚Ε͹ͳΒͳ͍ ֓ཁ - େ্ஈͷ໨త = ϦεΫ؅ཧ
146. - ର৅γεςϜ͕୲͏ۀ຿΍औΓѻ͏σʔλɺͦͯ͠৘ใγεςϜͷಛੑʹ Ԡͯ͡ҟͳΔϦεΫ͕͋Δ - อ༗͢ΔσʔλΛอޢ͢ΔͨΊʹ͸ɺજࡏతͳϦεΫΛಛఆ͠ɺͦͷϦε Ϋͷେ͖͞ΛධՁ͢Δ - ϦεΫͷ༏ઌॱҐΛ໌֬ʹ͠ɺద੾ͳରࡦΛߨ͡Δ͜ͱͰɺࣄۀܧଓੑΛ ߴΊΔ ֓ཁ - γεςϜϦεΫ؅ཧͷ໨త
147. γεςϜϦεΫ؅ཧͷαΠΫϧ https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-30r1.pdf
148. - Өڹ౓ͷ೺Ѳ - ੬ऑੑxڴҖͷ೺Ѳ - ରԠͷܾఆ ֓ཁ - ؅ཧͷྲྀΕ
149. - Өڹ౓ͷ೺Ѳ - ϦιʔεʢΞηοτʣͷચ͍ग़͠ͱධՁ - σʔλʢ৘ใࢿ࢈ʣ - ৘ใγεςϜ - ϦεΫʢ੬ऑੑxڴҖʣͷ೺Ѳʢ෼ੳʣ - ରԠͷ೺Ѳ ֓ཁ - ྲྀΕ
150. - Өڹ౓ͷ೺Ѳ - ϦεΫʢ੬ऑੑxڴҖʣͷ೺Ѳ - ҆શରࡦɾඪ४ͷચ͍ग़͠ɾબ୒ - ڴҖͷચ͍ग़͠ɾબ୒ - ϦεΫ෼ੳ - ରԠͷ೺Ѳ ֓ཁ - ྲྀΕ
151. - Өڹ౓ͷ೺Ѳ - ϦεΫʢ੬ऑੑxڴҖʣͷ೺Ѳʢ෼ੳʣ - ରԠͷ೺Ѳ - ϦεΫධՁ - ҆શରࡦɾඪ४ͷܾఆ ֓ཁ - ྲྀΕ
152. Өڹ౓ͷ೺Ѳ
153. - σʔλʢ৘ใࢿ࢈ʣ - ৘ใγεςϜ Өڹ౓ - ચ͍ग़͠ର৅
154. - σʔλʢ৘ใࢿ࢈ʣ - ৘ใγεςϜ Өڹ౓ - ચ͍ग़͠ର৅
155. - ӦۀൿີͳͲࣄۀʹඞཁͰ૊৫ʹͱͬͯՁ஋ͷ͋Δ৘ใ΍ɺސ٬΍ैۀһ ͷݸਓ৘ใͳͲ؅ཧ ੹೚Λ൐͏σʔλ - ߨٛͷจ຺Ͱ͸σʔλج൫Ͱอ؅ɾม׵ɾ෼ੳ͍ͨ͠ର৅ - ࣾ಺ͷϝλσʔλ౳΋ର৅ ৘ใࢿ࢈ - ৘ใࢿ࢈ͱ͸
156. - ۀ຿ͷதͰऔѻɾॲཧɾอ؅͢Δ৘ใ͔Βચ͍ग़͢ - ۀքελϯμʔυ͔Βચ͍ग़͢ - ϑϩʔσʔλ΍ετοΫσʔλ͔Βચ͍ग़͢ - ϑϩʔσʔλ: νϟοτɺձٞɺٞࣄ࿥ - ετοΫσʔλ: ϑΝΠϧαʔόʔ - Ϟμϯʁͳ΍Γํ - ϓϩμΫτͷDBεΩʔϜ͔Βચ͍ग़͢ - AIͰྨਪͤ͞Δɻศར - AIͷਖ਼֬ੑ͸͖ʹͳΔ͕ɺͦ΋ͦ΋ຊۀͰͳ͍ਓؒͷਫ਼౓΋ظ଴Ͱ͖ͳ͍ͷͰؾʹ͠ͳͯ͘Α͍ ৘ใࢿ࢈ͷચ͍ग़͠ํ
157. ৘ใࢿ࢈ - ચ͍ग़͠ํ DFDฤ - ॏཁͳ৘ใʯΑΓʮॏཁͳۀ຿ʯͷ΄͏͕ɺKPIɾOKRʹͻ΋෇͚΍͍͢ - ʮϩΠϠϧΧελϚʔͷ໰͍߹ΘͤͬͯͲ͏ॲཧͯ͠ΔΜͰ͔͢ʁʯ - ʮࠓظͷച্ʹࢸΔ·ͰͷӦۀύΠϓϥΠϯͬͯͲΜͳײ͡Ͱ͔͢ʁ
158. - Apache IcebergͰ͋Ε͹ Catalog - DBͰ͋Ε͹show tables; - mysqldump -u username -p --no-data --databases prd > schema.sql ৘ใࢿ࢈ - ચ͍ग़͠ํDBฤ
159. ৘ใࢿ࢈ - ચ͍ग़͠ํAIฤ - ϑϩʔ৘ใ΍ετοΫ৘ใΛ৯͏AIʹ ·ͱΊͤ͞Δ - NotionAIΛྫʹͩͯ͠Δ͕ɺChatGPT / Gemini ͳΜͰ΋͍͍
160. - ҆શੑ͕ଛͳΘΕͨ৔߹ͷӨڹ౓߹͍ - 3͔࣠Βߟ͑Δͷ͕ϝδϟʔ - 1. ػີੑ: ΞΫηεΛڐՄ͞Εͨऀ͚͕ͩ৘ใʹΞΫηεͰ͖Δ - 2. ׬શੑ: ਖ਼֬ੑٴͼอશੑɻ࠷ۙ͸ਅਖ਼ੑ΋ؚ·ΕΔؾ͕͢Δ - 3. Մ༻ੑ: ਖ਼౰ͳར༻ൣғͰܧଓͯ͠ΞΫηεͰ͖Δঢ়ଶ - ݸਓ৘ใ͔ - ಛఆݸਓ৘ใ͔ɺཁ഑ྀݸਓ৘ใ͔ɺػඍ৘ใ͔ ৘ใࢿ࢈ - Өڹ౓ධՁ
161. ৘ใࢿ࢈؅ཧ୆ாྫ https://www.ipa.go.jp/security/sme/f55m8k000000587z-att/outline_guidance_risk.pdf
162. - ৘ใࢿ࢈ - ৘ใγεςϜ Өڹ౓ - ચ͍ग़͠ର৅
163. - ؅ཧ͍ͯ͠ΔσʔλΛ࣮ࡍʹॲཧɾอ؅͢Δίϯϙʔωϯτ - ॲཧ: औѻɺม׵౳ - ߨٛͷจ຺Ͱ͸Iceberg TableɺΫΤϦΤϯδϯɺETL౳ΛؚΉ - ηΩϡϦςΟରࡦΛ࣮ࡍʹࢪ͢Օॴ - ʢݸਓతʹ͸ʣ৘ใࢿ࢈؅ཧΑΓ༏ઌ͢΂͖ͩͱࢥ͏ - ͳͥͳΒɺ৘ใ͸୯ͳΔ֓೦Ͱ͋ΓɺͦΕࣗମ͕Ձ஋Ͱ͸ͳ͍͔Β - ΠϯαΠτ͋Δ͍͸ΞΫγϣϯΛى͜͢ͷ͸৘ใγεςϜͱͦͷ্Ͱࣥߦ͞ΕΔۀ຿͔ͩΒ ৘ใγεςϜ
164. ৘ใγεςϜચ͍ग़͠ํ Beyond Corp -> https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/44860.pdf
165. - Ͳͷ৘ใγεςϜ͕Ͳͷ৘ใࢿ࢈ΛऔΓѻͬͯΔ͔Ϛοϐϯά - ۀ຿ςʔϒϧ΍DFDʢσʔλϑϩʔਤʣ͕͋ΔͱḿΔ ৘ใγεςϜͱ৘ใࢿ࢈ͷ͔͚͋Θͤ
166. ϦεΫ෼ੳ
167. - ओʹ̎ͭͷΞϓϩʔν͋Γ - ίϯϓϥΠΞϯεϕʔε vs ϦεΫϕʔε - ຊߨٛͷ࠲ֶ͸΍΍ίϯϓϥΠΞϯεΑΓ - ϫʔΫγϣοϓ͸ϦεΫϕʔεΑΓ - ੬ऑੑͷಛఆ - ڴҖͷಛఆ - ϦεΫʢൃੜ֬཰ʣΛ෼ੳ ϦεΫ෼ੳ
168. - “ ιϑτ΢ΣΞ͓Αͼϋʔυ΢ΣΞͷίϯϙʔωϯτʹଘࡏ͢Δɺܭࢉϩ δοΫʢྫ: ίʔυʣ্ͷऑ఺Ͱ͋Γɺѱ༻͞Εͨ৔߹ʹػີੑɺ׬શੑɺ ·ͨ͸Մ༻ੑʹѱӨڹΛٴ΅͢΋ͷ” - ҰํɺγεςϜϦεΫΞηεϝϯτʹ͓͚Δ੬ऑੑ͸ඞͣ͠΋ʮܭࢉϩ δοΫʯʹݶΒͳ͍ - ͦͷҙຯͰ͸NCSCͷVulnerability managementͷ͕ΑΓจ຺ʹ߹ͬͯΔ - NCSC: ӳࠃͷαΠόʔηΩϡϦςΟ౰ہ ੬ऑੑͱ͸ https://www.cve.org/Resources/Roles/Cnas/CNA_Rules_v3.0.pdf
169. - “੬ऑੑͱ͸ɺITγεςϜʹଘࡏ͢Δऑ఺Ͱ͋Γɺ߈ܸऀ͕ ߈ܸΛ੒ޭͤ͞ΔͨΊʹѱ༻͢ΔՄೳੑͷ͋Δ΋ͷͰ͢ɻ ੬ऑੑ͸ɺγεςϜͷܽؕɺҙਤ͞Εͨػೳɺ·ͨ͸Ϣʔ βʔͷաࣦʹΑͬͯൃੜ͠ɺ߈ܸऀ͸࠷ऴతͳ໨తΛୡ੒ ͢ΔͨΊʹɺ͜ΕΒͷ͏ͪҰͭ·ͨ͸ෳ਺Λ૊Έ߹Θͤͯ ѱ༻ΛࢼΈ·͢ɻ” ੬ऑੑͱ͸ by NCSC https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities
170. - “੬ऑੑͱ͸ɺITγεςϜʹଘࡏ͢Δऑ఺Ͱ͋Γɺ߈ܸऀ͕ ߈ܸΛ੒ޭͤ͞ΔͨΊʹѱ༻͢ΔՄೳੑͷ͋Δ΋ͷͰ͢ɻ ੬ऑੑ͸ɺγεςϜͷܽؕɺҙਤ͞Εͳ͍ػೳɺ·ͨ͸ ϢʔβʔͷաࣦʹΑͬͯൃੜ͠ɺ߈ܸऀ͸࠷ऴతͳ໨తΛ ୡ੒͢ΔͨΊʹɺ͜ΕΒͷ͏ͪҰͭ·ͨ͸ෳ਺Λ૊Έ߹Θ ͤͯѱ༻ΛࢼΈ·͢ɻ” ੬ऑੑͱ͸ by NCSC https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities ຊߨٛͰ͸ɺʮҙਤ͞Εͨػೳʯ΋ʮϢʔβʔͷաࣦʯ΋جຊɺ ࣄલ૝ఆ͞Εͳ͔ͬͨʮܽؕʯͰ͋Δલఏͱͯ͠ѻ͏ɻͦ͠ ͯɺʮܽؕʯͰ͋Δͱ͍͏͜ͱ͸ TobeͱAsIsͷDiffͰ͋Δ
171. - େ͖͘෼͚ͯɺίϯϓϥΠΞϯεϕʔε vs ϦεΫ ϕʔε - ຊষʢ੬ऑੑͷষʣ͸جຊతʹલऀΛ૝ఆ - ϦεΫϕʔε͸ڴҖΛߟ͑ͳ͕ΒɺӨڹ౓ͷߴ͍ Օॴʹूத͢Δͷ͕٢ ੬ऑੑͷચ͍ग़͠ํ https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities
172. - ʮܽؕʯͰ͋Δͱ͍͏͜ͱ͸ TobeͱAsIsͷDiffͰ͋ Δ - Q: ToBeͱ͸ʁ - A: ඪ४ɻϕεϓϥ - ʮܽؕʯͰ͋Δͱ͍͏͜ͱ͸ ཧ૝ͱݱঢ়ͷDiffͰ͋Δ ੬ऑੑͷચ͍ग़͠ํ - ToBeͷચ͍ग़͠ https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities
173. - Ұൠత: CIS Benchmarks - Ϋϥ΢υಛԽ: CSAΨΠμϯε - ϕϯμʔಛԽ: AWS Security Baseline - ۀքಛԽ: FISC ۚ༥ػؔ౳ίϯϐϡʔλγεςϜͷ҆શରࡦج४ - ߦ੓: ۚ༥෼໺ʹ͓͚ΔαΠόʔηΩϡϦςΟʹؔ͢ΔΨΠυϥΠϯ - ଞ: ࣗࣾάϧʔϓ༻ ੬ऑੑͷચ͍ग़͠ํ - ඪ४ྫ
174. - ʮԶͷߟ͑ͨ࠷ڧͷඪ४ʯ΋͋Γɹ - ͔͠͠ɺ਎ݩΛ֬ೝ͞Εͨෳ਺ਓͷઐ໳Ո͕ɺϨϏϡʔ Λड͚ͳ͕Βɺ૊৫಺ͰੵΈ্͖͛ͯͨ΋ͷͱൺֱ͢Δ ͱɺ٬؍తͳ৴པੑ͸ݶఆత - Ұํɺඪ४͸͖Γ͕ͳ͍ͷͰɺखஈͷ໨తԽʹཹҙ ੬ऑੑͷચ͍ग़͠ํ - ඪ४ͷ᠘
175. - ඪ४ͱͷΪϟοϓܭଌ͢ΔͨΊʹݱঢ়Λ೺Ѳ͢Δඞཁ͕͋Δɻ - ͔͠͠ɺݱ୅ͷγεςϜ͸૬౰ෳࡶͳͷͰࣗಈԽલఏͰͳ͍ͱݫ ͘͠ɺऩूɾอ؅ɾ੔ܗɾ෼ੳͳͲ΋ࣗಈԽલఏͰͳ͍ͱݫ͍͠ - ͜͜਺೥͸ґଘ෺΋ؚΊͨ੬ऑੑ؅ཧΛٻΊΒಘΒΕΔέʔε͋ Γ - αϓϥΠνΣʔϯͷ࿩ɺSBOMͷීٴ ੬ऑੑͷચ͍ग़͠ํ - ݱঢ়
176. ੬ऑੑͷચ͍ग़͠ํ - ྫ: AWS ୯७ʹ̍̑ˋ΄Ͳࣦഊ͍ͯ͠Δ९क߲໨͕͋Δͱ͍͏ҙຯ ↑͸ཧ૝ঢ়گͱͷΪϟοϓͳͷͰ੬ऑੑͱ͍͑Δ
177. ੬ऑੑͷચ͍ग़͠ํ - ྫ: AWS ͜͜Ͱ͸֤Ϧιʔεͷ͋Δ΂͖͕࢟ʮTitleʯʹهࡌ͞Ε͍ͯ Δɻͦͷ৚݅ΛΈ͍ͨͯ͠ͳ͍ͷͰControl Status͸౰વ ʮFailedʯʹͳΔɻ͜Ε͕੬ऑੑͰ͋Δɻ
178. ੬ऑੑͷચ͍ग़͠ํ - ྫ: AWS ಉ༷
179. - “γεςϜ΍૊৫ʹର͠ɺ֐Λ༩͑Δ๬·ͳ͍ΠϯγσϯτΛൃੜ͢Δ જࡏతݪҼ” - `ڴҖΠϯςϦδΣϯε`ɹ΍ `TLPTʢڴҖϕʔεϖωτϨʣ` ͳͲ࢖Θ ΕΔίϯςΫετ͕ଟذʹΘͨΓ࢝ΊͨͨΊɺ͔ͳΓ࢖͏৔໘ʹΑͬ ͯΏΒ͕͗͋Δʢࢲݟʣ - ͳͷͰɺ͜Ε͕ඪ४ʂͱ͍͏ͷ͸͋·Γͳ͍ʢࢲݟʣ - ͳͷͰɺຊίʔεͰ͸ͦ͜·Ͱਂ΅Βͳ͍ ڴҖͱ͸ https://www.iso.org/obp/ui/#iso:std:iso-iec:27000:ed-5:v1:en
180. ϋϯζΦϯ
181. ϫʔΫγϣοϓ - ηΩϡϦςΟ؂ࢹͱɺͦͷσʔλγεςϜʹ͓͚Δ੬ऑੑͱڴҖΛߟ͑ͯΈΑ͏ - ৘ใࢿ࢈ͷચ͍ग़͠ - ৘ใγεςϜͷચ͍ग़͠ - ੬ऑੑͷચ͍ग़͠ ※ඪ४͸Ұ୴ߟ͑ͳ͓̺ͯ͘Ͱ͢ - ڴҖͷચ͍ग़͠ - ͬ͘͟Γ̍ݸͣͭͰ͍͍͔ͳ… - ϫʔΫγϣοϓ: 15min - ൃද: ͳ͠
182. ϦεΫධՁ - ϦεΫ෼ੳ: ϦεΫ = ੬ऑੑ x ڴҖ - Өڹ౓ͱϦεΫʢՄೳੑʣ͔ΒϦε ΫΛӈਤΛ࢖ͬͯධՁ - ྫ: Өڹ౓͕ߴ͘ɺϦεΫ͕௿͍৔ ߹ɺอݥ౳ͷϦεΫҠస͕ߟ͑Β ΕΔ ڴҖΠϯςϦδΣϯεͷڭՊॻ p.375
183. - Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλ؅ཧ: 40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλ΢ΣΞϋ΢ε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 50min (11:30~12:20) - ·ͱΊ: 5min (12:20~12:25) λΠϜςʔϒϧ
184. ·ͱΊ
185. 1. AIʹ΋ඞཁͳσʔλΛ؅ཧ͢Δͱ͸Ͳ͏͍͏ঢ়ଶ͔ཧղ 2. ͦͷதͰApache IcebergΛ࢖ͬͨγεςϜͷಛੑͷཧղ 3. ηΩϡϦςΟͷઓུతͳରԠͷཧղ ࠓ೔ͷΰʔϧ
186. ηΩϡϦςΟઐ໳Ոͱͯ͠དྷΔAI࣌୅ ΍Δ͜ͱ͕໌֬ʹͳΓ·͔ͨ͠ʁ
187. ͳΔΘ͚͕ͳ͍ɻઓ͍͸࢝·ͬͨ͹͔Γ
188. - ඪ४తͳΧϦΩϡϥϜͳͲଘࡏ͠ͳ͍ - ಡΈ෺ΛࢭΊͯɺ࣮ફ͋ΔͷΈ - ίʔυΛॻ͚ - ίʔυΛյͤ - ஌ࣝΛڞ༗ͤΑ - ίϛϡχέʔγϣϯΛຏ͚ - ϋʔυϫʔΫͯ͠ɺͨ·ʹࣦഊ͠Α͏ - ʢͰ͖Δ͚ͩʣָ؍తͰ͍Α͏ - ॿ͚ΛٻΊΑ͏ So, you want to work in security? https://www.freecodecamp.org/news/so-you-want-to-work-in-security-bc6c10157d23/ ”Parisa Tabriz ͸ΞϝϦΧਓͷΤϯδχΞɺίϯϐϡʔλ ηΩϡϦςΟͷઐ໳ՈͰ͋Γɺ Google Ͱ Google Chrome ͷ෭ࣾ௕݉θωϥϧ ϚωʔδϟʔΛ຿ΊΔװ෦Ͱ͢ɻ൴ঁ͸ઐ ໳తʹ͸ʮηΩϡϦςΟϓϦϯηεʯ”by https://en.wikipedia.org/wiki/Parisa_Tabriz
189. Good luck and happy hacking!
190. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ DM (X): @ken5scal