Share
Embed
AI時代の大規模データ活用とセキュリティ戦略
データ管理とApacheIcebergについてセキュリティ監視の文脈から、ハンズオンしながら話すスライド。
近年、AIの発展とともに扱われるデータ量は飛躍的に増大しています。膨大なデータを効率的に活用するには、ストレージやOpen Table Format(データの構造を扱いやすくするための形式)、DataOps(データの開発・運用プロセスを管理する手法)などの技術を正しく理解し、運用する必要があります。これらの仕組みを把握しておくことで、高精度なAIモデル開発だけでなく、ビジネスのあらゆる場面でデータ利活用をスムーズに進められるようになります。
また、AIが業務全般に浸透し始めた今だからこそ、サイバーセキュリティ対策の重要性が増しています。本講義では、大規模データとAIを活用する際の注意点を体系的に学ぶとともに、組織的なセキュリティマネジメントにおいて留意すべきポイントを解説します。また、AI時代に求められる新人像やキャリア観にも触れ、これからのエンジニアが持つべき知識・姿勢をディスカッションしながら身につけていきます。
Kengo Suzuki
August 13, 2025
Transcript
189 slides
-
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- ॴଐ - LayerX Fintechࣄۀ෦ - ࡾҪ࢈σδλϧɾΞηοτϚωδϝϯτ CISOʢˢ͔Βग़ʣ - Dev
Ops, Corp Eng, ηΩϡϦςΟΛ·Δͬͱ - Φϧλφ৴ୗɹγεςϜϦεΫ౷ׅʢˢͷࢠձࣾʣ - σδλϧிɹηΩϡϦςΟϢχοτ - ݸਓ׆ಈ - िؒχϡʔεϨʔλʔɺPodCastɺಉਓࢽ - ηΩϡϦςΟຊͷ༁ - ηΩϡΩϟϯߨࢣʢBίʔεʣ: 2019, 2023 ɹ※ຊߨٛʹશৼΓͷͨΊɺ্هͯ͢Ұ࣌ఀࢭத… @ken5scal ʢླݚޗʣ χϡʔεϨλʔ: https://ken5scal.notion.site/54bda4932da14add9e9911ab3e9a6e5c podcast: https://open.spotify.com/show/73sFeKzUIkSYfCZWVBNO70 -
- 2011~2014: NRIηΩϡΞςΫϊϩδʔζ (NRI͔Βͷग़ʣ - 2014~2018: Money Forward - 2018~2020:
FOLIO - 2020~: ݱ৬ (ॴଐઌx4ʣ - ʢ݁ՌతʹʣελʔτΞοϓ+ ۚ༥ܥͬ ͔͠ɻ @ken5scal - དྷྺ χϡʔεϨλʔ: https://ken5scal.notion.site/54bda4932da14add9e9911ab3e9a6e5c podcast: https://open.spotify.com/show/73sFeKzUIkSYfCZWVBNO70 -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
ଞͷߨٛ ߨ໊ٛ ϥϕϧ ༰ ʢݸਓͷղऍʣ ࣮ફLLMΞϓϦέʔγϣϯɾηΩϡϦςΟ D1/D6 શମతͳ߈ܸɾޚ؍ ΫϥυιʔγϯάʹΑΔֶशσʔλ࡞ೖ D2
σʔλ࡞ ཧAIηΩϡϦςΟ D4 ը૾ೝࣝΛத৺ͱͨ͠ ཧαʔϏεಛԽηΩϡϦςΟ AIΤʔδΣϯτͰ࡞ΔηΩϡϦςΟࢧԉπʔϧ D5 γεςϜ։ൃ -
ຊߨٛͷείʔϓ σʔλཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว
Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲Ͱͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance -
- “࠲ֶͱϋϯζΦϯܗࣜͰ෯͘औΓѻ͍·͢” - ਐΊํ: ৽نੑͷ͋Δཁૉͱීวੑͷ͋ΔཁૉΛ࠲ֶͱϋϯζΦϯΛ௨ֶͯ͠श - ৽نੑ: Apache IcebergΛͬͨσʔλཧ -
ීวੑ: σʔλཧͷγεςϜʹର͢Δอޢ - σʔλར༻ͷϢʔεέʔε: ηΩϡϦςΟࢹ - ͏σʔλ: AWSͷࠪϩά - https://bit.ly/40Wiw1M <- ͪ͜ΒͷϦϯΫ͔ΒదٓϑΝΠϧDLΛ͓ئ͍͢ΔͷͰɺURLΛ ͓ؾʹೖΓʹ͍ΕͨΓ͢ΔͳͲ͓ئ͍͠·͢ - Discordνϟωϧͷઆ໌ཝʹ͋Γ·͢ɻ ຊߨٛΛͲ͏ਐΊΔ͔ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- γνϡΤʔγϣϯ: - ࢿՈͷࢿۚΛ༬͔Δূ݊αʔϏε - ҙਤ͠ͳ͍ग़ۚɾചങ͕͋ΔͱΫϦςΟΧϧ - Ұํɺਖ਼ͳۀͱͯ͠ޱ࠲͔Βग़ۚ͢Δ͜ͱ͕͋Δɻ͜ͷ߹ࣄલʹٞͰঝೝɾܾࡋΛಘΔ - ఆ͢ΔϦεΫɺͦ͏͍ͬͨਖ਼ͳۀΛߦ͏ࣾһͷۜߦΞΧϯτ͕ෆਖ਼ར༻͞Εͨɹ͘͠෦ෆਖ਼ΛߦΘΕͨ
- ଘࡏ͢Δϩά - ۜߦAͷग़ۚཤྺϩά - ۜߦBͷೖग़ۚཤྺϩά - ग़ۚͷঝೝɾܾࡋΛͱΔϫʔΫϑϩʔϩά - ※ঝೝɾܾࡋ = ग़ۚͱݶΓ·ͤΜ - ͜ΕΛ݄࣍Ͱʢࠪʣ͢ΔϩάੳͷϓϩηεΛߟ͍͑ͯͩ͘͞ - ※ͲͷϩάΛͲͷΑ͏ʹಥ߹͠ɺूܭͯ͠ɺԿΛ࣋ͬͯͳ͍ͱ͢Δ͔ɺɺɺͱ͍͏ϓϥϯΛͨͯΔ - ϋϯζΦϯ: 10min - ൃද: 7min? ϋϯζΦϯ01: ηΩϡϦςΟੳΛͬͯΈΑ͏ -
- ҎԼͷཧ༝Ͱ͍ʹ͍͘ - จࣈίʔυ - ΧϥϜ໊ͷݴޠ - ֤γεςϜͰҟͳΔεΩʔϚ - ࣌ؒϑΥʔϚοτ
- ݸਓใͷࠞೖ ϋϯζΦϯ: Ͳ͏͍͏લఏ͔ͩͬͨ -
- σʔλͱใࢿ࢈Λܭըɺཧɺ৴͢ΔϏδωεػೳ by DMBOK - σʔλཧͰɺҙࢥܾఆͷͨΊʹσʔλΛऩूͯ͠༻͢ΔͨΊͷϙϦ γʔͱखॱΛ੍ఆ by Tableau -
ʮσʔλͱใࢿ࢈Λऩूɾอɾར༻Ͱ͖ΔΑ͏ʹఏڙ͢Δػೳ ʯ σʔλཧʢData ManagementʣͱԿ͔ʁ DAMA DMBOK -
- ੜσʔλଟछଟ༷ - ѹॖํࣜ( `json.gz` , `zip`, ·ͨѹॖͳ͠) - ܗࣜ(
`jpg`, `csv`, `json`, `txt`, `mp3`, `md`) - εΩʔϚɺΧϥϜ໊ - (ྫ: גࣜձࣾͱ(ג)) - ߋ৽͞ΕΔεΩʔϚ - ੜσʔλɺͦͷ··Ͱ͑ͳ͍ʮૉࡐʯͷࢁ σʔλͦͷ··͑Δ͔ -
- ใΛར༻ऀͷϢʔεέʔεʹ͋Θͤͯ࠷దԽ͍ͯ͠Δ͜ͱ - ࠷దԽʮσʔλΛඋ͠ɺՃ͢Δʯ͜ͱ - తʹԊͬͨσʔλͷఏڙ - దͳൣғͷσʔλͷఏڙ - దͳܗࣜͷσʔλͷఏڙ
- ൃݟɾཧղɾॲཧͰ͖ΔڥΛ͑Δ͜ͱ - ʮσʔλͱใࢿ࢈Λऩूɾอɾར༻Ͱ͖ΔΑ͏ʹఏڙ͢Δػೳ ʯ - ར༻ऀ͕AIͰਓؒͰมΘΒͳ͍ - ͔͠͠ɺAI࣌͜ͷϓϩηεͷ࣭ͱεϐʔυ͕ɺڝ૪ྗΛࠨӈ͢Δ σʔλΛཧɾར༻Ͱ͖Δɺͱ -
σʔλཧͷཁૉ01 ཁૉ໊ આ໌ Data preparation ੜͷσʔλΛΫϦʔϯΞοϓ͠ɺม͢Δϓϩηε Data pipelines σʔλΛγεςϜ͔Βผͷஈ֊ͷγεςϜྲྀ͢Ұ࿈ͷγεςϜίϯϙʔωϯτ Data
ETL σʔλΛσʔλΣΞϋεʹϩʔυ͢ΔͨΊɺநग़ɾม͢ΔϓϩηεɻData preparationʹΘΕΔ͠ɺύΠϓϥΠϯ্ʹࡌΔ -
σʔλཧͷཁૉ02 ཁૉ໊ આ໌ Data catalog σʔλͷશମ૾ͷѲ͠ɺతͷσʔλΛൃݟ͘͢͢͠ΔΑ͏ɺϝλσʔλΛཧ͢Δ ίϯϙʔωϯτ Data warehouses ར༻͢ΔσʔλιʔεΛ౷߹͍ͯ͠Δίϯϙʔωϯτ
Data governance σʔληΩϡϦςΟͱίϯϓϥΠΞϯεΛҡ࣋͢ΔͨΊͷϙϦγʔͱखॱɺͦͯͦ͠ΕΛ ཧ͢Δଶ Data architecture σʔλϑϩʔΛཧ͢ΔεΩʔϚ Data security σʔλΛอޢ͢ΔͨΊͷηΩϡϦςΟ -
σʔλཧͷཁૉʢ࠶ܝʣ σʔλཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว
Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲Ͱͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance -
- ࢹڥͱσʔλΛܨ͙طଘͷίωΫλʹ੍ݶ͕͋Δɻ - ੍ݶ1: ίωΫλ͕ଘࡏ͠ͳ͍ - ࣗࣾͰ͏શͯͷαʔϏεʹରԠ͢ΔίωΫλ͕͋Δͱ ݶΒͳ͍ - ಛʹɺ֤ۜߦͷೖग़ۚ໌ࡉͷΑ͏ͳɺຊࠃͷಛఆۀ
ʹಛԽͨ͠ίωΫλ·ͣఏڙ͞Ε͍ͯͳ͍ - ੍ݶ2: ࣗ࡞ͷίεύ͕ྑ͍ͱݴ͑ͳ͍ - ࣗ࡞ίωΫλͷ։ൃɾอकίετɺσʔλΛࣗલͷετ Ϩʔδʹอ͢ΔΈΛΉͷͱಉఔ ίωΫλͷน ಛʹࣗࣾͷϦεΫ ϕʔεͰࢹͨ͠ ͍߲Ͱݦஶ ࡞ۀ -
- શͯͷσʔλ͕ɺ៉ྷͳAPI͔ΒऔಘͰ͖ΔΘ͚Ͱͳ͍ɻݱͰɺటष͍ํ๏ Ͱ͔͠खʹೖΒͳ͍σʔλଟ͍ɻ - API͕ͳ͍߹ͷऔಘํ๏ - ࢹ͞ΕΔଆͷγεςϜ͕Webhookࢦఆʢྫ: GitHub Enterprise -
Webը໘͔Βͷखಈμϯϩʔυ (ྫ: ۜߦͷCSV) - WebαΠτΛ८ճͯ͠ใΛूΊΔ (ΫϩʔϦϯά) - ࠷ѱͷ߹ɺը໘ͷεΫϦʔϯγϣοτ औಘํ๏ͷଟ༷ੑɾࠔ͞ - ᶃ -
- ϑΟϧλϦϯάͷ - ͜͏ͨ͠ੜσʔλʹɺੳʹෆཁͳ ใɺѻ͏ͷ͕ةݥͳݸਓใؚ͕·Ε ͍ͯΔ͜ͱ͕ଟʑ͋Δɻ - ʮ͏͔ͬΓݸਓใΛւ֎ͷαʔόʔʹ ૹͬͯ͠·ͬͨʯͰࡁ·ͳ͍ɻ -
σʔλΛੳπʔϧʹೖΕΔલͷʮࣄલ ϑΟϧλϦϯάʯΛ͍ͨ͠ɻ औಘํ๏ͷଟ༷ੑɾࠔ͞ - ᶄ -
- ϑΝΠϧܗࣜͷҧ͍ - จࣈίʔυͷҧ͍ - ݴޠͷҧ͍ - λΠϜκʔϯͷҧ͍ - ࣌ࠁϑΥʔϚοτͷҧ͍
- ΦϨΦϨσʔλ - ಛఆ࣌ظͷϑΥʔϚοτҧ͍ ϑΥʔϚοτ͕όϥόϥ ಛʹࣗࣾͷϦ ϕʔεͰࢹ͠ -
ྫ: ۜߦͷೖग़ۚϩά - ຊޠจࣈίʔυલఏ - CSV - ΧϥϜ͕ຊޠ - ۜߦ͝ͱͷṖใ
- `240002184 008 0000008` ͬͯͳʹ... - Time͕JSPલఏ - ݄࣌ - ଞͷγεςϜͱҧ͍ඵ͕ͳ͍ - ݸਓใ -
ྫ: ۜߦͷೖग़ۚϩά - ϦχϡʔΞϧʹ͏ഁյతͳεΩʔϚม ߋ - ͜͏͍͏ͷ͕͋ΔͱɺաڈͷσʔλΛ ৽εΩʔϚʹม͢ΔͳͲͷஅ͕ඞ ཁ -
ͦͯ͠ɺ͍͔ͭΒ͍ͭ·Ͱ͕چόʔδϣ ϯͰ͔͋ͬͨΛϝλతʹೝ͢Δඞཁ ͕Ͱͯ͘Δ... -
ηΩϡϦςΟࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ
ΤΫεϙʔτ γεςϜ ίωΫλ σʔλར༻ཧ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠιϦϡʔγϣϯಛ ༗… औΓࠐΊͳ͍… औΓࠐΊͳ͍… ᶃϑΥʔϚοτ͕ ᶄଓܗ͕ࣜ ᶅݸਓใͷআڈ͕ αϙʔτ͞Εͯͳ͍͔Β औΓࠐΊͳ͍… -
ࣗલ ίωΫλ ηΩϡϦςΟࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ
API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ ࣗ લ தܧ DB σʔλར༻ཧ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠιϦϡʔγϣϯಛ༗… ʢղܾ͞Εͳ͍…ʣ ࣗલͰதܧ͢ΕͳΜͱ͔ͳ ΔɻͰ͜͜·ͰΔͳΒ…. -
ࣗલ ίωΫλ ηΩϡϦςΟࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ
API γεςϜ ΤΫεϙʔτ γεςϜ ίωΫλ ࣗ લ தܧ DB σʔλར༻ཧ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠιϦϡʔγϣϯಛ༗… ʢղܾ͞Εͳ͍…ʣ ࣗલͰதܧ͢ΕͳΜͱ͔ͳ ΔɻͰ͜͜·ͰΔͳΒ…. ηΩϡϦςΟࢹͷՃՁΛ্͛ΔͨΊ σʔλΛʰίϯτϩʔϧʱͰ͖ΔൣғΛ͍͛ͨ ≒ σʔλཧΛऔΓࠐΊΔܗʢιϦϡʔγϣϯɺ࣮ʣʹ͍ͨ͠ -
લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ
DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ -
- ✅ ऩूɾอɾੳΛҰݩԽ͢Δ͚ͩͰͳ͘ɺίϯτϩʔϥϏϦςΟΛߴΊΒΕΔɻ - ✅ λΠϜκʔϯจࣈίʔυΛɺ౷ҰతͳϧʔϧͰܗɻ - ✅ ੳͨ͘͠ͳ͍ݸਓใͳͲΛɺࣄલʹআ֎ (ϑΟϧλ)ɻ
- ✅ ඞཁͳใʢྫɿࣾһϚελʣΛ༩ͯ͠ɺσʔλΛϦονʹ (ΤϯϦον)ɻ - ✅ తʹԠͨ͡৽͍͠σʔληοτΛɺࣗ༝ʹ࡞ɻ - ✅ ݁Ռతʹɺੳίετ࠷దԽͰ͖Δ σʔλཧ͕ͨΒ͢ϝϦοτ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
σʔλཧͷཁૉʢ࠶ܝʣ ཁૉ໊ આ໌ Data catalog σʔλͷશମ૾ͷѲ͠ɺతͷσʔλΛൃݟ͘͢͢͠ΔΑ͏ɺϝλσʔλΛཧ͢Δ ίϯϙʔωϯτ Data warehouses ར༻͢ΔσʔλιʔεΛ౷߹͍ͯ͠Δίϯϙʔωϯτ
Data governance σʔληΩϡϦςΟͱίϯϓϥΠΞϯεΛҡ࣋͢ΔͨΊͷϙϦγʔͱखॱɺͦͯͦ͠ΕΛ ཧ͢Δଶ Data architecture σʔλϑϩʔΛཧ͢ΔεΩʔϚ Data security σʔλΛอޢ͢ΔͨΊͷηΩϡϦςΟ -
લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ
DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ -
- AWSΞΧϯτ297323088823, 085773780922 ͷ౦ژϦʔδϣϯʹ͓͍ͯ2025/07/14 18:00 ~ 2025/07/15 09:00ʹ࣮ࢪ͞ΕͨAWS্ͷActionɺ࣮ߦͨ͠ΞΫλʔΛΧϯτ͢Δ ϓϩηεͱϩδοΫΛ࡞͍ͬͯͩ͘͞ɻ -
ͱͱͲ͜ʹ͋Δ͔Λಛఆ͢Δͷ͍͠ͱ͓͏ͷͰɺࣄલʹఏڙͨ͠ɹhttps:// bit.ly/40Wiw1M > ʮੜawsϩάʯΛετϨʔδͱΈͨͯͯΛ͍ͬͯͩ͘͞ - औಘ͔Βੳ·Ͱͷखॱ໌͍ࣔͯͩ͘͠͞ - πʔϧूܭϩδοΫΧϥϜ໊Λͬͯઆ໌Ͱ͖Ε͓̺Ͱ͢ɻਖ਼֬Ͱͳ͍͍ͯ͘Ͱ ͢ɻ ɹ※AIʹ͛ࠐΉ͋Γɻͦͷ߹ɺpython/sqlͷΫΤϦΛఏڙͯ͘͠Δͱ͓ΉͷͰɺͦͷΫ ΤϦͷʮूܭϩδοΫʯΛ͓͍ͩ͘͞ɻ - ࣮ߦ: 10min, ൃද: 1minx7 ϋϯζΦϯ02: ηΩϡϦςΟࢹΛͬͯΈΑ͏ -
ϋϯζΦϯ02-01: σʔλཧԼʹ͋ΔσʔλੳΛͯ͠ΈΑ͏ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯࢀߟ > ʮϋϯζΦϯ02-01: σ ʔλཧԼʹ͋ΔσʔλੳΛͯ͠ΈΑ͏ʯʹίϐϖͰ͖ΔΫΤϦ͕͋Γ
·͢ɻ ΫϥΠΞϯτduckDBͱ͍͏πʔϧ͕ͩɺඪ४ʹै͍ͬͯΕଞ ͷπʔϧϥΠϒϥϦͰՄೳ㲈૬ޓޓੑ -
ൺֱ ඇσʔλཧԼͷରԠ σʔλཧԼͷରԠ ෆཁʹͳͬͨʁ ετϨʔδ୳ࡧ aws s3όέοτΛ୳͢ ʢWeb, cliʣ ςʔϒϧ໊Λ୳͢
✗ ετϨʔδཧղ όέοτԼͷϑΥϧμߏ ΛѲ - ◦ ϑΝΠϧൃݟ json.gz` Λ औಘ - ◦ σʔλཧղ `jq`ͳͲΛͬͯߏѲ describeҰൃɻ ༷Λຬ͍ͨͯ͠Επʔ ϧΘͳ͍ ˚ɻΫΤϦͳͷͰΑΓѲ ͘͢͠ͳͬͨ σʔλར༻ `jq`ͳͲΛͬͯੳ ΫΤϦͰੳɻ ༷Λຬ͍ͨͯ͠Επʔ ϧΘͳ͍ ˚ɻΈʹΑΓσʔλऔ ಘ͕ޮԽ͞Ε͍ͯΔ -
ηΩϡϦςΟࢹιϦϡʔγϣϯ γεςϜ API DB UI + ΫΤϦ γεςϜ API γεςϜ
ΤΫεϙʔτ γεςϜ ίωΫλ औΓࠐΊͳ͍… औΓࠐΊͳ͍… ᶃϑΥʔϚοτ͕ ᶄଓܗ͕ࣜ ᶅݸਓใͷআڈ͕ αϙʔτ͞Εͯͳ͍͔Β औΓࠐΊͳ͍… σʔλར༻ཧ શ෦ιϦϡʔγϣϯ্… ΫΤϦݴޠιϦϡʔγϣϯಛ ༗… -
લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ
DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ ΫΤϦ ΫΤϦ σʔλཧଆʹґଘ͠ͳ͍ ׳ΕͯΔΫΤϦݴޠΛ ͑ΔΑ͏ʹͳͬͨʂ σʔλߏ؆୯ʹΘ͔Δʂ -
લॲཧ πʔϧ γεςϜ API γεςϜ API γεςϜ ΤΫε γεςϜ ੜσʔλ
DB લॲཧ πʔϧ લॲཧ πʔϧ લॲཧ πʔϧ લॲཧࡁΈ DB UI ΫΤϦ σʔλར༻ऀ ΫΤϦ ΫΤϦ σʔλཧଆʹґଘ͠ͳ͍ ׳ΕͯΔΫΤϦݴޠΛ ͑ΔΑ͏ʹͳͬͨʂ σʔλߏ؆୯ʹΘ͔Δʂ πʔϧʹޓੑ͕͋Δʂ -
- ݁: Open Table Formatͱ͍͏ඪ४ʹै͍ͬͯΔ͔Β - ͍͖ͳΓOTFʹೖΔલʹɺͦΕҎલͷ֓೦ʹ͍ͭͯ৮Ε ·͢ - OLAP
- σʔλΣΞϋε - σʔλϨΠΫ ͳͥ૬ޓޓੑ͕͋Δ͔ -
- Q: σʔλϕʔεͱҧ͏ͷʁ - A: DBOLTPʮOnLine Transaction Processingʯͱ͍͏τϥϯβΫγϣϯ୯ҐͰͷ҆ શͳॲཧ͕ಘҙɻେྔͷσʔλΛ͔ͭͬͨੳͱ ·ͨҟͳΔ
OLAP: Online Analytical Processing -
- Q: ͳΜͰOLTPͱOLAPΛΘ͚Δͷʁ - A: - OLAPͱϦιʔεΛୣ͍߹Θͳ͍Α͏ʹ - OLAPେྔͷσʔλΛରʹ͍ͨ͠ͷͰɺΧϥϜϕʔεͰ͕ͬ͞ͱੳ͍ͨ͠ -
ྫʮ͓ళͷϨδʢOLTPʣͱɺࡏݿཧݿʢOLAPʣׂ͕ҧ͏͔Βɻ - Ϩδߴͳձܭ͕ٻΊΒΕɺݿେྔͷࡏݿ͔ΒతͷͷΛ୳͢ೳྗ͕ٻΊΒΕΔ - OLAPෳσʔλΛͬ͞ͱ૬ؔͤ͞ΔηΩϡϦςΟࢹɺಛʹظؒͷੳʹ૬ੑ͕͍͍ OLAP: Online Analytical Processing -
OLAPͷίϯϙʔωϯτ ίϯϙʔωϯτ໊ ֓ཁ ίϯϐϡʔτΤϯδϯ σʔλॲཧɺ·ͨͦͷϫʔΫϩʔυͷ࣮ߦ Χλϩά ར༻͍ͨ͠σʔλͷൃݟɾಛఆΛޮԽ͢ΔͨΊʹɺσʔλͷϝλσʔ λʢςʔϒϧ໊ɺεΩʔϚɺཧతͳஔՕॴʣΛίϯϐϡʔτΤϯδ ϯʹఏڙ ςʔϒϧϑΥʔϚοτ
σʔλͷ࣮ମͰ͋ΔଟͷϑΝΠϧ܈͕ͲͷΑ͏ʹετϨʔδ্ʹஔ ͞Ε͍ͯΔ͔Λࣔ͢ϝλσʔλɺ·ͨཧతͳදͷΑ͏ͳͷɻ ετϨʔδΤϯδϯ ςʔϒϧϑΥʔϚοτʹैͬͯཧσʔλΛอଘɾߋ৽ɾআɻ ஔ࠷ దԽɺظؒΛ͑ͨσʔλͷআͷ࠷దԽ࣮ࢪ ετϨʔδ σʔλͷอଘɻΦϒδΣΫτετϨʔδͱ͍͏ͷ͕͍͍͔ɻσʔλʹ ΑͬͯɺϑΝΠϧϑΥʔϚοτͱςʔϒϧϑΥʔϚοτҟͳΔ ϑΝΠϧܗࣜ ཧతʹσʔλΛ֨ೲ͢ΔϑΝΠϧܗࣜɻߏ vs ඇߏ,͔Β͞Βʹ ߦ ࢥߟ vs ྻࢥߟͰ͞Βʹྨ͞ΕΔ -
- ࡉ͔͍͜ͱΛল͘ͱɺੜσʔλࣄʹҟͳΔෳͷϑΝΠϧϑΥʔϚοτΛϦʔζφ ϒϧʹอଘɾੳͰ͖ΔॊೈੑΛɺϫʔΫϩʔυ͝ͱʹҟͳΔίϯϐϡʔτΤϯδ ϯΛ࠾༻Ͱ͖ΔΑ͏ͳΈʹ͢Δ͜ͱͰ࣮ݱ - ίϯϐϡʔτͱετϨʔδͷʂ - ϑΝΠϧͷΦʔϓϯԽ! - …ͱ͍͏ίϯηϓτͰFacebook͕Hadoop্ͰSQLΛهड़͘͢͢͠ΔͨΊʹHive
ϑϨʔϜϫʔΫΛ։ൃ (2008) - σʔλΣΞϋεͷ͕ղܾ͢ΔͱͱʹɺҰ෦σάϨ·ͨΑΓਂࠁԽ σʔλϨΠΫొ https://www.borthakur.com/ftp/hadoopworld.pdf -
OLAPͷίϯϙʔωϯτʢ࠶ܝʣ ίϯϙʔωϯτ໊ ֓ཁ ίϯϐϡʔτΤϯδϯ σʔλॲཧɺ·ͨͦͷϫʔΫϩʔυͷ࣮ߦ Χλϩά ར༻͍ͨ͠σʔλͷൃݟɾಛఆΛޮԽ͢ΔͨΊʹɺσʔλͷϝλσʔ λʢςʔϒϧ໊ɺεΩʔϚɺཧతͳஔՕॴʣΛίϯϐϡʔτΤϯδ ϯʹఏڙ ετϨʔδΤϯδϯ
ςʔϒϧϑΥʔϚοτʹैͬͯཧσʔλΛอଘɾߋ৽ɾআɻ ஔ࠷ దԽɺظؒΛ͑ͨσʔλͷআͷ࠷దԽ࣮ࢪ ϑΝΠϧܗࣜ ཧతʹσʔλΛ֨ೲ͢ΔϑΝΠϧܗࣜɻߏ vs ඇߏ,͔Β͞Βʹ ߦ ࢥߟ vs ྻࢥߟͰ͞Βʹྨ͞ΕΔ ςʔϒϧϑΥʔϚοτ σʔλͷ࣮ମͰ͋ΔଟͷϑΝΠϧ܈͕ͲͷΑ͏ʹετϨʔδ্ʹஔ ͞Ε͍ͯΔ͔Λࣔ͢ϝλσʔλɺ·ͨཧతͳදͷΑ͏ͳͷɻ ετϨʔδ σʔλͷอଘɻΦϒδΣΫτετϨʔδͱ͍͏ͷ͕͍͍͔ɻσʔλʹ ΑͬͯɺϑΝΠϧϑΥʔϚοτͱςʔϒϧϑΥʔϚοτҟͳΔ O'Reilly-ApacheIceberg-ਤ1-5-දܗࣜΛͬͯσʔλϑΝΠϧΛදʹ·ͱΊΔ -
- ޮతͳσʔλΞΫηε: - Ϣʔβʔੳπʔϧɺσʔλ͕࣮ࡍʹͲͷϑΥϧμʹɺ͍ͭ͘ͷϑΝΠϧ ͱͯ͠อଘ͞Ε͍ͯΔ͔ͱ͍ͬͨཧతͳৄࡉΛҙࣝ͢Δඞཁ͕ͳ͘ͳΔɻ - ཧతͳҰͭͷදΛૢ࡞Ͱ͖Δɻ - Ұ؏ੑͷ୲อ: -
༷ʑͳπʔϧෳͷνʔϜ͕σʔλΛར༻͢Δࡍʹɺৗʹ࠷৽ͷσʔλ͕ࢀ র͞Ε͍ͯΔ͜ͱΛอূ - ٯʹݴ͏ͱɺίϨ͕อূ͞Εͳ͍ͱਓʹΑͬͯੳ݁Ռ͕ҟͳΔͱ͍͏ࣄଶʹ ςʔϒϧϑΥʔϚοτͷׂ O'Reilly-ApacheIceberg-ਤ1-5-දܗࣜΛͬͯσʔλϑΝΠϧΛදʹ·ͱΊΔ -
- σʔλϨΠΫ·Ͱ Apache Hiveςʔϒϧ ϑΥʔϚοτ - ςʔϒϧͷఆ͕ٛϑΝ ΠϧγεςϜͷཧత ͳσΟϨΫτϦߏʹ ڧ͘ґଘ
σʔλϨΠΫͷϑΥʔϚοτ https://estuary.dev/blog/explaining-data-lakes-lakehouses-catalogs/ -
- ॻ͖ࠐΈதʹϑΝΠϧΛಡΈऔΒΕΔͷͰɺੳ݁ՌʹҰ؏ੑ ͕ͳ͘ͳΔ - Ұ෦ͷϑΝΠϧ͕ॻ͖ࠐΈͳͲͰഁଛ͠ɺಡΈࠐΊͳ͘ͳΔ - εΩʔϚมߋ࣌ʹ৽چͰޓੑ͕୲อ͞Εͣɺݹ͍ϑΝΠϧ͕ಡ ΈࠐΊͳ͘ͳΔ - ϑΝΠϧͷϦωʔϜ͕Ͱ͖ͳ͍
- ཧతͳॻ͖͑ʹͳΔͷͰɺσόοά࠶ݱ͕ࠔʹͳΔ - ্هΛΧόʔ͢Δʹ૬Ԡͷίετ͕ඞཁ σʔλϨΠΫʹ͓͚ΔςʔϒϧϑΥʔϚοτͷ -
Open Table FormatʢOTFʣͱσʔλϨΠΫϋεͷొ - σΟϨΫτϦߏͰͳ͘ݸʑͷσʔλϑΝΠϧ ϑΝΠϧ୯Ґʹɺෳͷ֊ʹ͔Εͨϝλσʔ λΛςʔϒϧͱͯ͠ཧ - ϑΝΠϧॻࠐΈͷͨͼʹɺϝλσʔλ͕ߋ৽͞Ε Δ
- OTFΛ֩ͱͨ͠ΞʔΩςΫνϟ͕ʮσʔλϨΠΫϋ εʯ - ΦϒδΣΫτετϨʔδΛετϨʔδͱͯ͠ ར༻ͭͭ͠ɺDBMSͷΑ͏ͳػೳɺಛʹACIDಛ ੑΛσʔλϨΠΫʹͨΒͨ͠ - ۩ମతͳOTFͷ࣮ྫͷ͕̍ͭʮApache Icebergʯ https://estuary.dev/blog/explaining-data-lakes-lakehouses-catalogs/ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- 2015ʹNet fl ixͷΤϯδχΞͰ͋ΔDaniel WeeksࢯͱRyan Blueࢯ͕։ൃ ͨ͠ΦʔϓϯςʔϒϧϑΥʔϚοτ - Net fl
ixϖλόΠτϨϕϧͷσʔλΛѻ͏ͷͰɺͦ͜·ͰεέʔϧͰ͖ Δඞཁ͕͋Δ - ͭ·Γඪ४Ͱ͋ΓɺιϑτΣΞͦͷͷͰͳ͍ - https://iceberg.apache.org/spec/ - 2018ʹΦʔϓϯιʔεԽ & Apache Software FoundationʢASFʣʹد ଃ - ݱ࣌Ͱv4Λ։ൃத Apache Icebergͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓΛฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ -
ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔Βͷཧతͳ - Partition Evolution, Hidden Partitioning, Time travel, Version
Rollback, Schema Evolutionͷαϙʔτ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ϝλσʔλΛ׆༻ͨ͠ΫΤϦϓϥχϯάΛΫϥΠΞϯτଆͰ࣮ ݱ͠ɺϘτϧωοΫΛςʔϒϧଆʢαʔόʔଆʣʹ͠ͳ͍ - ύϑΥʔϚϯεʢSpeedʣ -
ಡΈऔΓฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ΫΤϦϓϥχϯάΛO1ʹ͢Δ - σʔλϨΠΫͷ߹ɺཧతͳσΟϨΫτϦߏΛςʔϒϧͷఆٛͱ͍ͯ͠Δͨ
ΊɺΫΤϦΛ࣮ࢪ͢ΔલʹύʔςΟγϣϯͷཧߏΛߟྀ͢Δඞཁ͕͋ͬͨɻ ͜Εɺσʔλར༻ऀͱΑΓΫΤϦΛ࣮ࢪ͢ΔϫʔΫϩʔυʹ༏͘͠ͳ͍ - ಡΈऔΓฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓΛฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ -
ACIDಛੑͷ֬อ, ߦ୯ҐͰͷςʔϒϧૢ࡞ - ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ -
ݎ࿚ͳܕʢDependable typesʣ - ໌֬ʹఆ͚ٛͮΒΕͨςʔϒϧܕ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτʢFormatrsʣ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
- ཧతετϨʔδ͔ΒͷཧతͳʢετϨʔδͷʣ - ֦େੑʢScaleʣ - ύϑΥʔϚϯεʢSpeedʣ - ಡΈऔΓฒߦ͢Δॻ͖ࠐΈ͔ΒͷʢSerializable Isolationʣ -
ݎ࿚ͳܕʢDependable typesʣ - ʢσʔλϑΝΠϧͷʣϑΥʔϚοτ - ಡΈऔΓ࠷దԽͱॻ͖ࠐΈ࠷దԽͷ྆ํͷܗ͕ࣜར༻Մೳ - ·ͨϑΝΠϧܗࣜʹ͔͔ΘΒͣɺఆٛ͞ΕͨσʔλܕεΩʔϚཧ ϓϩηεΛద༻Ͱ͖Δ Apache Iceberg͕࣮ݱͨ͜͠ͱ -
ຊߨٛͰσϞΛ·͑͡Δػೳ ػೳ໊ ༰ ର ACIDಛੑ- ָ؍తฒߦ੍ޚʹΑΔACIDಛੑ ॻ͖ࠐΈྃޙʹɺ࠷৽ͷϝλσʔλϑΝΠϧΛ࡞͢Δ͕ɺͦͷ࠷৽ͷϑΝΠ ϧʹϙΠϯλΛม͑Α͏ͱ͢ΔλΠϛϯάͰʢଞͷϥΠλʔ͕ଞͷมߋΛίϛο τ͍ͯ͠ͳ͍͔ʣڝ߹Λݕग़͢Δɻͳ͚Είϛοτ͞Εɺ͋Είϛοτ͞Ε ͳ͍
˚ HiddenύʔςΟγϣχϯά σʔλར༻ऀͷΫΤϦΛղੳ͢ΔࡍʹɺύʔςΟγϣχϯάʹө͞ΕΔΑ͏ಛ ఆͷॲཧΛ෦తʹద༻͠ΫΤϦΛมߋ͢Δɻ ◦ ύʔςΟγϣϯߋ৽ʢEvolutio ύʔςΟγϣϯཧΛཧతͳσΟϨΫτϦߏ͔ΒϝλσʔλϑΝΠϧ্Ͱ ཧ͢ΔܗࣜͱͳͬͨͨΊɺύʔςΟγϣϯΛมߋͯ͠ςʔϒϧͷ࠶࡞͕ෆཁ ͱͳΔɻ·ͨɺύʔςΟγϣϯόʔδϣϯΛҙࣝͤͣΫΤϦ͕Մೳ ˚ εΩʔϚߋ৽ʢEvolution ↑ͷςʔϒϧεΩʔϚόʔδϣϯ ◦ λΠϜτϥϕϧ ߋ৽࣌ʹϝλσʔλϑΝΠϧ͕࡞͞ΕΔͱͱʹɺΠϛϡʔλϒϧͳεφοϓ γϣοτੜ͞ΕΔͨΊɺಛఆͷλΠϛϯάͰͷσʔλΛΫΤϦͰ͖ΔΑ͏ʹ ◦ όʔδϣϯϩʔϧόοΫ εφοϓγϣοτ͕͋ΔͷͰʢࡶɺϝλσʔλϑΝΠϧͷ࠷৽ϙΠϯλΛߋ৽͢ ΔͳͲͯ͠ϩʔϧόοΫՄೳ ◦ -
- Net fl ix͕Ͳ͏ηΩϡϦςΟϦεΫΛఆྔԽ͍ͯ͠Δ͔ - https://www.youtube.com/watch?v=wgQkqPmDIAQ&t=1184s - Α͋͘ΔAE͕ͩɺͦ͜Net fl ixɺ͜ͷϏσΦͰHowΛ͕ͬͭΓઆ໌ͯ͠Δ
- ϞϯςΧϧϩγϛϡϨʔγϣϯͳͲ౷ܭϞσϦϯάख๏Λ͕ͬͭΓ͍ͬͯΔ - ͜ΕʹݶΒͣNet fl ixલ·ͰߴͳηΩϡϦςΟϒϩάΛൃ৴͍ͯͨ͠ͷͰ͓͢͢Ί - https://net fl ixtechblog.com/all?topic=security - ͦͷޙɺͲ͜ʹ͍ͬͨͷ͔.... ༨ஊ(ඈ͔͢ʣ -
πʔϧͷղઆ https://iceberg.apache.org/spec/#overview - ΫΤϦΤϯδϯ - DuckDB: OLAPϫʔΫϩʔυ͚ʹઃܭ͞ΕͨΠϯϝϞϦͳੳDBɻ ༷ʑͳ֦ுπʔϧ͕͋Δ - Athena:
AWSͷαʔόʔϨεͳϑϧϚωʔδυͳΫΤϦαʔϏε - Iceberg SDK: IcebergͷSDKɻݴޠ͝ͱʹ͋Δ - Lake Formation: Amazon S3 ςʔϒϧͱ౷߹͞ΕɺS3 ʹอଘ͞Εͨσʔ λʹର͢ΔूதతͳΞΫηε੍ޚͱΨόφϯεΛՄೳʹ͠·͢ɻ - S3 Tables: Apache Iceberg ܗࣜͷςʔϒϧϑΥʔϚοτσʔλͱͯ͠࠷ద Խ͞ΕͨS3 BucketετϨʔδΛఏڙ - σʔλϨΠϠͷS3҉తʹ࡞ΒΕΔ -
ΧλϩάϨΠϠʔ https://iceberg.apache.org/spec/#overview - σʔλར༻ऀ͕Icebergςʔϒϧͱର͢Δ࠷ॳͷϙΠϯτ - ݱࡏͷϝλσʔλͷϙΠϯλΛอ࣋ - ACIDಛੑͷAtomicityઈର֬อ͢Δ͘Μ - ͜ͷςʔϒϧʹͲͷΑ͏ͳσʔλؚ͕·Ε͍ͯΔ͔Λ࠷ॳʹఏڙ
- ಛఆͷλΠϛϯάͰͷςʔϒϧͷεΩʔϚɺύʔςΟγϣχϯάɺεφοϓγϣ οτཤྺͳͲͱ͍ͬͨςʔϒϧͷঢ়ଶʹؔ͢ΔΛϝλσʔλΛอ࣋ɻ - ࣮ࡍʹςʔϒϧଓΛ͢Δࡍɺ͜ͷΧλϩάΛࢦఆ͍ͯ͠Δ -
ΧλϩάͱͷଓʢDuckDB) $ cat duckdb_init.sql INSTALL httpfs; INSTALL iceberg; INSTALL aws;
LOAD iceberg; LOAD httpfs; LOAD aws; FORCE INSTALL iceberg FROM core_nightly; FORCE INSTALL avro FROM core_nightly; CREATE SECRET secret ( TYPE s3, PROVIDER credential_chain, REGION 'ap-northeast-1' ); ATTACH 'arn:aws:s3tables:ap-northeast-1:297323088823:bucket/security- camp-table-bucket' AS s3_tables ( TYPE iceberg, secret secret, ENDPOINT_TYPE s3_tables ); SET unsafe_enable_version_guessing = true; % export AWS_ACCESS_KEY_ID=hogehoge % export AWS_SECRET_ACCESS_KEY=fugafuga % duckdb --init duckdb_init.sql https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡΈࠐΈ > Χλϩάͱͷଓ -
ϝλσʔλϨΠϠʔ https://iceberg.apache.org/spec/#overview - Apache Iceberg͕ఏڙ͢Δػೳͷ؊ - ΤϯδϯͱετϨʔδͷؒͰϝλσʔλ/நԽϨΠϠʔͱͯ͠ػೳ - ۩ମతʹσʔλͱͦͷϝλσʔλɺ͓ΑͼͦΕΒͷૢ࡞Λ͢Δߏ -
Metadata fi leɺManifest listɺManifest fi leͱ͍ͬͨ̏छྨͷϑΝΠϧ͔Βߏ - ΫΤϦΤϯδϯ͕Χλϩάܦ༝ͰΞΫηε͢Δ߹ɺσϑΥϧτͰ࠷৽ͷ metadata fi le͕ಡΈࠐ·ͤΒΕΔ - ͜ΕΒ࣮ࡍͷϑΝΠϧͱಉ͡ετϨʔδʹอ͞ΕΔ -
ϝλσʔλϑΝΠϧΛಡΜͰΈΑ͏ - ΫΤϦɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ
> ʮϝλσʔλϑΝΠϧΛಡΜͰΈΑ͏ʯ͔Βऔಘɻ - Ͱ͖ͳ͚Εɺhttps://bit.ly/40Wiw1M > metadata-sample.json ͔Βऔಘ -
ϝλσʔλϑΝΠϧ͔ΒϚχϑΣετϦετ https://iceberg.apache.org/spec/#overview - `current-snapshot-id` ͷΛಛఆ - `snapshots` Ϧετ͔Βɺcurrent-snapshot-idͱಉ͡idΛ࣋ͭ `snapshot-id` Λ
snapshots͔Β୳ࡧ - ↑ͷ `manifest-list` ͷ = ϑΝΠϧͷύε ʢ `*.avro`ʣΛಛఆ - DuckDBͰΔ߹ - ΫΤϦɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ >ɹʮϝλσʔλϑΝΠϧ͔ΒϚχϑΣετϦετʯɻ - Ͱ͖ͳ͚Εɺhttps://bit.ly/40Wiw1M > snap-manifest-list-sample.avro Λऔಘ –ϚχϑΣετϦετϑΝΠϧͷύεͷऔಘ D SELECT s."manifest-list" FROM read_json('s3://a33d7730-dc61-4b7d-1rerceqph8j3ubw9fj6snbyfxtqyhapn1b--table-s3/metadata/00868- b975b279-8eac-4cfd-96db-0bb85bbdc1ea.metadata.json') t CROSS JOIN UNNEST(t.snapshots) AS u(s) WHERE s."snapshot-id" = t."current-snapshot-id"; –ϚχϑΣετϦετϑΝΠϧಡΈࠐΈ D SELECT * FROM read_avro('s3://a33d7730-dc61-4b7d-1rerceqph8j3ubw9fj6snbyfxtqyhapn1b--table-s3/metadata/ snap-8236464468384760848-0-f0f21f1d-d821-4b42-b969-7cde2eef7d59.avro'); -
- ಛఆλΠϛϯάʹ͓͚Δςʔϒϧͷ εφοϓγϣοτͱ1:1Ͱඥͮ͘Ϧ ετ - ໊લͷ௨ΓϚχϑΣετϑΝΠϧ ʢޙड़ʣͷϦετ - ଞʹϚχϑΣετϑΝΠϧͷύεɺ ύʔςΟγϣϯͷൣғ͕Θ͔Δ
- `where`۟ͳͲͰύʔςΟγϣϯ Λࢦఆ͢Εɺ͜ͷΛͱʹෆ ཁͳσʔλΛಡΈࠐ·ͳ͍ - ·ͨɺલճͷεφοϓγϣοτͱ ͷ͕ࠩΘ͔Δ ϚχϑΣετϦετ -
- Ϧετ - Ϧετ - Ϧετ - Ϧετͷڧௐจࣈ - Ϧετ
ϚχϑΣετϦετͷεΩʔϚ -
ϚχϑΣετϦετ͔ΒϚχϑΣετϑΝΠϧ https://iceberg.apache.org/spec/#overview - ֤ཁૉͷ `manifest_path` ͷ( `*-m0.avro`)ΛಡΈࠐΉ - ͱΓ͋͑ͣ࠷ॳͷཁૉΛಡΈࠐΜͰΈΑ͏ -
duckdbͷ߹ - ΫΤϦɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡΈࠐΈ >ɹʮϚχϑΣετϦετ͔ΒϚχϑΣετϑΝΠϧʯɻ - Ͱ͖ͳ͚Εɺhttps://bit.ly/40Wiw1M > manifests-samle-m0.avroΛऔಘ -
- ↑ϚχϑΣετϑΝΠϧ - ↓ϚχϑΣετϑΝΠϧͷ data_ fi leͷ - σʔλͷσʔλϑΝΠϧͷα ϒηοτΛ͠ɺ֤ϑΝΠϧ
ͷৄࡉ౷ܭใΛؚΉ - ͜͜Ͱ͍͏౷ܭใΧϥϜͷ ࠷େɾ࠷খɺnullͷ༗ແɺ ϨίʔυͳͲ - Γ`where` ۟ͰΑΈ͜Ήϑ ΝΠϧΛ੍ݶͰ͖Δ ϚχϑΣετϑΝΠϧ -
ϚχϑΣετϑΝΠϧ͔ΒσʔλϑΝΠϧ https://iceberg.apache.org/spec/#overview - ϚχϑΣετϑΝΠϧͷ `data_ fi le`.` fi le_path`ͷΛಡΉ -
data_ fi leΧϥϜࣗମϑΝΠϧͱ͍͏ΑΓɺ࣮ࡍͷσʔλͷύεͱͦͷϝλσ ʔλΛຒΊͨͷ - ࣮ࡍͷσʔλϑΝΠϧͷऔಘ࣍ͷΑ͏ʹ࣮ࢪʢDuckdb) - ΫΤϦɺhttps://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ> Apache Iceberg > ಡ ΈࠐΈ >ɹʮσʔλϑΝΠϧʯɻ - Ͱ͖ͳ͚Εɺhttps://bit.ly/40Wiw1M > data- fi le-sample.parquetΛऔಘ -
σʔλϨΠϠʔ https://iceberg.apache.org/spec/#overview - ࣮ࡍͷσʔλ͕֨ೲ͞ΕΔཧ - ࢄϑΝΠϧγεςϜɺΦϒδΣΫτετϨʔδʹͳΓ͕ͪ - ཧతʹσʔλར༻ऀ͔ΒӅṭ͞ΕͯΔͷͰҙࣝ͠ͳͯ͘ ྑ͍ -
…͕S3TableσʔλཧऀͰΞΫηεͰ͖ͳ͍ͷͰ ϝϯςతʹࠔΔʢཪʣ -
- ͍ͭʹੜσʔλʹ౸ୡʂ - ࣮ࡍͷσʔλͷϑΝΠϧ - AWSࠪϩά - ෳͷϑΝΠϧܗࣜΛαϙʔτ: parquet, avro,
orc - Parquet͔͠Έͨ͜ͱͳ͍͔ - εΩʔϚϩά͝ͱʹҟͳΔ σʔλϑΝΠϧ -
͓·͚: - ಛఆͷλΠϛϯάͷεφοϓγϣοτΛݟΔ(Time Travel) - Hidden Partitioning - ύʔςΟγϣϯΛઃఆ͍ͯ͠Δલఏ -
ࠓճͷ߹ͩͱɺregion, accountId, time_dt_dayΛఆٛͯ͠ ͍Δɻtime_dt_daytimestamp͔ΒΛࣗಈܭࢉ͢Δύʔ ςΟγϣϯʹ͍ͯ͠Δɻ - σʔλར༻ऀͦΕΛҙࣝͤͣλΠϜελϯϓʢྫ: current_date() - Interval 3 day)ͱ͔͢Ε͍͍ -
ॻ͖ࠐΈͷΈ ਤ3-2-Apache Icebergʹ͓͚Δॻ͖ࠐΈॲཧ - جຊΧλϩά͕ϙΠϯτ͢ΔϝλσʔλΛ ͬͯޮతͭ҆શʹσʔλΛॻ͖ࠐΉ - ಡΈࠐΈͱҧ͏ͷɺຖॲཧ͝ͱʹϝλσʔ λϑΝΠϧɺεφοϓγϣοτͷϑΝΠϧ ͕ੜ͞ΕΔ͜ͱ
- ͞Βʹָ؍తฒߦੑʹΑͬͯΧλϩά͕ࣔ͢ ϝλσʔλϑΝΠϧͷϙΠϯλઌ͕࠷৽ͷ ͷʹͳΔ - ͜ͷࡍʹɺॳΊͯڝ߹ʢίϯϑϦΫτʣν ΣοΫΛ͢ΔɻͷͰɺAtomicੑ͕୲อ͞Ε Δ -
- https://bit.ly/40Wiw1M > ʮtable-create.zip Λղౚ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ >
Apache Iceberg > ॻࠐΈ > ςʔϒϧ࡞ - ͦͷޙɺϝλσʔλΛಡΜͰΈΑ͏ ςʔϒϧ࡞ -
σʔλॻࠐΈ - https://bit.ly/40Wiw1M > ʮdata-ingest.zipΛ ղౚ - https://bit.ly/40Wiw1M > ʮ࣮ߦΫΤϦूʯ>
Apache Iceberg > ॻࠐΈ > σʔλೖ - ͦͷޙɺϝλσʔλΛಡΜͰΈΑ͏ - snapshots, snapshot-logͳͲͷมԽʹண -
- duckdb·ͩॻࠐΈૢ࡞Λαϙʔτͯ͠ͳ͍ͷͰɺ͜͜ͰαϯϓϧͷΈ - AthenaͰࡶʹআ - https://bit.ly/40Wiw1M > delete.metadata.json Λམͱ͠ಡΜͰΈΔ σʔλআ
DELETE FROM "system_logs"."tojitsu" WHERE class_uid = 3002; -
σʔλআ - ϝλσʔλͷsnapshotsͰҎԼΛ - added-position-delete- fi les: 7 - added-delete-
fi les: 7 - added-position-deletes: 36 - total-delete- fi les: 7 - total-position-deletes: 36 -
σʔλআ - manifest list - https://bit.ly/40Wiw1M > delete-snap.avro Λམͱ͠ಡΜͰΈΔ -
ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete-snap.avro'); - Manifest listͷσʔλ༷͔Β `content` `The type of fi les tracked by the manifest, either data or delete fi les`Λࣔ͢ɻ - `int` with meaning: `0: data`, `1: deletes` - Ճ͞Εͨdelete fi les ͕̓ͭͱ͍͏͜ͱ - Metadata ͱໃ६ͳ͠ https://iceberg.apache.org/spec/#manifest-lists -
σʔλআ - manifest fi le - https://bit.ly/40Wiw1M > delete-m0.avro Λམͱ͠ಡΜͰΈΔ
- ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete-m0.avro'); - `content` `Type of content stored by the data fi le: data, equality deletes, or position deletes`Λࢦ͠ɺ`1` ͷ߹͕ `position delete` ʹ֘͢Δ - `status` ͷ 1 added Λҙຯ͢Δ͕ɺdelete fi le ͱͯ͠Ճ͞Εͨɺͱ͍ ͏͜ͱ - Manifest-listͷηΫγϣϯͰॻ͍ͨ௨ΓɺίϨఆͲ͓Γɻ -
σʔλআ - delete fi le - https://bit.ly/40Wiw1M > delete- fi
le.parquetΛऔಘ - ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./delete- fi le.parquet'); - ࣮ࡍͷεΩʔϚΛΈΔͱɺී௨ͷcloudtrailϩάͱҟͳΔ delete fi leͷεΩʔϚʢ্ਤʣ - fi le_pathʹʢཧআͨ͠ʣσʔλ - posʹɹআͨ͠ΠϯσοΫε -
σʔλআ - ࣮ࡍͷσʔλ - https://bit.ly/40Wiw1M > deleted-logically-but-physical-data- fi le.parquetɹΛऔಘ -
ಡΈํ: DuckDBͰɺSELECT * FROM read_avro('./deleted-logically-but-physical-data- fi le.parquet'); - ࣮ࡍͷϑΝΠϧʢӈਤʣ - ΫΤϦͰআ͕ͨ͠ొ͢Δidx͕ɺdelete fi leʢԼਤʣͷposʹ߹க͍ͯ͠Δ - ↓আΫΤϦ DELETE FROM "system_logs"."ex04_with_partition" WHERE class_uid = 3002; -
σʔλআ - ࣮ࡍͷσʔλ - ࣮ࡍͷϑΝΠϧʢӈਤʣ - ΫΤϦͰআ͕ͨ͠ొ͢Δidx͕ɺdelete fi le ʢԼਤʣͷposʹ߹க͍ͯ͠Δ
- ↓আΫΤϦ DELETE FROM "system_logs"."ex04_with_partition" WHERE class_uid = 3002; ϝλσʔλϨΠϠʔ͕আΛཧతʹཧͨ͜͠ͱ͕Θ͔Δ -
- ͜͜ͰαϯϓϧͷΈ - AthenaͰࡶʹcommonΧϥϜՃɻ - ϝλσʔλϑΝΠϧʹεΩʔϚ (id =1ʣ͕Ճ ͞Ε͍ͯΔ -
εφοϓγϣοτࣗମ૿͑ͯແ͍ - εΩʔϚͷid = 0 εΩʔϚߋ৽ ALTER TABLE "system_logs"."ex04_with_partition" ADD COLUMNS (comment string) -
- AthenaͰࡶʹcommonΧϥϜՃɻ - ϝλσʔλϑΝΠϧʹεΩʔϚ (id =1ʣ͕ Ճ͞Ε͍ͯΔ - εφοϓγϣοτࣗମ૿͑ͯແ͍ -
εΩʔϚͷid = 0 εΩʔϚߋ৽ ALTER TABLE "system_logs"."ex04_with_partition" ADD COLUMNS (comment string) ϝλσʔλϨΠϠʔ͕εΩʔϚߋ৽Λཧతʹཧͨ͜͠ͱ͕ Θ͔ΔʢSchema Evolutionʣ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
ຊߨٛͷείʔϓ σʔλཧཁૉ Χόʔ ۩ମతͳରԠྫ ৽نੑ/ීวੑ Data preparation ˚ ݸਓใͷϚεΫɺڠఆੈք࣌ʢUTCʣԽɺΧϥϜͷӳޠԽ ීว
Data pipelines ✘ Data ETL ˚ Data catalogs ◦ Apache Iceberg ৽ن Data warehouses ◦ Apache Iceberg ৽ن Data governance ˚ Data architecture ✘ Data security ◦ γεςϜϦεΫධՁ ීว ຌྫ ◦: ͔ͬ͠Γղઆ + ϋϯζΦϯ ˚: ݸผ߲Ͱͳ͍͕ཁૉͱͯ͠αϥοͱ৮ΕΔ https://www.tableau.com/learn/articles/data-management-vs-data-governance -
- σʔλΛཧɾ׆༻͢Δత༷ʑͰ͋Δ - ͔͠͠ɺతୡͱͦͷӡ༻ʹਖ਼͍͠ܕͱखஈ͕ඞཁʹͳΔ - ਖ਼͘͠ͳ͍ܕɾखஈɺϚΠφεޮՌ㲈 **ϦεΫ** ʹͳΓ͏Δ - తୡ·ͰͷܭըʹϦεΫΛ৫ΓࠐΈɺରࡦ͠ɺৼΓฦ͑Γɺ࠶ݱੑΛ
࣋ͨͤͳ͚ΕͳΒͳ͍ ֓ཁ - େ্ஈͷత = ϦεΫཧ -
- ӨڹͷѲ - ϦεΫʢ੬ऑੑxڴҖʣͷѲ - ҆શରࡦɾඪ४ͷચ͍ग़͠ɾબ - ڴҖͷચ͍ग़͠ɾબ - ϦεΫੳ
- ରԠͷѲ ֓ཁ - ྲྀΕ -
- ۀͷதͰऔѻɾॲཧɾอ͢Δใ͔Βચ͍ग़͢ - ۀքελϯμʔυ͔Βચ͍ग़͢ - ϑϩʔσʔλετοΫσʔλ͔Βચ͍ग़͢ - ϑϩʔσʔλ: νϟοτɺձٞɺٞࣄ -
ετοΫσʔλ: ϑΝΠϧαʔόʔ - ϞμϯʁͳΓํ - ϓϩμΫτͷDBεΩʔϜ͔Βચ͍ग़͢ - AIͰྨਪͤ͞Δɻศར - AIͷਖ਼֬ੑ͖ʹͳΔ͕ɺͦͦຊۀͰͳ͍ਓؒͷਫ਼ظͰ͖ͳ͍ͷͰؾʹ͠ͳͯ͘Α͍ ใࢿ࢈ͷચ͍ग़͠ํ -
- Apache IcebergͰ͋Ε Catalog - DBͰ͋Εshow tables; - mysqldump -u
username -p --no-data --databases prd > schema.sql ใࢿ࢈ - ચ͍ग़͠ํDBฤ -
- ҆શੑ͕ଛͳΘΕͨ߹ͷӨڹ߹͍ - 3͔࣠Βߟ͑Δͷ͕ϝδϟʔ - 1. ػີੑ: ΞΫηεΛڐՄ͞Εͨऀ͚͕ͩใʹΞΫηεͰ͖Δ - 2.
શੑ: ਖ਼֬ੑٴͼอશੑɻ࠷ۙਅਖ਼ੑؚ·ΕΔؾ͕͢Δ - 3. Մ༻ੑ: ਖ਼ͳར༻ൣғͰܧଓͯ͠ΞΫηεͰ͖Δঢ়ଶ - ݸਓใ͔ - ಛఆݸਓใ͔ɺཁྀݸਓใ͔ɺػඍใ͔ ใࢿ࢈ - ӨڹධՁ -
- ཧ͍ͯ͠ΔσʔλΛ࣮ࡍʹॲཧɾอ͢Δίϯϙʔωϯτ - ॲཧ: औѻɺม - ߨٛͷจ຺ͰIceberg TableɺΫΤϦΤϯδϯɺETLΛؚΉ - ηΩϡϦςΟରࡦΛ࣮ࡍʹࢪ͢Օॴ
- ʢݸਓతʹʣใࢿ࢈ཧΑΓ༏ઌ͖ͩ͢ͱࢥ͏ - ͳͥͳΒɺใ୯ͳΔ֓೦Ͱ͋ΓɺͦΕࣗମ͕ՁͰͳ͍͔Β - ΠϯαΠτ͋Δ͍ΞΫγϣϯΛى͜͢ͷใγεςϜͱͦͷ্Ͱࣥߦ͞ΕΔۀ͔ͩΒ ใγεςϜ -
- ओʹ̎ͭͷΞϓϩʔν͋Γ - ίϯϓϥΠΞϯεϕʔε vs ϦεΫϕʔε - ຊߨٛͷ࠲ֶίϯϓϥΠΞϯεΑΓ - ϫʔΫγϣοϓϦεΫϕʔεΑΓ
- ੬ऑੑͷಛఆ - ڴҖͷಛఆ - ϦεΫʢൃੜ֬ʣΛੳ ϦεΫੳ -
- “ ιϑτΣΞ͓ΑͼϋʔυΣΞͷίϯϙʔωϯτʹଘࡏ͢Δɺܭࢉϩ δοΫʢྫ: ίʔυʣ্ͷऑͰ͋Γɺѱ༻͞Εͨ߹ʹػີੑɺશੑɺ ·ͨՄ༻ੑʹѱӨڹΛٴ΅͢ͷ” - ҰํɺγεςϜϦεΫΞηεϝϯτʹ͓͚Δ੬ऑੑඞͣ͠ʮܭࢉϩ δοΫʯʹݶΒͳ͍ -
ͦͷҙຯͰNCSCͷVulnerability managementͷ͕ΑΓจ຺ʹ߹ͬͯΔ - NCSC: ӳࠃͷαΠόʔηΩϡϦςΟہ ੬ऑੑͱ https://www.cve.org/Resources/Roles/Cnas/CNA_Rules_v3.0.pdf -
- “੬ऑੑͱɺITγεςϜʹଘࡏ͢ΔऑͰ͋Γɺ߈ܸऀ͕ ߈ܸΛޭͤ͞ΔͨΊʹѱ༻͢ΔՄೳੑͷ͋ΔͷͰ͢ɻ ੬ऑੑɺγεςϜͷܽؕɺҙਤ͞Εͨػೳɺ·ͨϢʔ βʔͷաࣦʹΑͬͯൃੜ͠ɺ߈ܸऀ࠷ऴతͳతΛୡ ͢ΔͨΊʹɺ͜ΕΒͷ͏ͪҰͭ·ͨෳΛΈ߹Θͤͯ ѱ༻ΛࢼΈ·͢ɻ” ੬ऑੑͱ by NCSC
https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities -
- “੬ऑੑͱɺITγεςϜʹଘࡏ͢ΔऑͰ͋Γɺ߈ܸऀ͕ ߈ܸΛޭͤ͞ΔͨΊʹѱ༻͢ΔՄೳੑͷ͋ΔͷͰ͢ɻ ੬ऑੑɺγεςϜͷܽؕɺҙਤ͞Εͳ͍ػೳɺ·ͨ ϢʔβʔͷաࣦʹΑͬͯൃੜ͠ɺ߈ܸऀ࠷ऴతͳతΛ ୡ͢ΔͨΊʹɺ͜ΕΒͷ͏ͪҰͭ·ͨෳΛΈ߹Θ ͤͯѱ༻ΛࢼΈ·͢ɻ” ੬ऑੑͱ by NCSC
https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities ຊߨٛͰɺʮҙਤ͞ΕͨػೳʯʮϢʔβʔͷաࣦʯجຊɺ ࣄલఆ͞Εͳ͔ͬͨʮܽؕʯͰ͋Δલఏͱͯ͠ѻ͏ɻͦ͠ ͯɺʮܽؕʯͰ͋Δͱ͍͏͜ͱ TobeͱAsIsͷDiffͰ͋Δ -
- େ͖͚ͯ͘ɺίϯϓϥΠΞϯεϕʔε vs ϦεΫ ϕʔε - ຊষʢ੬ऑੑͷষʣجຊతʹલऀΛఆ - ϦεΫϕʔεڴҖΛߟ͑ͳ͕ΒɺӨڹͷߴ͍ Օॴʹूத͢Δͷ͕٢
੬ऑੑͷચ͍ग़͠ํ https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities -
- ʮܽؕʯͰ͋Δͱ͍͏͜ͱ TobeͱAsIsͷDiffͰ͋ Δ - Q: ToBeͱʁ - A: ඪ४ɻϕεϓϥ
- ʮܽؕʯͰ͋Δͱ͍͏͜ͱ ཧͱݱঢ়ͷDiffͰ͋Δ ੬ऑੑͷચ͍ग़͠ํ - ToBeͷચ͍ग़͠ https://www.ncsc.gov.uk/collection/vulnerability-management/understanding-vulnerabilities -
- Ұൠత: CIS Benchmarks - ΫϥυಛԽ: CSAΨΠμϯε - ϕϯμʔಛԽ: AWS
Security Baseline - ۀքಛԽ: FISC ۚ༥ػؔίϯϐϡʔλγεςϜͷ҆શରࡦج४ - ߦ: ۚ༥ʹ͓͚ΔαΠόʔηΩϡϦςΟʹؔ͢ΔΨΠυϥΠϯ - ଞ: ࣗࣾάϧʔϓ༻ ੬ऑੑͷચ͍ग़͠ํ - ඪ४ྫ -
- “γεςϜ৫ʹର͠ɺΛ༩͑Δ·ͳ͍ΠϯγσϯτΛൃੜ͢Δ જࡏతݪҼ” - `ڴҖΠϯςϦδΣϯε`ɹ `TLPTʢڴҖϕʔεϖωτϨʣ` ͳͲΘ ΕΔίϯςΫετ͕ଟذʹΘͨΓ࢝ΊͨͨΊɺ͔ͳΓ͏໘ʹΑͬ ͯΏΒ͕͗͋Δʢࢲݟʣ -
ͳͷͰɺ͜Ε͕ඪ४ʂͱ͍͏ͷ͋·Γͳ͍ʢࢲݟʣ - ͳͷͰɺຊίʔεͰͦ͜·Ͱਂ΅Βͳ͍ ڴҖͱ https://www.iso.org/obp/ui/#iso:std:iso-iec:27000:ed-5:v1:en -
ϫʔΫγϣοϓ - ηΩϡϦςΟࢹͱɺͦͷσʔλγεςϜʹ͓͚Δ੬ऑੑͱڴҖΛߟ͑ͯΈΑ͏ - ใࢿ࢈ͷચ͍ग़͠ - ใγεςϜͷચ͍ग़͠ - ੬ऑੑͷચ͍ग़͠ ※ඪ४Ұ୴ߟ͑ͳ͓̺ͯ͘Ͱ͢
- ڴҖͷચ͍ग़͠ - ͬ͘͟Γ̍ݸͣͭͰ͍͍͔ͳ… - ϫʔΫγϣοϓ: 15min - ൃද: ͳ͠ -
ϦεΫධՁ - ϦεΫੳ: ϦεΫ = ੬ऑੑ x ڴҖ - ӨڹͱϦεΫʢՄೳੑʣ͔ΒϦε
ΫΛӈਤΛͬͯධՁ - ྫ: Өڹ͕ߴ͘ɺϦεΫ͕͍ ߹ɺอݥͷϦεΫҠస͕ߟ͑Β ΕΔ ڴҖΠϯςϦδΣϯεͷڭՊॻ p.375 -
- Πϯτϩ: 10min (08:30~08:40) - ຊߨٛͷഎܠ: 15min (08:40~08:55) - σʔλཧ:
40min (08:55~09:35) - ٳܜ݉ηοτΞοϓϔϧϓ: 15min (09:35~09:50) - σʔλΣΞϋε: 40min (09:50~10:30) - Apache Iceberg: 50min (10:30~11:20) - ٳܜ: 10min (11:20~11:30) - γεςϜϦεΫΞηεϝϯτ: 40min (11:30~12:10) - ·ͱΊ: 5min (12:10~12:15) λΠϜςʔϒϧ -
- ඪ४తͳΧϦΩϡϥϜͳͲଘࡏ͠ͳ͍ - ಡΈΛࢭΊͯɺ࣮ફ͋ΔͷΈ - ίʔυΛॻ͚ - ίʔυΛյͤ - ࣝΛڞ༗ͤΑ
- ίϛϡχέʔγϣϯΛຏ͚ - ϋʔυϫʔΫͯ͠ɺͨ·ʹࣦഊ͠Α͏ - ʢͰ͖Δ͚ͩʣָ؍తͰ͍Α͏ - ॿ͚ΛٻΊΑ͏ So, you want to work in security? https://www.freecodecamp.org/news/so-you-want-to-work-in-security-bc6c10157d23/ ”Parisa Tabriz ΞϝϦΧਓͷΤϯδχΞɺίϯϐϡʔλ ηΩϡϦςΟͷઐՈͰ͋Γɺ Google Ͱ Google Chrome ͷ෭ࣾ݉θωϥϧ ϚωʔδϟʔΛΊΔװ෦Ͱ͢ɻ൴ঁઐ తʹʮηΩϡϦςΟϓϦϯηεʯ”by https://en.wikipedia.org/wiki/Parisa_Tabriz