Share
Embed
DeNA流データエンジニアリングの極意
Developers Summit 2017 における「DeNA の機械学習基盤と分析基盤」の発表から「データエンジニアリングの極意」のパートを抜き出したもの。「機械学習基盤」のパートについては https://speakerdeck.com/sonots/dena-aisisutemubu-niokerukuraudowohuo-yong-sitaji-jie-xue-xi-ji-pan-falsegou-zhu を参照。
Naotoshi Seo
February 17, 2017
Transcript
42 slides
-
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ʲ#ʳ%F/"ͷ ػցֶशج൫ͱੳج൫ EFWTVNJ#
1 Feb 17, 2017 Developer Summit 2017 Naotoshi Seo AI System Division DeNA Co., Ltd. -
˞%F/"ͷػցֶशج൫ͱੳج൫ ͷࢿྉ͔Βੳج൫ͷύʔτ ʮσʔλΤϯδχΞϦϯάͷۃҙʯ Λൈ͖ग़ͨ͠ͷͰ͢ 2 Copyright (C) DeNA Co.,Ltd. All
Rights Reserved. ˞ػցֶशج൫ͷύʔτͪ͜Β IUUQTTQFBLFSEFDLDPNTPOPUTEFOB BJTJTVUFNVCVOJPLFSVLVSBVEPXPIVPZPOHTJUBKJKJF YVFYJKJQBOGBMTFHPV[IV -
ࣗݾհ 3 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ɾඌར!TPOPUT
ɾ"*γεςϜ෦ چੳج൫෦ ɾ"OBMZUJDT5PPMTνʔϜ $MPVE*OGSBνʔϜ ϦʔυΤϯδχΞ ɾ'MVFOUE3VCZίϛολ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 1
2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ -
σʔλϨΠΫͱσʔλΣΞϋε 5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. σʔλϨΠΫͱDWHʹ͚Δ͠
DB Log Data Lake ౷߹DWH Data Source શͯҰσʔλ ϨΠΫʹೖΔ ҙࢥܾఆͷͨΊʹ తผʹฤ͞Εɺ ౷߹͞Εͨ࣌ܥྻ σʔλϕʔε ※ ͞ΒʹΞϓϦέʔγϣϯ༻ʹ༻ҙͨ͠ Data MartΛ࣋ͭ৫͋Δׂ͕Ѫ -
6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. σʔλϨΠΫͱDWHʹ͚Δ͠ DB
Log Data Lake ౷߹DWH Data Source HDFS %F/"ͷ߹ Google BigQuery (BQʹҠߦத) -
7 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. σʔλϨΠΫͱDWHʹ͚Δ͠ •
ऩू͚ͩͰࠔ͕͋Δ • σʔλ͕͚͍ܽͯͳ͍͔ • ٯʹɺσʔλ͕ॏෳ͍ͯ͠ͳ͍͔ • ଞͷ͜ͱ·Ͱಉ࣌ʹߟ͑Δͷ͍͠ • ϑΥʔϚοτͷ౷Ұ • ΧϥϜ໊ͷ౷Ұ • σʔλόϦσʔγϣϯ • ΦϯϥΠϯεΩʔϚมߋ • ूΊΔ͜ͱ͚ͩʹઐ೦͢Δ • શͯͷσʔλͦ͜ʹ͋Δ ͱʹ͔͘σʔλϨΠΫʹूΊΔ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
ετϦʔϛϯάऩू ͱ όονऩू 9 Copyright (C) DeNA Co.,Ltd. All Rights
Reserved. ετϦʔϛϯάऩूʹόοϑΝΛઃ͚Δ͠ -
10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ετϦʔϛϯάऩूͱόονऩू •
ϦΞϧλΠϜʹ͍ۙ ετϦʔϛϯάऩू • ӡ༻͕͍͠ • ϓϩηεΛσʔϞϯԽ͠ৗறԽͤ͞ΔͨΊϝϞ ϦϦʔΫͷΛҾ͖ى͍͜͢͠ • Ұ෦ܽଛͨ͠߹ͷ࠶సૹ͕ࠔ • ߴτϥϑΟοΫͳ࣌ؒͰͷෛՙ্ঢʹΑΔసૹ Ԇͷൃੜ ϦΞϧλΠϜੑͷ୲อ ❌ܽ ⭕ར -
11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ετϦʔϛϯάऩूͱόονऩू όονऩू
• ϓϩηε͕ৗற͠ͳ͍ͨΊɺϝϞϦϦʔΫॏେͳ ͱͳΒͣπʔϧͷ࣮͕༰қ • σʔλͷऔΓ͠Ұൠతʹ࠶࣮ߦ͢Δ͚ͩͰ༰қ • ϦΞϧλΠϜੑ͕ͳ͍ • ϚγϯϦιʔεΛ࠷େʹར༻͠Α͏ͱ͢ΔͨΊɺଞ ͷγεςϜʹӨڹΛ༩͑ͯ͠·͏͜ͱ͕͋Δ • ॲཧ͕ऴΘͬͨΒશ͘ϦιʔεΛΘͳ͘ͳΔ %BJMZ͘͠)PVSMZͰόονπʔϧΛىಈ ⭕ར ❌ܽ -
%F/"ͰͷετϦʔϛϯάͱόονͷ͍͚ 12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ετϦʔϛϯάऩूͱόονऩू
• ϩάऩू ετϦʔϛϯά όον • .Z42-ʹ֨ೲ͞Ε͍ͯΔϚελσʔλͷऔΓࠐΈ • σʔλసૹ )%'4͔Β#2ͷऔΓࠐΈ FUD -
13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Web Web
Web Cassandra Storm HBase ログコレクタ Hadoop HDFS ログAPI イベント ログ %F/"ͷϩάऩूج൫ • qVFOUE ·ͨIBEPPQMPHEBFNPOͱ͍ ͏πʔϧ ͕ϩά"1*ʹσʔλΛૹΔ • )%'4ʹ֨ೲ͞ΕΔ ετϦʔϛϯάऩूͱόονऩू -
14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DB (MySQL)
Sqoop Importdb MapReduce ジョブ Map HDFS Map HDFS Map HDFS 起動 *NQPSUECϚελσʔλͷόονऩू • ϢʔβใͳͲͷϚελσʔλΛ࣋ͭαʔϏεͷσʔ λϕʔε .Z42- ͔Β࣌Ͱऔಘ • 4RPPQΛϕʔεʹͨ͠*NQPSUECπʔϧɺ .BQ3FEVDFͰ)%'4ʹσʔλΛ֨ೲ ετϦʔϛϯάऩूͱόονऩू -
Rails (Web UI) Embulk Sidekiq Worker Hadoop HDFS Vertica アナリスト
設定情報 (MySQL) ジョブキュー (Redis) Embulk Sidekiq Worker 15 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. .FEKFEόονͰσʔλసૹ • )%'4ͷσʔλΛ7FSUJDBPS#JH2VFSZʹҠ͢ • &NCVMLΛόοΫΤϯυʹͨ͠3BJMTΞϓϦ έʔγϣϯ.FEKFEͰσʔλΛసૹ ετϦʔϛϯάऩूͱόονऩू -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 16 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
17 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Web Web
Web Cassandra Storm HBase ログコレクタ Hadoop HDFS ログAPI イベント ログ %F/"ͷϩάऩूج൫ • $BTTBOESB͕όοϑΝϨΠϠʔ • ࡉ͔͘ॻ͖ࠐΉͱɺ*0ෛՙ͕ߴ·Γɺ·ͨஅยԽ ͕ى͜Γ͍͢ • ૹ৴ଆ qVFOUE ʹόοϑΝϨΠϠʔ͕͋Γɺ σʔλΛ·ͱΊͯૹΔ͜ͱͰޮԽ͍ͯ͠Δ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 18 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
19 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Retryableʹ͢͠ 3
3FUSZBCMFʹ͢͠ • खಈͰ࠶࣮ߦͨ͠߹ͷߟ͑ࣄΛগͳ͘ • ࣗಈϦτϥΠΛೖΕΒΕΔΑ͏ʹ -
20 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Retryableʹ͢͠ 3
3FUSZBCMFʹ͢Δۃҙ ᶃ ͖ʹ͢͠ ᶄ BUPNJDʹ͢͠ ᶅ VVJEͰॏෳআ֎͢͠ -
21 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶃ͖ʹ͢͠ •
ϦτϥΠͰಉ͡σʔλΛೋॏʹೖΕͯ͠·͏ Retryableʹ͢͠ ❌ϦτϥΠػೳͰΑ͘ൃੜ͢Δ ⭕ରॲ๏ ͋Δૢ࡞Λ1ճߦͬͯෳճߦͬͯ݁Ռ͕ಉ͡Ͱ͋Δ͜ͱ ႈੑ • هͰͳ্͘ॻ͖͕ݪଇ • ྫ #2ͷ߹83*5&@"11&/%Ͱͳ͘83*5&@536/$"5& • ͦΕͰه͍ͨ͠߹ϢχʔΫͳKPC@JEΛৼΓɺϦτϥ Π͕ͬͯॏʹ࣮ߦ͞Εͳ͍Α͏ʹ͢Δ -
22 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. FNCVMLPVUQVUCJHRVFSZ ࡞
• BQQFOEϞʔυͱSFQMBDFϞʔυ • 83*5&@"11&/%ΛͬͨBQQFOEϞʔυ • 83*5&@536/$"5&ΛͬͨSFQMBDFϞʔυ • SFQMBDFϞʔυΛ͏‷ ŋŷŋ و ̑̑ • ෦తʹVOJRVFͳKPC@JEΛੜ • #JH2VFSZଆͰෆҙʹϦτϥΠ͕ͬͯσʔλ ͕ॏෳ͠ͳ͍Α͏ʹ͍ͯ͠Δ ͖ʹ͢͠ -
23 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶄBUPNJDʹ͢͠ •
్த·Ͱ͔͠σʔλ͕֨ೲ͞Ε͍ͯͳ͍ঢ়ଶͰޙଓ ͷδϣϒΛ࣮ߦͯ͠͠·͏ • ϦτϥΠ͢Δͱɺ֨ೲઌσʔλ͕Ұ࣌తʹফ͞Εͯ ͠·͍ɺσʔλ͕ଘࡏ͍ͯ͠ͳ͍ॠ͕ؒൃੜ͢Δ τϥϯβΫγϣϯʹؚ·ΕΔλεΫ͕શ࣮ͯߦ͞ΕΔ ͔ɺ͋Δ͍શ࣮͘ߦ͞Εͳ͍͜ͱΛอূ͢Δੑ࣭ Retryableʹ͢͠ atomicity ❌Α͘ൃੜ͢Δ -
24 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. %F/"ͷ߹ •
BUPNJDͳૢ࡞ΛετϨʔδ͝ͱʹௐࠪ • )%'4 • UNQσΟϨΫτϦʹσʔλΛॻ͍ͯશͯྃ ͨ͠ΒSFOBNF • 7FSUJDB • UNQςʔϒϧʹσʔλΛॻ͍ͯɺ"-5&3 5"#-&3&/".&50ͰTXBQ • #JH2VFSZ • UNQςʔϒϧʹσʔλΛॻ͍ͯɺDPQZKPC atomicʹ͢͠ -
25 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶅVVJEͰॏෳআ֎͢͠ •
ଓ͕ෆҙʹΕͯϦτϥΠ͕Γ͕ͪ • ಉ͡σʔλ͕ೋॏʹೖͬͯ͠·͏ Retryableʹ͢͠ ❌ΫϥΠΞϯτϩάͰΑ͘ൃੜ͢Δ • ϩάʹVVJEͳΧϥϜΛઃ͚ͯσʔλॏෳআ֎͢Δ ⭕ରॲ๏ -
26 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Web Web
Web Cassandra Storm HBase ログコレクタ Hadoop HDFS ログAPI イベント ログ %F/"ͷ߹ • ΫϥΠΞϯτϩάߦͣͭʹVVJE • ϩάऩूج൫Ͱॏෳআڈ • %8)Ͱ%*45*/$5 Ͱॏෳআڈ uuid Ͱॏෳআ֎͢͠ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 27 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
28 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 7BMJEBUJPO͢͠ •
ͳʹΤϥʔ͕ى͜ΒͣͲ͔͜Ͱσʔλ͕ /6--ʹมΘ͍ͬͯͨ • ྫ %8)ͷΧϥϜܕΛJOUܕʹ͍͕ͯͨ͠ɺ σʔλϨΠΫʹ ͷΑ͏ͳσʔλ͕ೖΔ͜ ͱ͕͋ͬͨ ❌Α͘ൃੜ͢Δ • 7BMJEBUJPOͯ͠ૣΊʹؾͮ͘͠ ⭕ରॲ๏ Validation͢͠ 4 -
29 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Spreadsheet্ͷςʔϒϧఆٛΛWeb UI
͔ΒऔΓࠐΈɺϝλใͱͯ͠ѻ͏ SpreadsheetͰఆٛ͞Εͨ ςʔϒϧఆٛॻ .FEKFE(44ͰΧϥϜͷܕఆٛ Validation͢͠ औΓࠐΈ࣌ʹܕνΣοΫΛߦ͏ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 30 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
31 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 42-Ͱੳ͢͠ ❌υϦϧΞοϓɺυϦϧμϯͳͲ42-ࣗಈੜ
ܥͷ#*πʔϧͷ • ෦Ͱෳࡶ͕ΫΤϦ͕࡞ΒΕͯҟৗʹॏ͍ • πʔϧ͕αϙʔτ͓ͯ͠ΒͣΓ͍ͨ͜ͱ͕ Ͱ͖ͳ͍߹͕͋Δ ⭕ੳऀ͕ࣗͰ42-Λهड़͢Δ͠ • ѹతͳࣗ༝ SQLͰੳ͢͠ 5 -
32 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. "SHVTϨϙʔτπʔϧ 1.
ΫΤϦΛॻ͘ 2. νϟʔτԽͷઃ ఆΛߦͳ͏ 3. Ϩϙʔτ͕ ग़དྷΔʂ ΞʔΨε ͘Μ! SQLͰੳ͢͠ -
σʔλϨΠΫͱ%8)ʹ͚Δ͠ 34 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DeNAྲྀσʔλΤϯδχΞϦϯάͷۃҙ
1 2 ετϦʔϛϯάऩूʹόοϑΝΛ ઃ͚Δ͠ 3 3FUSZBCMFʹ͢͠ 4 7BMJEBUJPO͢͠ 5 42-Ͱੳ͢͠ 6 ࠶औΓࠐΈΛ༰қʹ͢͠ -
35 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࠶औΓࠐΈΛ༰қʹ͢͠ •
࠶औΓࠐΈසൟʹൃੜ͠͏Δ • σʔλԆ • σʔλෆਖ਼ • औΓࠐΈࣦഊ #JH2VFSZ͕ෆ҆ఆ • ༰қʹͰ͖Δ͜ͱͰӡ༻ָ͕ʹͳΔ ࠶औΓࠐΈΛ༰қʹ͢͠ 6 -
36 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࠶औΓࠐΈΛ༰қʹ͢Δۃҙ ࠶औΓࠐΈΛ༰қʹ͢͠
6 ᶃ 3FUSZBCMFʹ͢͠ • ͖ʹ͢͠ • BUPNJDʹ͢͠ • VVJEͰॏෳഉআ ᶄ ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢͠ ᶅ σʔλൃੜ࣌ࠁͰσʔλΛ͚Δ͠ ᶆ σʔλͷͭͳ͕ΓΛཧ͢Δ͠ -
37 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶄ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢͠ •
࣌ؒύϥϝʔλΛEBUFίϚϯυ/08 ͷΑ ͏ͳ42-ؔͰࢦఆ • ࣮ߦλΠϛϯάͰύϥϝʔλ͕ܾ·ͬͯ͠·͍ɺ࠶ ࣮ߦͰ͖ͳ͍ ❌Α͋͘Δ • σϑΥϧτͱͯ͠ͷ/08 Ҏ֎ʹɺ࠶࣮ߦ ࣌ʹɺ࣌ؒύϥϝʔλΛࢦఆͰ͖ΔΑ͏ʹ͓ͯ͘͠ ⭕ରॲ๏ ࠶औΓࠐΈΛ༰қʹ͢͠ -
38 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢͠ ϘλϯҰͭͰςʔϒϧ
࡞/ߋ৽ Λࢦఆͯ͠ ࠶औΓࠐΈ࣮ߦ ΧϨϯμʔͰऔΓࠐΈ ঢ়گΛ֬ೝग़དྷΔ औΓࠐΈͷৄࡉใ͕ ֬ೝग़དྷΔɻ .FEKFE8FC6*͔Β࠶࣮ߦ ࠶औΓࠐΈΛ༰қʹ͢͠ -
39 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶅσʔλൃੜ࣌ࠁͰσʔλΛ͚Δ͠ •
σʔλൃੜ࣌ࠁͰσΟϨΫτϦΛׂ • ϩάൃੜ࣌ࠁɺߦߋ৽࣌ࠁ ·ͨऔΓࠐΈ࣌ࠁ • ಛఆσΟϨΫτϦͷΈͷ࠶औΓࠐΈ‷ ŋŷŋ و ̑̑ • )%'4ͷσΟϨΫτϦΛ࣌Ͱ͚͍ͯͳ͍ • ࠶औΓࠐΈ͕શσʔλऔΓࠐΈʹͳΔ • σʔλճऩ࣌ࠁͰσΟϨΫτϦΛׂ • ੳରͷσʔλΛ࠶औΓࠐΈ͢Δͷʹ ϑΟϧλϦϯάͱϚʔδ͕ඞཁʹͳΔ ❌Α͋͘Δ ⭕ରॲ๏ ࠶औΓࠐΈΛ༰қʹ͢͠ -
40 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. %F/"ͷ߹ •
σʔλൃੜ࣌ࠁͰσʔλׂ • )%'4:NEͰσΟϨΫτϦׂ • 7FSUJDBE EBUF ΧϥϜͰׂ • #JH2VFSZ@:NEαϑΟοΫεͰςʔϒ ϧׂ͢Δ͔:NEͰύʔςΟγϣϯ σʔλൃੜ࣌ࠁͰσʔλΛ͚Δ͠ -
41 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. ᶆσʔλͷͭͳ͕ΓΛཧ͢Δ͠ •
࠶࣮ߦ͍͕ͨ͠ɺͲͷδϣϒ͕Ͳͷσʔλʹґଘ͠ ͍ͯΔͷ͔Α͔͘Βͳ͍ • δϣϒΛఆٛͨ͠ਓ͔͠Θ͔Βͳ͍ଐਓঢ়ଶ ࠶औΓࠐΈΛ༰қʹ͢͠ ❌Α͋͘Δ ⭕ରॲ๏ • ϫʔΫϑϩʔπʔϧ+FOLJOTύΠϓϥΠϯͰ ґଘؔΛཧ͢Δ -
42 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • σʔλϨΠΫͱ%8)ʹ͚Δ͠
• ετϦʔϛϯάऩूʹόοϑΝΛઃ͚Δ͠ • 3FUSZBCMFʹ͢͠ • ͖ʹ͢͠ • BUPNJDʹ͢͠ • VVJEͰॏෳഉআ͢͠ • 7BMJEBUJPO͢͠ • 42-Ͱੳ͢͠ • ࠶औΓࠐΈΛ༰қʹ͢͠ • 3FUSZBCMFʹ͢͠ • ࣌ؒύϥϝʔλͷࢦఆΛՄೳʹ͢͠ • σʔλൃੜ࣌ࠁͰσʔλΛ͚Δ͠ • σʔλͷͭͳ͕ΓΛཧ͢Δ͠ ·ͱΊ 1 2 3 4 5 6