Share
Embed
TD Tech Talk 2018: DeNA AI&Analytics Infrastructure
#tdtech https://techplay.jp/event/669345
Hidenori Matsuki
May 22, 2018
Transcript
45 slides
-
© DeNA Co., Ltd. May 22, 2018 TD Tech Talk
2018 Hidenori MATSUKI AI System Division DeNA Co., Ltd. ʹ͚ͨ %F/"ͷੳج൫ © DeNA Co., Ltd. -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ࣗݾհ দ लݑ DeNA γεςϜຊ෦ AIγεςϜ෦
AI&ੳج൫Λ୲ [ུྺ] ΤϯλʔϓϥΠζ։ൃΛܦݧޙɺΑΓར༻ऀʹۙ ͍αʔϏεʹܞΘΔͨΊɺεϚʔτϑΥϯήʔϜ ͷόοΫΤϯυ։ൃʹैࣄɻ ͦͷޙେنࢄج൫ͷߏஙӡ༻ܦݧͱελʔτ ΞοϓͰͷ։ൃϚωδϝϯτܦݧΛܦͯDeNAʹ ೖࣾɻ AI&ੳج൫ͷߏஙͱϚωδϝϯτΛߦ͏ɻ 2 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾհ DeNAͱAIγεςϜ෦ͷ͝հ DeNAੳج൫ͷ"ࠓ" ⁃ DeNAੳج൫ͷಛ
⁃ ੳج൫ͷγεςϜߏ ⁃ ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020ʹ͚ͯ ~ AI&ੳج൫ͱͯ͠ ~ 3 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. ) -
Game Automotive EC Sports New! Entertainment Other Health care ©
DeNA Co., Ltd. ࣄۀϙʔτϑΥϦΦ 5 • ଟछଟ༷ͳࣄۀྖҬͰϏδωεɾαʔϏεల։ • ύʔτφʔاۀͱڞʹࣾձӨڹͷେ͖ͳαʔϏεΛग़ • େنΠϯλʔωοταʔϏεϓϥοτϑΥʔϜͷߏஙɾӡ༻ྗ -
© DeNA Co., Ltd. DeNAͰͷσʔλ׆༻ 6 AI Service Data ࠷৽ݚڀΛϕʔεʹݚڀ։ൃ
AIΛ׆༻ͨ͠৽ͨͳՁ͋Δ αʔϏεઃܭɾ࣮ αʔϏεɾσʔλͷαΠΫϧ͔Β ੜ·ΕΔڧݻͳϏδωεߏ -
© DeNA Co., Ltd. AIγεςϜ෦ͷׂʢੳฤʣ 7 DeNA Customers Game Analyst
AIγεςϜ෦ AI&ੳج൫ ITPF CERT ΞφϦετ͕༰қʹࣗ༝ʹੳͰ͖ΔΠϯϑϥΛߏஙӡ༻͢Δ Sports Analyst (other) Analyst AIݚڀ։ൃ (other) -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾհ DeNAͱAIγεςϜ෦ͷ͝հ DeNAੳج൫ͷ"ࠓ" ⁃ DeNAੳج൫ͷಛ
⁃ ੳج൫ͷγεςϜߏ ⁃ ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020ʹ͚ͯ ~ AI&ੳج൫ͱͯ͠ ~ 8 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. ) -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNAੳج൫ͷಛ • 2010ʹHadoopʹΑΔେنσʔλੳج൫Λߏஙɺӡ༻։࢝ • ͋ΒΏΔαʔϏεͰϩάΛੵ͠ੳ͢Δ͜ͱ͕ͨΓલ •
αʔϏε։ൃͱฒߦͯ͠ϩάઃܭΛߦ͏ • ͍ΘΏΔETL෦͕ఆܕԽ͞Ε͓ͯΓҰఆͷ࣭Ͱੳ͕ߦ͑Δ • ͚ࣾϩάϑΥʔϚοτ͕ఆٛ͞Ε͍ͯΔ • PBΫϥεͷσʔλྔͱ60Ҏ্ͷར༻෦/αʔϏε • ήʔϜʹݶΒͣBtoC/BtoBଟ࠼ͳࣄۀͷੳσʔλΛѻ͏ • σʔλΞφϦετΛத৺ʹར༻ऀ͕ࣾʹวࡏ • Ϗδωε৬͍ΘΏΔόοΫΦϑΟεͷํΫΤϦΛ͛Δ 10 -
ੳج൫ ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNAੳج൫ͷશମ૾ 12 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫ DeNAੳج൫ͷશମ૾ 13 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫: App -> LogGateway • ΞϓϦέʔγϣϯϩά
• ΞϓϦέʔγϣϯαʔόʔͰൃੜͨ͠ϩάΛFluentdͰऩू • https://github.com/civitaspo/fluent-plugin-cat-sweep • ϩάதܧαʔόʔΛհͯ͠શࣾڞ௨ͷLogGatewayʹHTTPS POST • ೝূͨ͠αʔϏε͔ΒͷΈϩάΛड͚औΔ • DBεφοϓγϣοτ • εφοϓγϣοτऩूαʔόʔ͔ΒLogGatewayʹHTTPS POST • DB͕ΦϯϓϨϛεͷ߹SqoopʹΑΔεφοϓγϣοτऩू 14 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Fluentdͷ͕͜͜ੌ͍ • མͪͳ͍ • མͪͯ࠶ىಈ͢ΕϩάΛ࠶ૹͯ͘͠ΕΔ •
ಋೖ࣌Ҏ֎ͷ͖᪴΄΅ͳ͠ • PluginʹΑΓॊೈʹΧελϚΠζͰ͖Δ • HTTP POST͢Δ߹ͳͲ • ετϦʔϜσʔλͷHubͱͯ͠ػೳ͢Δ • ࠷ऴతʹFluentdʹͨͲΓண͘༏ΕͨΞʔΩςΫνϟ 15 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫ DeNAੳج൫ͷશମ૾ 16 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫: LogGateway -> HDFS • 2018ݱࡏHadoopΛओʹData
Lakeͱͯ͠༻͍ͯ͠Δ • ͨͩ͠HiveQLΛ༻͍ͯ͠Δ෦/αʔϏε·ͩଘࡏ͢Δ • ͦͷͨΊHueΛఏڙ • LogGatewayʹू·ͬͨϩάΛҎԼΛߦ͍ͳ͕ΒHDFSʹॻ͖ࠐΉ • αʔϏε͝ͱͷྨ • ॏෳഉআ 17 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫ DeNAੳج൫ͷશମ૾ 18 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP Ver=ca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫: HDFS -> BigQuery/Vertica • 2018ݱࡏͷDeNAਪΫΤϦΤϯδϯGoogle
BigQuery • 2013ࠒHP VerticaಋೖɺݱࡏBigQueryॱ࣍Ҡߦத • 2013ҎલHiveQL • Bulk Loadπʔϧ"Medjed"ͰHDFS͔ΒΫΤϦΤϯδϯʹσʔλసૹ • Medjed = Web UI + Embulk manager • ΞφϦετ͕yamlΛॻ͔ͳͯ͘ࡁΉ • ΞφϦετ͕Google Spread SheetΛฤू͢Εyaml͕ੜ͞ΕΔ • ΞφϦετ͕SchemaཧΛߦ͑Δ • EmbulkࣗମෳͷWorkerϊʔυͰ࣮ߦ͞ΕΔͨΊεέʔϧՄೳ 19 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Bulk Loadπʔϧ"Medjed" 21 input / output
ͷఆٛΛGoogle Spread SheetͰߦ͑Δ -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Embulkͷ͕͜͜ศར • FluentdͷΑ͏ͳpluginΞʔΩςΫνϟ • pluginΛ༻/࣮͢Δ͜ͱͰ͋ΒΏΔsource/destinationʹରԠՄೳ •
https://github.com/medjed • https://github.com/sonots/embulk-output-vertica • https://github.com/sonots/embulk-input-vertica • ઃఆΛyamlͰॻ͚Δ • ϓϩάϥϛϯάݴޠͱਓؒ(ΤϯδχΞ)ʹ༏͍͠ • (ΞφϦετ͚ʹMedjedΛఏڙ) 22 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫ DeNAੳج൫ͷશମ૾ 23 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP Ver=ca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫: μογϡϘʔυ for BigQuery/Vertica • DeNAͰ2014ࠒ͔ΒμογϡϘʔυπʔϧ"Argus"Λ
• ݱࡏͷରԠΫΤϦΤϯδϯ: BigQuery, Vertica • ࠷ۙͷଟ͘ͷμογϡϘʔυಉ༷ΫΤϦΛॻ͍ͯάϥϑΛදࣔͰ͖Δ • ͷϝϦοτ • ͳͷͰࡉ͔ͳཁʹૉૣ͘Ԡ͑ΒΕΔ • 2018ݱࡏͰճ/िͷϦϦʔε • ࣾʹ߹Θͤͨࡉ͔ͳݖݶཧ͕ߦ͑Δ • ADάϧʔϓͱ࿈ܞͯ͠ཧ/ฤू/ӾཡݖݶΛ༩Ͱ͖Δ 24 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫ DeNAੳج൫ͷશମ૾ 27 Log Gateway Hadoop
Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ੳج൫: Job Scheduler/Workstation • Job Schedulerͱͯ͠JenkinsΛ༻
• Web UI͕ΞφϦετʹͱ͍͍ͬͯ͢ • େͷ͜ͱ͕Ͱ͖ͯ͠·͏ • ݖݶཧॊೈʹߦ͑Δ • ਖ਼ӡ༻େม • Web UIͰ݁͠ͳ͍ۀͷͨΊʹLinux WorkstationΛఏڙ • ΞφϦετ͕ϩάΠϯͰ͖ΔLinuxαʔόʔΛఏڙ͍ͯ͠Δ • Hadoop CLIͷ࣮ߦPythonεΫϦϓτͷ։ൃʹ༻͞Ε͍ͯΔ 28 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Digdagͱ͍͏༗ྗީิ • ಋೖ͍ͨ͠ϙΠϯτ • Embulkͱಉ͘͡yamlͰઃఆͰ͖Δ •
WorkflowΛίʔυͰཧͰ͖Δ • Shell/Ruby/PythonεΫϦϓτΛॻ͚Δ • Workflow, Taskͷࡉ͔͍ఆ͕ٛͰ͖Δ: ݅ذ, ґଘ, ܁Γฦ͠, etc. • HAͱฒྻॲཧ͕༰қ • ಋೖͰ͖ͣʹ͍Δཧ༝ • ݖݶཧ͕ߦ͍ͮΒ͍ 29 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ओͳӡ༻ۀ • ArgusʢμογϡϘʔυʣ/ MedjedʢBulk Loadπʔϧʣͷઃఆ •
BigQueryͷ伴ొ • ݖݶͷॳظઃఆ • Jenkinsͷઃఆ • ݖݶઃఆ • αʔόʔӡ༻ • ͍ΘΏΔී௨ͷΠϯϑϥӡ༻ • ֤λεΫGitHub IssueͰཧ • Issueݪଇશैۀһ͔ΒӾཡՄೳ 31 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ओͳαϙʔτۀ • ࣾͷੳج൫ར༻ऀ͚ٕज़αϙʔτ • ϙϦγʔʮੳج൫ΑΖͣ૯ड͚ʯ •
ੳج൫ʹ͍ͭͯͷ૬ஊͰ͋Εʮ·ͣड͚Δʯ • αϙʔτରओʹΞφϦετ • ͔͠͠ΞφϦετ৬Ͱͳ͍ਓ͔Βͷ߹ͤ͋Δ • 2018ݱࡏSlackͰͷαϙʔτ͕7ׂ΄Ͳ • ͍࣍Ͱର໘ϛʔςΟϯάɺϝʔϧ 32 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. SlackʹΑΔαϙʔτํ • ୭ͰೖΕΔαϙʔτઐ༻Slack WorkspaceΛ༻ҙ͍ͯ͠Δ • DeNAͰશैۀһ͕Slack
Enterprise GridΞΧϯτΛ͍࣋ͬͯΔ • ࠔͬͨΒmentionͯ͠Β͏ @support ͱ͍͏άϧʔϓΛ༻ҙ͍ͯ͠Δ • @support ʹAI&ੳج൫ϝϯόʔؚΊෳ໊͕ొ͞Ε͍ͯΔ 33 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNAͷAIج൫ • AWS/GCPʹΑΔػցֶशج൫Λߏங/ӡ༻ • GPUΠϯελϯεΛ͍ͭͰඞཁͳ͚ͩ •
Infra as CodeΛపఈ͍ͯ͠Δ • Terraform • Itamae ( RubyϓϩϏδϣχϯάπʔϧ ) • ϓϩδΣΫτൃ͔ΒظؒͰΠϯϑϥఏڙͰ͖͍ͯΔ • ֤छਃؚΊͯ࠷1िؒ 36 -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾհ DeNAͱAIγεςϜ෦ͷ͝հ DeNAੳج൫ͷ"ࠓ" ⁃ DeNAੳج൫ͷಛ
⁃ ੳج൫ͷγεςϜߏ ⁃ ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020ʹ͚ͯ ~ AI&ੳج൫ͱͯ͠ ~ 38 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. ) -
© DeNA Co., Ltd. ͜Ε͔ΒͷDeNAੳج൫͕ࢦ࢟͢ • ΫϥυωΠςΟϒ • αʔϏεͷΫϥυԽʹ߹Θͤͯੳج൫ΫϥυԽ •
ಉ࣌ʹΞφϦετʹطଘͱಉͷUXΛఏڙ • ੳج൫ʹཹ·Βͳ͍AI&ੳج൫ • ੳͷߴԽͱAIٕज़ͷಋೖͷͨΊศརͳUXΛఏڙ • σʔλੳ / AI / σʔλαΠΤϯε / Kaggle ѻ͏σʔλͷଟ༷Խ • ΑΓ҆શͳݖݶ • AWSΞΧϯτ/GCPϓϩδΣΫτࣗମΛσʔλͷൿಗੑʹԠͯ͡ 40 -
طଘੳج൫ © DeNA Co., Ltd. ΫϥυωΠςΟϒͳੳج൫ͷྫ 41 Argus Dashboard Medjed
+Embulk Jenkins & WorkstaIon Analyst Web UI Web UI & SSH Service Google BigQuery Log DB Snapshot αʔϏε͕Ϋϥυ্ʹ͋Δ ߹ɺ͋͑ͯΦϯϓϨϛεʹ Data LakeΛ࣋ͭඞཁͳ͍ɻ ҰํɺΞφϦετ׳ΕͨUX ͰੳΛߦ͍͍ͨɻ -
© DeNA Co., Ltd. Amazon SageMaker ( Jupyter Notebook )
43 Web UI্ͰPythonίʔυ͕࣮ߦͰ͖Δ -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ϚϧνϝσΟΞσʔλΛѻ͏AIج൫ͷྫ • ࣮Ҋ݅ͷAIٕज़ಋೖͰ੩ࢭը૾ಈը૾Λѻ͏έʔε͕૿͖͑ͯͨ • S3ͷϚϧνϝσΟΞσʔλೖΛτϦΨʔʹॲཧΛ࣮ߦ͢Δ •
ੳج൫ͱѻ͏σʔλॲཧͷཻҟͳΔ 44 GPUΠϯελϯε (ը૾/ಈըॲཧ) -
ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ͜Ε͔ΒͷDeNAੳج൫ • ΫϥυωΠςΟϒ • αʔϏεʹ߹Θͤͯੳج൫ΫϥυωΠςΟϒʹͳ͍ͬͯ͘ •
ੳج൫ʹཹ·Βͳ͍AI&ੳج൫ • ੳରͷσʔλ࣭ɾྔͱʹ૿͍͑ͯ͘ • ػցֶशΞϧΰϦζϜͷ࣮αʔϏεͰͷӡ༻Λࢧ͑ΔML Ops͕ඞཁ • 1ͭͷϓϩδΣΫτͰAIٕज़͏͠ੳߦ͏ • ʮAI͔ੳʯͰͳ͘ʮAIੳʯߦ͑Δج൫͕ॏཁ • ΑΓ҆શͳݖݶ • ѻ͏σʔλଟ༷ʹͳ͖͍ͬͯͯΔ • దʹݖݶͨ͠ΑΓηΩϡΞͳڥߏங͕ॏཁ 45