Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TD Tech Talk 2018: DeNA AI&Analytics Infrastruc...

TD Tech Talk 2018: DeNA AI&Analytics Infrastructure

Hidenori Matsuki

May 22, 2018
Tweet

More Decks by Hidenori Matsuki

Other Decks in Technology

Transcript

  1. © DeNA Co., Ltd. May 22, 2018 TD Tech Talk

    2018 Hidenori MATSUKI AI System Division
 DeNA Co., Ltd. ೥୅ʹ޲͚ͨ %F/"ͷ෼ੳج൫ © DeNA Co., Ltd.
  2. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ࣗݾ঺հ দ໦ लݑ DeNA γεςϜຊ෦ AIγεςϜ෦

    AI&෼ੳج൫Λ୲౰ [ུྺ] ΤϯλʔϓϥΠζ։ൃΛܦݧޙɺΑΓར༻ऀʹۙ ͍αʔϏεʹܞΘΔͨΊɺεϚʔτϑΥϯήʔϜ ͷόοΫΤϯυ։ൃʹैࣄɻ ͦͷޙେن໛෼ࢄج൫ͷߏஙӡ༻ܦݧͱελʔτ ΞοϓͰͷ։ൃϚωδϝϯτܦݧΛܦͯDeNAʹ ೖࣾɻ AI&෼ੳج൫ͷߏஙͱϚωδϝϯτΛߦ͏ɻ 2
  3. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾ঺հ DeNAͱAIγεςϜ෦ͷ͝঺հ DeNA෼ੳج൫ͷ"ࠓ" ⁃ DeNA෼ੳج൫ͷಛ௃

    ⁃ ෼ੳج൫ͷγεςϜߏ੒ ⁃ ෼ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020೥ʹ޲͚ͯ ~ AI&෼ੳج൫ͱͯ͠ ~ 3 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. )
  4. Game Automotive EC Sports New! Entertainment Other Health care ©

    DeNA Co., Ltd. ࣄۀϙʔτϑΥϦΦ 5 • ଟछଟ༷ͳࣄۀྖҬͰϏδωεɾαʔϏεల։ • ύʔτφʔاۀͱڞʹࣾձӨڹͷେ͖ͳαʔϏεΛ૑ग़ • େن໛ΠϯλʔωοταʔϏεϓϥοτϑΥʔϜͷߏஙɾӡ༻ྗ
  5. © DeNA Co., Ltd. DeNAͰͷσʔλ׆༻ 6 AI Service Data ࠷৽ݚڀΛϕʔεʹݚڀ։ൃ

    AIΛ׆༻ͨ͠৽ͨͳՁ஋͋Δ αʔϏεઃܭɾ࣮૷ αʔϏεɾσʔλͷαΠΫϧ͔Β ੜ·ΕΔڧݻͳϏδωεߏ଄
  6. © DeNA Co., Ltd. AIγεςϜ෦ͷ໾ׂʢ෼ੳฤʣ 7 DeNA Customers Game Analyst

    AIγεςϜ෦ AI&෼ੳج൫ ITPF CERT ΞφϦετ͕༰қʹࣗ༝ʹ෼ੳͰ͖ΔΠϯϑϥΛߏஙӡ༻͢Δ Sports Analyst (other) Analyst AIݚڀ։ൃ (other)
  7. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾ঺հ DeNAͱAIγεςϜ෦ͷ͝঺հ DeNA෼ੳج൫ͷ"ࠓ" ⁃ DeNA෼ੳج൫ͷಛ௃

    ⁃ ෼ੳج൫ͷγεςϜߏ੒ ⁃ ෼ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020೥ʹ޲͚ͯ ~ AI&෼ੳج൫ͱͯ͠ ~ 8 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. )
  8. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNA෼ੳج൫ͷಛ௃ • 2010೥ʹHadoopʹΑΔେن໛σʔλ෼ੳج൫Λߏஙɺӡ༻։࢝ • ͋ΒΏΔαʔϏεͰϩάΛ஝ੵ͠෼ੳ͢Δ͜ͱ͕౰ͨΓલ •

    αʔϏε։ൃͱฒߦͯ͠ϩάઃܭΛߦ͏ • ͍ΘΏΔETL෦෼͕ఆܕԽ͞Ε͓ͯΓҰఆͷ඼࣭Ͱ෼ੳ͕ߦ͑Δ • ࣾ಺޲͚ϩάϑΥʔϚοτ͕ఆٛ͞Ε͍ͯΔ • PBΫϥεͷσʔλྔͱ60Ҏ্ͷར༻෦໳/αʔϏε • ήʔϜʹݶΒͣBtoC/BtoBଟ࠼ͳࣄۀͷ෼ੳσʔλΛѻ͏ • σʔλΞφϦετΛத৺ʹར༻ऀ͕ࣾ಺ʹวࡏ • Ϗδωε৬΍͍ΘΏΔόοΫΦϑΟεͷํ΋ΫΤϦΛ౤͛Δ 10
  9. ෼ੳج൫ ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNA෼ੳج൫ͷશମ૾ 12 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  10. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫ DeNA෼ੳج൫ͷશମ૾ 13 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  11. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫: App -> LogGateway • ΞϓϦέʔγϣϯϩά

    • ΞϓϦέʔγϣϯαʔόʔͰൃੜͨ͠ϩάΛFluentdͰऩू • https://github.com/civitaspo/fluent-plugin-cat-sweep • ϩάதܧαʔόʔΛհͯ͠શࣾڞ௨ͷLogGatewayʹHTTPS POST • ೝূͨ͠αʔϏε͔ΒͷΈϩάΛड͚औΔ • DBεφοϓγϣοτ • εφοϓγϣοτऩूαʔόʔ͔ΒLogGatewayʹHTTPS POST • DB͕ΦϯϓϨϛεͷ৔߹SqoopʹΑΔεφοϓγϣοτऩू 14
  12. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Fluentdͷ͕͜͜ੌ͍ • མͪͳ͍ • མͪͯ΋࠶ىಈ͢Ε͹ϩάΛ࠶ૹͯ͘͠ΕΔ •

    ಋೖ࣌Ҏ֎ͷ͖᪴΄΅ͳ͠ • PluginʹΑΓॊೈʹΧελϚΠζͰ͖Δ • HTTP POST͢Δ৔߹ͳͲ • ετϦʔϜσʔλͷHubͱͯ͠ػೳ͢Δ • ࠷ऴతʹFluentdʹͨͲΓண͘༏ΕͨΞʔΩςΫνϟ 15
  13. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫ DeNA෼ੳج൫ͷશମ૾ 16 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  14. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫: LogGateway -> HDFS • 2018೥ݱࡏ͸HadoopΛओʹData

    Lakeͱͯ͠࢖༻͍ͯ͠Δ • ͨͩ͠HiveQLΛ࢖༻͍ͯ͠Δ෦໳/αʔϏε΋·ͩଘࡏ͢Δ • ͦͷͨΊHueΛఏڙ • LogGatewayʹू·ͬͨϩάΛҎԼΛߦ͍ͳ͕ΒHDFSʹॻ͖ࠐΉ • αʔϏε͝ͱͷ෼ྨ • ॏෳഉআ 17
  15. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫ DeNA෼ੳج൫ͷશମ૾ 18 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP Ver=ca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  16. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫: HDFS -> BigQuery/Vertica • 2018೥ݱࡏͷDeNA಺ਪ঑ΫΤϦΤϯδϯ͸Google

    BigQuery • 2013೥ࠒHP VerticaಋೖɺݱࡏBigQuery΁ॱ࣍Ҡߦத • 2013೥Ҏલ͸HiveQL • ಺੡Bulk Loadπʔϧ"Medjed"ͰHDFS͔ΒΫΤϦΤϯδϯʹσʔλసૹ • Medjed = Web UI + Embulk manager • ΞφϦετ͕yamlΛॻ͔ͳͯ͘ࡁΉ • ΞφϦετ͕Google Spread SheetΛฤू͢Ε͹yaml͕ੜ੒͞ΕΔ • ΞφϦετ͕Schema؅ཧΛߦ͑Δ • Embulkࣗମ͸ෳ਺ͷWorkerϊʔυͰ࣮ߦ͞ΕΔͨΊεέʔϧՄೳ 19
  17. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Embulkͷ͕͜͜ศར • FluentdͷΑ͏ͳpluginΞʔΩςΫνϟ • pluginΛ࢖༻/࣮૷͢Δ͜ͱͰ͋ΒΏΔsource/destinationʹରԠՄೳ •

    https://github.com/medjed • https://github.com/sonots/embulk-output-vertica • https://github.com/sonots/embulk-input-vertica • ઃఆΛyamlͰॻ͚Δ • ϓϩάϥϛϯάݴޠͱਓؒ(ΤϯδχΞ)ʹ༏͍͠ • (ΞφϦετ޲͚ʹ͸MedjedΛఏڙ) 22
  18. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫ DeNA෼ੳج൫ͷશମ૾ 23 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP Ver=ca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  19. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫: μογϡϘʔυ for BigQuery/Vertica • DeNAͰ͸2014೥ࠒ͔ΒμογϡϘʔυπʔϧ"Argus"Λ಺੡

    • ݱࡏͷରԠΫΤϦΤϯδϯ: BigQuery, Vertica • ࠷ۙͷଟ͘ͷμογϡϘʔυಉ༷ΫΤϦΛॻ͍ͯάϥϑΛදࣔͰ͖Δ • ಺੡ͷϝϦοτ • ಺੡ͳͷͰࡉ͔ͳཁ๬ʹૉૣ͘Ԡ͑ΒΕΔ • 2018೥ݱࡏͰ΋਺ճ/िͷϦϦʔε • ࣾ಺ʹ߹Θͤͨࡉ΍͔ͳݖݶ؅ཧ͕ߦ͑Δ • ADάϧʔϓͱ࿈ܞͯ͠؅ཧ/ฤू/ӾཡݖݶΛ෇༩Ͱ͖Δ 24
  20. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫ DeNA෼ੳج൫ͷશମ૾ 27 Log Gateway Hadoop

    Argus Dashboard Medjed +Embulk HP VerIca Hue Jenkins & WorkstaIon Services Analyst Log DB Snapshot Web UI Web UI & SSH Query Query Google BigQuery
  21. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ෼ੳج൫: Job Scheduler/Workstation • Job Schedulerͱͯ͠JenkinsΛ࢖༻

    • Web UI͕ΞφϦετʹͱͬͯ࢖͍΍͍͢ • େ఍ͷ͜ͱ͕Ͱ͖ͯ͠·͏ • ݖݶ؅ཧ΋ॊೈʹߦ͑Δ • ਖ਼௚ӡ༻͸େม • Web UIͰ׬݁͠ͳ͍ۀ຿ͷͨΊʹLinux WorkstationΛఏڙ • ΞφϦετ͕ϩάΠϯͰ͖ΔLinuxαʔόʔΛఏڙ͍ͯ͠Δ • Hadoop CLIͷ࣮ߦ΍PythonεΫϦϓτͷ։ൃʹ࢖༻͞Ε͍ͯΔ 28
  22. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Digdagͱ͍͏༗ྗީิ • ಋೖ͍ͨ͠ϙΠϯτ • Embulkͱಉ͘͡yamlͰઃఆͰ͖Δ •

    WorkflowΛίʔυͰ؅ཧͰ͖Δ • Shell/Ruby/PythonεΫϦϓτΛॻ͚Δ • Workflow, Taskͷࡉ͔͍ఆ͕ٛͰ͖Δ: ৚݅෼ذ, ґଘ, ܁Γฦ͠, etc. • HAͱฒྻॲཧ͕༰қ • ಋೖͰ͖ͣʹ͍Δཧ༝ • ݖݶ؅ཧ͕ߦ͍ͮΒ͍ 29
  23. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ओͳӡ༻ۀ຿ • ArgusʢμογϡϘʔυʣ/ MedjedʢBulk Loadπʔϧʣͷઃఆ •

    BigQuery౳ͷ伴ొ࿥ • ݖݶͷॳظઃఆ • Jenkinsͷઃఆ • ݖݶઃఆ • αʔόʔӡ༻ • ͍ΘΏΔී௨ͷΠϯϑϥӡ༻ • ֤λεΫ͸GitHub IssueͰ؅ཧ • Issue͸ݪଇશैۀһ͔ΒӾཡՄೳ 31
  24. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ओͳαϙʔτۀ຿ • ࣾ಺ͷ෼ੳج൫ར༻ऀ޲͚ٕज़αϙʔτ • ϙϦγʔ͸ʮ෼ੳج൫ΑΖͣ૯ड͚ʯ •

    ෼ੳج൫ʹ͍ͭͯͷ૬ஊͰ͋Ε͹ʮ·ͣ͸ड͚Δʯ • αϙʔτର৅͸ओʹΞφϦετ • ͔͠͠ΞφϦετ৬Ͱ͸ͳ͍ਓ͔Βͷ໰߹ͤ΋͋Δ • 2018೥ݱࡏ͸SlackͰͷαϙʔτ͕7ׂ΄Ͳ • ͍࣍Ͱର໘ϛʔςΟϯάɺϝʔϧ 32
  25. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. SlackʹΑΔαϙʔτํ਑ • ୭Ͱ΋ೖΕΔαϙʔτઐ༻Slack WorkspaceΛ༻ҙ͍ͯ͠Δ • DeNAͰ͸શैۀһ͕Slack

    Enterprise GridΞΧ΢ϯτΛ͍࣋ͬͯΔ • ࠔͬͨΒmentionͯ͠΋Β͏ @support ͱ͍͏άϧʔϓΛ༻ҙ͍ͯ͠Δ • @support ʹ͸AI&෼ੳج൫ϝϯόʔؚΊෳ਺໊͕ొ࿥͞Ε͍ͯΔ 33
  26. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. DeNAͷAIج൫ • AWS/GCPʹΑΔػցֶशج൫Λߏங/ӡ༻ • GPUΠϯελϯεΛ͍ͭͰ΋ඞཁͳ਺͚ͩ •

    Infra as CodeΛపఈ͍ͯ͠Δ • Terraform • Itamae ( Ruby੡ϓϩϏδϣχϯάπʔϧ ) • ϓϩδΣΫτൃ଍͔Β୹ظؒͰΠϯϑϥఏڙͰ͖͍ͯΔ • ֤छਃ੥ؚΊͯ࠷୹1िؒ 36
  27. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. Agenda ࣗݾ঺հ DeNAͱAIγεςϜ෦ͷ͝঺հ DeNA෼ੳج൫ͷ"ࠓ" ⁃ DeNA෼ੳج൫ͷಛ௃

    ⁃ ෼ੳج൫ͷγεςϜߏ੒ ⁃ ෼ੳج൫ͷӡ༻ ⁃ AIج൫ͷ"ࠓ" 2020೥୅ʹ޲͚ͯ ~ AI&෼ੳج൫ͱͯ͠ ~ 38 ( 1 min. ) ( 4 min. ) ( 25 min. ) ( 10 min. ) ( 5 min. ) ( 5 min. ) ( 5 min. ) ( 10 min. )
  28. © DeNA Co., Ltd. ͜Ε͔ΒͷDeNA෼ੳج൫͕໨ࢦ࢟͢ • Ϋϥ΢υωΠςΟϒ • αʔϏεͷΫϥ΢υԽʹ߹Θͤͯ෼ੳج൫΋Ϋϥ΢υԽ •

    ಉ࣌ʹΞφϦετʹ͸طଘͱಉ౳ͷUXΛఏڙ • ෼ੳج൫ʹཹ·Βͳ͍AI&෼ੳج൫ • ෼ੳͷߴ౓ԽͱAIٕज़ͷಋೖͷͨΊศརͳUXΛఏڙ • σʔλ෼ੳ / AI / σʔλαΠΤϯε / Kaggle ౳ѻ͏σʔλͷଟ༷Խ • ΑΓ҆શͳݖݶ෼཭ • AWSΞΧ΢ϯτ/GCPϓϩδΣΫτࣗମΛσʔλͷൿಗੑʹԠͯ͡෼཭ 40
  29. طଘ෼ੳج൫ © DeNA Co., Ltd. Ϋϥ΢υωΠςΟϒͳ෼ੳج൫ͷྫ 41 Argus Dashboard Medjed

    +Embulk Jenkins & WorkstaIon Analyst Web UI Web UI & SSH Service Google BigQuery Log DB Snapshot αʔϏε͕Ϋϥ΢υ্ʹ͋Δ ৔߹ɺ͋͑ͯΦϯϓϨϛεʹ Data LakeΛ࣋ͭඞཁ͸ͳ͍ɻ ҰํɺΞφϦετ͸׳ΕͨUX Ͱ෼ੳΛߦ͍͍ͨɻ
  30. © DeNA Co., Ltd. Amazon SageMaker ( Jupyter Notebook )

    43 Web UI্ͰPythonίʔυ౳͕࣮ߦͰ͖Δ
  31. ɹɹɹɹɹɹɹɹɹɹɹɹɹ© DeNA Co., Ltd. ͜Ε͔ΒͷDeNA෼ੳج൫ • Ϋϥ΢υωΠςΟϒ • αʔϏεʹ߹Θͤͯ෼ੳج൫΋Ϋϥ΢υωΠςΟϒʹͳ͍ͬͯ͘ •

    ෼ੳج൫ʹཹ·Βͳ͍AI&෼ੳج൫ • ෼ੳର৅ͷσʔλ͸࣭ɾྔͱ΋ʹ૿͍͑ͯ͘ • ػցֶशΞϧΰϦζϜͷ࣮αʔϏεͰͷӡ༻Λࢧ͑ΔML Ops͕ඞཁ • 1ͭͷϓϩδΣΫτ಺ͰAIٕज़΋࢖͏͠෼ੳ΋ߦ͏ • ʮAI͔෼ੳʯͰ͸ͳ͘ʮAI΋෼ੳ΋ʯߦ͑Δج൫͕ॏཁ • ΑΓ҆શͳݖݶ෼཭ • ѻ͏σʔλ͸ଟ༷ʹͳ͖͍ͬͯͯΔ • ద੾ʹݖݶ෼཭ͨ͠ΑΓηΩϡΞͳ؀ڥߏங͕ॏཁ 45