フリマアプリ「メルカリ」の急成長を支えるエンジニアリング

Transcript

1. ϑϦϚΞϓϦ ʮϝϧΧϦʯͷ ٸ੒௕Λࢧ͑Δ ΤϯδχΞϦϯά גࣜձࣾϝϧΧϦ ΤϯδχΞ ٱอୡ඙ bokko@mercari.com PHPΧϯϑΝϨϯε 2015
2. @cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc. • Infrastructure Engineering • Skills • C, Go, Lua, nginx,… • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, …
3. ISUCON5༧બ 2ҐͰಥഁ͠·ͨ͠ ݴޠ͸PerlͰ͢ νʔϜGoBold @kazeburo @shmorimo @cubicdaiya
7. ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 2000ສDL(JP+US) ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্
8. 4ਓ 30ਓऑ ΠϯϑϥνʔϜ ૯ΤϯδχΞ਺
9. 100+ 13,000+ αʔό୆਺ ඵؒϦΫΤετ਺ (ϐʔΫ࣌) JP only
10. Πϯϑϥ͸͘͞ΒͱAWSͷϋΠϒϦουߏ੒ • ͘͞Βͷઐ༻αʔό • ͘͞ΒͷΫϥ΢υ • Amazon Web Services • EC2, S3, SES౳
11. ઐ༻αʔόͱΫϥ΢υͷ࢖͍෼͚ • ઐ༻αʔό͸ίετύϑΥʔϚϯε͕ߴ͍ • 1୆͋ͨΓͷੑೳ͕ߴ͍ͷͰ୆਺ΛগͳΊʹͰ͖Δ • MySQL on ioDrive • Ϋϥ΢υ͸ॊೈੑ͕ߴ͍ • ಥൃతͳෛՙ΁ͷରԠ • ࢼݧɾ࢖͍ࣺͯ༻ʹαʔόΛ֬อ͠΍͍͢
12. PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ
13. PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ • ֤ॴͰΧϦΧϦʹνϡʔχϯά • ܰྔͳϑϨʔϜϫʔΫΛར༻(dietcake) • Ωϟογϡ(memcached, Redis) • ඇಉظॲཧ(Q4M + php-Parallel-Prefork) • ۚͷ஄ؙMySQL on ioDrive
14. http://dietcake.github.io/
15. php-Parallel-Prefork • prefork server framework for PHP • https://github.com/travail/php-parallel-prefork • ඇಉظॲཧʹར༻ • Q4M͔ΒδϣϒΛσΩϡʔɺϚϧνϓϩηεͰฒྻʹॲཧ
16. New RelicʹΑΔϞχλϦϯά
17. PHP5.3 -> 5.6ʢ2015೥7݄ʣ
18. nginxʹΑΔϋΠύϑΥʔϚϯε ωοτϫʔΩϯά
19. TLSλʔϛωʔγϣϯɺSPDYήʔτ΢ΣΠ ɾɾɾ ɾɾɾ "1*
    "QBDIF NPE@QIQ OHJOY OHJOY HTTPS or SPDY HTTP "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ OHJOY
20. HTTPSऴ୺αʔόͱͯ͠ͷnginx • େྔͷಉ࣌઀ଓ(਺ສ)Λܰշʹࡹ͚Δ • APIαʔόͷϩʔυόϥϯε΋݉ͶΔ • ߴ଎ͳHTTPS௨৴ͷఏڙ • TLS Session (Cache|Tickets), OCSP Stapling • SPDY/3.1ɺHTTP/2ͷήʔτ΢ΣΠ • ϝϧΧϦ͸SPDY/3.1Λར༻
21. L7(HTTP)ϩʔυόϥϯε "1* "QBDIF
    4PMS search "QBDIF
    4PMS "QBDIF
    4PMS OHJOY "1* "1* search
22. ݕࡧ݁ՌͷΩϟογϡ "1* OHJOY "QBDIF
    4PMS search
23. ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕ͳ͍৔߹) "1* OHJOY "QBDIF
    4PMS 1:search 2:proxy 3:response 5:response 4:cached response
24. ݕࡧ݁ՌͷΩϟογϡ(Ωϟογϡ͕͋Δ৔߹) "1* OHJOY "QBDIF
    4PMS 1:search 2:response
25. https://speakerdeck.com/cubicdaiya/nginxfalsepahuomansutiyuningu ଓ͖͸WebͰ
26. SlackͰChatOps
27. SlackͰͰ͖Δ͜ͱʢϝϧΧϦͷ৔߹ʣ • σϓϩΠ • ۈଵ؅ཧ • Ξϥʔτ௨஌ • CIͷखಈ࣮ߦ • etc…
28. σϓϩΠ
29. ۈଵ؅ཧ
30. Ξϥʔτ௨஌
31. CIͷखಈ࣮ߦ
32. ࣦഊ͢Δͱɺ
33. SlackϓϩΩγαʔόܦ༝Ͱ Ͳ͔͜ΒͰ΋ϙετ
34. ͍ΖΜͳαʔό͔ΒSlackʹ௨஌͢Δࡍͷ໰୊఺ • ֤αʔό্ͷΫϥΠΞϯτ͕Incomming WebhooksͷURLΛ஌͍ͬͯΔඞཁ͕͋Δ • ΫϥΠΞϯτͷར༻ݴޠ͕όϥόϥ • ௨஌ॲཧΛॻ͘ͷ͕໘౗ • ϓϩΩγཱͯͯϓϩΩγͱ௨৴͢ΔΫϥΠΞ ϯτΛ֤αʔόʹ഑ஔ͠Α͏ʂ
35. Πϝʔδ
36. slackboard • A slack proxy server in Go • https://github.com/cubicdaiya/slackboard • ߏ੒ϓϩάϥϜ • slackboardʙSlackϓϩΩγαʔόʙ • slackboard-cliʙΫϥΠΞϯτ for slackboardʙ • slackboard-logʙΫϥΠΞϯτ like cronlogʙ
37. slackboard-cliͰSlack΁௨஌ $ echo mercari | \ slackboard-cli \ -c tech-test \ -s slackboard-server:29800 TMBDLCPBSE POST /notify-directly Slack΁ϙετ
38. slackboard-logͰSlack΁௨஌ $ ls $ slackboard-log \ -c tech-test \ -s slackboard-server:29800 -- ls hoge TMBDLCPBSE POST /notify-directly Slack΁ϙετ
39. θϩμ΢ϯλΠϜσϓϩΠ
40. "1*
    "QBDIF NPE@QIQ ҎલͷσϓϩΠखॱ "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU rsync rsync rsync
41. Pros / Cons • Pros • ؆୯Ͱͦͦ͜͜଎͍(appαʔό͸30୆͘Β͍) • Cons • ϊϯΤϥʔͰσϓϩΠͰ͖ͳ͍ • Too many 500 errors!!! • ࣌ʑΦϖίʔυΩϟογϡ͕յΕΔʢ࠶ىಈ͕ඞཁʣ
42. ɾɾɾ ɾɾɾ "1*
    "QBDIF NPE@QIQ OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ
43. ɾɾɾ ɾɾɾ "1*
    "QBDIF NPE@QIQ OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ
44. ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • nginxͷΞοϓετϦʔϜΛ HTTPϕʔεͷAPIͰಈతʹมߋͰ͖Δ • rsyncͷલޙͰΞοϓετϦʔϜΛॻ͖׵͑
45. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ
46. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
47. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
48. θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ
49. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
50. θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ
51. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
52. θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
53. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
54. θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
55. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
56. θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
57. "1*
    "QBDIF NPE@QIQ θϩμ΢ϯλΠϜσϓϩΠ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
58. θϩμ΢ϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
59. ϓογϡ௨஌ج൫
60. 1೥લ
61. ൒೥લ
62. ݱࡏ
63. Gaurun • A general push notification server in Go • https://github.com/mercari/gaurun • APNS΍GCM΁ͷϓογϡ௨஌ϦΫΤετΛϓϩΩγ • Q4M+php-Parallel-Preforkͩͱੑೳग़ͳ͔ͬͨͷͰ ϑϧεΫϥονͰॻ͍ͨ • ଞͷδϣϒϫʔΧʔͷҰ෦΋GoͰϦϥΠτத
64. ϝϧΧϦͷ੒௕Λࢧ͑Δϩά෼ੳج൫
65. 200GB / day ϩά༰ྔ
66. ֤αʔόͰऩूͨ͠ϩάσʔλΛFluentdͰసૹ "QQ 8PSLFS #BUDI ,JCBOB #JH2VFSZ /PSJLSB .BDLFSFM
67. Mercari\DataTrack • Various purpose logging framework for PHP • αʔόαΠυͰى͜ΔΠϕϯτΛॊೈʹϩΪϯά • ϝʔϧɺϓογϡͷૹ৴ • A/Bςετ • Πϕϯτͷܦա • Pascal(ޙड़)ͱૄʹ࿈ܞ
68. /PSJLSB ϩάσʔλసૹ ఆظతͳΠϕϯτͷऔಘ 3ඵຖ <match access_log> type norikra norikra norikra-server:26571 target_string access_log </match> <source> type norikra norikra localhost:26571 <fetch> method sweep target target_name tag query_name tag_prefix norikra.query interval 3s </fetch> </source> SQLΛొ࿥ /** * access.log͔ΒHTTPεςʔλείʔυͷׂ߹(1෼ؒ)Λूܭ */ SELECT COUNT(1, status like "5%")/COUNT(1)*100 AS rate_5xx, COUNT(1, status like "4%")/COUNT(1)*100 AS rate_4xx, COUNT(1, status like "3%")/COUNT(1)*100 AS rate_3xx, COUNT(1, status like "2%")/COUNT(1)*100 AS rate_2xx FROM access_log.win:time_batch(1 min) NorikraʹΑΔϩάετϦʔϛϯάॲཧ
69. Πϕϯτͷऔಘ fluent-plugin-norikra Mackerel΁సૹ ֤छϝτϦΫεΛάϥϑԽ ྫ:APIͷฏۉϨεϙϯελΠϜ౳ ݕ஌ͨ͠Τϥʔ಺༰Λ௨஌ Slack΁సૹ /PSJLSB SQLͰநग़ͨ࣌͠ܥྻσʔλΛ͞Βʹసૹ
70. σʔλͷूܭɺՄࢹԽɺϨϙʔςΟϯά • Ϩϙʔτϝʔϧ • Web UIʹΑΔ౷ܭμογϡϘʔυ • ֎෦ͷ෼ੳπʔϧར༻
71. ϩάσʔλ෼ੳ্ͷ՝୊ • ֤छKPI΍෼ੳʹඞཁͳݩσʔλ͕෼ࢄ • ϩά͕෼ੳ༻ʹઃܭ͞Εͯͳ͍ͷͰɺ෼ੳ͢ Δͷʹ͸޲͍ͯͳͯ͘࢖͍ͮΒ͍(ίπ͕ཁΔ) • ֎෦ͷ෼ੳπʔϧͩͱखܰʹूܭɾՄࢹԽͰ ͖Δ͕ɺଞͷσʔλ΍πʔϧͱ૊Έ߹ΘͤΔ ͷ͕೉͍͠
72. ͱ͍͏Θ͚Ͱɺ • ෼ੳʹదͨ͠ϩάΛҰ͔Βઃܭɾूܭ • ͦͷ্Ͱ෼ੳπʔϧͱ૊Έ߹Θͤͯ࢖͑ΔΑ ͏ʹ͠Α͏ ৽͍͠ϩά෼ੳج൫Λߏங͢Δ͜ͱʹ
73. PascalʙMercari analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF
    #JH
    2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer
74. Puree • ΫοΫύου͕ࣾ։ൃɾެ։͍ͯ͠ΔεϚϗ ΞϓϦ޲͚ͷϩάίϨΫλϥΠϒϥϦ • https://github.com/cookpad/puree-android • https://github.com/cookpad/puree-ios • Android, iOSͰ΄΅ಉ͡৚݅ͰϩάσʔλΛ औಘՄೳ
75. OpenResty • nginxϕʔεͷWebΞϓϦέʔγϣϯϑϨʔϜϫʔΫ • bundled ngx_lua, Lua, LuaJIT, ͦͷଞศརͳ Ϟδϡʔϧ • ͋Δҙຯnginxͷεʔύηοτ
76. PascalʙMercari analysis baseʙ • جຊ͸BigQueryͰΞυϗοΫʹ෼ੳ • ϩάͷϑΥʔϚοτ͕෼ੳʹదͨ͠ܗʹ • Chartio౳ͷ෼ੳπʔϧͱ΋૊Έ߹Θͤͯར༻ • ΞϓϦ಺ΞΫγϣϯ΍A/Bςετ෼ੳΛ͸͡Ί ద༻ൣғΛ֦େத
77. ·ͱΊ • ৭ʑ޻෉ͯ͠PHPͰߴ଎ͳAPIαʔόΛ࣮ݱ͍ͯ͠·͢ • དྷ೥͸7ରԠʁ • όοΫΤϯυͷํͰ͸GoΛ࠾༻͢Δ৔໘͕૿͑ͯ·͢ • γεςϜͷύϑΥʔϚϯε΍ࣗಈԽɺσϓϩΠɺϩά෼ ੳͳͲٸ੒௕Λࢧ͑ΔͨΊͷج൫࡞Γʹۈ͠ΜͰ͍·͢
78. We are hiring!