Share
Embed
フリマアプリ「メルカリ」の急成長を支えるエンジニアリング
Tatsuhiko Kubo
October 03, 2015
Transcript
78 slides
-
@cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc.
• Infrastructure Engineering • Skills • C, Go, Lua, nginx,… • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, … -
ઐ༻αʔόͱΫϥυͷ͍͚ • ઐ༻αʔόίετύϑΥʔϚϯε͕ߴ͍ • 1͋ͨΓͷੑೳ͕ߴ͍ͷͰΛগͳΊʹͰ͖Δ • MySQL on ioDrive •
Ϋϥυॊೈੑ͕ߴ͍ • ಥൃతͳෛՙͷରԠ • ࢼݧɾ͍ࣺͯ༻ʹαʔόΛ֬อ͍͢͠ -
PHPͰߴͳAPIαʔόΛ࣮ݱ • ֤ॴͰΧϦΧϦʹνϡʔχϯά • ܰྔͳϑϨʔϜϫʔΫΛར༻(dietcake) • Ωϟογϡ(memcached, Redis) • ඇಉظॲཧ(Q4M
+ php-Parallel-Prefork) • ۚͷؙMySQL on ioDrive -
php-Parallel-Prefork • prefork server framework for PHP • https://github.com/travail/php-parallel-prefork •
ඇಉظॲཧʹར༻ • Q4M͔ΒδϣϒΛσΩϡʔɺϚϧνϓϩηεͰฒྻʹॲཧ -
TLSλʔϛωʔγϣϯɺSPDYήʔτΣΠ ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY OHJOY HTTPS or
SPDY HTTP "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ OHJOY -
HTTPSऴαʔόͱͯ͠ͷnginx • େྔͷಉ࣌ଓ(ສ)Λܰշʹࡹ͚Δ • APIαʔόͷϩʔυόϥϯε݉ͶΔ • ߴͳHTTPS௨৴ͷఏڙ • TLS Session
(Cache|Tickets), OCSP Stapling • SPDY/3.1ɺHTTP/2ͷήʔτΣΠ • ϝϧΧϦSPDY/3.1Λར༻ -
slackboard • A slack proxy server in Go • https://github.com/cubicdaiya/slackboard
• ߏϓϩάϥϜ • slackboardʙSlackϓϩΩγαʔόʙ • slackboard-cliʙΫϥΠΞϯτ for slackboardʙ • slackboard-logʙΫϥΠΞϯτ like cronlogʙ -
slackboard-cliͰSlack௨ $ echo mercari | \ slackboard-cli \ -c tech-test
\ -s slackboard-server:29800 TMBDLCPBSE POST /notify-directly Slackϙετ -
slackboard-logͰSlack௨ $ ls $ slackboard-log \ -c tech-test \ -s
slackboard-server:29800 -- ls hoge TMBDLCPBSE POST /notify-directly Slackϙετ -
"1* "QBDIF NPE@QIQ ҎલͷσϓϩΠखॱ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ
ChatOps with Slack yes EFQMPZCPU rsync rsync rsync -
Pros / Cons • Pros • ؆୯Ͱ͍ͦͦ͜͜(appαʔό30͘Β͍) • Cons •
ϊϯΤϥʔͰσϓϩΠͰ͖ͳ͍ • Too many 500 errors!!! • ࣌ʑΦϖίʔυΩϟογϡ͕յΕΔʢ࠶ىಈ͕ඞཁʣ -
ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY
OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ -
ɾɾɾ ɾɾɾ "1* "QBDIF NPE@QIQ OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY
OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ -
ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • nginxͷΞοϓετϦʔϜΛ
HTTPϕʔεͷAPIͰಈతʹมߋͰ͖Δ • rsyncͷલޙͰΞοϓετϦʔϜΛॻ͖͑ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream -
θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream -
θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream -
θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream -
θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU -
θϩμϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ -
"1* "QBDIF NPE@QIQ θϩμϯλΠϜσϓϩΠ "1* "QBDIF NPE@QIQ ChatOps with Slack
yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU -
θϩμϯλΠϜσϓϩΠ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY
OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ "1* "QBDIF NPE@QIQ -
Gaurun • A general push notification server in Go •
https://github.com/mercari/gaurun • APNSGCMͷϓογϡ௨ϦΫΤετΛϓϩΩγ • Q4M+php-Parallel-Preforkͩͱੑೳग़ͳ͔ͬͨͷͰ ϑϧεΫϥονͰॻ͍ͨ • ଞͷδϣϒϫʔΧʔͷҰ෦GoͰϦϥΠτத -
Mercari\DataTrack • Various purpose logging framework for PHP • αʔόαΠυͰى͜ΔΠϕϯτΛॊೈʹϩΪϯά
• ϝʔϧɺϓογϡͷૹ৴ • A/Bςετ • Πϕϯτͷܦա • Pascal(ޙड़)ͱૄʹ࿈ܞ -
/PSJLSB ϩάσʔλసૹ ఆظతͳΠϕϯτͷऔಘ 3ඵຖ <match access_log> type norikra norikra norikra-server:26571
target_string access_log </match> <source> type norikra norikra localhost:26571 <fetch> method sweep target target_name tag query_name tag_prefix norikra.query interval 3s </fetch> </source> SQLΛొ /** * access.log͔ΒHTTPεςʔλείʔυͷׂ߹(1ؒ)Λूܭ */ SELECT COUNT(1, status like "5%")/COUNT(1)*100 AS rate_5xx, COUNT(1, status like "4%")/COUNT(1)*100 AS rate_4xx, COUNT(1, status like "3%")/COUNT(1)*100 AS rate_3xx, COUNT(1, status like "2%")/COUNT(1)*100 AS rate_2xx FROM access_log.win:time_batch(1 min) NorikraʹΑΔϩάετϦʔϛϯάॲཧ -
PascalʙMercari analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist
Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer -
PascalʙMercari analysis baseʙ • جຊBigQueryͰΞυϗοΫʹੳ • ϩάͷϑΥʔϚοτ͕ੳʹదͨ͠ܗʹ • ChartioͷੳπʔϧͱΈ߹Θͤͯར༻ •
ΞϓϦΞΫγϣϯA/BςετੳΛ͡Ί ద༻ൣғΛ֦େத