High Performance Backend For Mercari

1. High Performance Backend For Mercari Tatsuhiko Kubo@cubicdaiya ࣌Ӎࡇ@2015/03/21
2. ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in Infrastructure Engineering • Mercari, Inc.
3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ
6. ओཁKPIσʔλ μ΢ϯϩʔυ਺ ߪೖֹۚ ग़඼਺ ਺஋ 1000ສDLಥഁ ݄ؒ਺ेԯԁ 1೔਺ेສ඼Ҏ্
7. ͷେࡶ೺ͳγεςϜߏ੒ ͘͞ΒͱAWSͷϋΠϒϦου
8. ৄ͘͠͸ͪ͜Β http://bit.ly/18Os1Dc
9. Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go
10. Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go
11. ͱ͋ΔAPIαʔόͷฏۉϨεϙϯελΠϜ(1೔ؒ) ϐʔΫ࣌Ͱ΋30ms୆ʂ
12. ࣮ࡍʹ͸΋ͬͱ࣮ߦ࣌ؒͷ௕͍API΋͋Γ·͢ e.g. ߪೖτϥϯβΫγϣϯ
13. Why API server very fast? • ܰྔͰߴ଎ͳPHPϑϨʔϜϫʔΫ(DietCake) • Ωϟογϡ • ඇಉظॲཧ • ۚͷ஄ؙMySQL on ioDrive • ͦ΋ͦ΋Ϩεϙϯε͕JSON onlyͳͷͰ͍ܰ
14. http://dietcake.github.io/
16. Ωϟογϡ • memcachedͱRedisΛซ༻ • ௨ৗͷΩϟογϡ͸جຊతʹmemcached • Redis͸Ϧετܕ͕ޮՌతͳՕॴ͕͋ͬͯͦ͜ Ͱར༻͍ͯ͠Δ • ͦͷલஈͰ͞ΒʹmemcachedͰΩϟογϡ
17. Ωϟογϡͷ΢ΥʔϜΞοϓ • ΞΫηε͕ଟ͍ & ߋ৽ස౓͕ߴ͍ & Ωϟογϡͷύλʔϯ͕ଟ͍ & ݁ߏॏ͍API • Ϣʔβ͔ΒͷΞΫηεͷࡍʹΩϟογϡΛੜ੒ͯ͠ ΋ޮՌ͕ബ͍ • ཪͰͻͨ͢ΒΩϟογϡΛੜ੒͢ΔσʔϞϯΛಈ͔͢ • ޙड़
18. ඇಉظॲཧ • APIαʔό͸ͳΔ͸΍ͰϨεϙϯεΛฦ͍ͨ͠ • ͔͠͠ɺதʹ͸Ͳ͏ͯ͠΋ॏ͍ॲཧ͕͋Δˣ • ΩϟογϡɺݕࡧΠϯσοΫεͷߋ৽ • ϝʔϧɺSMSͷૹ৴ • δϣϒΩϡʔ&ϫʔΧʔͰඇಉظʹॲཧ͢Δ
19. δϣϒΩϡʔͱϫʔΧʔ • δϣϒΩϡʔ͸Q4M • ϫʔΧʔ͸php-Parallel-Prefork • PHPͰpreforkσʔϞϯ͕؆୯ʹ࡞ΕΔ
21. Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go
22. OHJOY
23. ϝϧΧϦʹ͓͚Δnginxͷओͳ໾ׂ • APIϦΫΤετͷϩʔυόϥϯε • SSLλʔϛωʔγϣϯ & SPDYήʔτ΢ΣΠ • େྔಉ࣌઀ଓ΁ͷඋ͑(C10K) • HTTPS௨৴ͷߴ଎Խ • TLS Session (Cache|Tickets)
24. TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ • CPUϦιʔεͷ࡟ݮ΍ϨΠςϯγղফͷޮՌ͕͋Δ
25. TLS Session Cache with nginx
26. TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥ ΠΞϯτʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆ͰηογϣϯΩϟογϡ Λڞ༗Ͱ͖Δ • αϙʔτ͍ͯ͠Δ୺຤͕গͳ͍ͷ͕೉఺
27. TLS Session Tickets with nginx
28. SPDY • ࠷৽ͷiOS΍AndroidͰ͸αʔό͕ରԠͯ͠ ͍Ε͹ࣗಈతʹSPDY/3.1͕ར༻͞ΕΔ • ΫϥΠΞϯτଆͰରԠ͠ͳͯ͘΋Α͍
29. SPDY with nginx
30. TLS Session (Cache|Tickets) ಋೖλΠϛϯά
31. TLS Session (Cache|Tickets) ಋೖλΠϛϯά
32. SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ
33. Agenda • High performance API server with PHP!!! • High performance networking with nginx • High performance Application with Go
34. mercari.go • ϝϧΧϦͷϝΠϯ։ൃݴޠ͸PHP • ΠϯϑϥνʔϜͩͱGoɺNodeɺRubyɺPerl౳͍ Ζ͍Ζ(ΈΜͳಘҙ෼໺͕ҧ͏) • ࠷ۙ͸ͪΐͬͱͨ͠CLI΍σʔϞϯɺύϑΥʔϚϯ ε͕ཁٻ͞ΕΔίϯϙʔωϯτΛGoͰ։ൃ͢Δػ ձ͕૿͑ͯΔ
35. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache
36. cɺconsul΋࢖ͬͯΔ͔Β(਒͑੠)
37. ͳ͓ɺόάͬͯΔ໛༷ ઌि͜ͷόάΛճආ͢Δίʔυॻ͍ͯ·ͨ͠
38. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache
39. Gaurun • ൚༻ϓογϡ௨஌αʔό • ୹࣌ؒͰ਺ेɺ਺ඦສ݅ͷϓογϡ௨஌͕Մೳ • HTTPͱJSONΛϕʔεʹͨ͠γϯϓϧͳAPI • ϝϧΧϦ಺ͷϓογϡ௨஌ɺେମGaurunͰஔ ͖׵͑ͨ
40. ൚༻ϓογϡ௨஌γεςϜͷΞʔΩςΫνϟ PS
    CBUDIαʔό
41. Gaurun HTTP API "1* ղઆ 1045
    QVTI ϓογϡ௨஌ϦΫΤετड෇ (&5
    TUBUHP (PϥϯλΠϜͷ׆ಈঢ়گΛऔಘ (&5
    TUBUBQQ ಺෦Ωϡʔ νϟωϧ ͷ࢖༻ྔ΍
    ϓογϡͷ੒ޭࣦഊ਺ͷऔಘ (&5
    DPOpHBQQ αʔόઃఆ 50.- Λऔಘ
42. POST /pushͷϦΫΤετϘσΟ
43. GET /stat/go $ curl -s http://127.0.0.1:1056/stat/go { "time": 1423270089195474419, "go_version": "go1.4", "go_os": "linux", "go_arch": "amd64", "cpu_num": 24, "goroutine_num": 55, "gomaxprocs": 24, "cgo_call_num": 124016, "memory_alloc": 10309424, "memory_total_alloc": 295474683712, "memory_sys": 25557240, "memory_lookups": 214058, "memory_mallocs": 2118264299, "memory_frees": 2118230111, ɾ ɾ
44. GET /stat/app $ curl -s http://127.0.0.1:1056/stat/app { "queue_max": 40960, // ಺෦ΩϡʔͷαΠζ "queue_usage": 0, // ಺෦Ωϡʔͷ࢖༻ྔ(͙͢ࡹ͚ΔͷͰେମ0) "ios": { "push_success": 31804, // iOSϓογϡ௨஌ͷ੒ޭ਺ "push_error": 17 // iOSϓογϡ௨஌ͷࣦഊ਺ }, "android": { "push_success": 18683, // Androidϓογϡ௨஌ͷ੒ޭ਺ "push_error": 233 // Androidϓογϡ௨஌ͷࣦഊ਺ } $
45. GaurunΛ࣮ߦ͍ͯ͠Δ༷ࢠͰ͢
46. mercari.go • nginx-build • cachectl • slackboard • Ϋϩʔζυ • Gaurun, refreshcache
47. refreshcache • ࠷ॳͷํͰݴͬͯͨಛఆAPIͷϨεϙϯεͷ ΩϟογϡΛੜ੒͢ΔσʔϞϯ • ҰఆִؒͰฒྻʹ݁ߏͳྔͷHTTPϦΫΤετ ΛૹΓଓ͚Δ • ݩʑRubyͰॻ͔Ε͍ͯͨ(஗͔ͬͨ)
48. ݩͷRubyίʔυ Parallel.map(params_list, :in_processes => in_processes) do |params| (தུ) res = http_request('GET', url, params) (தུ) end.compact params_listͷαΠζ͕େ͖͍ ↓ େྔʹfork͢ΔͷͰॏ͍
49. GoͰॻ͖௚ͨ͠ done := make(chan error) size := len(paramCombs.Params) var errors []error for i := 0; i < size; i++ { go makeCache(URL, paramCombs.Params[i], done) } for i := 0; i < size; i++ { err := <-done if err != nil { errors = append(errors, err) } }
50. ࣮ߦαʔόͷάϥϑ(CPU) ϦϥΠτͨ͠λΠϛϯά
51. ࣮ߦαʔόͷάϥϑ(LA) ϦϥΠτͨ͠λΠϛϯά
52. ·ͱΊ • ϝϧΧϦͷߴ଎ͳόοΫΤϯυͷ࢓૊Έʹ͍ͭͯ ͍͔ͭ͘ղઆ͠·ͨ͠ • ϝΠϯ͸PHP͚ͩͲ࠷ۙ͸Goͷར༻ࣄྫ૿͑ͯΔ • Go͸ૣ͍ɾ଎͍ɾ͍ܰ • Gaurun͸ͦͷ͏ͪOSSԽ༧ఆ