Software Engineer, Infrastructure

1. Tatsuhiko Kubo@cubicdaiya mercari.go#2 2018/08/10 Software Engineer, Infrastructure
2. @cubicdaiya / Tatsuhiko Kubo Principal Software Engineer, SRE @ Mercari, Inc.
4. Agenda • Software Engineer, Infrastructure ϙδγϣϯͷ঺հ • ϝϧΧϦͷཪଆͰՔಇ͢ΔGoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ
5. Software Engineer, Infrastructure https://careers.mercari.com/job/
6. Software Engineer, Infrastructure https://careers.mercari.com/job/
7. Software Engineer, Infrastructure • Role • ιϑτ΢ΣΞΤϯδχΞϦϯάʹΑΔߴ͍ύϑΥʔϚϯε΍৴པੑͷߴ͍Πϯϑϥε τϥΫνϟͷ࣮ݱ • ΞϓϦέʔγϣϯ΍ΠϯϑϥपΓͰڞ௨͢Δٕज़త՝୊Λղܾ͢ΔͨΊͷπʔϧ΍ϛ υϧ΢ΣΞͷ։ൃɾӡ༻ • Software Engineer, Site Reliability (SRE)ͱͷҧ͍ • ਖ਼௚ʹݴ͏ͱɺͦ͜·Ͱ໌֬ʹ۠ผ͞ΕΔΘ͚Ͱ͸ͳ͍ʢॴଐ͸SREνʔϜʣ • ҰํͰɺSREΑΓ΋։ൃۀ຿ͷൺॏ͕େ͖͍ʢ7:3͔8:2͘Β͍ʣ
8. Software Engineer, Infrastructure • ։ൃݴޠ͸Go͕த৺ • ωοτϫʔΫ΍γεςϜϓϩάϥϛϯάدΓͷ஌ࣝ΍εΩϧ͕ཁٻ͞ΕΔ৔໘͕ଟΊ • ΠϯϑϥετϥΫνϟͷج൫΍Τοδέʔε޲͚ͷαʔόαΠυϛυϧ΢ΣΞͷ ։ൃɾӡ༻ • ֤छΦϖϨʔγϣϯͷࣗಈԽ΍ϞχλϦϯά޲͚πʔϧ౳ͷ։ൃɾϝϯς • ύϑΥʔϚϯενϡʔχϯά΍ϘτϧωοΫͷௐࠪɾվળ౳ɺ໨ͷલʹ͋Δٕज़త՝ ୊Λιϑτ΢ΣΞΤϯδχΞϦϯάͷྗʹΑͬͯൃݟɾղܾ͢Δͷ͕ϛογϣϯʢ͜ Ε͸SRE΋Ұॹʣ
9. ಥવͰ͕͢ɺϚΠΫϩαʔϏε͕ྲྀߦΓͰ͢Ͷ
11. https://tech.mercari.com/entry/2018/07/11/180500
12. https://tech.mercari.com/entry/2018/07/11/180500
13. ࣮ࡍͷͱ͜Ζɺ • ϝϧΧϦͰGo͕ར༻͞Ε࢝Ίͨͷ͸2014೥͔Β • αʔϏεͷػೳ։ൃͰ͸౰͔࣌ΒPHP͕ϝΠϯͩͬͨ • ҰํɺཪଆͰՔಇ͢Δπʔϧ΍ϛυϧ΢ΣΞͳΜ͔͸ ౰͔࣌ΒGo΍Node.js΋ͪ´ • ࢝·Γ͸ӡ༻޲͚πʔϧͷ։ൃ΍ύϑΥʔϚϯε͕ཁٻ͞ΕΔՕॴͰͷར༻ • GKE+GoΛத৺ʹਾ͑ͯϚΠΫϩαʔϏεԽΛਪਐ <- ΠϚίί
14. ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨPHP • PHP͸7ʹͳͬͯେ෼଎͘ͳͬͨ • CPU࢖༻཰͸൒෼ҎԼ • ॲཧ͕࣌ؒ਺े%ݮ • ͔͠͠ɺ୯ମੑೳ͕޲্͚ͨͩ͠Ͱ΢ΟʔΫϙΠϯτ͸ͦͷ·· • γϯάϧεϨουͳͷͰϚϧνίΞΛ׆͔ͤͳ͍ • ඇಉظॲཧ͕ۤख PHP5.6 -> PHP7.1 Response time avg PHP5.6 -> PHP7.1
15. • े෼ͳੑೳ • ϐʔΫ࣌Ͱඵؒ਺ઍreq/secΛࡹ͍ͯΔ ͱ͋ΔGoͷHTTP APIαʔόͷ CPU࢖༻཰ͷάϥϑˠ • ϨεϙϯελΠϜ͸avgͰ4msɺ90~99percentileͰ10~20msલޙ • ඇಉظॲཧ΍ฒߦੑͷߴ͍ϓϩάϥϜͷ࣮ݱ͕Մೳ • ΰϧʔνϯ΍νϟωϧʹΑΔڧྗͳαϙʔτ ύϑΥʔϚϯε΍εέʔϥϏϦςΟͷ؍఺͔ΒݟͨGo
16. • γϯάϧόΠφϦͳͷͰ഑෍ָ͕ • γεςϜӡ༻ऀతʹָ͘͢͝ • ύοέʔδϯά΋ָ πʔϧ։ൃݴޠͱͯ͠ͷGo
17. • DockerͰͲ͔ͬʔΜ • RPM • πʔϧ΍ϛυϧ΢ΣΞྨ͸ΞϓϦέʔγϣϯͱൺ΂ͯͦΜͳʹසൟʹߋ ৽͠ͳ͍ͷͰ΄ͱΜͲͷ৔߹͸͜ΕͰे෼ ϝϧΧϦͰͷ࣮ࡍͷ഑෍ํ๏
18. DockerͱMakeʹΑΔRPMύοέʔδͷϏϧυγεςϜ https://tech.mercari.com/entry/2016/08/15/163219
19. ෳ਺ͷOS޲͚ʹDockerΛىಈͯ͠Ϗϧυ
20. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package & upload to S3
21. ࣮ࡍʹyum installͰ͖ΔΑ͏ʹͳΔ·ͰͷྲྀΕ Merge PR to chage spec Build package & upload to S3 JP US UK aws s3 sync yum repo yum repo yum repo
22. ϝϧΧϦͷཪଆͰՔಇ͢Δ GoʹΑΔπʔϧ΍ϛυϧ΢ΣΞͷ঺հ
23. slackboard
24. slackboard • Slack Incoming WebhookϓϩΩγ݉ΫϥΠΞϯτ • github.com/cubicdaiya/slackboard • Slack΁ͷ௨஌ॲཧ΍ඞཁͳ৘ใΛҰݩ؅ཧԽ • ௨஌ઌ΍Webhook URLઃఆͷू໿ • ΫϥΠΞϯτϓϩάϥϜͷ౷ҰʹΑΔϝϯςφϏϦςΟͷ޲্ • ౰࣌(2014೥ࠒ)ࣾ಺ͰPHP΍Ruby΍PythonͷSlackΫϥΠΞϯτ͕ཚཱͯͨ͠
25. slackboard configuration [core] # listen port port = “29800” # URL for Incoming Webhooks slack_url = “https://hooks.slack.com/services/…” [log] # access log path or redirector access_log = “stdout” # error log path or redirector error_log = “stderr” # log level level = “error”
26. Usage of slackboard-cli $ slackboard -c /path/to/slackboard.toml Server Client $ echo mercari | slackboard-cli -c test -s 127.0.0.1:29800
27. Usage of slackboard-log $ slackboard -c /path/to/slackboard.toml Server Client $ slackboard-log -c test -s 127.0.0.1:29800 — ls notfound
28. ࣮ࡍͷγεςϜߏ੒ by slackboard asia region us region eu region eu region asia region us region Cloud Load Balancing GCP TMBDLCPBSE TMBDLCPBSE TMBDLCPBSE
29. ͳ͓ɺϝϧΧϦͰ࣮ࡍʹ࢖ΘΕ͍ͯΔΫϥΠΞϯτ͸Perl੡ͷslacklog $ slacklog -c test --log-channel test -- ls notfound
30. pvpool
31. pvpool • ϝϧΧϦͷ঎඼Ӿཡ਺ΛΧ΢ϯτΞοϓ͢ΔͨΊͷ࢓૊Έ • ࢦఆ͞Εͨ঎඼IDͷӾཡ਺ͷΧ΢ϯτΞοϓ͓Αͼ औಘͷͨΊͷAPIΛఏڙ • ※ ը໘͸։ൃ༻ͷ΋ͷͰ͢
32. ঎඼Ӿཡ਺Χ΢ϯτΞοϓͷϫʔΫϩʔυͱٕज़తͳཁ݅ • ঎඼Ӿཡʹؔ͢ΔAPI΁ͷϦΫΤετ਺͸ϝϧΧϦͷશAPIதTOP3ʹೖΔ • ϝϧΧϦAPI΁ͷશϦΫΤετ਺͸ϐʔΫ࣌Ͱ໿56,000req/sec • ੑೳཁ݅ • ग़དྷΔݶΓ௿஗ԆͰԠ౴Ͱ͖ΔAPIΛఏڙ͢Δʢ໨҆ɿ਺ᶡඵҎ಺ʣ • ग़དྷΔݶΓϦΞϧλΠϜʹӾཡ਺Λ࠷৽ͷঢ়ଶʹ൓ө͢Δʢ໨҆ɿ਺ඵ Ҏ಺ʣ
33. System architecture of pvpool • nginx • pvpoold • Powered by Go (net/http) • MySQL pvpoold pvpoold nginx MySQL pvpool.local by consul DNS : HTTP protocol : MySQL protocol The gopher was designed by Renée French
34. ఏڙ͢ΔAPI͸2ͭ • POST /items/pvc • ༩͑ΒΕͨ঎඼IDຖͷPVΛΧ΢ϯτΞοϓ • POST /items/pv • ༩͑ΒΕͨ঎඼IDຖͷPVΛฦ͢ • ঎඼ID͸ෳ਺ࢦఆՄೳ
35. pvpoold Internals POST /items/pvc { "item_ids" [ ... ] } PVCHandler Sharding Item IDs MySQL Slot Slot Slot Slot Slot Slot Pooling query Flushing periodically synchronous processing asynchronous processing
36. pvpoold Internals • PVCHandler • HTTPϋϯυϥؔ਺ • Slot • Χ΢ϯτΞοϓ༻ϓϩηεϢχοτ by ΰϧʔνϯ
37. pvpoold Internals • Slot಺ͷొ৔ਓ෺ • Slot -> chan types.ItemID • PVMap -> map[types.ItemID]int • Storer -> Slot͔Β঎඼IDΛϑΣονͯ͠PVMap಺ͷ஋Λߋ৽ • Flusher -> ఆظతʹPVMap಺ͷ஋ΛऔΓग़ͯ͠MySQLʹߋ৽ΫΤϦΛൃߦ • Storer ͱ Flusher ΋·ͨΰϧʔνϯ
38. Slot overview Slot Storer Flusher PVMap dequeue item ID Store pv per item Fetch pv per item periodically MySQL Issue SQL to count up
39. Storer func (slot *PVCSlot) storer() { for { itemID := <- slot.Slot slot.Lock() slot.PVMap[itemID]++ slot.Unock() } }
40. Flusher func (slot *PVCSlot) flusher(interval time.Duration) { ticker := time.NewTicker(interval) for { <- ticker.C slot.Lock() if err := slot.flush(); err != nil { // transaction // error handling } slot.Unlock() } }
41. StorerͱFlusherͷ໾ׂ • Storer • MySQL΁ൃߦ͢ΔSQLΛ঎඼IDຖʹ·ͱΊΔ • eg. UPDATE pv = pv + 5 instead of five of UPDATE pv = pv +1 • Flusher • ҰఆִؒͰMySQLʹߋ৽༻SQLΛൃߦ͢Δ • ঎඼͕Ӿཡ͞ΕΔ౓ʹUPDATE͕૸ΔͷΛආ͚ͯෛՙΛ࿨Β͛Δ
42. go-httpstats
43. go-httpstats • ʢػೳ͕গͳ͍ͱ͍͏ҙຯͰʣখن໛ͳHTTPαʔό޲͚ͷϝτϦΫεऔಘϥΠϒϥϦ • ϦΫΤετ਺ʢτʔλϧͱεςʔλείʔυຖʣ • ϨεϙϯελΠϜʢmaxɺminɺavgɺpercentileຖʣ • Not published • ։ൃͨ͠ͷ͸ࠓ೥৽ଔೖࣾͷSoftware Engineer, Infrastructure • ͢Ͱʹ͋Δ΋ͷΛ୳͚ͨ͠Ͳɺͪΐ͏Ͳ͍͍΋ͷ͕ͳ͔ͬͨͷͰࣗ࡞
44. Getting Started - go-httpstats mw := stats.New() handler := mw.WrapHandleFunc( http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(“Hello, World”)) })) statsHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { d, err := mw.Data() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } }) http.Handle(“/“, handler) http.Handle(“/stats”, statshandler) http.ListenAndServe(“:9999”, nil)
45. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } }
46. Getting Started - go-httpstats $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
47. Sampling factor const ( // DefaultBufferSize is buf size sampling response time for percentile and average DefaultBufferSize = 1000 // DefaultSamplingFactor is factor to sample response time for percentile and average DefaultSamplingFactor = 1 ) // … // mw := stats.New() // NewCapa returns a new statistics structure // buffer size of HTTP requests is allocated length of bufsize // If bufsize is less than 2, NewCapa returns error mw, err := stats.NewCapa(bufsize, factor)
48. Introducing sampling factor • percentileͷܭࢉ͸ͦͦ͜͜ॏ͍ • avgͷܭࢉͱҧͬͯιʔτ͕ඞཁ • ϨεϙϯελΠϜܥ͸౷ܭσʔλͳͷͰඞͣ͠΋શ෦औΔඞཁ͸ͳ͍ • ϫʔΫϩʔυͷܹ͍͠ՕॴͩͱόοϑΝॻ͖ࠐΈͷMutexڝ߹͕ܹͯ͘͠ ໰୊ʹͳͬͨͷͰɺͦͷϫʔΫΞϥ΢ϯυͱͯ͠ಋೖ
49. ࠷ऴతʹ͸MackerelͰάϥϑԽ $ curl -s http://127.0.0.1:9999/stats | jq ‘.request’ { “count”: 13821031, “status_count”: { “200”: 13820000, “400”: 31, “401”: 0, “403”: 0, “404”: 1000, “500”: 0, … } } $ curl -s http://127.0.0.1:9999/stats | jq ‘.response’ { “max_time”: 0.0238210, “min_time”: 0.00015, “average_time”: 0.0035, “percentiled_time”: { “90”: 0.0109, “95”: 0.0130, “99”: 0.0215 } }
50. mackerelϓϥάΠϯ΋GoͰ։ൃ • MackerelϓϥάΠϯ։ൃ༻ϔϧύʔΛ࢖͏ͱָ • github.com/mackerelip/go-mackerel-plugin • specϑΝΠϧ࡞੒ޙɺmake mackerel-plugin-nameͰRPMύοέʔδΛϏ ϧυ & ഑෍ • ฐࣾͷಠࣗMackerelϓϥάΠϯ͸Go & Perl੡ • https://speakerdeck.com/kazeburo/mackerel-day
51. mfc
52. mfc • In-house Fastly CLI tool at Mercari • Early development phase • ϝϧΧϦͰ͸2017೥͔ΒFastlyΛར༻த • ػೳͷ໢ཏੑࣗମ͸ͦΜͳʹߴ͘ͳ͍ • ඞཁʹͳͬͨΓɺ͋Δͱศརͦ͏ͳػೳΛ౎౓ΠϯςάϨʔτͯ͠Δײ͡ • ΏΔ;Θʹ։ൃͯ͠·͢
53. mfc configuration $ cat ~/.fastly/conf.ini service = xxx apikey = yyy …
54. Usage of mfc • $ mfc Usage of mfc: config the utility for mfc configuration service the utility for fastly service acl the utility for fastly ACL (etc…) • ػೳྖҬຖʹαϒίϚϯυΛఆٛ • ACL, Service, Version౳ • ౰ॳ͸ผʑͷϓϩάϥϜ͚ͩͬͨͲɺ૿͖͑ͯͨͷͰ౷߹ switch args[1] { case “config”: return config.NewCLI().Run(args) case “service”: return service.NewCLI().Run(args) case “acl”: return acl.NewCLI().Run(args) case “…” … } ಈ࡞Πϝʔδ
55. mfcͷػೳ(Ұ෦) $ mfc config -h Usage of config: mfc config list output mfc configuration $ mfc service -h Usage of service: mfc service vcldiff from_version to_version output difference between from and to mfc service versions output active and latest versions $ mfc acl -h Usage acl: mfc acl create create acl mfc acl remove remove acl mfc acl add add given entry to acl mfc acl del delete given entry in acl mfc acl show output acl mfc acl list output acl entries mfc acl sync sync acl with given provider file
56. imageflux-cli
57. imageflux-cli • In-house ImageFlux CLI tool for Mercari • github.com/mercari/imageflux-cli • ImageFlux ~ Ϋϥ΢υը૾ม׵αʔϏε • https://www.sakura.ad.jp/services/imageflux/ • Reference • ImageFluxΛར༻ͨ͠ը૾഑৴ͷ࠷దԽʙಈతϦαΠζͱWebPม׵ʙ • https://tech.mercari.com/entry/2018/01/30/161001
58. imageflux-cli configuration $ cat ~/.imageflux/conf.ini token = xxx
59. Usage of imageflux-cli $ imageflux-cli -h Usage: imageflux-cli cache.lookup -k $url imageflux-cli cache.delete -k $url imageflux-cli signature -s $secret -p $path
60. ΄͔ʹ΋͍Ζ͍Ζ • Gaurun - Building high performance push notification server in Go • https://speakerdeck.com/cubicdaiya/building-high-performance-push-notification-server-in- go • chocon - Simple proxy server for persisting connection between upstream servers • https://speakerdeck.com/kazeburo/cloud-connect-the-world-as-a-glue-aws-dev-day-2017? slide=53 • Mercari API Gateway • https://go-talks.appspot.com/github.com/tcnksm/talks/2018/07/mercarigo/microservices-api- gateway.slide#1 • etc… ( There are too many works and products in Go at Mercari)
61. ·ͱΊ • Software Engineer, Infrastructureͷ঺հ • ओʹGoͰαʔόαΠυϛυϧ΢ΣΞ΍ΒCLIπʔϧͷ։ൃΛ΍ͬͯ·͢ • ݱࡏ3ਓ(͏ͪ1ਓ͸SREͱ݉຿)
62. We are hiring Go Software Engineer! https://careers.mercari.com/job/