SRE大全 メルカリ編 後半 #hbstudy 75 / SRE Taizen Mercari 2 hbstudy#75

SRE大全 メルカリ編 後半 hbstudy#75

1. SREେશ: ϝϧΧϦฤ [ޙ൒] 2017/08/17 hbstudy#75 Masahiro Nagano @kazeburo
2. AGENDA • ޙ൒ • PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ • ηΩϡϦςΟͷऔΓ૊Έ(ύεϫʔυϦετ߈ܸࣄྫ) • ϝϧΧϦSREͷࠓޙ • SREͷ໾ׂͷݱࡏͱະདྷɺϚΠΫϩαʔϏε
3. PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ
4. PHP7.1Խ͠·ͨ͠
5. CPU Usage 1)1
   Y 1)1
   Y CPU࢖༻཰൒෼!!
6. Response time(95percentile) ‒PHP 7.1.x ‒PHP 5.6.x Response Time 20-30%վળ!!!
7. PHP 7.1Խ • 5.6ܥ͔ΒͷΞοϓάϨʔυ • CIͰͷςετɺΫϥΠΞϯτͷࣗಈςετɺQAʹΑΔखಈςετΛܦͯ׬શҠ ߦ • ͦͷଞҠߦͷޮՌ • CIʹ͔͔Δ࣌ؒ΋୹ॖ
8. ΊͰͨ͠ΊͰͨ͠
 Ͱ͸ͳͯ͘..
9. PHP7.1ԽҎ֎ͷ PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ
10. JSONಡΈࠐΈ • ػೳͷ֦ॆʹΑΓɺࠃࡍԽରԠͷmessage֨ೲϑΝΠϧ͕਺ेKBʹ๲ΒΉ • messageϑΝΠϧ͸΄ͱΜͲͷϦΫΤετͰಡ·ΕΔͨΊෛՙ͕૿େ
11. JSONಡΈࠐΈ • 1୆ͷαʔόͰར༻͞Εͨmessage keyΛϩάʹه࿥ɺΑ͘ར༻͞ΕΔ messageΛಛఆ • grep | sort | uniq -c | sort -n • Α͘ར༻͞ΕΔmessage͚ͩΛूΊͨ essential.json Λ࡞੒ • ϕϯνϚʔΫ All: 9msec => Essential: 1msec ҎԼ
12. JSONಡΈࠐΈ: ߴ଎Խ ਺msecͷվળ!!
13. େྔͷྫ֎Ϋϥε • app_exception.php ͱ͍͏ϑΝΠϧʹ330ݸͷྫ֎Ϋϥε͕ఆٛ • ػೳ͕૿͑Ε͹ྫ֎Ϋϥε΋૿͑Δ • ຖϦΫΤετͰಡΈࠐ·ΕΔ
14. େྔͷྫ֎Ϋϥε • αʔό্Ͱى͖͍ͯΔ࣮ࡍʹى͖͍ͯΔྫ֎Λूܭ • ൃੜ݅਺ͷଟ͍΋ͷΛ app_exception_base.php ʹ੾Γग़͢ • ϕϯνϚʔΫ 10msec => 1msec • try-catchͰϩάʹͰͳ͍΋ͷ΋͋ͬͨ
15. େྔͷྫ֎Ϋϥε: ରࡦޙ ਺msecͷվળ!!!
16. PHP(Opcache) ಈ࡞Πϝʔδ memory Destroy!! start end DB API shared mem php php php php php php php php php Response Request OPCODE Copy ࣮ߦ ࣮ߦ ࣮ߦ ίϯύΠϧ݁ՌͰ͸ͳ͘ɺOPCODEΛcache͓ͯ͠Γɺhash΍class͸౎౓ߏங = ௿଎ ϦΫΤετΛ௒͑ͯcache͢Δͷ͸೉͍͠
17. PHPͷಈ࡞ ֎෦ͱͷ௨৴ PHP಺෦ͷॲཧ hash΍classͷߏஙͳͲΛؚΉPHPͷಈ࡞͕࣌ؒ௕͍ ֎෦΁ͷ໰͍߹Θͤͱ͋Θͤͯνϡʔχϯά͢ΔϙΠϯτ
18. ϘτϧωοΫΛ୳͢ํ๏ • NewRelic • σʔλϕʔεɺmemcachedɺ֎෦APIͷݺͼग़͠ճ਺ͱ࣌ؒɻN+1΍ɺpreload࿙Εͷൃݟ • PHP಺෦ͷτϨʔε΋Ͱ͖Δ • strace • System Call ͷτϨʔε • PHP಺෦Ͱى͖͍ͯΔ͜ͱ͸τϨʔεͰ͖ͳ͍ • System CallͱιʔείʔυΛݟൺ΂ͯϘτϧωοΫΛ୳͢ => Ͳ͏΍ͬͯʁ
19. strace $ sudo strace -tt -s 256 -p $(pgrep -n -f httpd) |& tee strace.txt 15:29:31.122566 read(20, “GET /...) 15:29:31.122844 stat( ... 15:29:31.123914 getcwd("/", 4095) = 2 15:29:31.123936 chdir(“/var/www/vhosts/app/webroot") = 0 15:29:31.123966 setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={60, 0}}, NULL) = 0 15:29:31.123987 fcntl(18, F_SETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1}) = 0 15:29:31.125310 open(“/var/www/vhosts/app/foo/bar.json”) = 0 ## Ctrl-C $ less strace.txt PHP ಺෦ͷಈ࡞͸traceͰ͖ͳ͍ ιʔείʔυͱরΒ͠߹ΘͤΔʹ͸ώϯτͱͳΔsystem call͕΄͍͠ => OpcacheͷઃఆΛΈΔ
20. php.ini in Production opcache.enable = 1 opcache.revalidate_freq = any opcache.validate_timestamps = 0 ࠷େੑೳ opcache.enable = 1 opcache.revalidate_freq = 1 opcache.validate_timestamps = 1 ϝϧΧϦӡ༻த ϑΝΠϧͷߋ৽͸νΣοΫ͠ͳ͍ ϑΝΠϧͷߋ৽͸νΣοΫ͢Δ 1ඵ͝ͱʹνΣοΫ nginx dynamic upstreamΛ࢖͍ɺbalancer͔Β֎ͯ͠ɺ rsync => 1ඵsleep ͯ͠balancerʹ໭͢
21. php.ini for strace opcache.enable = 1 opcache.revalidate_freq = 0 opcache.validate_timestamps = 1 TUSBDF࣮ߦ༻ ຖϦΫΤετͰνΣοΫ͢Δ 18:21:01.253488 stat("/var/www/current/app/webroot/index.php", {st_mode=S_IFREG|0644, st_size=356, ...}) = 0 18:21:01.253572 stat("/var/www/current/dietcake/dietcake.php", {st_mode=S_IFREG|0644, st_size=1013, ...}) = 0 18:21:01.253697 stat("/var/www/current/dietcake/core/exception.php", {st_mode=S_IFREG|0644, st_size=46, ...}) = 0 18:21:01.253775 stat("/var/www/current/dietcake/core/inflector.php", {st_mode=S_IFREG|0644, st_size=289, ...}) = 0 .... 18:21:01.254005 stat("/var/www/current/dietcake/core/dispatcher.php", {st_mode=S_IFREG|0644, st_size=1300, ...}) = 0 18:21:01.254044 stat(“/var/www/current/app/aaaaaaa.php", {st_mode=S_IFREG|0644, st_size=5457, ...}) = 0 18:21:01.254135 stat("/var/www/current/app/app_xxx.php", {st_mode=S_IFREG|0644, st_size=10423, ...}) = 0 18:21:01.254618 stat("/var/www/current/app/app_exception.php", {st_mode=S_IFREG|0644, st_size=45680, ...}) = 0 18:21:01.257390 stat("/var/www/current/app/app_yyy.php", {st_mode=S_IFREG|0644, st_size=1133, ...}) = 0
22. App࠷దԽͷϞνϕʔγϣϯͱλΠϛϯά • Appαʔό͕૿͑Δͱ • ނো཰͕૿͑ɺσϓϩΠ࣌ؒ΋৳ͼΔ • AppαʔόΛ૿΍ͯ͠΋஗͘ͳΔͷ͸๷͛Δ͕ɺϨεϙϯε଎౓͕มΘΒͳ͍ • ϝϧΧϦͰ͸AppαʔόͷCPU࢖༻཰͕ఆظతʹ50%Λ௒͑࢝ΊͨΒରࡦΛߦ͏ • αʔόͷ௥Ճ or ΞϓϦέʔγϣϯͷίʔυͷ࠷దԽ • قઅతཁҼͰΞΫηε͕૿͑ΔͷͰɺ࣌ؒͷ͋Δͱ͖ʹ४උɾ࠷దԽͷωλΛ΋͓ͬͯ͘ • αʔόαΠυΤϯδχΞͱऔΓ૊Ή͜ͱ͕Ͱ͖Ε͹ͳ͓ྑ͍ • SREݚमͰऔΓ૊ΉͳͲ
24. ηΩϡϦςΟͷऔΓ૊Έ ύεϫʔυϦετ߈ܸͷࣄྫ
25. ύεϫʔυϦετ߈ܸ • ͓٬͞·ͷΞΧ΢ϯτʹରͯ͠ɺϥϯμϜͳύεϫʔυ΋͘͠͸ผͰ࿙Ӯ ͨ͠ύεϫʔυจࣈྻΛ࢖͍ϩάΠϯΛࢼߦ • ͞·͟·ͳن໛ͷ߈ܸ͕ߦΘΕ͍ͯΔ • ·ͣ͸ؾ෇͘͜ͱ͕Ͱ͖Δ࢓૊Έͷߏங͕ॏཁ
26. ύεϫʔυϦετ߈ܸͷݕ஌ • ϩάΠϯࣦഊΛAPIͷϩάͱͯ͠࢒͢ • ΞΧ΢ϯτͳ͠/ύεϫʔυҧ͍ • ϩάΛnorikraͰूܭɺmackerelͰՄࢹԽ
 ͱ؂ࢹ • ϝϧΧϦ͕TVʹऔΓ্͛ΒΕΔͱΞϥʔτ͕དྷΔ͜ͱ΋
27. ύεϫʔυϦετ߈ܸ͔Βͷ๷ޚ • ൺֱత୯७ͳ߈ܸ͸ΞϓϦέʔγϣϯ಺Ͱࣗಈతʹ๷ޚ • ಉҰͷϝʔϧΞυϨεʹΑΔϩάΠϯࢼߦ • ಉҰͷIPΞυϨεʹΑΔϩάΠϯࢼߦ • ߈ܸͱ൑அ͞Εͨ৔߹͸֘౰IPɺ֘౰ΞΧ΢ϯτΛҰఆظؒڋ൱ • ͓٬͞·ʹύεϫʔυͷϦηοτଅ͢
28. େن໛ͳύεϫʔυϦετ߈ܸ • ੈքத͔Β߈ܸ͕ߦΘΕΔ • 2016೥ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫη εݩͷࠃ • 1IPʹ͖ͭɺ਺ճͷϩάΠϯࢼߦͰ ͋ΓɺࣗಈͰ͸๷͛ͳ͍ ͦͷଞ 18% Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30%
29. େن໛ͳύεϫʔυϦετ߈ܸ΁ͷඋ͑ • ߈ܸ͞Ε΍͍͢Web൛Ͱ͸CAPTCHAΛಋೖ • Client Reputation ͷར༻ • GeoIP • Anonymous proxy detection • IP Reputation
30. Client reputation http://www.cyren.com/security-center/ip-reputation-check https://www.ip2location.com/demo
31. ߈ܸݩIP Reputation -PX  .JEEMF  )JHI  4ׂ͸๷͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ* ༷ʑͳ৘ใϦιʔεΛ૊Έ߹ΘͤͯαʔϏεͷ҆શੑΛߴΊ͍ͯ·͢ * ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰ͸ͳ͍ͷͰଟগζϨ͕͋Γ·͢
32. ϝϧΧϦSREͷࠓޙ SREͷ໾ׂͷݱࡏͱະདྷɺMicroservice
33. Mercari SRE (࠶ܝ) • ͍ͭͰ΋շద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯά͸શ෦΍Δʯ • ʮ։ൃʯͱʮӡ༻ʯͷ෼཭ • Production؀ڥͷ࡞ۀશൠΛSRE͕ड͚࣋ͭ
34. Mercari SREͷҐஔ͚ͮ ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗɺमਖ਼ґཔ
35. SREͷ՝୊ • ૿͑Διϑτ΢ΣΞΤϯδχΞɺαʔϏε/ػೳʹରͯ͠SREͷਓ਺ͷ૿Ճ ͕௥͍͔ͭͳ͍ • Production؀ڥͱιϑτ΢ΣΞΤϯδχΞͷҙࣝͷဃ཭
36. SREͷ՝୊ ೥લ ൒೥લ ݱࡏ 4& 43&
37. MicroserviceԽ • ։ൃऀʹ Ownership • ։ൃ૊৫ͷεέʔϥϏϦςΟͱॊೈੑΛߴΊɺαʔϏεΛΑΓ޿͍͛ͯ͘ • ʮϝϧΧϦνϟϯωϧʯͳͲͷ৽͍͠ػೳ • Atte΍Χ΢ϧͳͲͷ࿈ܞΞϓϦ • ஍Ҭ͝ͱͷಠࣗαʔϏε => US/UKͰ͸ਐΜͰ͍Δ
38. Microservice in US ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) Task Mechanical Turk Non-Service Specific backend for frontends GKE Core/طଘAPI protobuf JSON over HTTPs GRPC Service A Service B Spanner GKE GKE Deployج൫ SaaS PubSub
39. MicroserviceԽʹΉ͚ͯ • Requirements(ཁ݅)ͷࡦఆ • Microserviceج൫ͷߏங • Deploy, Log, Database, Monitoring... • ຊ൪ͷΞʔΩςΫνϟͱͯ͠ͷ࠾༻ • લਐͭͭ͠՝୊Λݟ͚ͭɺͻͱͭͻͱͭղܾ͍ͯ͘͠
40. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻
41. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ Microserviceج൫ ϚΠΫϩαʔϏεԽ͕ਐΉ͜ͱͰ ͚ͩ͜͜ʹͳ͍ͬͯ͘ͷ͔ʁ
42. ൱ ͱߟ͍͑ͯ·͢
43. ࠓޙͷMercari SRE ։ൃνʔϜ SRE BOT Infrastructure σʔλϕʔε࡞ۀ/ௐࠪґཔ εΩʔϚ/ΞʔΩςΫνϟ૬ஊ ߏஙʗӡ༻ σʔλϕʔε࡞ۀ ௐࠪ ߏஙɾར༻ σϓϩΠ Log ಗ໊DB ར༻ ར༻ ൓ө/ࢀর ໰୊ௐࠪͷڠྗ मਖ਼ґཔ Microserviceج൫/Requirement Mircoservice Infrastructure σϓϩΠ ߏங ࣗಈԽ͞Εͨӡ༻ ੵۃతʹؔΘ͍ͬͯ͘
44. ࠓޙͷMercari SRE • Microserviceͷج൫ߏஙɾӡ༻ • SREͷϊ΢ϋ΢Λ΋ͬͯαʔϏεʹੵۃతؔ༩ • εέʔϥϏϦςΟɺՄ༻ੑͷվળ • ࣗಈԽͷਪਐɺΦϖϨʔγϣϯͷվળ • ηΩϡϦςΟͷ࣮ࢪ • Globalͳnetwork/infrastructureͷޮ཰ӡ༻
45. SRE More!!! https://twitter.com/kazeburo/status/890131903529054210
46. Mercari SRE More!!! Microservice, Automation, Performance, Scalability Database, Network, Distributed System, OS, Cloud, Hardware, Security SWE + ༷ʑͳ஌ࣝɾܦݧΛੜ͔ͯ͠ ʮ৽ͨͳՁ஋ΛੜΈग़͢ੈքతϚʔέοτΛ૑Δʯ ৴པੑΛ࣋ͬͯࢧ͑Δ શํ໘ͰΑΖ͓͘͠ئ͍͠·͢!!
47. Ҏ্