Share
Embed
成長し続けるインフラストラクチャとメルカリの挑戦/mercari infrastructure and software
2018/07/29 @ July Tech Festa 2018
kazeburo
July 30, 2018
Transcript
62 slides
-
Me • Masahiro Nagano • twitter.com/kazeburo, github.com/kazeburo • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ
Site Reliability Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ -
Me • 2000ࠒ ~ 2006: ژͰελʔτΞοϓࢀՃ • ΤϯδχΞ1-2໊ɻ։ൃΛ͠ͳ͕ΒΠϯϑϥͷ໘ΛݟΔɻDC࡞ۀͬͨ • ΞϓϦέʔγϣϯͷνϡʔχϯάɺۭ͍ͨϦιʔεͰ৽ػೳͷՃͱ͍͏αΠΫϧ
• 2006 ~: mixi • ʮΞϓϦέʔγϣϯӡ༻νʔϜʯॴଐɻDCʹߦ͔ͳ͍ӡ༻ΤϯδχΞ • σʔληϯλʔνʔϜ͕༻ҙͨ͠αʔόͷೳྗΛҾ͖ग़͠ɺΞϓϦέʔγϣϯΤϯδ χΞ͕࡞ͨ͠ίʔυΛ࠷ߴͷܗͰಈ͔͢ͷ͕ࣗΒ(νʔϜ)ͷׂ -
Me • 2010 ~: livedoor => NHN Japan => LINE
• livedoorLINEϑΝϛϦʔͷαʔϏεΛԣஅͯ͠ΠϯϑϥετϥΫνϟ ύϑΥʔϚϯεͷվળ • livedoor Blog ͷMySQLνϡʔχϯά • LINEϑΝϛϦʔαʔϏεͷಥൃΞΫηεͷରԠ • 2015/02 ~ : Mercari -
SREͱͷग़ձ͍ • 2012/7 ༑ਓͱͷIRCͰͷձ͔Βڭ͑ͯΒ͏ • GoogleͷڊେͳΠϯϑϥͱαʔϏεͷՔಇɺ҆ఆੑΛ୲͢ΔνʔϜ͕SRE • ʮSite Reliability Engineers:
“solving the most interesting problems”ʯهࣄ͕ެ։͞Εͨࠒ https://research.googleblog.com/2012/07/site-reliability-engineers-solving-most.html • twitter ͷbioൃදεϥΠυʹʮSite ReliabilityʯΛՃͯ͠ҙࣝ https://www.slideshare.net/kazeburo/yapc2102mysql/2 (2012/9) • 2015/11 ϝϧΧϦʹͯνʔϜ໊ͱͯ͠ఏҊ -
࠷ۙͷ׆ಈ • ొஃ • AWS Dev Day Tokyo 2017 •
YAPC::Fukuoka 2017, YAPC::Okinawa 2018 • Manabiya Teratail Developer Days • هࣄ • WEB+DB PRESS Vol.88, Vol.92-97 ࿈ࡌ, Vol.100 -
AGENDA • ࣗݾհ • ϝϧΧϦʹ͍ͭͯ • ϝϧΧϦͷ Infrastructure History #1
- αʔϏε֦େ • Infrastructure Λࢧ͑Δ Software ࣄྫ * 2 • ϝϧΧϦͷ Infrastructure History #2 - Microservices -
ϝϧΧϦ • ຊ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3)
ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ(͓ۚͷΓͱΓ͕ࣾؒʹհࡏ) • ಗ໊ૹ -
Infrastructure in 2017 DNS: Amazon Route53 CDN: Akamai, CloudFront Storage:
Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel ! " # -
! " # Infrastructure in 2018 DNS: Amazon Route53 CDN:
Akamai, Fastly, ImageFlux(JP) Storage: Amazon S3 Analysis: Google BigQuery / Monitoring: Mackerel, DataDog + + -
Infrastructure History (1) • 2013/07 JP ϦϦʔε • ͘͞ΒΠϯλʔωοτͷʮ͘͞ΒͷVPSʯ1ʹWebDBࡌͤͨߏͰ։࢝ •
Infrastructure ઐऀ͕͍ͳ͍தͰɺ։ൃऀʹۙͳج൫Λબ • ϦϦʔεޙ2ϲ݄Ͱʮ͘͞ΒͷΫϥυʯʮઐ༻αʔόʯҠߦ -
ʮ͘͞Βͷઐ༻αʔόʯ • Metal as a Service • ཧαʔόͳΒͰͷύϑΥʔϚϯε • ωοτϫʔΫͱϋʔυΣΞͷอक
͘͞ΒΠϯλʔωοτ͕୲ • ʮ͘͞ΒͷΫϥυʯͱଓ͕Մೳ -
Infrastructure History (2) • 2014/09 " US ϦϦʔε • AWS
(Oregon) ʹͯαʔϏεߏங • JPϦϦʔε͔Β͠Β͘ܦͪɺ։ൃऀʹAWSܦݧऀ͕૿Ճ • ͦΕͰ Infrastructure ઐऀগͳ͘ɺRDSElastiCacheϚωʔδυαʔϏεΛ ར༻ͯ͠αʔϏεΛߏங • USࠃͷ MaaS Λݕ౼͕ͨ͠ɺUSͰͷαʔϏεͷ༧͕͘͠ɺΫϥυͷॊ ೈ͞Λ JP ΑΓॏཁࢹ -
Infrastructure History (3) • 2015/11 SREνʔϜൃ • JP/US ͷΞʔΩςΫνϟΛվળ͠ɺαʔϏεͷ৴པੑͱεέʔϥϏϦςΟͷ ্ʹऔΓΉ
• memcachedͷ༗ޮ׆༻ɺN+1ରࡦɺSQLνϡʔχϯάɺMaster/Slave͍͚ • HTTP/2ԽɺPHP upgrade/࠷దԽ -
Architecture nginx nginx nginx DNS-RR App App App App App
App MySQL MySQL memcached memcached util util cloud cloud •γϯϓϧͳ3ߏ •ΫϥυͰEC2/GCE (αʔό) Λத৺ʹߏ • ՄೳͳݶΓڞ௨ͷΞʔΩςΫνϟΛ࠾༻ • ϚωʔδυαʔϏεΛαʔόʹϦϓϨΠε -
Managed to Server (USͷྫ) • ELB • Internal ELB •
ElastiCache • RDS DNS-RR, NGINX, OpenResty Internal DNS (BIND), Consul memcached on EC2 MySQL on EC2 ngx_dynamic_upstreamʹΑΔແఀࢭɾߴdeploy ಥൃతͳΞΫηεੑɾઃఆөվળ OS/Kernel tuningʹΑΔScalability্ όοΫΞοϓखॱͷඪ४ԽɾRolling Schema Upgrade -
Infrastructure History (3) • 2017/03 UK ϦϦʔε • ৽͍ٕ͠ज़ͱͯ͠ʮGCPʯ্ͰαʔϏεΛߏங •
৽͍͠ΠϯϑϥͰ͋Δ͕ɺΞʔΩςΫνϟAnsible Playbookͷ࠶ར༻ʹΑ Γ࣮ͷ͋Δج൫ͰαʔϏε։࢝ -
Consul • Service Discovery • DNSΠϯλʔϑΣΠεΛར༻͢Δ͜ͱͰ෦LBͱͯ͠ར༻ • ෦͚API, SMTPͳͲ෦͚αʔϏεͷΤϯυϙΠϯτ •
Configuration Deployment • Distributed Lock -
keyword suggest ͷࣄྫ docker Consul check & register service Index
docker Consul check & register service Index App App App Consul ΛͬͨService Discovery http://keyword-suggest.service.dc.consul:8080/ # service.json { "service": { "name": "keyword-suggest", "tags": ["production"], "port": 8080, "checks": [ { "script": "/usr/local/bin/check-http -u 'http://127.0.0.1:8080/hc'", "interval": "10s", "timeout": "5s" } ] } } -
keyword suggest ͷࣄྫ docker Consul Index # cron consul lock
-verbose keyword_suggest_reload /path/to/check_and_reload gcloud docker -- pull $IMAGE_PATH:$LATEST_TAG |& tee $PULL_LOG echo -n $LATEST_TAG > $CDIR/tag/latest if grep "Downloaded newer image" $PULL_LOG > /dev/null 2>&1; then echo "[[[FOUND NEW IMAGE]]]" else echo "NO UPDATE. exit.." exit fi consul maint -enable -service suggest sleep 2 restart keyword-suggest sleep 5 consul maint -disable -service suggest Consul Λͬͨࢄlock + ϝϯςφϯεϞʔυʹΑΔҰ࣌తͳαʔϏεఀࢭ cron ࠶ىಈ ϝϯςφϯεϞʔυ ແఀࢭͷIndexΞοϓσʔτ -
OpenResty in Mercari nginx nginx DNS-RR OpenResty Apache mod_php OpenResty
Apache mod_php •ApplicationαʔόʹOpenRestyΛىಈ •άϩʔόϧଆnginxͱOpenRestyͷؒHTTP KeepAlive༗ޮ •TCPଓճΛݮΒ͢͜ͱͰύϑΥʔϚϯε্ •ΈࠐΈluaʹΑΓߴػೳͳL7ϩʔυόϥϯγϯά •ಥൃతͳେྔΞΫηεΛΒ͛Δ keepalive -
Concurrency management with OpenResty OpenResty Apache mod_php nginx •ಛఆͷͷߪೖॲཧ͕ूதʹΑΓDBͷෛՙ্ঢ •SELECT
FOR UPDATE ͕ඦཹ •Busy LockʹΑΓCPUͷ༻͕ٸ্ঢ •DBͷ͍߹ΘͤΑΓલʹฒྻͷௐ •ͨͩ͠PHPͰwaitΛ͍ΕΔͱApacheͷϓϩηε͕ރׇ ͢ΔՄೳੑ SELECT FOR UPDATE -
Concurrency management with OpenResty OpenResty Apache mod_php memcached add lock_key
ࣦഊͨ͠ΒԿճ͔retry nginx location = /buy { rewrite_by_lua_block { ngx.req.read_body() local args, err = ngx.req.get_post_args() local lock_key = "buy/" .. args.item_id -- #memcachedʹadd local locked = lock_with_memcached(lock_key) -- #addʹࣦഊͨ͠Β503 if not locked then ngx.exit(ngx.HTTP_SERVICE_UNAVAILABLE) end ngx.ctx.lock_key = lock_key } log_by_lua_block { local lock_key = ngx.ctx.lock_key -- #ϦΫΤετऴΘͬͨΒcacheΛআ if lock_key then ngx.timer.at(0,unlock_with_memcached,lock_key) end } proxy_pass http://127.0.0.1; } lock͔ͯ͠Β upstreamʹproxy ϦΫΤετྃޙʹআ -
Microservices • αʔϏεͷ Resilience Λ্ͤ͞Δ • ࡉ͔͍୯ҐͰͷεέʔϦϯάɺোͷ • νʔϜɾ৫ͷ Scalability
ΛߴΊΔ • αʔϏε։ൃͷΛ͞Βʹ͍͋͛ͯͨ͘Ί • 1000໊Ҏ্ͷΤϯδχΞ৫Λࢤ • Microservice PlatformΛ࡞ΓMicroservicesΛਐΊΔ -
Microservices Backend Team A Backend Team B Backend Team C
Develop Service A Develop Service B Develop Service C QA Deploy Operation QA Team SRE -
Microservices Backend Team A Backend Team B Backend Team C
Develop Service A Develop Service B Develop Service C QA Deploy Operation QA Deploy Operation QA Deploy Operation -
US Re-Architecture • US marketʹΑΓ࠷దԽ͘͢ Client ΛFull Renewal • MicroservicesͷroutingΛߦ͏API
GatewayΛGolangͰ࣮ • AWS্ͷMonolith APIΛWrap • ؇͔ͳҠߦΛ࣮ݱ API Gateway search personalization offer gRPC JSON over HTTPs Protocol Buffers over HTTPs gRPC gRPC Monolith API -
API Fork • 3ͭͷRegionͰڞ༗͍ͯͨ͠Monolith APIͷίʔυΛ US,UK ͱ JP Ͱ •
ࣗregionͷมߋ͕ଞregionʹӨڹ͢Δ͜ͱΛ͑ΔɻௐɾQAίετݮ • ΑΓ֤ࠃͷࣄʹ͋ͬͨ։ൃΛ֤ࠃͰߦ͏ • US,UKͷݱ࠾༻ਐల -
API Gateway in JP • JPͰMicroservicesΛਐΊΔͨΊ API GatewayΛಋೖ • GolangͰ࣮͍ͯ͠Δ͕USͱผ࣮
• Clientͷมߋͳ͘Protocolجຊҡ࣋ • Golangͷnet/httpΛ༻ͭͭ͠ɺ DNS cacheɺRequest bufferingͳͲͷՃ API Gateway JSON over HTTPs JSON over HTTPs ProtoBuf over HTTPs ServiceA ServiceC ServiceB gRPC -
Microservices Tech Stack • Container / Docker • Kubernetes •
Spinnaker • Terraform Software is Infrastructure -
Container / Docker • Container • Ϧιʔεͷɾ੍ޚ • Docker •
DockerfileʹΑΔҰ؏ͨ͠Πϝʔδͷ࡞ • ϙʔλϏςΟͷ࣮ݱ -
Kubernetes • Container ͷ Orchestration Platform • ࣗಈScalingɺࣗಈhealing • Container
ӡ༻ίετͷݮ • GKE(Google Kubernetes Engine) Λத৺ʹར༻ -
Spinnaker • Continuous Delivery Platform • Developed by Netflix •
googleͳͲͷڠྗɾOSSԽ • Deploy pipelineͷఆٛɾࣗಈ࣮ߦ • Continuous Delivery for Microservices with Spinnaker at Mercari https://speakerdeck.com/tcnksm/continuous-delivery-for-microservices-with-spinnaker-at-mercari -
Terraform • Infrastructure as a code / DevOps • MicroservicesΛૉૣ্ཱͪ͛͘ΔStarter-
KitΛఏڙ • GCP Project • kubernetes Namespace • GitHub teams -
ping (ੴङ → ౦ژ) $ ping -c 5 example.jp PING
example.jp (130.211.x.x) 56(84) bytes of data. 64 bytes from 129.31.x.x.bc.googleusercontent.com (130.211.x.x): icmp_seq=1 ttl=51 time=16.9 ms 64 bytes from 129.31.x.x.bc.googleusercontent.com (130.211.x.x): icmp_seq=2 ttl=51 time=16.9 ms 64 bytes from 129.31.x.x.bc.googleusercontent.com (130.211.x.x): icmp_seq=3 ttl=51 time=16.8 ms 64 bytes from 129.31.x.x.bc.googleusercontent.com (130.211.x.x): icmp_seq=4 ttl=51 time=16.9 ms 64 bytes from 129.31.x.x.bc.googleusercontent.com (130.211.x.x): icmp_seq=5 ttl=51 time=16.8 ms -— example.jp ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4020ms rtt min/avg/max/mdev = 16.846/16.918/16.966/0.150 ms -
HTTPs (ੴङ → ౦ژ) $ ./httpstat.sh https://example.jp/hc HTTP/1.1 200 OK
Server: nginx/1.13.3 Date: Wed, 11 Oct 2017 01:59:15 GMT Content-Type: application/json; charset=utf-8 Content-Length: 22 Expires: Wed, 11 Oct 2017 02:59:15 GMT Cache-Control: max-age=3600 Cache-Control: public Via: 1.1 google Alt-Svc: clear DNS Lookup TCP Connection SSL Handshake Server Processing Content Transfer [ 1ms | 19ms | 165ms | 20ms | 0ms ] | | | | | namelookup:1ms | | | | connect:20ms | | | pretransfer:185ms | | starttransfer:205ms | total:205ms -
SoftwareʹΑΔվળ • 3 way handshakeΛආ͚ΔɻTLS ͷ handshake ආ͚Δ • HTTP/1,
HTTP/2 ͷKeepAlive Λ׆༻͢Δ • PHPͰϦΫΤετؒͰͷHTTPίωΫγϣϯͷ࠶ར༻͕Ͱ͖ͳ͍ • ϛυϧΣΞͰConnection Aggregation -
chocon % curl -H ‘Host: example.jp.ccnproxy-https’ http://10.0.0.1/v1/foo chocon Web/API Client
https://example.jp/ http http/https with keepAlive Private Network proxy -
Chocon & Consul Chocon Consul check & register service App
App App Chocon Consul check & register service *.ccnproxy-https IN CNAME production.chocon.service.dc.consul. $ curl http://example.jp.ccnproxy-https/v1/foo DNS Consulͱ෦DNSΛΈ߹Θͤͯར༻͘͢͠ ੑͷ࣮ݱ https://example.jp/v1/foo -
https with chocon (ੴङ → ౦ژ) $ ./httpstat.sh /dev/null https://example.jp.ccnproxy-https/hc
HTTP/1.1 200 OK Cache-Control: max-age=3600,public Content-Length: 22 Content-Type: application/json; charset=utf-8 Date: Thu, 01 Jun 2017 00:43:49 GMT Expires: Thu, 01 Jun 2017 01:43:49 GMT Server: nginx/1.11.5 X-Chocon-Req: bSCzJrCMZ9wbRN8TYhZ3wV Body stored in: /tmp/httpstat-body.390174181496278775 DNS Lookup TCP Connection Server Processing Content Transfer [ 1ms | 1ms | 19ms | 0ms ] | | | | namelookup:1ms | | | connect:2ms | | starttransfer:21ms | total:21ms -
ϝϧΧϦͷInfrastructure • SoftwareʹΑͬͯՄ༻ੑɾύϑΥʔϚϯεΛ্ɺΠϯϑϥΛఆٛ • Consul, OpenResty • chocon • Microservices
Stack • ଞʹଟ͘ͷSoftware͕SRE͔Βੜ·Ε͍ͯΔ • https://github.com/mercari/ -
(Cloud + Network + Hardware + N) × Software =
Infrastructure