Share
Embed
Site::Reliability::Engineering - YAPC::Hakkaido 2016 Sapporo
Site::Reliability::Engineering - YAPC::Hakkaido 2016 Sapporo
kazeburo
December 12, 2016
Transcript
76 slides
-
Me • Masahiro Nagano • @kazeburo • Mercari, Inc Principal
Engineer Site Reliability Engineering (SRE) Team • BASE, Inc Technical Advisor -
CPAN/Perl • Gazelle • Cookie::Baker(::XS) • WWW::Form::UrlEncoded(::XS) • HTTP::Entity::Parser •
Apache::LogFormat::Compiler • Plack::Middleware::ServerStatus::Lite • GrowthForecast -
Agenda • Site Reliability Engineering (SRE) ͱ • MercariͱMercariͷSRE •
Mercari SREͷࣄྫ • Wrap-up & -
Site Reliability Engineering • Google ͕ఏএ • Google ͷ༷ʑͳϓϩμΫτɾαʔϏεΛԣஅ͠ ͯɺαΠτͷ৴པੑΛ্ͤ͞Δ
Software Engineer/Team -
Google SRE • શһ͕ Software Engineer • ։ൃνʔϜͱSREͱ࠾༻ͷ۠ผͳ͍ • 6ϲ݄ؒͷSREݚमड͚Δ͜ͱͰ͖ɺ͔ͦ͜ΒSREʹͳΔྫ
• SLA ͱ Error Budget • ։ൃνʔϜͱSREͷؒͰSLAɺError BudgetΛڞ༗͢Δ͜ͱͰɺ৽نػೳ ͷՃͱαʔϏεͷ҆ఆͷڝ߹Λղܾ͢Δ • SLAαʔϏε͝ͱʹܾఆ͞ΕΔ • Error Budget͕ෆͦ͠͏Ͱ͋Εɺ৴པੑΛ্ͤ͞Δ։ൃʹઐ೦͢ Δ͜ͱ͕ٻΊΒΕΔ -
Google SRE • OnCall (൪) • SREνʔϜͩΕͰఆظతʹ୲͢Δ • ӡ༻ʹ͋ͨΔ࣌ؒΛ 50%
ʹ੍ݶ • Γͷ࣌ؒ৴པੑ্ͷͨΊͷιϑτΣΞ։ൃ ʹ͋ͯΔ • ࣗಈԽෛՙࢄͷιϑτΣΞ͕ੜ·ΕΔʹ -
Google SRE “what happens when a software engineer is tasked
with what used to be called operations” https://cloudplatform.googleblog.com/2016/07/adventures-in-SRE-land-welcome-to-Google-Mission-Control.html Google Vice President of Engineering Ben Treynor Sloss, who coined the term SRE “our work is like being a part of the world’s most intense pit crew. We change the tires of a race car as it’s going 100mph” https://students.googleblog.com/2012/06/site-reliability-engineers-worlds-most.html -
SRE in JP • ʮΠϯϑϥΤϯδχΞ/DevOpsʯશ • 2015/11 Mercari Tech Blog
ʹͯհͯ͠Ҏ͕߱ू·Δ • “ΠϯϑϥνʔϜվΊ Site Reliability Engineering (SRE) νʔϜʹ ͳΓ·ͨ͠” http://tech.mercari.com/entry/2015/11/18/153421 • 2016/06 SRE Tech Talk#1 • https://connpass.com/event/34825/ -
SRE in JP • αΠϘζ͞Μ “SRE νʔϜΛઃཱ͠·͢” • http://blog.cybozu.io/entry/2016/09/01/080000 •
Cookpad͞Μ “࠷ۙɺElastic BeanstalkECSͱઓ͍ͬͯΔSREνʔ ϜͷੁݪͰ͢” • http://techlife.cookpad.com/entry/2016/10/06/000000 • Retty͞Μ “ΠϯϑϥͰͳ͍ʂ৴པੑΛߴΊΔΤϯδχΞ ʮSREʯͱʁ” • https://www.wantedly.com/companies/retty/posts/17568 -
Architecture nginx nginx nginx ©2011 Amazon Web Services LLC or
its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached JP util util cloud cloud ઐ༻αʔόʹΑΔߏ͕ϕʔε -
Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC
or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP util util cloud cloud util util EC2 EC2 -
Architecture US nginx nginx nginx ©2011 Amazon Web Services LLC
or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers DNS-RR App App App App App App MySQL MySQL memcached memcached nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia C d Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific DNS-RR App App App App App App MySQL MySQL memcached memcached EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 EC2 JP ΫϥυͰ ಉ͡ߏ util util cloud cloud util util EC2 EC2 -
Architecture • ϝϯςϯεϏϦςΟɾεέʔϥϏϦςΟઓུͷڞ௨Խ • > গਓͰͷ Operation • > US
Ͱͷ App Store ϥϯΩϯά3Ґʹ͑Δ • αʔϏεͷల։࣌ʹ࠷Bestͳ Infrastructure Λબ • > UKGCP -
SREʹมߋͨ͠ཧ༝ • ʮΠϯϑϥʯ͔ΒΠϝʔδ͢ΔۀͷΛ͑Δ • ϝϧΧϦΛ5ɾ10ͱ͍ͬͯͨͩͨ͘͘Ίͷʮ৴པੑʯ • εϚϗͰ24͖࣌ؒͳ࣌ʹɺ͖ͳॴ͔ΒΞΫηε͕དྷ Δɻ͍ͭͰշదʹΞΫηε͕Ͱ͖ΔʮՄ༻ੑʯͱ ʮύϑΥʔϚϯεʯ •
ιϑτΣΞΤϯδχΞϦϯάͰ͜ΕΒΛղܾ͍ͯ͘͠νʔϜ • (ւ֎/USͰ௨͡Δ໊শ) -
OSS by SRE • Gaurun - General push notification server
in Go • https://github.com/mercari/gaurun • WideBullet - an API gateway with JSON-RPC • https://github.com/mercari/widebullet • Mackerel Plugins • https://github.com/kazeburo/custom-mackerel-plugins -
Mercari SREͷۀ • APIαʔόɺϛυϧΣΞͷՄ༻ੑɺύϑΥʔϚϯεͷ ҡ࣋ɾ্ • OnCall (োରԠͷ൪) • ϩάऩूɾੳج൫ͷߏஙɺӡ༻
• αʔόϓϩϏδϣχϯάɾσϓϩΠͷඋ • ηΩϡϦςΟͷ୲อ • ։ൃڥͳͲͷඋ -
OnCall • ༵0:00͔Β༵23:59·Ͱͷ1िؒ͝ͱަ • Ξϥʔτͷड͚औΓͱۓٸରԠ • SREશһ͕௨ۈిंʹ͍Δ͜ͱΛආ͚ΔͨΊɺฏϝ ϯόʔ͕ग़ࣾ͢Δ·Ͱࣗػ • ٳ15-20ҎʹରԠ։࢝Ͱ͖Δ͜ͱ͕·͘͠ɺߦ
ಈʹ੍ݶ͋Δ ਂͷΞϥʔτɺٳͷରԠͳͲɺՈͷෛ୲͋Γ·͢ɻ Ԟ͞Μଉࢠ່ͷڠྗʹ͍ͭײँ -
Nginx as a Internal LB App nginx nginx Solr Solr
Solr Solr App App clustering with consul nginx nginx DNS-RR Services LLC or its affiliates. All rights reserved. Multimedia Corporate data center Traditional server Mobile Client Example: IAM Add-on ment/ k Requester Workers private network -
Nginx as a Useful L7-LB App nginx nginx Solr Solr
Solr Solr App App clustering with consul nginx nginx DNS-RR Services LLC or its affiliates. All rights reserved. Multimedia Corporate data center Traditional server Mobile Client Example: IAM Add-on ment/ k Requester Workers private network Cache!! Here -
Cache is NOT silver bullet • Cache͕ଘࡏ͠ͳ͍߹͍ɻϢʔβମݧͷѱԽ • Cache Thundering
Problem • eg) proxy_cache_lock • => Cache ΛઌಡΈ͢Δ͜ͱͰ͖ͳ͍͔ • ύλʔϯ͕ଟ͘batchͰcacheΛ࡞͢Δͷ͍͠ • සൟʹߋ৽͍ͨ͠ -
make Search pre-cacher App nginx nginx Solr Solr Solr Solr
App App log daemon tail access.log http req -
#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; while(1){ open(my $fh,
"-|", "tail","-5000","/var/log/nginx/access.log") or die $!; my %path_query; while(<$fh>){ if ( m!pre-cacher! ) { next } if ( $_ !~ m!(cachemode=readahead|group.limit)! ) { next } if ( m!\suri:(.+?)\s! ){ $path_query{"$1"} = 1; } } my $ua = LWP::UserAgent->new(agent=>”pre-cacher"); $ua->timeout(20); for my $path_query ( keys %path_query ) { my $req = HTTP::Request->new( GET => ‘http://localhost' . $path_query ); $req->header('Host'=>'lb-search'); $ua->request($req); } sleep 5; } -
Search pre-cacher • ͱ͋Δ Perl Monger (Me) ͷSRE͕3Ͱॻ͍ͨ • ॏ͍Solr
queryΛ͛Δ࣌query_stringʹ readadheadͱ։ൃऀʹ͚ͭͯ͑ࣗಈͰઌಡ ΈରԠՄೳ -
JobWorker in Mercari App Q4M Q4M enqueue child child child
child Parent fork(2) php-parallel-prefork dequeue clustering with consul -
JobWorker in Mercari • Q4M + Parallel::Prefork • େਓͷࣄͰ max-request-per-child
ʮ1ʯ • workerͷॲཧͷߴԽ͕՝ -
Copy on Write (CoW) • fork(2) Λߦͬͨࡍʹɺϓϩηε͔Βࢠϓϩη εͷϝϞϦʔίϐʔߦΘͣɺมߋͯ͠ॳΊͯ ίϐʔ͞ΕΔɻ •
ϝϞϦʔΛϓϩηεͱڞ༗͢Δ͜ͱͰઅ • ܭࢉ݁ՌΛڞ༗͢Δ͜ͱͰෛՙΛԼ͛Δ • mod_perl mongerͳΒྦΛྲྀͯ͠ޠΔ -
CoW in PHP • mod_php ͰޮՌͳ͠ ϓϩηεىಈ࣌ʹԿ͔ΔΑ͏ͳΦϓγϣϯ͕ଘࡏ͠ ͳ͍ • CLI
Ͱ͑Δ • PHPͰClassΛར༻ͨ࣌͠ʹautoload͢ΔจԽ • > ࣗવͱCoWʹͳΓʹ͍͘ • ϕϯνϚʔΫͰূ໌ͯ͠औΓࠐΜͰΒ͏ -
function t_sendmail() { $smtp = new SimpleMailWithSwift(); $smtp->send($params); } //
t_sendmail(); // ϓϩηεͰϝʔϧΛૹΔ͜ͱͰclassͷpreloadΛߦ͏ for ( $i=0; $i < 300; $i++ ) { $pid = pcntl_fork(); if ($pid == -1) { die('fork Ͱ͖·ͤΜ'); } else if ($pid) { pcntl_wait($status); } else { // ࢠϓϩηεͷ߹ t_sendmail(); exit(0); } } 28.639s => 22.048s -
In-house URL shortener • େنϝʔϧ৴࣌ʹར༻͍ͨ͠ • > URLੜͰ͖Δmsec୯ҐͷύϑΥʔϚϯε • JP/US/UK
Ͳ͔͜ΒͰԆͰ͑Δ • URIͷhost෦શRegionͰڞ௨ • https://example.ly/abcd1234 -
RTT between regions ੴ AWS ౦ GCP 18ms 110ms 140ms
GCP 6ms ͍͍ͩͨͷ -
RTT between regions ੴ AWS ౦ GCP 18ms 110ms 140ms
GCP 6ms ͍͍ͩͨͷ γεςϜΛࢄͯ͠ஔ͢Δඞཁ͕͋Δ ࢀরՕॴʹ·ͱΊΔ -
In-house URL shortener • Regional API • Go + MySQL
• Private Networkʹઃஔ • Global API • GAE/Go / OperationͷলྗԽ • US region ʹઃஔ -
• ଵଦ • > Infrastructure, Operation ͷࣗಈԽ • ؾ •
> ߴ͍ύϑΥʔϚϯεΛ͍ٻΊΔ • ၗຫ • > ଟ͘ͷϢʔβɺେنͳΞΫηεɺ༷ʑͳσʔλΛѻ͏ • > ۀͷϓϥΠυ -
• TMTOWTDI • ڧྗͳจࣈྻॲཧ • One liner • ༷ʑͳڥͰ͑Δ •
CPAN • ਓͷॻ͍ͨίʔυ͕ಡΊͳ͍ -
DSL for System Call • CݴޠϑϨϯυϦʔ • System callͱ1:1ͰରԠ •
> fork, waitpid, syswrite, sysreadͳͲ • Socket·ΘΓૉ • Ruby, PHPͱൺͯΫη͕ͳ͍ • OSNetworkΛֶͿͷʹ͍͍ݴޠ(ͩͱࢥ͏) -
࠷ߴͷڭՊॻ @kazuho Starlet, Server-Starter, Parallel::Prefork HTTP::Parser::XS, Furl, kaztools and many!!
Perl Mongerৗʹͷղܾͷߴಓ࿏ͷ্ʹ͍Δ