Share
Embed
SRE大全 メルカリ編 前半 #hbstudy 75 / SRE Taizen Mercari 1 hbstudy#75
SRE大全 メルカリ編 hbstudy#75
kazeburo
August 21, 2017
Transcript
53 slides
-
AGENDA • ޙ • PHP ΞϓϦέʔγϣϯͷ࠷దԽࣄྫ • ηΩϡϦςΟͷऔΓΈ(ύεϫʔυϦετ߈ܸࣄྫ) • ϝϧΧϦSREͷࠓޙ
• SREͷׂͷݱࡏͱະདྷɺϚΠΫϩαʔϏε -
Me • Masahiro Nagano / խ • @kazeburo • Mercari,
Inc Principal Engineer Site Reliability Engineering (SRE) Team • BASE, Inc Technical Advisor -
Me • ~ 2006: ژͰελʔτΞοϓࢀՃ • ։ൃΛ͠ͳ͕ΒΠϯϑϥͷ໘ΛݟΔɻDC࡞ۀͬͨ • ΞϓϦέʔγϣϯͷνϡʔχϯάɺۭ͍ͨϦιʔεͰ৽ػೳͷՃͱ͍͏αΠΫϧ •
mod_perlɺSquidʹΑΔReverse Proxy • 2006 ~: mixi • ʮΞϓϦέʔγϣϯӡ༻νʔϜʯ / DCʹߦ͔ͳ͘ͳͬͨ • େنͳը૾৴/memcached/Q4M -
Me • 2010 ~: livedoor (NHN Japan => LINE) •
livedoorLINEϑΝϛϦʔͷαʔϏεΛԣஅͯ͠Πϯϑϥύ ϑΥʔϚϯεͷվળ • livedoor blog ͷMySQLνϡʔχϯά • GrowthForecast/HRForecast/Plack Optimization/MHA • 2015/02: mercari -
࠷ۙͷ׆ಈ • ొஃ • AWS Dev Day Tokyo 2017 •
YAPC::Fukuoka 2017, YAPC::Hokkaido 2016 • هࣄ • WEB+DB PRESS Vol.88, Vol.92-97 ࿈ࡌ • ܦSYSTEMS 2017 7݄߸, ITPro -
ࣗশ `!ΠϯϑϥΤϯδχΞ` • (ݸਓతʹ) ΠϯϑϥΤϯδχΞͷٙ • 2006ҎདྷDC࡞ۀ͍ͯ͠ͳ͍ɻDC࡞ۀΛߦ͏νʔϜଞʹ͍Δ • σʔληϯλʔνʔϜ͕༻ҙͨ͠αʔόͷೳྗΛҾ͖ग़͠ɺΞϓϦέʔγϣ ϯΤϯδχΞ͕࡞ͨ͠ίʔυΛ࠷ߴͷܗͰಈ͔͢ͷ͕ࣗΒ(νʔϜ)ͷׂ
• αʔϏεͷՄ༻ੑϋʔυΣΞͷνʔϜͰͳ͘ɺιϑτΣΞΛѻ͏ νʔϜͷ -
ΦϖϨʔγϣϯΤϯδχΞ • 2010ग़൛ʮWeb Operationsʯ • ʮΣϒΦϖϨʔγϣϯٕܳͰ͋ΓՊֶͰͳ͍ʯ • ܧଓతσϓϩΠɺDevOpsɺࣗಈԽɺࢹͳͲΦϖ Ϩʔγϣϯʹؔ͢ΔΤοηΠ •
ͨͩ͠ɺΦϖϨʔγϣϯ=ӡ༻ΛϧʔνϯϫʔΫͱଊ ͑Δਓଟ͍ -
SREͱͷग़ձ͍ • 2012/7 ༑ਓͱͷIRCͰͷձ͔Β • ΠϯϑϥͱαʔϏεͷՔಇɺ҆ఆੑΛ୲͢ΔνʔϜ͕SRE • https://research.googleblog.com/2012/07/site-reliability-engineers-solving-most.html ͜ͷهࣄ͕ެ։͞Εͨࠒ •
twitter ͷbioൃදεϥΠυʹʮSite ReliabilityʯΛՃͯ͠ҙࣝ • https://www.slideshare.net/kazeburo/yapc2102mysql/2 (2012/9) • 2015/11 ϝϧΧϦʹͯνʔϜ໊ͱͯ͠࠾༻ -
ϝϧΧϦͰSREΛ࠾༻ͨ͠ཧ༝ • ϝϧΧϦΛ͓٬͞·ʹͬͯ͘Β͏ʹʮ͍ͭͰշదʹ҆શʹ ͑Δʯ৴པੑ͕ॏཁ • Πϯϑϥ=ϞϊɺΦϖϨʔγϣϯ=ࣄͰͳ͘ɺ৴པੑʹΛ࣋ͭ͜ ͱΛ໌Β͔ʹ͢Δ • ւ֎Ͱ௨͡Δ໊শ •
JP/US/UKͰͷల։ɻάϩʔόϧͰͷ࠾༻ৗʹҙࣝ • ઌਐతͳऔΓΈͱͯ͠ -
Mercari SRE • ͍ͭͰշద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷΤϯδχΞϦϯάશ෦Δʯ • ݱࡏϝϯόʔʮ9ਓʯ • શһ౦ژۈ
• ࣾπʔϧͷӡ༻վળɺAIܥͷۀʹܞΘΔϝϯόʔ͍Δ -
Mercari SRE ͷۀൣғ Operations Software Eng. ج൫ߏங OnCall (োରԠ) Automation
εέʔϥϏϦςΟɾՄ༻ੑվળ DBAɺϛυϧΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾੳج൫ͷߏஙɺӡ༻ αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεج൫ͷඋ ηΩϡϦςΟʗෆਖ਼ར༻ݕग़ -
SRE൪/OnCall • ༵0͔࣌Β༵24࣌·Ͱ1िؒͰަ • Ξϥʔτͷड͚औΓͱҰ࣌ରԠ • ฏνʔϜϝϯόʔ͕ग़ࣾ͢Δ·Ͱࣗػ • 9͔࣌Βࣗػ͠ɺUS͔ΒͷґཔͳͲʹରԠ •
UK͔ΒͷରԠͰؒ࡞ۀ͋Δ • ٳ15-20ҎʹରԠ։࢝Ͱ͖Δ͜ͱ͕·͘͠ɺߦಈʹ੍ݶ͋Δ • ਂٳͷରԠͳͲɺՈͷڠྗඞཁ -
൪/OnCall Λࢧ͑Δٕज़ • ࢹ • Mackerel, slacklog • ௨/Scheduling •
Slack, PagerDuty, Twilio -
mackerel Worker Batch App App MySQL cron mackerel-agent fluent-plugin-mackerel mkr
ՄࢹԽ/ᮢͷઃఆ Metricsऩू ௨ -
[ઢ] ࠷ۙ࡞ͬͨmackerel-plugin #!/usr/bin/perl use HTTP::Date; my $NUM_LOG_WATCH = 1000; my
$CHECK_RANGE = 300; #5min my $exceptions = 0; my $now = time; open( my $messages_tail, "-|", "tail","-$NUM_LOG_WATCH","/var/log/messages") or die $!; while (<$messages_tail>) { if ( $_ !~ m!Machine Check Exception! ) { next; } if ( my ($time) = ($_ =~ m!^(\w{3}\s+\d{1,2}\s\d{2}:\d{2}:\d{2})\s!) ) { $time = str2time($time); if ( $now - $time < $CHECK_RANGE ) { $exceptions++; } } } if ( $exceptions > 0 ) { print "CRITICAL: Machine Check Exception Found in this 5 minutes\n"; exit 2; } print "OK: No Machine Check Exception found\n"; exit 0; % dmesg | tail sbridge: HANDLING MCE MEMORY ERROR CPU 0: Machine Check Exception: 0 Bank 8: cc0427c000010090 TSC 0 ADDR 37805ac0 MISC 45048ce86 PROCESSOR 0:406f1 TIME 1495654896 SOCKET 0 APIC 0 [Hardware Error]: Machine check events logged EDAC MC1: CE row 0, channel 0, label "CPU_SrcID#0_Ha#0_Channel#0_DIMM": 4255 Unknown error(s): memory read on FATAL area OVERFLOW: cpu=0 Err=0001:0090 (ch=0), addr = 0x37805ac0 => socket=0, ha=1, Channel=0(mask=1), rank=0 DIFDLNBDIJOFFYDFQUJPOT ϝϞϦʔΤϥʔΛݟ͚ͭΔ -
[ઢ] ࠷ۙ࡞ͬͨmackerel-plugin #!/bin/sh set -e if [ ! -f /opt/MegaRAID/MegaCli/MegaCli64
]; then exit fi if ( /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state'|grep -v "Online, Spun Up" > /dev/null 2>&1 ); then /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state' exit 2 fi /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL|grep 'Firmware state' exit 0 DIFDLSBJEEJTL .FHB$MJΛ͔ͭͬͯ3"*%ͷঢ়ଶΛࢹ -
slacklog ίϚϯυ $ slacklog -t alert-information --notify -- perl -e
'die "TEST!"' !LB[VIP͞ΜͷDSPOMPHΛࢀߟʹ࡞ CBUDICBDLVQͷࣦഊΛݕ slackboard ௨ͷू IUUQTHJUIVCDPNDVCJDEBJZBTMBDLCPBSE -
PagerDuty • ༷ʑͳखஈͰ௨Λߦ͏͜ͱ͕Ͱ͖Δ • mail • SMS • App •
ి • 12ʹҰిΛೖΕΔϧʔϧͰӡ༻ • Appͷ௨͕ศར -
γεςϜ͔ΒΈͨϝϧΧϦ ©2011 Amazon Web Services LLC or its affiliates. All
rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corp data c Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ߴʹେྔͷτϥϯβΫγϣϯΛѻ͏ ը૾ ܾࡁ AI -
ΠϯϑϥετϥΫνϟ JP US UK DNS: Amazon Route53 CDN: Akamai, Fastly,
ImageFlux Storage: Amazon S3 Analysis: Google BigQuery ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε -
ΞʔΩςΫνϟ • ࡾ+ΞϧϑΝͳΞʔΩςΫνϟ • Reverse Proxy(nginx) Application(Apache+mod_php) Database(MySQL) Cache(memcached) Search(Solr)
• ଟ͘Λʮઐ༻αʔόʯʹͯߏ • εέʔϧΞτͱεέʔϧΞοϓΛಉ࣌ʹߦ͏Diagonal Scale • ओʹ24ίΞ~56ίΞ·ͰͷαʔόΛར༻ • Databaseʹ ioMemory NVMe Λࡌͨ͠αʔόΛ࠾༻ nginx nginx nginx ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Traditional server Mobile Client WS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers ic DNS-RR App App App App App App MySQL MySQL memcached memcached util util cloud cloud JP Solr Solr -
Timeline • 6/22 • 9:41ɹ CDNͷΓସ͑Λ࣮ࢪʢൃੜʣ • 14:41ɹΧελϚʔαϙʔτʹ͓ͯ٬͞·͔Βͷ͍߹ΘͤΛ֬ೝ͠ɺࣾใࠂ • 15:05ɹCDNͷΓସ͑Λதࢭ͠ɺैདྷͷCDN͢
• 15:16ɹWeb൛ͷϝϧΧϦΛϝϯςφϯεϞʔυΓସ͑ • 15:38ɹΓସ͑ઌCDNͷઃఆΛdeactivate͠ɺΞΫηεΛःஅ • 15:47ɹWeb൛ͷϝϧΧϦϝϯςφϯεϞʔυΛऴྃ • 17:55 ίʔϙϨʔταΠτʹ͓ΒͤΛܝࡌ • 20:45 Tech blogʹͯৄࡉެ։ -
Timeline • 6/28 • Γସ͑ઌCDNͷઃఆΛ࠶Activate • CacheΛແޮԽ͢ΔઃఆΛߦ͍ɺΓସ͑ઌCDNͷΤϯδχΞʹϨϏϡʔΛґཔ • nginxͷઃఆΛߋ৽͠ɺcacheʹؔ͢ΔϔομΛมߋ •
ࣾͰݕূ • 6/29 • Γସ͑ઌCDNͷΤϯδχΞػͷͱɺΓସ͑Λ࣮ࢪ -
Γସ͑ઌCDNʹ͓͚Δcacheͷಈ࡞ • CacheΛແޮԽ͢ΔͨΊʹ “Cache-Control: private” ͘͠ "Set-Cookie" ͕ඞཁ • ”Cache-Control:
no-cache” “no-store” ແࢹ͞ΕΔ • Expiresϔομར༻͞ΕΔ͕ɺͷղऍʹࣦഊ͋Δ͍աڈͷ߹ “0ඵ” ͱͯ͠ѻΘΕΔ • ʮ0ඵͷΩϟογϡ͕ଘࡏ͢Δʯ • (্هઃఆʹΑΓΧελϚΠζ͕Մೳ) -
0ඵͷcache • CDN͔ΒΦϦδϯͷϦΫΤετͷॲཧதʹɺಉ ͡URLʹରͯ͠ϦΫΤετ͕ൃੜ͢Δͱɺ࠷ॳͷ ϨεϙϯεΛͬͯɺ2ͭҎ߱ͷϦΫΤετʹ ಉ͡Ϩεϙϯε͕ฦ͞ΕΔ • ੩తίϯςϯπͰඇৗʹ༗ޮͰ͋Δ • ͜ͷ༷ʹؔ͢ΔѲ͕Ͱ͖ͯͳ͔ͬͨ
©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All r User Users Client Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Workers Amazon Mechanical Turk Non-Service Specific origin CDN (1) (2) (3) (4) (5) (5) -
cache aware nginx configuration • ExpiresϔομΘͳ͍ • ݹ͍ϒϥβ͚ʹPragmaϔομͰରԠ • Cache-ControlҎ֎ʹΩϟογϡΛආ͚ΔͨΊ͚ͩͷ
Set-Cookieૹ৴ more_clear_headers 'Expires'; more_set_headers "Cache-Control: private, no-cache, no-store, must-revalidate" "Pragma: no-cache"; add_header Set-Cookie "merCtx=\"\"; HttpOnly" always; OHJOYDPOG