On-call Engineering

Transcript

1. Tatsuhiko Kubo@cubicdaiya SRE Tech Talks#2 2017/01/30 On-call Engineering
2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.
4. SRE mission @ Mercari •Operation •γεςϜͷ໰୊఺Λൃݟɾղܾ͠ɺɹɹɹɹɹ αʔϏεͷ৴པੑΛ޲্ͤ͞Δ •Software Engineering •αʔϏεΛεέʔϧͤ͞ΔͨΊͷπʔϧɺ ϛυϧ΢ΣΞɺγεςϜج൫ͷ։ൃɾӡ༻
5. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
6. Mercari Engineering BlogͰ΋Ұ෦ެ։த • PascalʙPuree + ngx_lua + Fluentd + BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ • http://tech.mercari.com/entry/2015/09/09/163007 • DockerͱMakeΛར༻ͨ͠RPMύοέʔδͷϏϧυ؀ڥ • http://tech.mercari.com/entry/2016/08/15/163219 • ϋΠύϑΥʔϚϯεGaurunʙϝϧΧϦͷେن໛ϓογϡ഑৴Λࢧ͑Δϛυϧ΢ΣΞʙ • http://tech.mercari.com/entry/2016/11/08/170343 • ؂ࢹ͚ͩ͡Όͳ͍ʂσϓϩΠʹMackerelΛ࢖͏࿩ • http://tech.mercari.com/entry/2016/11/14/120000 refs: http://tech.mercari.com/
7. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
8. SREͷۀ຿಺༰ @ Mercari • ֤छAPIɺϛυϧ΢ΣΞͷՄ༻ੑɺ ύϑΥʔϚϯεͷҡ࣋ɾ޲্ • On-call౰൪ • αʔόϓϩϏδϣχϯά౳ͷ֤छࣗಈԽ • ։ൃɺσϓϩΠɺϩά෼ੳ౳ͷͨΊͷج൫੔උ • ηΩϡϦςΟͷ୲อ
9. On-call(Φϯίʔϧ)
10. On-call • ۓٸ࣌(ྫɿγεςϜো֐)ʹඋ͑ͯి࿩౳Ͱ ࿈བྷ͕औΕΔঢ়ଶΛอͭ͜ͱ
11. WebαʔϏεͱOn-call • WebαʔϏε͸24/7ͰՔಇ • ۓٸ࣌ʹඋ͑ͯ࠷௿1~2ਓ͸଴ػ͓ͯ͘͠ ඞཁ͕͋Δ
12. On-call౰൪ @ Mercari • SREνʔϜͰϩʔςʔγϣϯʢ̍िؒຖʹަ୅ʣ • ೔༵೔0:00͔Β౔༵೔23:59 • ΞϥʔτͷҰ࣍ड͚औΓͱۓٸରԠ • ٳ೔Ͱ΋15-20෼Ҏ಺ʹରԠͰ͖Δ͜ͱ͕๬·͍͠ • ฏ೔த͸ଞSRE͕ग़ࣾ͢Δ·Ͱࣗ୐଴ػ(SREશһ͕ిं௨ۈதͰ͋Δ ͜ͱΛආ͚Δ) • ౰൪த͸ߦಈʹ੍ݶ͕͔͔Δ
13. େܕ࿈ٳத͸มଇϩʔς
14. On-call౰൪ @ Mercari • On-call౰൪͸৭ʑͱෛ୲͕͔͔Δ • ਫ਼ਆతͳϓϨογϟʔɺಉډਓ΁ͷෛ୲ • ਂ໷ʹΞϥʔτɺւͷ޲͜͏͔Βి࿩ • ౰൪೔਺ʹԠͨ͡ख౰΍ৼٳΛࢧڅ
15. On-callʹؔΘΔॾ໰୊ͷྫ • ϩʔςʔγϣϯ΍౰൪ͷۈଵɺৼସٳՋͷ؅ཧ͸ʁ • खಈͰ΍ΔͷΊΜͲ͍ • On-call౰൪ͷి࿩ͬͯԿ൪͚ͩͬʁ • ౰൪ʹ࿈བྷ͢Δଆͷෛ୲Λܰݮ͢Δඞཁ͕͋Δ • On-call౰൪͕ి࿩ʹग़ͳ͔ͬͨΒʁ • ΤεΧϨʔγϣϯ͕ඞཁ Software EngineeringͰղܾ͠Α͏ʂ
16. On-call Engineering
17. bot hello
18. bot hello • ۈଵଧࠁ༻ͷίϚϯυ in Slack • On-call౰൪SREͷࣗ୐଴ػελʔτͷ߹ਤ • ଞSRE͕ग़ࣾͨ͠Β౰൪SRE΋ग़ࣾ͢Δ • bot bye΋͋Δ
19. bot touban call
20. bot touban call • On-call౰൪ʹి࿩Λ͔͚ΔίϚϯυ in Slack • Powered by PagerDuty • allΛ଍͢ͱSREશһʹి࿩ • Slack͕μ΢ϯͯ͠Δ৔߹ʹඋ͑ͯผ్ۓٸ࿈ བྷઌͷϦετΛwikiʹ·ͱΊͯ͋Δ
21. bot touban help
22. On-call౰൪ϩʔςͷ ؅ཧͱΤεΧϨʔγϣϯ • PagerDutyͷػೳΛ׆༻ • On Call Schedules • Escalation Policy • https://www.pagerduty.com/
23. PagerDuty API & Google Apps Script • On-call౰൪ͷ༧ఆ΍࣮੷ΛGoogleͷεϓ Ϩουγʔτ΍ΧϨϯμʔʹಉظ • ఆظతʹूܭͯ͠څ༩΍ٳՋ೔਺ʹ൓ө
24. ·ͱΊ • ۓٸ࣌ʹඋ͑ͯOn-callମ੍Λ੔͑Α͏ • On-callۀ຿͸ෛ୲͕େ͖͍ • ੍౓΍͘͠ΈͰΧόʔ • On-call΋Software Engineeringʂ
25. We are hiring! https://www.mercari.com/jp/jobs/