Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SREcon19 Americas 参加レポート #srefukuoka / srecon19...

SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report

Manabu Matsuzaki

June 12, 2019
Tweet

More Decks by Manabu Matsuzaki

Other Decks in Technology

Transcript

  1. About me • Nameɿ Manabu Matsuzaki • Work atɿ LINE

    Fukuoka Corporation • Roleɿ SRE • Twitterɿ @matsumana
  2. ΧϯϑΝϨϯε֓ཁ • ΦϑΟγϟϧαΠτɿ
 https://www.usenix.org/conference/srecon19americas • ೔ఔɿ 2019/03/25~27 • ձ৔ɿ New

    York Marriott (Brooklyn, NewYork) • ηογϣϯ਺ɿ ໿50
 ʢࢿྉͱಈը͕ެ։͞Εͯ·͢ʣ
 https://www.usenix.org/conference/srecon19americas/program
  3. ΧϯϑΝϨϯε֓ཁ • ࢀՃऀ਺ɿ 646ਓ • AM: Americas • AP: Asia/Pacific

    • Europe/Middle East/Africa see also: https://www.usenix.org/conferences/byname/925
  4. What Breaks Our Systems: A Taxonomy of Black Swans •

    ηογϣϯ֓ཁ
 https://www.usenix.org/conference/srecon19americas/presentation/nolan-taxonomy • εϐʔΧʔ
 Laura Nolan, Slack • εϥΠυ
 https://www.usenix.org/sites/default/files/conference/protected-files/sre19amer_slides_nolan.pdf
  5. What Breaks Our Systems: A Taxonomy of Black Swans •

    Black swanͱ͸ʁ • ҟৗͳΠϕϯτ • ༧ଌ͢Δͷ͕೉͍͠ • γϏΞͳΠϯύΫτ • ͜ͷηογϣϯͰ͸ɺ͍͔ͭ͘ͷ࣮ࡍͷαʔϏεো֐Λྫʹͯ͠ɺ
 ͦͷΑ͏ͳো֐Λ๷͙ύλʔϯ͕঺հ͞Ε·ͨ͠ • ঺հ͞Εͨख๏Λಋೖͨ͠ͱͯ͠΋ɺ༧ଌ͕೉͍͠ҟৗͳΠϕϯτΛશͯ๷͙ͷ͸೉͍͠ ͱࢥ͍·͕͢ɺࢀߟʹ͸ͳΔͱࢥ͍·͢
  6. What Breaks Our Systems: A Taxonomy of Black Swans •

    black swanͷछྨͱकΓํ • Hitting limits • load and capacity testing • Monitoring • Spreading Slowness • Fail fast • Use dashboards • Thundering Herds • Plan and test
  7. What Breaks Our Systems: A Taxonomy of Black Swans •

    black swanͷछྨͱकΓํ • Automation interactions • controll • Cyberattacks • Smaller blast radius • Dependency problems • Layer and test
  8. Keeping the Balance:
 Internet-Scale Loadbalancing Demystified • ηογϣϯ֓ཁ
 https://www.usenix.org/conference/srecon19americas/presentation/ nolan-loadbalancing

    • εϐʔΧʔ
 Laura Nolan, Slack
 Murali Suriar, Google • εϥΠυ
 https://www.usenix.org/sites/default/files/conference/protected-files/ sre19amer_slides_nolan-load-balancing.pdf
  9. Keeping the Balance:
 Internet-Scale Loadbalancing Demystified • LBͷجຊΛ঺հ • DNS

    ϥ΢ϯυϩϏϯ • Proxyํࣜ • L2DSR • L3DSR • DNS geo ϩʔυόϥϯγϯά • ΫϥΠΞϯταΠυ ϩʔυόϥϯγϯά
  10. Aperture: A Non-Cooperative, Client-Side Load Balancing Algorithm • ηογϣϯ֓ཁ
 https://www.usenix.org/conference/srecon19americas/presentation/oanta

    • εϐʔΧʔ
 Ruben Oanta, Twitter • εϥΠυ
 https://www.usenix.org/sites/default/files/conference/protected-files/ sre19amer_slides_oanta.pdf
  11. Keeping the Balance:
 Internet-Scale Loadbalancing Demystified • TwitterͰ։ൃ͞Ε͍ͯΔFinagle(Web Framework)ͷ࿩ •

    ͍͔ͭ͘ͷϩʔυόϥϯγϯάΞϧΰϦζϜ͕બ୒Մೳ • P2C • Aperture + Least Loaded • etc • ެࣜυΩϡϝϯτ
 https://twitter.github.io/finagle/guide/Clients.html#load- balancing
  12. Keeping the Balance:
 Internet-Scale Loadbalancing Demystified • Aperture Load BalancersΛ࣮૷ͯ͠αʔόϦιʔεΛվળͨ͠

    • 78% reduction in standard deviation for requests/sec • 91% drop in aggregate connections (~280k to ~25k) • 75% fewer failures • ~20% reduction in latency at 99.9%tile • 20~25% less CPU used • Total GC time cut in half
  13. Tracing, Fast and Slow: Digging into and Improving Your Web

    Service's Performance • ηογϣϯ֓ཁ
 https://www.usenix.org/conference/srecon19americas/presentation/root • εϐʔΧʔ
 Lynn Root, Spotify • εϥΠυ
 https://www.usenix.org/sites/default/files/conference/protected-files/ sre19amer_slides_root.pdf
  14. What I Wish I Knew before Going On-call • ηογϣϯ֓ཁ


    https://www.usenix.org/conference/srecon19americas/presentation/shu • εϐʔΧʔ
 Chie Shu and Wenting Wang, Yelp • εϥΠυ
 https://www.usenix.org/sites/default/files/conference/protected-files/ srecon19americas_slides_wang.pdf
  15. ͳͥʁ • Afraid of unknown situations • Lack of confidence

    • Poor understanding of systems • Lack of protocol • Afraid of asking for help • etc
  16. • ΧϦΩϡϥϜΛ࡞Δ • ৘ใΛ٧ΊࠐΈ͗͢ͳ͍ • ΠϯτϩμΫγϣϯ • γϯϓϧͳਤ • γεςϜͷ֓ཁ

    • ԿΛ͍ͯ͠ΔγεςϜͳͷ͔ • Կʹґଘ͍ͯ͠Δͷ͔ τϨʔχϯάϓϩάϥϜͷ࡞Γํ
  17. • ٕज़తͳ಺༰ • ΠϯύΫτධՁ • ࣮ߦίϚϯυ • ඇٕज़తͳ಺༰ • ֤ࣗͷ໾ׂ෼୲

    • ίϛϡχέʔγϣϯํ๏ • ΤεΧϨʔγϣϯϙϦγʔ खॱॻʹؚΊΔ΋ͷ