Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Own your reliability - Tech Summit 2016

Own your reliability - Tech Summit 2016

Who do you trust? What do you control? What are your dependencies? Reliability in the Internet is an adrenaline adventure but we all want a good night sleep and working service. Let’s take a closer look at some of the reliability nightmares and how they could be dealt with.

Adam Surak

April 13, 2016
Tweet

More Decks by Adam Surak

Other Decks in Technology

Transcript

  1. @AdamSurak #id Algolia since 2014 (team of 8) DevOps &

    Security Engineer Responsible for infrastructure I like to sleep and break things
  2. @AdamSurak Algolia Today 15 regions, 35+ datacenters 1300+ customers in

    100+ countries 20B+ Write operations per month 12B+ User-generated queries per month
  3. @AdamSurak What is SLA? “A service level agreement (SLA) is

    a contract between a service provider (either internal or external) and the end user that defines the level of service expected from the service provider.” by Palo Alto Networks Mostly uptime In advanced environments - response time, error rate
  4. @AdamSurak Common SLA levels SLA Downtime per month Cost ($95/min,

    $50M/year) Cost per year 99 % 7 hours and 18 minutes $41 610 $500 000 99,9 % 43 minutes $4 085 $49 000 99,95 % 21 minutes $1 995 $24 000 99,99 % 4 minutes $380 $4 560 99,9999 % 2,6 seconds $4 $48
  5. @AdamSurak SLA tricks “100% uptime, 5% refund after each 0.05%”

    ➡ 99.95% “99.9% SLA, downtime counts if backend responds with error during 2 consecutive 90s intervals” ➡ 99.8%
  6. @AdamSurak Underestimated dependencies Power/network segments • are two adjacent racks

    really independent? • how protected is the network? Rogue DHCP? IP hijack? • can you choose rack with your provider? • what happens if you order 3 servers at once? A/C • influences a set of racks Network cables and interfaces are always broken • is your 1Gbit interface really in 1Gbit mode?
  7. @AdamSurak Network related issues top of the rack switch network

    maintenance • unplanned • planned - but they forget to tell you • failing - planned, they told you, but you have downtime
  8. @AdamSurak Clouds are not error-proof AWS has outages GCP has

    outages Azure has outages iCloud has outages Verizon has outages …you name it you can deploy multi-cloud! => APIs!
  9. @AdamSurak Network related issues Transit corruption • Algolia - LeaseWeb

    - ISP - AWS - Customer AWS AZs ISP edge LeaseWeb edge Algolia - LW ISP2 edge Algolia - ISP2 - proxy
  10. @AdamSurak Network related issues Transit corruption • AWS Dublin broke

    Direct Connect with OVH BGP related • ISP received 0.0.0.0/0 from a new peer => 75% traffic lost • Malaysia Telecom announced AWS prefixes => US-East-1 unavailable • ISP of CloudFlare misconfigured router and started to receive all CloudFlare’s worldwide traffic in Doha, Qatar TCP proxy becomes your best friend
  11. @AdamSurak DNS Essential service No DNS, no new connections Packet

    loss prone Latency of DNS is counted to timeouts DNS providers are popular DDoS targets Having two DNS providers is perfectly doable -> APIs!
  12. @AdamSurak Software design TCP checksum is not 100% safe DNS

    resolving is not 100% working HTTP calls don’t always succeed or return 200 • what is the default timeout of your HTTP client?
  13. @AdamSurak Software operations Package repositories can get broken or out-of-sync

    Can you deploy when GitHub is down? Invest in introducing mistakes! iptables -A INPUT -p udp --dport 33434:33523 -j REJECT
  14. @AdamSurak People Who holds the knowledge about the system? Do

    people know what to do? How do you escalate?
  15. –Sidney Dekker “Everything that can break will work and then

    we will make wrong assumptions about the reliability.”