Lessons Learned from Five Years of Multi-Cloud at PagerDuty

Arup Chakrabarti Director of Engineering, PagerDuty Five Years of Multi-Cloud
at PagerDuty A ROMANTIC AND COMPLICATED LOVE STORY SRECON AMERICAS 2018 @arupchak

@arupchak Disclaimers and Context

@arupchak What is PagerDuty?

@arupchak I work with smrt smart people

@arupchak You are not PagerDuty

@arupchak We get this wrong sometimes

@arupchak You will not get an easy answer

@arupchak Not a vendor endorsement

@arupchak Slides will be posted online afterward

@arupchak Terminology

@arupchak Multi-Cloud

@arupchak Having Active or Passive Infrastructure in Multiple Cloud Providers

@arupchak Having the same product or service spread across Multiple
Cloud Provider

@arupchak What every Procurement Manager thinks they want

@arupchak Active / Active

@arupchak Running the same workload across multiple datacenters

@arupchak “Distributed Systems”

@arupchak Active / Passive

@arupchak Running a workload in one datacenter with a standby
datacenter

@arupchak History Lesson

@arupchak PagerDuty Early 2012 • Cloud Native • Used Failover
for High Availability • MySQL Master/Slave Topology based on DRBD • Stateless Rails app behind Load Balancers • AWS us-east-1 and failover site in New Jersey

@arupchak

@arupchak 2012: Cloud is Unreliable

@arupchak Minutes of downtime is unacceptable

@arupchak Only way to achieve Reliability is through distinct Regions

@arupchak PagerDuty Late 2012 • Started teasing apart PagerDuty into
separate Services • Starting using Quorum based systems • Cassandra and Zookeeper • Favored Durability over Performance • Still needed Regions or Datacenters within 50ms • Tried AWS us-east-1, us-west-1, us-west-2

@arupchak

@arupchak Remember that 50ms requirement?

@arupchak 20ms 75ms 100ms

@arupchak Had to go Multi-Cloud due to latency requirement

@arupchak 20ms 5ms 20ms

@arupchak PagerDuty Early 2018 • Software deployed to AWS us-west-1,
us-west-2 and Azure Fresno • ~50 Services across ~10 Engineering teams • Each team owns the entire vertical stack

@arupchak What went well

@arupchak Reliability Benefits

@arupchak

@arupchak Reliability: Hard to measure

@arupchak Portability Benefits

@arupchak Portability Benefits • Everything is treated as Compute •
If there is a base Ubuntu image, we can secure and use it • Actually helped in pricing

@arupchak Engineering Culture Benefits

@arupchak Engineering Culture Benefits • Teams built for Reliability early
in the SDLC • Teams had deep expertise in their technical stacks (double-edged sword) • Failure Injection / Chaos Engineering

@arupchak What did not go well

@arupchak Right sizing is hard

@arupchak Pinned to limiting system resource

@arupchak AWS m3.large =   Azure Standard F4

@arupchak 8GB / 2 Cores ≠   8GB / 4
Cores

@arupchak $112 ≠   $182

@arupchak Deep Technical Expertise Required

@arupchak Deep Technical Expertise Required • Forced to only use
common Compute across providers • Every engineer needs to know how to run their own: • Load Balancers • Databases • Applications • HA systems

@arupchak Complexity Overhead

@arupchak Abstract away providers via Chef

@arupchak

@arupchak Even Less Control Over Network

@arupchak

@arupchak The farther apart your datacenters, the less control you
have

@arupchak Cannot use hosted services

@arupchak The Big Question

@arupchak Should you go Multi-Cloud?

@arupchak “It Depends” -Arup on almost everything

@arupchak What to consider • Business requirements first, technical requirements
second • Company buy-in • Engineering staff capabilities • What do your customers care about?

@arupchak “Understand your customer’s problems better than they do” -Andrew
Miklas, PagerDuty Co-Founder

Arup Chakrabarti Director of Engineering, PagerDuty Thank You WE ARE
HIRING! PAGERDUTY.COM/CAREERS @arupchak

Lessons Learned from Five Years of Multi-Cloud ...

Lessons Learned from Five Years of Multi-Cloud at PagerDuty

More Decks by Arup Chakrabarti

Other Decks in Technology

Featured

Transcript