You show up to work in the morning, and your boss says, “Alright, starting today, go reduce incidents.” …What would you start with? That’s the kind of story I’ll be sharing today.
insights from hands-on incident response 2. How I analyzed incidents and turned them into action 3. How we pushed a unified process across the org Disclaimer 1: Disclaimer 1: Super context-dependent, N=1 case! My story from the trenches of a special task force.
Assumes Dev handles ops + incidents) 2.Applying “Best Practices” of SRE/DevOps … All about what I actually learned in the field Disclaimer 2: I’m not a pro SRE or DevOps guru!
How do we even define “a lot” of incidents? • Ikuo: Are we actually making that many changes? • Are changes even the root cause of these incidents? • And what kind of changes are we talking about? At this point—it was all just gut feeling. Hold up! (Though I’ve learned to respect a senior engineer’s nose for trouble.) 1-1. The Beginning
SmartView (Article) Ads Ikuo! News & Push D! Ranking R! CoreSystem T! Mobile M! SmartView T! VPoE K! * Let me call myself an “all-star” just for the sake of this story 🙏 (Manager) CTO Report To 1-2. Assemble the Strongest Team We pulled in the all-stars from every division…
SmartView (Article) Ads Ikuo! News & Push D! Ranking R! CoreSystem T! Mobile M! SmartView T! VPoE K! (Manager) CTO Report To SRE wizard * Let me call myself an “all-star” just for the sake of this story 🙏 1-2. Assemble the Strongest Team We pulled in the all-stars from every division…
was real. Pulling aces from every team showed how serious the company was about this. At the same time... we had no excuses. The downside of top-down 1-2. Assemble the Strongest Team
challenge • “Reduce incidents.” Sounds simple—turns out, it's massive. • Where do We even start? • What’s the real problem? What actually helps? • And... are there even that many incidents?
really mean? • Reduce critical incidents • Install SRE best practices into the org • Key KPIs to improve: • Mean Time Between Failure(MTBF) / Change Failure Rate(CFR) = # of incidents • Mean Time to Recover(MTTR) = Recovery time The “why are we here?” got crystal clear Thanks to our awesome VPoE 1-3. Guiding the Team
Jump into every incident! • Whenever something went down, someone's PagerDuty in ACT went off. • Eventually, we invited the whole ACT squad to every active incident. • If it was in one’s home domain—they’d help fight the fire. • If not, they’d step in for status updates, bring in the right people, or handle comms with the business side. Brutal!!
started thinking: Incident = ACT. And we earned a lot of trust. ACT shows up when there’s trouble. ACT’s got your back during incidents. ACT’s getting stuff done! 2-1. P0: Supporting Incident Handling
action items There were definitely ones that just got… forgotten • We had a culture of writing incident reports— • and even listing action items for prevention. • Awesome! • But those items weren’t being tracked. • No assignees. No due dates. No status. • WHAT?? • And the report format Different for each division… • Sometimes even different per person.
Jira tickets and track them. • Maybe even send reminders. • But… each report had a totally different format. • Now what? Help me, ChatGPT… That’s not happening. 2-2. P1. Crush unresolved critical action items
#1: Always store data in machine-readable format!! Heck yeah! we manually moved a year’s worth of incident AIs into a Notion database *Once it’s a database, you can pull it via API. Easy mode. Lesson #2: Don’t be afraid to get your hands dirty if it serves the mission. 2-2. P1. Crush unresolved critical action items
Response Process • Each division had its own way of handling incidents. • We’d tried to build a company-wide protocol before— • the “Incident Response Framework (IRF)”. • But... it never really caught on. • It only reflected the needs of one division. Why didn’t we already have a unified company-wide process?
So we rebuilt it as a company-wide framework, • backed by domain knowledge from our all-star team. • And it had to be lightweight. • We also pulled ideas from public frameworks • e.g. Pager Duty Incident Response How we build a company-wide process and framework? Because our team covered every domain, we could actually make it happen. 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
3. Workflow 4. Communication Guideline 5. Incident Report Template, Postmortem Let me walk you through the key parts Please check the slide later for the details! 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
engineer on call. Triages alerts and escalates to the IC if necessary, initiating the IRF (declaring the incident). • Incident Commander(IC) • Leads the incident response. Brings in necessary people and organizes information. May also act as the CL (Communication Lead). • Usually a Tech Lead or Engineering Manager. • Their responsibility is not to directly fix the issue, but to organize and make decisions. • Responder • Handles the actual work—such as rollbacks, config changes, etc. • Communication Lead(CL) • Handles communication with external stakeholders (i.e., non-engineers). Key point: Separate responsibilities between IC and Responder 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
severity judgment when declaring the incident. The final severity level is determined during the postmortem. • 🔥 SEV-1 • Complete failure of core UX features (e.g., news reading becomes unavailable) • 🧨 SEV-2 • Partial failure of core UX features, or complete failure of sub-UX features • 🕯 SEV-3 • Partial failure of sub-UX features It's crucial to estimate severity early on— severe incidents should be resolved faster 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
of an incident. 🩸 = Bleeding status (ongoing impact) 1. 🩸 Occurrence • An issue arises. Common triggers include deployments or config changes. 2. 🩸 Detection • The on-call engineer detects the issue via alerts. Triage begins. 3. 🩸 Declaration • The incident is officially declared. IRF begins under the IC's lead. External communication starts as needed. • While bleeding, updates must be continuously provided. 4. ❤🩹 Mitigation • Temporarily eliminate the cause (e.g., rollback) and stop further impact. 5. Resolution • Permanently fix the issue (e.g., bug fix, data correction). Bleeding is fully stopped. 6. Postmortem • Investigate root causes and discuss recurrence prevention based on the incident report. 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
place (Slack channels): • #incident • Used for status updates to the entire company and for communication with external stakeholders. • #incident-irf-[incidentId]-[title] • or technical communication to resolve the issue. • All relevant discussions and information are gathered here. Having all discussions and info in one place makes writing the report much easier later 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
company-wide template includes: • Summary • Impact • Direct Cause, Mitigation • Root Cause Analysis (5-whys) • It’s crucial to analyze direct and root causes separately. • Based on root causes, define action items to prevent recurrence • Timeline • Use a machine-readable format!!!! We standardized templates across divisions (super important!) and centralized all postmortems. 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
into every incident “Hello there, it’s me, the IRF guy 😎 Alright, I’ll be the Incident Commander this time! Everyone else, focus on firefighting!” Lesson #4: In an emergency, no one has time to learn a new protocol. Just do it! Lesson #5: Use it ourselves first, and build a feedback loop! 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
run /incident in Slack. I’ll take care of the rest Automatically creates incident tickets, dedicated Slack channels, and posts links to playbooks. Automate what works well after you’ve done it manually! 2-3. P2: Root Fixes/ Rolling Out a Unified Incident Response Process
incidents did we handle this month? How about last month…? How long did it take to resolve each one? 2-4. P2: Root Fixes/ Enhancing Incident Clarity No clue!!
Title • Status • State Machine.(we’ll get to this later) • Severity • SEV 1~3(IRF2.0) • Direct Cause • (explained later) • Direct Cause System • Group of components defined at the microservice level • Direct Cause Workload • Online Service, Offline Pipeline, … Define as many fields as possible using Enums! 46 Free-form input leads to high cardinality and makes proper analysis impossible 2-4. P2: Root Fixes/ Enhancing Incident Clarity
through different states. Model it as a State Machine and clean it up. Then you can define time metrics for each state transition! 48 2-4. P2: Root Fixes/ Enhancing Incident Clarity
include the defined data fields. If the data definition is solid, the source can be flexible. (As long as the data is trustworthy, of course.) 49 2-4. P2: Root Fixes/ Enhancing Incident Clarity • Make required fields into mandatory attributes. • Add a Notion Database for timelines. • Have people record when states change. • Make it machine-readable!!!!!
unified format) • Different formats across Divisions • Timelines as bullet points, missing data, and more… Give it three months and we’ll have plenty of data. Right? We only have six months!! 2-4. P2: Root Fixes/ Enhancing Incident Clarity
Heck yeah! We manually migrated one year’s worth of incident reports We split the work across the team and powered through over a week 2-4. P2: Root Fixes/ Enhancing Incident Clarity Re: Lesson #2: Don’t be afraid to get your hands dirty if it serves the mission.
Detected 3. Declared 4. Mitigated 5. Resolved Time To Detect Time To Mitigate Now we know where the time is going—and where we stand! Time To Resolve 2-4. P2: Root Fixes/ Enhancing Incident Clarity
caused by incidents e.g. Revenue, Reputation, Effectiveness…. Right? Especially, Revenue Loss… No… it’s the What we really want to reduce… The number of incidents? 🤔
the impact of an incident Severity Factor (impact level of an incident) Number of Incident × × In toC and ads, this pretty much defines the revenue impact • Shorten the time • Relatively easy to improve and quick to act on • Reduce severity • Ideal, but hard to control • Reduce incident count • Requires long-term efforts 3-1. What Does It Mean to Reduce Incidents?
the impact of an incident Severity Factor (impact level of an incident) Number of Incident × × In toC and ads, this pretty much defines the revenue impact • Shorten the time • Relatively easy to improve and quick to act on • Reduce severity • Ideal, but hard to control • Reduce incident count • Requires long-term efforts 3-1. What Does It Mean to Reduce Incidents? We’re starting here It also aligns with the KPIs we set when ACT was first formed! But a few months in, we gained much better clarity.
Declared 4. Mitigated 5. Resolved Time To Detect Time To Resolve Time To Mitigate Each one has different significance — and needs a different solution! Mainly in the alerting domain. Bleeding, Most critical — but also the easiest to improve. This is where IRF comes in. Time spent on root fixes and data correction. Bleeding has stopped — now it’s about accuracy, not speed. 3-2. Approaching Incident Resolution Time
slapping alerts on detected issues doesn’t work • “over-monitoring is a harder problem to solve than under-monitoring.” — SRE: How Google Runs Production System • Too many false positives → real alerts get buried • Shift to SLO + Error Budget–based alerting • Ideal: if it pages, it’s an incident • Not something you can fix overnight Still an open challenge Come back to it in Chapter 4 3-2. Approaching Incident Resolution Time
2.0 • Clear definition of what qualifies as an incident • Standardized response flow and communication guidelines • Separation of Responder and Commander roles • Training, with aces leading by example Deploying top aces + rolling out IRF 2.0 had a huge impact! 3-2. Approaching Incident Resolution Time
before production? Postmortem discussion… • Why was it deployed without testing? • → Because it could only be tested in production. • Why only in production? • → Lack of data, broken staging environment, etc… • …. Alright! Let’s fix the staging environment! 3-3. Approaching the Number of Incidents
Environments Still in Progress: Way harder than we thought! • There are tons of components. • Each division—News, Ads, Infra—has different needs and usage. • Ads is B2B, tied directly to revenue → needs to be solid and stable • News is B2C, speed of feature delivery is key Trying to build staging for everything? Not realistic, not even useful. So we started with Ads, where the demand was highest 3-3. Approaching the Number of Incidents
Tests? • Why didn’t we catch it with unit tests? • Because we didn’t have any… • … 😭 Alright! Let’s collect test coverage! 3-3. Approaching the Number of Incidents
Coverage • We charged into systems with no coverage tracking, and cranked out PRs to generate coverage reports. • Then we plotted unit test coverage vs. number of incidents by system # of Incident Ave. Coverage 3-3. Approaching the Number of Incidents
incidents? → There was a correlation. • But… does higher coverage actually reduce incidents? • Not sure. Correlation ≠ causation. • Still, when we looked at individual systems and teams with domain knowledge, the places with low coverage and high incidents did have clear reasons: • Hard to write tests / no testing culture / etc… Alright! Let’s storm the low-coverage zones and crank out unit tests! Approach #2 to Lack of Testing: Analyzing Unit Test Coverage 3-3. Approaching the Number of Incidents
Tests Get our hands dirty! add tests to everything we can get our hands on We hit 3–4 components… but it barely moved the needle We thought once we provided a few examples, others would follow… 3-3. Approaching the Number of Incidents 1. Use SonarQube to find files with high LOC and low coverage 2. Use LLMs to help generate tests 3. Repeat until the entire component hits 50% + coverage
them? • Well… the accuracy just isn’t there yet. • Teams need a habit of writing unit tests continuously. • No incentive or shared sense of value • Everyone’s under deadline pressure • — no time left for tests(!) This is a team culture and organizational challenge. To be continued in Chapter 4… 3-3. Approaching the Number of Incidents Approach #2 to Lack of Testing: Building Out Unit Tests
• Mechanisms that control app behavior dynamically • A/B Testing and Feature Flags • Both rely on our custom-built platforms — but they’re complicated • Unintended A/B assignments and misconfigurations caused frequent issues Alright, let’s clean up A/B testing and feature flags! 3-3. Approaching the Number of Incidents
Establish usage guidelines for feature flags • Strengthen validation logic • (We were able to input configs that literally caused parse errors…) Teamed up with the A/B testing platform group, Pushed for major improvements including UX! 3-3. Approaching the Number of Incidents Approaching Config Changes
jobs that are actually streaming • Server → Kafka → Flink → Scylla, ClickHouse, etc. • A custom-built platform by a specialized team • Few Flink experts on the app teams • Led to frequent issues with performance and restarts Alright! Let’s revamp the Flink platform! 3-3. Approaching the Number of Incidents
deployments, and more • Spread best practices • Cleaned up documentation • Released template projects (including tests!) • Sent direct refactor PRs to various components • Implemented best practices and tests Collaborated with the platform teamTogether, Improved the platform and its docs! 3-3. Approaching the Number of Incidents Approaching Offline Batch
December was off the charts • Last-minute changes before the holidays? • A side effect of rolling out IRF2.0? • More sensitive detection due to tighter definitions • Maslow’s Hammer: • “If all you have is IRF, everything starts to look like an incident” • But after January, things started trending down Continuous improvement is still key 3-4. Results: Did We Really “Halve” Incidents?
improvement in MTTMitigate • Thanks to the power of IRF2.0 • But MTTDetect barely improved • Detection is still an open challenge — Working on better alerting Definitely felt the momentum of change! 3-4. Results: Did We Really “Halve” Incidents?
levels showed a slight decrease We didn’t quite halve incidents… However, our challenges are clear, and we’ve laid the foundation for improvement! Let’s make it happen 3-4. Results: Did We Really “Halve” Incidents?
be real—Not happening To truly minimize incidents… • Just stop releasing? • A slow death 😇 • Throw infinite cost (people, time) at it? • More cost likely correlates with fewer incidents… • So do we keep testing until it feels 100% “safe”? 4-1. Open Challenges: Risk Management & Alerting
incident risk. • But the “right” balance is different for each system or project. • Required speed & release frequency • Cost we can throw in • Acceptable level of risk (≒ number of incidents, failure rate) • Ads is B2B and tied directly to revenue → needs to be rock-solid • News is B2C → speed of delivery comes first! Quantify our risk tolerance And use that to control how many incidents we accept. 4-1. Open Challenges: Risk Management & Alerting
SLO = Service Level Objective ~ How much failure is acceptable? • e.g. 99.9% available -> means 0.1% failure is “allowed” • Attach objectives to SLIs—metrics that reflect real UX harm • Error Budget = How much failure room we have left • When budget remains → we can step on the gas • Even bold releases are fair game • When budget runs out → UX is already suffering • No more risk—time to slow down — Ref: Implementing SLOs — Google SRE Error Budgets let us express risk tolerance—numerically. And in theory… this sounds pretty solid. 4-1. Open Challenges: Risk Management & Alerting
Time to get those SLOs in place! • Alert based on how fast your Error Budget is burning: • Trigger alerts when the budget is being consumed too quickly • If you ignore it, you’ll run out — violate your SLO • That means real UX damage! • Don’t ignore it — it is an incident! — Ref: Alerting on SLOs Sounds Good 4-1. Open Challenges: Risk Management & Alerting
rolling out SLOs in some places, but… • Defining effective SLOs isn’t easy • Biz and PdMs don’t always have the answers • There’s no time to implement them • They can’t even find time to write tests! • And even if we set them up— • if no one respects them, what’s the point? There’s no silver bullet…
need buy-in across the org • It’s not just about engineers—biz and PdMs have to be in too • Need a deliberate approach to culture • Ultimately, it’s about what we truly value • Do we believe that balancing cost and risk with SLOs is worth it? We want to embed SLOs— and ultimately, the mindset of SRE—into our engineering culture. 4-2. Remaining Challenge: Shaping Org & Culture
• Educate and train engineers, Biz, PdMs—get everyone involved • Top-Down: • Win support and direction from leadership SRE and DevOps are culture — They don’t take root in a day. It takes sweat, patience, and steady effort. 4-2. Remaining Challenge: Shaping Org & Culture How do we make SLOs actually work?
was coming to an end. • The challenge remained: Install SRE into SmartNews’s engineering culture • Implement and uphold SLOs • And more… • Boost observability • Track and act on DORA metrics, etcetc… These require ongoing effort How can we keep the momentum going— and tackle the remaining challenges even after ACT disbands?
“Distributed SRE Team” After ACT ends, ex-members return to their teams and continue SRE work using X% of their time It sounded reasonable to me… maybe? 4-3. What Comes Next…
their full-time job. • And allocating “X%” of time… yeah, that never really works. • Our decision: • Ex-ACTors would keep helping and promoting SRE, but we’d take the time to build a dedicated SRE team. We made that call as a team. There’s still plenty left unfinished—but no regrets! How should we disband ACT? — Team’s Call 4-3. What Comes Next…
has ended. Did we truly create an “Awesome Change”? Honestly… I don’t know. But we do feel like “We’ve taken the first step on a long journey toward SRE.” 4-3. What Comes Next… And a huge thanks to my teammates for fighting through these past six months!
back to work after this conference— and your boss says, “Alright, starting today, go reduce incidents.” …What would you start with? 4-3. What Comes Next…
at Scale • Site Reliability Engineering: How Google Runs Production Systems • SRE Google Workbook • Effective DevOps: Building a Culture of Collaboration, Affinity, and Tooling at Scale • Fearless Change: Patterns for Introducing New Ideas
martin_lover
wa6sn
rinchoku
gawa
yukukotani
terryu16
xgouchet
tomokusaba
ug
slobodan
satohiroyuki
hollycummins
karszawa
jcasabona
dougneiner
csswizardry
pedronauck
addyosmani
cassininazir
sachag
hannesfritz
samanthasiow
trishagee
colly
jlugia
