Scale Security Programs with Scorecarding

Security teams increasingly take a collaborative, partnership-based approach to securing their applications and organizations. Scaling these efforts requires thoughtfully distributing awareness and ownership of security risk. Scorecarding is used at leading companies to make security posture visible, actionable, and engaging across the entire organization.

In this session, we dive into how companies like Netflix, Chime, GitHub, and DigitalOcean use scorecarding to distribute security ownership, drive continuous improvement, and align risk management with business goals. You’ll walk away with practical, tool-agnostic strategies for implementing your own scorecarding program that not only enhances security posture but fosters a culture of shared responsibility and proactive risk management.

Transcript

1. Rami McCarthy Principal Security Research @ Wiz Scale Security Programs with Scorecarding
2. Scale Security Programs with Scorecarding Partnership Based Security
3. Scale Security Programs with Scorecarding
4. Scale Security Programs with Scorecarding Partnership Based Security… • Avoids being the ‘Department of No’ • Takes a consultative approach • Leverages “Security Champions” or “Security Partners”
5. Scale Security Programs with Scorecarding Partnership Based Security struggles to scale
6. Scale Security Programs with Scorecarding Challenges in Scaling Partnership •Keeping aligned with partner teams •Tracking and chasing issues •Providing visibility outwards and upwards •Security champions as a crutch
7. Scale Security Programs with Scorecarding Challenges in Scaling Partnership •The security team is not omniscient •The security team relies on engineering teams •The security team must avoid asymmetric work
8. Scale Security Programs with Scorecarding Ways Security Causes Asymmetric Work • FUD driven requests • Throwing vulnerabilities over the fence • Instituting blanket requirements, without paved roads • Lack of integration
9. Scale Security Programs with Scorecarding Hi, I’m Rami ! • Principal Security Researcher at https://ramimac.me Previously: Figma, Cedar, NCC Group Advisor:
10. Scale Security Programs with Scorecarding Introducing: Scorecarding •A data driven approach to managing security •Focused on creating a shared ground truth
11. Scale Security Programs with Scorecarding Types of Scorecarding • vulnerability focused • security holistic • comprehensive
12. Scale Security Programs with Scorecarding Case Studies
13. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Twilio (Segment) [2022] Eric Ellet - Embracing Risk Responsibly: Moving beyond inflexible SLAs and exception hell by treating security vulnerabilities and risk like actual debt
14. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Uber [2017] Lindsey Glovin - An Inside Look: What Happens to Bug Reports at Uber?
15. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Dominos [2018] Lebin Cheng, Michael Sheppard - Domino’s Delivery of a Faster Response was no Standard Order AppSec Metrics Dashboard – Executive View Health Score Health Score Health Score
16. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Dominos [2018] Lebin Cheng, Michael Sheppard - Domino’s Delivery of a Faster Response was no Standard Order AppSec Metrics Dashboard – Execu Health Score Health Score
17. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Dominos [2018] Lebin Cheng, Michael Sheppard - Domino’s Delivery of a Faster Response was no Standard Order KPI Metrics Drill-down
18. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Dominos [2018] Lebin Cheng, Michael Sheppard - Domino’s Delivery of a Faster Response was no Standard Order AppSec KPI Metrics Dashboard
19. Scale Security Programs with Scorecarding “Isn’t this just {GARTNER_ACRONYM}”
20. Scale Security Programs with Scorecarding “Isn’t this just {GARTNER_ACRONYM}” A Scorecard: • should be agnostic to the data sources integrated • must support roll-up and drill-down • must be tied to an integrated culture of risk ownership • doesn’t need to integrate with remediation -> but can!
21. Case Study: Security Scorecard Scale Security Programs with Scorecarding Chime - Monocle Atomic fixes doable by engineers observed by leaders
22. Case Study: Security Scorecard Scale Security Programs with Scorecarding Chime [2022-23] David Trejo – Monocle at Chime
23. Case Study: Security Scorecard Scale Security Programs with Scorecarding Chime [2022-23] David Trejo – Monocle at Chime
24. Case Study: Security Scorecard Scale Security Programs with Scorecarding Chime [2022-23] David Trejo – Monocle at Chime
25. Case Study: Security Scorecard Scale Security Programs with Scorecarding Chime [2022-23] David Trejo – Monocle at Chime
26. Case Study: Security Scorecard Scale Security Programs with Scorecarding Wiz4Wiz
27. Case Study: Security Scorecard Scale Security Programs with Scorecarding Wiz4Wiz
28. Case Study: Security Scorecard Scale Security Programs with Scorecarding Netflix [2020] Scott Behrens, Esha Kanekar - A Pragmatic Approach for Internal Security Partnerships
29. Case Study: Security Scorecard Scale Security Programs with Scorecarding Netflix [2020] Scott Behrens, Esha Kanekar - A Pragmatic Approach for Internal Security Partnerships
30. Case Study: Security Scorecard Scale Security Programs with Scorecarding Netflix [2020] Scott Behrens, Esha Kanekar - A Pragmatic Approach for Internal Security Partnerships
31. Case Study: Security Scorecard Scale Security Programs with Scorecarding Netflix [2020] Scott Behrens, Esha Kanekar - A Pragmatic Approach for Internal Security Partnerships
32. Case Study: Comprehensive Scorecard Scale Security Programs with Scorecarding GitHub • Accessibility (A11Y) + Security + Availability • Service metadata: • Tier • QoS • Type • Ownership [2024] Deepthi Rao Coppisetty - GitHub’s Engineering Fundamentals program: How we deliver on availability, security, and accessibility
33. Scale Security Programs with Scorecarding Security Debt
34. Scale Security Programs with Scorecarding Security Debt “An obligation for future security work” [2024] Jack Danger - Technical Debt Financing
35. Scale Security Programs with Scorecarding How to Discuss Security Debt • In the context of investments, principal, interest rates, or ROI • Debt should fundamentally be a result of strategic decisions • Debt appears when the company incentivizes (or allows) shortcuts for immediate value payouts • Focus on High Interest Debt [2024] Jack Danger - Technical Debt Financing
36. Scale Security Programs with Scorecarding Security Debt The optimal amount of security debt is non-zero [2024] Patrick McKenzie - The optimal amount of fraud is non-zero
37. Scale Security Programs with Scorecarding Security Debt: Case Studies
38. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Twilio (Segment) [2022] Eric Ellet - Embracing Risk Responsibly: Moving beyond inflexible SLAs and exception hell by treating security vulnerabilities and risk like actual debt
39. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Twilio (Segment) [2022] Eric Ellet - Embracing Risk Responsibly: Moving beyond inflexible SLAs and exception hell by treating security vulnerabilities and risk like actual debt
40. Case Study: Vulnerability Scorecard Scale Security Programs with Scorecarding Twilio (Segment) [2022] Eric Ellet - Embracing Risk Responsibly: Moving beyond inflexible SLAs and exception hell by treating security vulnerabilities and risk like actual debt
41. Case Study: Security Scorecard Scale Security Programs with Scorecarding DigitalOcean [2024] Ari Kalfus, Tim Lisko - Contextual Vulnerability Management With Security Risk As Debt
42. Case Study: Security Scorecard Scale Security Programs with Scorecarding Carta • A Debt System modeled after a credit card • Card balance • carried risk • Credit limit • risk tolerance • Credit score • control adherence [2021] Garret Held - Owning security risk
43. Scale Security Programs with Scorecarding Failure Modes in Fixing Security Debt • “Failmode: 20% time allocated to debt” • “Failmode: A Technical Debt Team” • “Failmode: Locally-visible debt reductions” [2024] Jack Danger - Technical Debt Financing
44. Scale Security Programs with Scorecarding Starting Scorecarding
45. Scale Security Programs with Scorecarding Pre-Work • Service/application discovery • Durable ownership • Business criticality • Centralized Data • Organizational alignment on risk RACI • Risk Acceptance / Deferral
46. Scale Security Programs with Scorecarding Elements of a Scorecard • Vulnerabilities vs. Risk vs. Debt • Roll-up and Drill-down • (Minimal) Golden Metrics • Democratized ownership • Embedded in intra and inter team processes
47. Scale Security Programs with Scorecarding Scorecard Go-To-Market • Start with a listening tour • Dogfood • Find a design partner • Limit initial data sources, • prioritizing Signal-to-Noise Ratio • Celebrate wins • Build goodwill (bribery!)
48. Scale Security Programs with Scorecarding How to get bugs fixed • Validate prioritization • Improve clarity (how, impact, next steps, ownership) • Improve motivation (why does security matter) • “you are in the small minority of people who have not fixed your open security bug" • Empathize, and integrate to other workflows • Gentle reminders • Clean escalations [2017] Collin Greene - Fixing security bugs
49. Scale Security Programs with Scorecarding courtesy of Chris Hughes Questions?