Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Crafting an Effective Security Organisation (Ki...

Rich Smith
December 11, 2014

Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Rich Smith

December 11, 2014
Tweet

More Decks by Rich Smith

Other Decks in Technology

Transcript

  1. @iodboi $whoami • Rich Smith - Brooklyn, NYC • Director

    of Security at Etsy • Co-Founder of Syndis in Reykjavík, Iceland • Background in offense: pen-testing, attack frameworks, post-exploitation, Goal Oriented Attack methodologies…
  2. @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35

    Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
  3. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture
  4. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  5. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  6. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security All work in progress !
  7. @iodboi From this perspective it’s easy to see that people

    need to be considered alongside technology for effective security
  8. @iodboi (Some) Core Engineering Principles • Empower the edges •

    Trust but verify • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you • Every engineer can push to prod at any time
  9. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  10. @iodboi Why Do This ? • Continuous Deployment/Delivery/Integration • Build

    your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Developers iterate in production, A/B experimentation • Improve Mean Time To Recovery
  11. Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  12. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events
  13. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone
  14. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches
  15. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches • Security engineers push fixes directly to production
  16. @iodboi A security team’s success should be measured by what

    they enable not by what they block Enabling
  17. @iodboi A security team that is open as to what

    it does, and why, spreads understanding and is embraced Transparent
  18. @iodboi Security failures will happen, only without blame will you

    be able to understand the true causes Blameless
  19. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers
  20. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers • Central to this focus on good Communication & Collaboration
  21. @iodboi ‘DevOpsSec’ Dev Ops Sec • Natural extension of DevOps

    • Security faces many of the same challenges as Ops did • Removing barriers between Security, Developers and Operations
  22. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world
  23. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you
  24. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you • Save your ‘No’s’ as the very last resort
  25. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely
  26. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security
  27. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security • Chase solutions to difficult challenges
  28. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships
  29. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security
  30. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security • Early visibility, input & deeper insight
  31. ‘You’re only a blocker if you’re the last to know’

    John Allspaw, ! Some meeting room ,somewhere at Etsy
  32. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem
  33. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture
  34. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical
  35. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical • Great culture depends on great people
  36. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts
  37. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires
  38. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires • ‘Cultural fit’ critically important
  39. @iodboi The more diverse a security team is, the more

    approachable it is to more people
  40. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation
  41. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs
  42. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs • Assign budget to this, it will be the best ROI you see
  43. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns
  44. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships
  45. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships • Seed champions back out to the organization
  46. @iodboi Security Candy! • Biggest source of security pod ‘drive

    bys’ • IRC bot command so people can see what’s in stock • Graph consumption!
  47. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away
  48. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable
  49. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can
  50. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can • Have your security dashboards front & centre
  51. @iodboi ‘We must strive to understand that accidents don’t happen

    because people gamble and loose. Accidents happen because the person believes that what is about to happen: - Is not possible - Has no connection to what they are doing - The intended outcome is worth the risk’ ! Erik Hollnagel Blameless Postmortems
  52. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame
  53. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution
  54. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories
  55. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories • Applies to security failures as much as Ops failures
  56. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey
  57. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector
  58. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector • Gather data to support security decisions and let it lead you to the correct shade of grey
  59. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway …….
  60. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries
  61. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming
  62. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming • Generates good metric sets about security (data driven)
  63. @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring
  64. @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring • Security engineers should be in amongst the codebase • Aim to own the entire fix process themselves
  65. @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots
  66. @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots • Save your NOs for when you need them - they are a finite resource
  67. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected
  68. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding
  69. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue ….
  70. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue …. • ‘Input validation was insufficiently applied’ does not
  71. @iodboi Recognises & Rewards • Rewarding folks in the org

    who reach out to Security • We do this is a number of ways: • Pins and patches • T-Shirts • Etsy gift vouchers • IRC Pluses & Value Awards
  72. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up
  73. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye
  74. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team
  75. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team • Get your consumers to buy in to this security shit
  76. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology
  77. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset
  78. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset • Enable don’t block, else you’ll make security a NOP