Crafting an Effective Security Organisation (QCon NYC)

An updated version of the talk I gave at KiwiCon 8.

Blurb:
Understanding people, and not just technology, is critical in building a successful Security team. Much has been spoken about Etsy's engineering culture, and how continuous deployment and 'devops' have been embraced and developed, but how does security operate in such an environment? This presentation will discuss the progressive approaches taken by the Etsy security team to provide security while not destroying the freedoms of the Etsy engineering culture that are loved so much.

Discussion will cover the building of an effective security organisation that is people rather than technology centric, and one that positions security to facilitate problem solving with fellow engineers rather than blocking progress through the fear of increased risk. The aim of this discussion is to start a dialogue that will hopefully result in a more honest and inclusive security environment, in contrast to the more common scenario where a false perception of security exists that becomes increasingly divergent from reality as the imposed constraints are actively circumvented.

The approaches discussed are those that we have found work for Etsy but should not be seen as a one-size-fits-all solution. Every organisation is different and has its own cultural needs, but it is hoped attendees will be able to adapt our learnings to best meet their own organisation and in doing so share these experiences back with the wider community.

Transcript

1. Crafting An Effective Security Organisation QCon NYC 2015 Rich Smith (@iodboi)
2. @iodboi $ whoami • Rich Smith - Brooklyn, NYC • Director of Security at Etsy • Co-Founder of Syndis in Reykjavík, Iceland • Background in breaking not building: Vuln Research, Exploit Dev, Pen-Testing, Attack Framework Dev …
3. @iodboi Who? • etsy.com - Craft and vintage marketplace • Gross Marketplace Sales (GMS) $1.93 Billion in 2014 • 20.8M active buyers, 1.4M active sellers* • Buying & selling from almost every country in the world • Offices in 7 countries*, HQ in Brooklyn NYC • 717 Full Time Employees*, 14 in the tech security team *As of March 2015
4. @iodboi Focus Of Today - Lessons Learnt & Where We Came From - Security Mindset & Motivations - Fostering & Growth of Security Culture Copyright: www.etsy.com/shop/EclipticMediaPhoto
5. Disclaimer A + B != Culture
6. Security from 50,000 ft
7. Technology Society Technology Security
8. @iodboi From this perspective it’s easy to see that people need to be considered alongside technology for effective security
9. @iodboi Security Ego It doesn’t diminish your security cred to value people as much as technology, it just means you will have greater impact & effectiveness. 
 You will have more tools to work with.
10. Etsy Engineering Culture
11. @iodboi (Some) Core Engineering Principles • Empower the edges • Trust but verify • ‘If it moves graph it’ - Let the data lead you • ‘Just Ship’ - Get things done • Every engineer can push to prod at any time
12. What does Continuous Deployment at Etsy look like?
13. @iodboi Very end of 2009 Today Pushes Per Day
17. @iodboi But how do you ‘security’ this anarchy¿
18. @iodboi In such an environment classical security approaches don’t apply well
19. @iodboi Classical == Restrictions
20. @iodboi Classical == Blocking
21. @iodboi If Security introduces blocking to the org, it will be ignored, not embraced
22. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • With CD, no such things as ‘out of cycle’ patches • Security engineers push fixes directly to production
23. ‘DevOpsSec’
24. @iodboi …. or ‘Lessons Security can learn from DevOps’
25. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim: Remove silos & organizational blockers between Ops and Developers • Central to this focus on good Communication & Collaboration
26. @iodboi ‘DevOpsSec’ Dev Ops Sec • Natural extension of DevOps • Security faces many of the same challenges as Ops does/did • Remove barriers between Security, Developers and Ops
27. @iodboi The time when a single person or team can be responsible for an orgs security is long over….
28. ….it is up to EVERYONE
29. @iodboi Security as a Blocker • Lazy and plain ‘bad’ security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you • No’s are a Finite Resource - use them wisely
30. @iodboi Security as a Enabler • Assisting teams to do their new crazy ideas - securely • Chase solutions to difficult challenges • If your security engineers don’t like hard problems and novel solutions you have the wrong ones • Incentivises proactive engagement with Security
31. @iodboi Designated Hackers • Security engineers assist multiple teams • ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security • Early visibility, input & deeper insight
32. ‘You’re only a blocker if you’re the last to know’ John Allspaw, Some meeting room, somewhere at Etsy
33. Principles of Effective Security
34. @iodboi 3 Principles of Effective Security 1. Enabling 2. Transparent 3. Blameless
35. @iodboi A security team’s success should be measured by what they enable not by what they block Enabling
36. @iodboi A security team that is open as to what it does, and why, spreads understanding and is embraced Transparent
37. @iodboi Security failures will happen, only without blame will you be able to understand the true causes Blameless
38. Progressive Security Culture
39. @iodboi Progressive Security Culture • Understanding that security is as much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical • Great culture depends on great people
40. @iodboi Security Team Hiring Number 1 rule …… Don’t Hire Assholes
41. @iodboi Security Team Hiring If you inadvertently do, or you inherit one…… Remove them ASAP
42. @iodboi Great culture needs great people • Abrasive members will be the single biggest factor undermining your progressive security efforts • Value social skills as highly as technical skills when making your security hires • ‘Cultural fit’ critically important
43. @iodboi The more diverse a security team, the more approachable it will be to more people
44. @iodboi Security Outreach • Distinct from security education • Focus on building relationships • Removes barriers / reduces intimidation • Can be as simple as buying cakes or beer! • Assign budget to this, it will be the best ROI you see
45. @iodboi ‘Sociable conversation is the inevitable product of socializing. Sociable conversation is the way that human beings establish trusted relationships among themselves’ Cory Doctorow - Information doesn’t want to be free
46. @iodboi Security Candy! • Biggest source of security pod ‘drive bys’ • IRC bot command so people can see what’s in stock • Graph consumption!
47. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security • Embracing transparency • Provides insight to daily security issues and concerns • Build strong personal relationships • Seed champions back out to the organization
48. Securgonomics
49. er·go·nom·ics ˌərɡəˈnämiks/ noun the study of people's efficiency in their working environment.
50. secur·go·nom·ics /səˈkyo͝or/ ɡəˈnämiks/ noun the study of the efficiency of people's security interactions in their working environment.
51. @iodboi Securgonomics • Lowering the barrier to interact with security • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can • Have your security dashboards front & centre
52. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories • Applies to Security failures as much as Ops failures
53. @iodboi ‘We must strive to understand that accidents don’t happen because people gamble and loose. Accidents happen because the person believes that what is about to happen: - Is not possible - Has no connection to what they are doing - The intended outcome is worth the risk’ Erik Hollnagel Blameless Postmortems
54. @iodboi Blameless postmortem 
 blog post by John Allspaw:
 codeascraft.com/2012/05/22/blameless-postmortems
55. Indicators of an Effective Security Team
56. @iodboi Is Data Driven • Too often security is explained with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector • Gather data to support security decisions and let it lead you to the correct shade of grey
57. @iodboi Runs a Bug Bounty • Continuous Assessment of your security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming • Generates good metric sets about security (data driven)
58. @iodboi Doesn’t Cry Wolf • Verify issues before raising them to developers • They will only chase their tail a few times before ignoring • Security engineers should be in amongst the codebase • Aim to own the entire fix process themselves
59. @iodboi Makes Realistic Tradeoffs • Not everything is critical • Understand impact • Let low risk issues ship & getting commitments to a reasonable remediation window buys you lots • No’s are a Finite Resource - use them wisely
60. @iodboi Provides Context & Impact • Explaining why something is an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful, starts dialogues …. • ‘Input validation was insufficiently applied’ doesn’t
61. @iodboi Recognises & Rewards • Rewarding folks in the org who reach out to Security • We do this is a number of ways: • Pins and patches • T-Shirts • Etsy gift vouchers • IRC Pluses & Value Awards • Thanking people for raising issues
62. @iodboi Etsy Value Awards
63. @iodboi Treats Security as a BRAND • Your security culture has real value • Work long & hard to build it up • Can however be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team org wide • Get your peers to buy into security
64. Wrap up
65. @iodboi Final thoughts • Building an effective security organisation takes effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset • Enable don’t block, else you’ll make security a NOP
66. @iodboi Enabling. Transparent. Blameless
67. We’re Hiring! etsy.com/careers (Conditions apply, see slide 40….!)
68. <link /> Blog sin-ack.co.uk Presentations speakerdeck.com/iodboi Tweetz twitter.com/iodboi Code github.com/mynameismeerkat