Share
Embed
Microservices Manchester 2016: Security, Microservices & Vault
Slides for talk done at Microservices Manchester conference http://www.microservicesmanchester.com/speakers/nick-watt/
Nicki Watt
July 05, 2016
Transcript
117 slides
-
About Me • Hands on Lead consultant at OpenCredo •
Co-author Neo4j In Action • Twitter: @techiewatt 2 -
9 from Silo’d teams with manual release processes image credit:
http://kittypluscoco.blogspot.co.uk/ 2011/04/day-at-dog-park.html Teams -
10 image credit: http://www.notey.com/@coolshitibuy/external/10054533/ ruffwear-approach-dog-backpack.html to agile teams with fast,
automated software delivery DevOps! Teams -
12 What do you mean “It’s going live today” ?
image credit: https://www.facebook.com/EarltheGrump/photos Security ? -
13 image credit: https://www.facebook.com/EarltheGrump/photos SECURITY BOLTED ON AT THE END!
# FAIL! Security ? What do you mean “It’s going live today” ? -
15 image credit: http://www.beauswish.org/wp-content/uploads/2016/04/arianna.jpg DevSecOps! agile teams (with security as
a 1st class citizen) practicing fast, secure, automated software delivery -
Delivery Pipeline 17 http://www.devsecops.org/blog/2016/5/20/-security <— Shifting Security to the Left
Shannon Lietz DEV TEST OPS SECURITY -
Delivery Pipeline 17 http://www.devsecops.org/blog/2016/5/20/-security <— Shifting Security to the Left
Shannon Lietz DEV TEST OPS SECURITY -
Microservice example: Big retail store selling goods which includes a
typical “web store” 19 -
22 user service product service Example: web store external system
XXX sensitive data passwords, keys -
23 Example: web store external system XXX store api store
front user service product service sensitive data passwords, keys -
24 sensitive data store api store front user service product
service external system XXX passwords, keys Example: web store -
Know thy playground! • What infrastructure? • What tech stacks?
• What databases? • What type of delivery channels? 26 -
27 sensitive data store api store front user service product
service external system XXX passwords, keys Example: web store -
28 sensitive data store api store front user service product
service external system XXX passwords, keys Example: web store -
29 sensitive data store api store front user service product
service external system XXX passwords, keys Example: web store -
33 IDENTIFY PROTECT DETECT RESPOND RECOVER What stuff needs protecting?
What can I do to protect it? How will I know if bad stuff happens? What should I do when bad stuff happens? How can I get my system back up and running after bad stuff has happened? -
38 IDENTIFY sensitive data external system XXX store api store
front passwords, keys user service product service steal sensitive user data -
store api store front sensitive data passwords, keys user service
product service external system XXX 39 IDENTIFY gain access to internal network steal sensitive user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB social engineering sniff non encrypted traffic -
external system XXX sensitive data passwords, keys user service product
service 40 IDENTIFY store api store front attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data steal sensitive user data modify data in DB -
external system XXX 41 IDENTIFY store api store front sensitive
data passwords, keys user service product service gain access to internal network steal sensitive user data social engineering sniff non encrypted traffic -
store api store front sensitive data passwords, keys user service
product service external system XXX 42 IDENTIFY gain access to internal network steal sensitive user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB social engineering sniff non encrypted traffic -
Security, and actually being able to do things, always requires
a trade off! 43 -
store api store front sensitive data passwords, keys user service
product service external system XXX 44 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall -
store api store front sensitive data passwords, keys user service
product service external system XXX 45 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall -
store api store front sensitive data passwords, keys user service
product service external system XXX 46 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall cfssl -
store api store front sensitive data passwords, keys user service
product service external system XXX 47 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall -
store api store front sensitive data passwords, keys user service
product service external system XXX 48 DETECT Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS -
store api store front sensitive data passwords, keys user service
product service external system XXX 49 gain access to internal network infect employee computer install malware via email sniff non encrypted traffic compromise user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS HTTPS Firewall antivirus Use prepared statements IDS Log suspicious queries Log HTTP requests Log HTTP requests build web app vuln verification into CI/CD DETECT Distributed logging capability Container level logging Alerting capability Infrastructure level logging Serverless logging ??? -
store api store front sensitive data passwords, keys user service
product service external system XXX 50 gain access to internal network infect employee computer install malware via email sniff non encrypted traffic compromise user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS HTTPS Firewall antivirus Use prepared statements IDS Log suspicious queries Log HTTP requests Log HTTP requests build web app vuln verification into CI/CD DETECT Distributed logging capability Container level logging Alerting capability Infrastructure level logging Serverless logging ??? -
store api store front sensitive data passwords, keys user service
product service external system XXX 52 RESPOND Redirect to HTTPS Block consistent offenders Adjust firewall rules Block attackers Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS Change DB Password Reset users passwords Inform users Redirect to HTTPS -
store api store front sensitive data passwords, keys user service
product service external system XXX 53 Log suspicious queries Block consistent offenders RECOVER Redirect to HTTPS Block consistent offenders Adjust firewall rules Block attackers Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS Change DB Password Reset users passwords Inform users Redirect to HTTPS Restore from backup Fix Code, Blue/Green deploys: redeploy microservice(s) redeploy infrastructure -
• Due diligence: know thy playground • Think holistically: identify,
protect, detect, respond, recover Summary 55 Make security a 1st class citizen in your thinking process! -
• Multiple, diverse, interconnected services • More varied attack surfaces
• Harder to track what’s going on (distributed, multi facetted logging capabilities) • Transient components • Dynamic transport level encryption (HTTPS) • Authentication & Authorisation (see David’s talk :) • Trash & burn recovery strategies Microservice security challenges 56 -
61 • Unified API to access multiple backends • ACL
policies - who can access what • Audit Logs -
62 Unseal Init service 1 service 2 Allow token to
be used by tools to access secrets Acquire policy constrained token Create microservice mount or area, add secrets System X -
63 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key
2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 -
64 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key
2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 -
65 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key
2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0 -
67 Unseal Create segregated area, policies, add secrets Init Acquire
policy constrained token Allow token to be used by tools to access secrets service 1 service 2 System X -
68 $ vault mount -path=usersvc generic Successfully mounted 'generic' at
‘usersvc'! $ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... usersvc/ generic system system Vault create new mount -
69 $ vault write usersvc/db-password value=ASDKJ234SF*2 Success! Data written to:
usersvc/db-password $ vault read usersvc/db-password Key Value lease_duration 2592000 value ASDKJ234SF*2 Vault write, then read back secret -
70 $ cat usersvc.policy path "usersvc/*" { policy = "read"
} $ vault policy-write usersvc usersvc.policy Policy 'usersvc' written. Vault create custom policy -
71 Unseal Allow token to be used by tools to
access secrets Init Acquire policy constrained token service 1 service 2 Create segregated area, add secrets System X -
74 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java
Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1 -
75 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java
Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1 Separate Code and Config - Especially Secrets!! -
76 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java
Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1 Separate Code and Config - Especially Secrets!! DETECT https://github.com/michenriksen/gitrob https://github.com/awslabs/git-secrets -
77 Options • Push secrets in • Pull secrets out
• Variations of the above … -
78 Push secrets in … user service db1 1 authenticate
2 orchestration / deployment platform 3 provide value as environment variables read secret/db-password -
79 user service db1 1 authenticate 2 read secret/db-password orchestration
/ deployment platform 3 provide value as environment variables $ vault auth e2d0a065-xxxx-yyyy-zzzz Successfully authenticated! You are… token_policies: [default, usersvc] $ vault read usersvc/db-password Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword 1 2 -
80 user service db1 1 authenticate 2 read secret/db1 orchestration
/ deployment platform 3 provide value as environment variables $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="MyClearTextPassword" -d usersvc:v1 3 -
81 Steal Sensitive User Data IDENTIFY steal sensitive user data
steal sensitive user data gain access to internal network gain access to user DB gain access to running user microservice(s) dump startup config steal plaintext password social engineering -
the-machine$ docker ps CONTAINER ID IMAGE ... CREATED STATUS NAMES
9950ea8e3c59 product-service:v1 ... 4 days ago Up 4 days prodsvc 29b9ebca6dab user-service:v2 ... 5 days ago Up 5 days usersvc 82 gain access to running user microservice(s) -
83 gain access to internal network find a disgruntled employee
dump startup config the-machine$ docker inspect 29b9ebca6dab [ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ] -
84 gain access to internal network find a disgruntled employee
the-machine$ docker inspect 29b9ebca6dab [ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ] steal plaintext password -
85 gain access to internal network gain access to user
DB gain access to running user microservice(s) dump startup config social engineering PROTECT don’t expose as plain text steal sensitive user data steal plaintext password limit user access -
87 Push secrets in … user service db1 1 authenticate
2 orchestration / deployment platform 3 provide value as environment variables read secret/db-password -
87 user service db1 1 authenticate 2 read wrapped secret
orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap Push wrapped secrets in … -
88 user service db1 1 authenticate 2 read wrapped secret
orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault read -wrap-ttl=60s usersvc/db-password Key Value --- ----- wrapping_token: 57ccef32-471d-869 wrapping_token_ttl: 60 wrapping_token_creation_time: 2016-06-28 22:.. 2 -
89 user service db1 1 authenticate 2 read wrapped secret
orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="57ccef32-471d-869" -d usersvc:v1 3 -
90 user service db1 1 authenticate 2 read wrapped secret
orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault unwrap 57ccef32-471d-869 Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword 4 -
91 dump startup config the-machine$ docker inspect 29b9ebca6dab [ {
"Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=57ccef32-471d-869", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ] -
92 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data steal plaintext password don’t expose as plain text gain access to user DB limit user access -
93 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data steal plaintext password don’t expose as plain text gain access to user DB limit user access -
94 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data don’t expose as plain text gain access to user DB steal wrapped password get real password limit user access -
95 user service db1 1 authenticate 2 read wrapped secret
orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault unwrap 57ccef32-471d-869 error reading cubbyhole/response: Error making API request. URL: GET https://vault:8200/v1/cubbyhole/response Code: 400. Errors: * permission denied 4 -
96 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access limit user access -
97 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text RESPOND gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access change DB password limit user access -
98 gain access to internal network gain access to running
user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text RESPOND gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access change DB password Expect secrets to change. Make a habit of changing them regularly. It will naturally force you to put measures in place. limit user access -
• Dynamic Secrets: Auto generate credentials on the fly Other
handy options 99 -
100 user service db1 1 authenticate 2 read dynamic password
orchestration / deployment platform 3 provide value as environment variables 0 Human / Other System Users -
101 user service db1 1 authenticate 2 orchestration / deployment
platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0 -
102 user service db1 1 authenticate 2 orchestration / deployment
platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0 -
103 user service db1 1 authenticate 2 orchestration / deployment
platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0 -
104 user service db1 1 authenticate 2 orchestration / deployment
platform 3 provide value as environment variables $ vault read postgresql/creds/usersvc-ro Key Value lease_id postgresql/creds/usersvc-ro/ c888a097-b0e2-26a8-b306-fc7c84b98f07 lease_duration 3600 password 34205e88-0de1-68b7… username vault-14301-usersvc-ro read dynamic password Human / Other System Users 0 2 -
105 user service db1 1 authenticate 2 orchestration / deployment
platform 3 provide value as environment variables $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="vault-14301-usersvc-ro" -e DB_PASSWORD="34205e88-0de1-68b7" -d usersvc:v1 read dynamic password -
• Dynamic Secrets: Auto generate creds on the fly •
Ability to combine security primitives dynamic secrets + resource wrapping Other handy options 106 -
107 gain access to internal network gain access to user
DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password -
108 gain access to internal network gain access to user
DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password use time limited dynamic creds -
109 gain access to internal network gain access to user
DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password use time limited dynamic creds compromise orchestration platform -
111 gain access to internal network gain access to user
DB gain access to running user microservice(s) dump startup config compromise orchestration platform find a disgruntled employee steal sensitive user data steal vault token get db password 1 2 3 4 Defense in Depth -
Put enough hurdles in the way of attackers for you
to stop when you can, but if not, to be able to … - realise what’s going on - react before too much damage is done 112 -
• Centralised Secrets Management • API - helps with automation
• Tries to address concerns across full security lifecycle • But still very new & maturing Vault Summary 113 -
• Encryption as a service: offload responsibility to Vault •
PKI: Generates X.509 certificates dynamically based on configured roles • SSH: Dynamically generates SSH credentials for remote hosts Other Handy Features 114 -
116 Make security a first class citizen! Don’t try and
just bolt it on at the end!