Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microservices Manchester 2016: Security, Micros...

Microservices Manchester 2016: Security, Microservices & Vault

Slides for talk done at Microservices Manchester conference http://www.microservicesmanchester.com/speakers/nick-watt/

Nicki Watt

July 05, 2016
Tweet

More Decks by Nicki Watt

Other Decks in Technology

Transcript

  1. About Me • Hands on Lead consultant at OpenCredo
 •

    Co-author Neo4j In Action
 • Twitter: @techiewatt 2
  2. 9 from Silo’d teams with manual release processes image credit:

    http://kittypluscoco.blogspot.co.uk/ 2011/04/day-at-dog-park.html Teams
  3. 12 What do you mean “It’s going live today” ?

    image credit: https://www.facebook.com/EarltheGrump/photos Security ?
  4. 13 image credit: https://www.facebook.com/EarltheGrump/photos SECURITY BOLTED ON AT THE END!

    # FAIL! Security ? What do you mean “It’s going live today” ?
  5. 23 Example: web store external system XXX store api store

    front user service product service sensitive data passwords, keys
  6. 24 sensitive data store api store front user service product

    service external system XXX passwords, keys Example: web store
  7. Know thy playground! • What infrastructure? • What tech stacks?

    • What databases? • What type of delivery channels? 26
  8. 27 sensitive data store api store front user service product

    service external system XXX passwords, keys Example: web store
  9. 28 sensitive data store api store front user service product

    service external system XXX passwords, keys Example: web store
  10. 29 sensitive data store api store front user service product

    service external system XXX passwords, keys Example: web store
  11. 33 IDENTIFY PROTECT DETECT RESPOND RECOVER What stuff needs protecting?

    What can I do to protect it? How will I know if bad stuff happens? What should I do when bad stuff happens? How can I get my system back up and running after bad stuff has happened?
  12. 38 IDENTIFY sensitive data external system XXX store api store

    front passwords, keys user service product service steal sensitive user data
  13. store api store front sensitive data passwords, keys user service

    product service external system XXX 39 IDENTIFY gain access to internal network steal sensitive user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB social engineering sniff non encrypted traffic
  14. external system XXX sensitive data passwords, keys user service product

    service 40 IDENTIFY store api store front attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data steal sensitive user data modify data in DB
  15. external system XXX 41 IDENTIFY store api store front sensitive

    data passwords, keys user service product service gain access to internal network steal sensitive user data social engineering sniff non encrypted traffic
  16. store api store front sensitive data passwords, keys user service

    product service external system XXX 42 IDENTIFY gain access to internal network steal sensitive user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB social engineering sniff non encrypted traffic
  17. store api store front sensitive data passwords, keys user service

    product service external system XXX 44 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall
  18. store api store front sensitive data passwords, keys user service

    product service external system XXX 45 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall
  19. store api store front sensitive data passwords, keys user service

    product service external system XXX 46 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall cfssl
  20. store api store front sensitive data passwords, keys user service

    product service external system XXX 47 PROTECT attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall
  21. store api store front sensitive data passwords, keys user service

    product service external system XXX 48 DETECT Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS
  22. store api store front sensitive data passwords, keys user service

    product service external system XXX 49 gain access to internal network infect employee computer install malware via email sniff non encrypted traffic compromise user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS HTTPS Firewall antivirus Use prepared statements IDS Log suspicious queries Log HTTP requests Log HTTP requests build web app vuln verification into CI/CD DETECT Distributed logging capability Container level logging Alerting capability Infrastructure level logging Serverless logging ???
  23. store api store front sensitive data passwords, keys user service

    product service external system XXX 50 gain access to internal network infect employee computer install malware via email sniff non encrypted traffic compromise user data attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS HTTPS Firewall antivirus Use prepared statements IDS Log suspicious queries Log HTTP requests Log HTTP requests build web app vuln verification into CI/CD DETECT Distributed logging capability Container level logging Alerting capability Infrastructure level logging Serverless logging ???
  24. store api store front sensitive data passwords, keys user service

    product service external system XXX 52 RESPOND Redirect to HTTPS Block consistent offenders Adjust firewall rules Block attackers Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS Change DB Password Reset users passwords Inform users Redirect to HTTPS
  25. store api store front sensitive data passwords, keys user service

    product service external system XXX 53 Log suspicious queries Block consistent offenders RECOVER Redirect to HTTPS Block consistent offenders Adjust firewall rules Block attackers Log suspicious queries Log HTTP requests Log HTTP requests attack store front / API sniff non encrypted traffic SQL Injection Alter query to get data modify data in DB HTTPS Use prepared statements build web app vuln verification into CI/CD gain access to internal network social engineering sniff non encrypted traffic steal sensitive user data HTTPS Firewall IDS Change DB Password Reset users passwords Inform users Redirect to HTTPS Restore from backup Fix Code, Blue/Green deploys: redeploy microservice(s) redeploy infrastructure
  26. • Due diligence: know thy playground • Think holistically: identify,

    protect, detect, respond, recover Summary 55 Make security a 1st class citizen in your thinking process!
  27. • Multiple, diverse, interconnected services • More varied attack surfaces

    • Harder to track what’s going on 
 (distributed, multi facetted logging capabilities)
 • Transient components • Dynamic transport level encryption (HTTPS) • Authentication & Authorisation (see David’s talk :) • Trash & burn recovery strategies Microservice security challenges 56
  28. 61 • Unified API to access multiple backends • ACL

    policies - who can access what • Audit Logs
  29. 62 Unseal Init service 1 service 2 Allow token to

    be used by tools to access secrets Acquire policy constrained token Create microservice mount or area, add secrets System X
  30. 63 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key

    2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0
  31. 64 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key

    2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0
  32. 65 $ vault init -key-shares=3 -key-threshold=2 Key 1: 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Key

    2: 3fd762583cc9755222fa20f2a78770aca5ecba3b16d8c Key 3: a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Initial Root Token: 57dfce17-08c4-d042-91a3-68082965367b Vault initialized with 3 keys and a key threshold of 2. Please securely distribute the above keys. When the Vault is re-sealed, restarted, or stopped, you must provide at least 2 of these keys to unseal it again. Vault does not store the master key. Without at least 2 keys, your Vault will remain permanently sealed. $ vault unseal 8573410dd211cc9b5ea6b426b19d6d668e0184c39d4ba Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 1 Vault init & unseal $ vault unseal a0428a6b6681eb15ffcea5be5c787beabcb7599a6fa91 Sealed: true Key Shares: 3 Key Threshold: 2 Unseal Progress: 0
  33. 67 Unseal Create segregated area, policies, add secrets Init Acquire

    policy constrained token Allow token to be used by tools to access secrets service 1 service 2 System X
  34. 68 $ vault mount -path=usersvc generic Successfully mounted 'generic' at

    ‘usersvc'! $ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... usersvc/ generic system system Vault create new mount
  35. 69 $ vault write usersvc/db-password value=ASDKJ234SF*2 Success! Data written to:

    usersvc/db-password $ vault read usersvc/db-password Key Value lease_duration 2592000 value ASDKJ234SF*2 Vault write, then read back secret
  36. 70 $ cat usersvc.policy path "usersvc/*" { policy = "read"

    } $ vault policy-write usersvc usersvc.policy Policy 'usersvc' written. Vault create custom policy
  37. 71 Unseal Allow token to be used by tools to

    access secrets Init Acquire policy constrained token service 1 service 2 Create segregated area, add secrets System X
  38. 74 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java

    Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1
  39. 75 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java

    Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1 Separate Code and Config - Especially Secrets!!
  40. 76 # Embedded Config spring.datasource.url=jdbc:mysql://localhost/test spring.datasource.username=dbuser spring.datasource.password=dbpass spring.datasource.driver-class-name= com.mysql.jdbc.Driver Java

    Code @Component public class MyBean { private final JdbcTemplate jdbcTemplate; @Autowired public MyBean(JdbcTemplate jdbcTemplate) { this.jdbcTemplate = jdbcTemplate; } // ... } Starting point … user service db1 Separate Code and Config - Especially Secrets!! DETECT https://github.com/michenriksen/gitrob https://github.com/awslabs/git-secrets
  41. 78 Push secrets in … user service db1 1 authenticate

    2 orchestration / deployment platform 3 provide value as environment variables read secret/db-password
  42. 79 user service db1 1 authenticate 2 read secret/db-password orchestration

    / deployment platform 3 provide value as environment variables $ vault auth e2d0a065-xxxx-yyyy-zzzz Successfully authenticated! You are… token_policies: [default, usersvc] $ vault read usersvc/db-password Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword 1 2
  43. 80 user service db1 1 authenticate 2 read secret/db1 orchestration

    / deployment platform 3 provide value as environment variables $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="MyClearTextPassword" -d usersvc:v1 3
  44. 81 Steal Sensitive User Data IDENTIFY steal sensitive user data

    steal sensitive user data gain access to internal network gain access to user DB gain access to running user microservice(s) dump startup config steal plaintext password social engineering
  45. the-machine$ docker ps CONTAINER ID IMAGE ... CREATED STATUS NAMES

    9950ea8e3c59 product-service:v1 ... 4 days ago Up 4 days prodsvc 29b9ebca6dab user-service:v2 ... 5 days ago Up 5 days usersvc 82 gain access to running user microservice(s)
  46. 83 gain access to internal network find a disgruntled employee

    dump startup config the-machine$ docker inspect 29b9ebca6dab [ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ]
  47. 84 gain access to internal network find a disgruntled employee

    the-machine$ docker inspect 29b9ebca6dab [ { "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=MyClearTextPassword", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ] steal plaintext password
  48. 85 gain access to internal network gain access to user

    DB gain access to running user microservice(s) dump startup config social engineering PROTECT don’t expose as plain text steal sensitive user data steal plaintext password limit user access
  49. 87 Push secrets in … user service db1 1 authenticate

    2 orchestration / deployment platform 3 provide value as environment variables read secret/db-password
  50. 87 user service db1 1 authenticate 2 read wrapped secret

    orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap Push wrapped secrets in …
  51. 88 user service db1 1 authenticate 2 read wrapped secret

    orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault read -wrap-ttl=60s usersvc/db-password Key Value --- ----- wrapping_token: 57ccef32-471d-869 wrapping_token_ttl: 60 wrapping_token_creation_time: 2016-06-28 22:.. 2
  52. 89 user service db1 1 authenticate 2 read wrapped secret

    orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="MyDBName" -e DB_PASSWORD="57ccef32-471d-869" -d usersvc:v1 3
  53. 90 user service db1 1 authenticate 2 read wrapped secret

    orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault unwrap 57ccef32-471d-869 Key Value --- ----- refresh_interval 2592000 value MyClearTextPassword 4
  54. 91 dump startup config the-machine$ docker inspect 29b9ebca6dab [ {

    "Id": “29b9ebca6dab147991201a5c61b72d3d546885ca8fd…”, "Created": "2016-06-27T21:26:16.126414991Z", "Args": [ "-jar", "UserService" ], "Config": { "Hostname": "29b9ebca6dab", "Env": [ “DB_USER=MyUserName", “DB_PASSWORD=57ccef32-471d-869", “VAR1=something-else“ ], "Cmd": [ "java", "-jar", "UserService" ], ... } ]
  55. 92 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data steal plaintext password don’t expose as plain text gain access to user DB limit user access
  56. 93 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data steal plaintext password don’t expose as plain text gain access to user DB limit user access
  57. 94 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data don’t expose as plain text gain access to user DB steal wrapped password get real password limit user access
  58. 95 user service db1 1 authenticate 2 read wrapped secret

    orchestration / deployment platform 3 provide wrapped value as environment variables 4 unwrap $ vault unwrap 57ccef32-471d-869 error reading cubbyhole/response: Error making API request. URL: GET https://vault:8200/v1/cubbyhole/response Code: 400. Errors: * permission denied 4
  59. 96 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access limit user access
  60. 97 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text RESPOND gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access change DB password limit user access
  61. 98 gain access to internal network gain access to running

    user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT don’t expose as plain text RESPOND gain access to user DB steal wrapped password get real password Raise TOFU alarm Audit access change DB password Expect secrets to change. Make a habit of changing them regularly. It will naturally force you to put measures in place. limit user access
  62. 100 user service db1 1 authenticate 2 read dynamic password

    orchestration / deployment platform 3 provide value as environment variables 0 Human / Other System Users
  63. 101 user service db1 1 authenticate 2 orchestration / deployment

    platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0
  64. 102 user service db1 1 authenticate 2 orchestration / deployment

    platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0
  65. 103 user service db1 1 authenticate 2 orchestration / deployment

    platform 3 provide value as environment variables $ vault mount postgresql Successfully mounted 'postgresql' at 'postgresql'! $ vault write postgresql/config/connection connection_url="postgresql:// vault:somepassword@yourhost:5432/postgres" $ vault write postgresql/roles/usersvc-ro \ sql="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD ‘{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA users TO \"{{name}}\";" Success! Data written to: postgresql/roles/ read dynamic password Human / Other System Users 0 0
  66. 104 user service db1 1 authenticate 2 orchestration / deployment

    platform 3 provide value as environment variables $ vault read postgresql/creds/usersvc-ro Key Value lease_id postgresql/creds/usersvc-ro/ c888a097-b0e2-26a8-b306-fc7c84b98f07 lease_duration 3600 password 34205e88-0de1-68b7… username vault-14301-usersvc-ro read dynamic password Human / Other System Users 0 2
  67. 105 user service db1 1 authenticate 2 orchestration / deployment

    platform 3 provide value as environment variables $ # Start docker container,pass in vars docker run --name usersvc -e DB_USER="vault-14301-usersvc-ro" -e DB_PASSWORD="34205e88-0de1-68b7" -d usersvc:v1 read dynamic password
  68. • Dynamic Secrets: Auto generate creds on the fly •

    Ability to combine security primitives
 dynamic secrets + resource wrapping Other handy options 106
  69. 107 gain access to internal network gain access to user

    DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password
  70. 108 gain access to internal network gain access to user

    DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password use time limited dynamic creds
  71. 109 gain access to internal network gain access to user

    DB gain access to running user microservice(s) dump startup config find a disgruntled employee PROTECT steal sensitive user data DETECT steal wrapped password don’t expose as plain text get real password Raise TOFU alarm Audit access RESPOND change DB password use time limited dynamic creds compromise orchestration platform
  72. 111 gain access to internal network gain access to user

    DB gain access to running user microservice(s) dump startup config compromise orchestration platform find a disgruntled employee steal sensitive user data steal vault token get db password 1 2 3 4 Defense in Depth
  73. Put enough hurdles in the way of attackers for you

    to stop when you can, but if not, to be able to … - realise what’s going on - react before too much damage is done 112
  74. • Centralised Secrets Management • API - helps with automation

    • Tries to address concerns across full security lifecycle • But still very new & maturing Vault Summary 113
  75. • Encryption as a service: offload responsibility to Vault •

    PKI: Generates X.509 certificates dynamically based on configured roles • SSH: Dynamically generates SSH credentials for remote hosts Other Handy Features 114