HashiConf EU 2016: Building secure environments in clouds using HashiCorp tools

Talk given at HashiConf EU 2016 (Amsterdam). Associated video can be found here: https://www.youtube.com/watch?v=tbrQnrLExow

Transcript

1. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Building secure environments in clouds using HashiCorp tools Nicki Watt @techiewatt HashiConf EU - 12/06/2016 1
2. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2 About Me • Hands on Lead consultant at OpenCredo
 • Co-author Neo4j In Action
 • Currently working with a UK government department on cloud automation project
 • Twitter: @techiewatt 

3. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3 Agenda • What is the problem
 • What are the options
 • How: Principles, challenges, lessons, tools • Conclusion 

4. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 4 What problem are we trying to address?
5. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 5 Act 1 : “Take advantage of cloud computing”
6. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 6 Act 2 : “Efficiently Take advantage of more cloud computing”
7. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 7
8. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 8
9. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 9 How to create fast, repeatable, secure environments capable of running in different clouds!
10. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 10 An example requirement: Team1 needs a Kubernetes based development environment
11. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 11 Input: Environment prefix: team1 Number K8s Slaves: 3 Environment domain suffix: t1tools.domain.io Initial SSH keys: AAAEFF user1@team1 Cloud: AWS
12. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 12
13. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13 K8S Master K8S Node-1 K8S Node-2 K8S Node-3 Postgres DB Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Environment “Contract” •Completely isolated, need to VPN in to access •Agreed customisable IaaS layout •Agreed base software installed
14. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 14 What are our options?
15. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 15 you need our cloud management platform !
16. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 16 vs
17. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 17
18. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 18 Principles
19. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 19 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP
20. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 20 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP
21. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 21 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP
22. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 22 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP
23. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 23 • Automate everything
 • Separate config from code • API driven clouds & tools • Prefer modular, open source tools
 ASAP
24. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 24 • Self service functionality • Automated environment creation 
 (under the hood) functionality
25. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 25 K8S Master K8S Node-1 K8S Node-2 K8S Node-3 Postgres DB Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Environment “Contract”
26. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 26 K8S Master K8S Node-1 K8S Node-2 K8S Node-3 Postgres DB Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Environment “Implementation” building blocks Configuration management tool
27. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 27 Core User Solution requirements (Intra-cloud) Supporting Services Environment “Implementation” building blocks
28. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 28 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services Environment “Implementation” building blocks
29. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 29 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services Environment “Implementation” building blocks
30. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 30 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Configuration management tool Environment “Implementation” building blocks
31. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 31 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Configuration management tool Environment “Implementation” building blocks
32. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 32 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Approach style: Mutable infrastructure Configuration management tool
33. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 34 How?
34. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ^ 35 • Automated Image creation
 • Automated IaaS Provisioning • Automated Instance Management • Securing all the things! Bootstrap
35. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 36 Core User Solution requirements (Intra-cloud) Supporting Services Challenge #1 Automated Image Provisioning
36. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 37
37. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services 38 Challenge #2 Automated IaaS Provisioning
38. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 39 • Networks • Firewall Rules • Routers • Compute 
 Resources • Public / Floating 
 IP Addresses
39. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 41 Lesson There is NO single common cloud API
40. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 42 Tool: Terraform Automated IaaS Provisioning
41. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 43 Creates, manages, and manipulates infrastructure resources.
42. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 44 Multiple Cloud Providers
43. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 45 Multiple Cloud Providers
44. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 46 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = "${var.env-prefix}-ovpn" image_name = "${var.image_name}" flavor_name = "${var.openvpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = "${var.public-ip-pool}" ... } terraform.tf Declarative DSL (AWS) ## AWS Compute instance resource "aws_instance" "ovpn" { ami = “${var.ovpn-ami}" instance_type = “${var.m-openvpn-instance-type}" vpc_security_group_ids = [ "${aws_security_group.ovpn.id}"] subnet_id = “${aws_subnet.dmz.id}" ... } ## DMZ network exposing Public IP resource "aws_subnet" "dmz" { vpc_id = "${aws_vpc.core.id}" cidr_block = “${var.m-dmz-net-cidr}" map_public_ip_on_launch = 1 ... } terraform.tf
45. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 47 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = “${var.m-env-prefix}-ovpn" image_name = “${var.m-ovpn-imgname}” flavor_name = “${var.m-ovpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = “${var.m-public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)
46. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 48 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = “${var.m-env-prefix}-ovpn" image_name = “${var.m-ovpn-imgname}” flavor_name = “${var.m-ovpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } variable “m-env-prefix" { default = "team1" } variable “m-ovpn-imgname” { default = "centos7-001"} variable “m-ovpn-flavour-name"{ default = "x1.medium" } terraform.tf inputs.tf Variables
47. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 49 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "ovpn" { name = “${var.m-env-prefix}-ovpn" image_name = “${var.m-ovpn-imgname}” flavor_name = “${var.m-ovpn-flavour-name}" floating_ip = “${openstack_compute_floatingip_v2. openvpn.address}" ... } ## OpenVPN Public IP resource "openstack_compute_floatingip_v2" "openvpn" { region = "" pool = “${var.m-public-ip-pool}" ... } terraform.tf Declarative DSL (OpenStack)
48. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 50 Configurable modules are your friend!
49. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 51 # External DNS (AWS Route 53) resource "aws_route53_record" "dns" { zone_id = “${var.m-route-53-domain-id}” name = “${var.m-dns-name}” type = "${var.m-type}" ttl = "${var.m-ttl}" records = [“${var.m-public-ip}"] } core.tf DNS Module
50. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 52 module "mgt" { source = “.../openstack/mgt” m-public-ip-pool = “${var.tf_public_ip_pool}” m-ovpn-flavour-name = “${var.tf_ovpn_flav_name}” m-ovpn-imgname = “${var.tf_ovpn_imgname}” … } module “ext-dns" { source = “.../aws/dns” m-public-ip = “${module.mgt.ovpn-public-ip}” m-dns-name = “${var.tf_ext_dns_name}” … } terraform.tf Modules are your friend!
51. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 53 module "mgt" { source = “.../openstack/mgt” m-public-ip-pool = “${var.tf_public_ip_pool}” m-ovpn-flavour-name = “${var.tf_ovpn_flav_name}” m-ovpn-imgname = “${var.tf_ovpn_imgname}” … } module “ext-dns" { source = “.../aws/dns” m-public-ip = “${module.mgt.ovpn-public-ip}” m-dns-name = “${var.tf_ext_dns_name}” … } terraform.tf Modules are your friend!
52. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 54 module "mgt" { source = “.../openstack/mgt” m-public-ip-pool = “${var.tf_public_ip_pool}” m-ovpn-flavour-name = “${var.tf_ovpn_flav_name}” m-ovpn-imgname = “${var.tf_ovpn_imgname}” … } module “ext-dns" { source = “.../aws/dns” m-public-ip = “${module.mgt.ovpn-public-ip}” m-dns-name = “${var.tf_ext_dns_name}” … } terraform.tf Modules are your friend!
53. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 55 cohesive multi-provider management is often required a lot sooner than you may think!
54. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 56 “Other” Infrastructure Providers
55. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 57 External DNS “Other” Infrastructure Providers
56. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 58 External DNS “Other” Infrastructure Providers
57. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 59 module "mgt" { source = “.../openstack/mgt” m-public-ip-pool = “${var.tf_public_ip_pool}” m-ovpn-flavour-name = “${var.tf_ovpn_flav_name}” m-ovpn-imgname = “${var.tf_ovpn_imgname}” … } module “ext-dns" { source = “.../aws/dns” m-ovpn-public-ip = “${module.mgt.ovpn-public-ip}” m-dns-name = “${var.tf_ext_dns_name}” … } terraform.tf Composed terraform file
58. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 60 • Adhered to all our ASAP principles • Handle multiple infrastructure providers
 • Compose/Generate Terraform from modular definitions • Infrastructure as code IaaS provisioning Summary
59. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 62 ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Configuration management tool Challenge #3 Automated Instance Management
60. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 63 initial bootstrap vs. longer term maintenance
61. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- boot time cloud/VM instance customisation 64 configuration management tool Lesson: conscious de-coupling is your friend —>
62. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 65 —> Approach & Tools: Automated (Bootstrap) Instance Management
63. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadat a service management subnet app subnet public IP
64. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 67 Boot time customisation of cloud instances (VMs)
65. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 68 Hooks into cloud provider’s metadata service cloud provider metadat a service
66. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 69 Accesses user supplied data for VM it is running on cloud provider metadat a service #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.$ {domain}" runcmd: - until curl -ksf https://${env- prefix}-ipa.${domain}:443/ca/admin/ ca/getStatus ; do sleep 30 ; done ; ipa-client-install —domain=${domain} ... --unattended --force-join - export COUNT=0 ; until puppet agent -t ; do echo "`date` - Attempting to run puppet agent for $COUNT time" ; if [[ $COUNT -eq 3 ]] ; then break ; fi ; sleep 30 ; ((COUNT++)) ; done
67. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 70 Example (user-data) cloud config #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}"
68. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 71 #cloud-config hostname: ${env-prefix}-jm fqdn: ${env-prefix}-jm.${domain} manage_etc_hosts: true puppet: conf: agent: server: "${env-prefix}-ipa.${domain}" Example (user-data) cloud config
69. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "km" { name = "${var.env-prefix}-km" image_name = "${var.image_name}" flavor_name = "${var.km-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } 72 Passing user-data via terraform
70. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 73 ## OpenVPN Compute instance resource "openstack_compute_instance_v2" "km" { name = "${var.env-prefix}-km" image_name = "${var.image_name}" flavor_name = "${var.km-flavour-name}" user_data = "${template_file.clientconfig.rendered}" ... } ## UserData as input to cloud-init resource "template_file" "clientconfig" { filename = "${path.module}/clientconfig.template" vars { domain = "${var.domain}" env-prefix = "${var.env-prefix}" ... } } Passing user-data via terraform
71. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet app subnet public IP
72. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet app subnet
73. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service management subnet app subnet
74. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-orch management subnet app subnet
75. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn team1-orch team1-k8sm team1-k8sn1 team1-k8sn2 team1-db team1-elk management subnet app subnet
76. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadata service team1-ovpn team1-orch team1-k8sm team1-k8sn1 team1-k8sn2 team1-db team1-elk management subnet app subnet
77. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 81 Automated Instance Mgt Summary • Decoupled, async bootstrap process 
 • Infrastructure as code • Adhered to all our ASAP principles
 • Tool swap possible: Ansible —> Puppet

78. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ovpn orchestrator k8smaster k8snode01 k8snode02 k8snode03 db01 Core User Solution requirements (Intra-cloud) Supporting Services VPN DNS Configuration management tool 83 Challenge #4 Securing all the things!
79. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 84 Secrets Management • For terraform provisioning
 • For configuration management • For anything needing access to sensitive stuff
 

80. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 85 Lesson: don’t roll your own!
81. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 86 Tool: Secure secrets management
82. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 87 • Unified API to access multiple backends • ACL policies - who can access what • Audit Logs
83. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Anything Else
84. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 90 Part 1: Securing the automated IaaS provisioning process
85. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init
86. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init Configure Global Static Secrets
87. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 93 Vault write, then read back secret $ vault write iaas/cloud-provider-password value=ASDKJ234SF*2 Success! Data written to: iaas/cloud-provider-password
 $ vault read iaas/cloud-provider-password Key Value lease_duration 2592000 value ASDKJ234SF*2
88. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init The brokering framework/ services Configure Global Static Secrets 94
89. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init The brokering framework/ services Configure Global Static Secrets Create new environment 95
90. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init The cloud-env mgmt App The brokering framework/ services Configure Global Static Secrets Create specific mount, policy & add secrets Create new environment 96
91. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init The cloud-env mgmt App The brokering framework/ services Configure Global Static Secrets Spin up environment Create specific mount, policy & add secrets Create new environment 97
92. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Unseal Init Get IaaS creds The brokering framework/ services Configure Global Static Secrets Spin up environment Create new environment Create specific mount, policy & add secrets 98
93. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The brokering framework/ services Unseal Init Get IaaS creds Configure Global Static Secrets Spin up environment Create new environment Create specific mount, policy & add secrets 99
94. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The brokering framework/ services Unseal Init Get IaaS creds Encrypt tfstate (terrahelp) Configure Global Static Secrets Spin up environment Create new environment Create specific mount, policy & add secrets https://github.com/opencredo/terrahelp 100
95. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 101 Benefits • Centralised secure storage solution • Flexible backends - “The right security for the job” 

96. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 102 Part 2: Securing the automated VM bootstrap process —>
97. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider cloud provider metadat a service management subnet app subnet ? 103
98. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App Unseal Init Get IaaS creds Encrypt tfstate (terrahelp) Spin up environment Create new environment Create specific mount, policy & add secrets 104
99. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App Spin up environment Get IaaS creds, /team1 + gitcred1 = x + gitcred2 = z Create specific mount, policy & add secrets Create new environment 105
100. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 106 $ vault mount -path=team1 generic Successfully mounted 'generic' at ‘team1'! $ vault mounts Path Type Default TTL Max TTL Description cubbyhole/ cubbyhole n/a n/a per-token private secr ... secret/ generic system system generic secret storage sys/ system n/a n/a system endpoints used f... team1/ generic system system Vault create new mount
101. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 107 Vault write, then read back secret $ vault write team1/git-password value=ASDKJ234SF*2 Success! Data written to: team1/git-password
 $ vault write team1/postgres-pwd value=S98KDJS#mvs3 Success! Data written to: team1/postgres-pwd
102. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 108 $ cat team1-vm-bootstrap.policy path "team1/*" { policy = "read" } $ vault policy-write team1-vm-bootstrap team1-vm-bootstrap.policy Policy ‘team1-vm-bootstrap' written. Vault create custom policy
103. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App Spin up environment Create new environment /team1 + gitcred1 = x + gitcred2 = z Create specific mount, policy & add secrets Get IaaS creds, generate real token 109
104. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App Get IaaS creds, generate real token & OTP /team1 + gitcred1 = x + gitcred2 = z Spin up environment Create new environment Create specific mount, policy & add secrets 110
105. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App Get IaaS creds, generate real token & OTP TOKEN: USES: 1 REAL TOKEN /cubbyhole /team1 + gitcred1 = x + gitcred2 = z Spin up environment Create new environment Create specific mount, policy & add secrets 111
106. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The cloud-env mgmt App TOKEN: USES: 1 REAL TOKEN /cubbyhole /team1 + gitcred1 = x + gitcred2 = z Get IaaS creds, generate real token & OTP Spin up environment Create new environment Create specific mount, policy & add secrets 112
107. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- TOKEN: USES: 1 REAL TOKEN /cubbyhole cloud provider management subnet dev subnet orch-vm /team1 + gitcred1 = x + gitcred2 = z 113
108. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet dev subnet orch-vm TOKEN: USES: 1 REAL TOKEN /cubbyhole /team1 + gitcred1 = x + gitcred2 = z 114
109. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet dev subnet orch-vm TOKEN: USES: 0 /cubbyhole REAL TOKEN /team1 + gitcred1 = x + gitcred2 = z 115
110. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet dev subnet orch-vm /team1 + gitcred1 = x + gitcred2 = z 116
111. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet dev subnet orch-vm /team1 + gitcred1 = x + gitcred2 = z 117
112. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 118 $ cat team1-vm-bootstrap.policy path "team1/*" { policy = "read" } Real token - policy restricted
113. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /team1 + gitcred1 = x + gitcred2 = z cloud provider management subnet dev subnet orch-vm 119
114. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /team1 + gitcred1 = x + gitcred2 = z cloud provider management subnet dev subnet orch-vm 120
115. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- cloud provider management subnet dev subnet orch-vm /team1 + gitcred1 = x + gitcred2 = z 121
116. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- /team1 + secret1 = x + secret2 = z cloud provider management subnet dev subnet orch-vm https://github.com/jsok/hiera-vault 122
117. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 123 Secrets Management Summary • Vault embodies our ASAP principles • Centralised secure storage solution • Flexible backends - “The right security for the job”
 • Granular access control 

118. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 125 Conclusion
119. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 126 How to create fast, repeatable, secure environments capable of running in different clouds!
120. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 127 • Developers can create environment in minutes
 • Addressed concerns moving towards cloud • Start leverage promise of cloud • Right cloud for the job 

121. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 128 • Combined “the right tools for the job”
 • Flexible and adaptive moving forward
122. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 129 “The only thing constant in life is change.” — François de La Rochefoucauld

123. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 130 Be true to your principles, but flex your tools (and approach) as required

124. ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 131 Thanks Questions? @techiewatt