Terraform at Honestbee

https://www.meetup.com/devops-singapore/events/248013807/

Tips after using Terraform to provision k8s clusters, manage service dependencies and provide developer workflow

Transcript

1. at
2. Agenda 1. Who am I? 2. What is Honestbee? 3. Scaling Honestbee 4. Adoption of Terraform at Honestbee 5. Reproduce Environments using Workspaces 6. Empower engineers using Modules 7. Adopt Git-Ops with Atlantis
3. Who am I? Vincent De Smet DevOps Engineer @ Honestbee Singapore Co-organiser for Kubernetes, Cloud Native and GCP Singapore user groups in Singapore vincentdesmet so0k
4. What is Honestbee? Groceries, Food, Medical, Movie / Attraction Tickets, ...
5. What is Honestbee? … many more services
6. Scaling Honestbee
7. Initial Honestbee Set Up Born in the cloud (AWS)
8. Initial Honestbee Set Up - Small team, move fast - Monolithic code base Ruby on Rails - Platform as a Service AWS Elastic Beanstalk
9. Scaling your company in the cloud - Grow engineering headcount Monolithic code base - Stay agile - move towards - Loosely Coupled Services - Frequent / incremental updates - Elastic cloud resources - but… keep cloud resources and costs under control Reference: http://docker.com - Agile evolution
10. Scaling your company in the cloud How to adopt loosely coupled services … Immutable deployments, maximise resource utilisation, resilient architecture, ...
11. Scaling your company in the cloud Kubernetes on AWS in Singapore …. Not GA (yet)
12. Adoption of Terraform at Honestbee
13. Adopting Terraform Kubernetes (single-az) on AWS: https://engineers.sg/v/2225 https://speakerdeck.com/so0k/singapore-kubernetes-meetup-cluster-bootstrap
14. Adopting Terraform Hosted services on AWS:
15. Adopting Terraform High Availability:
16. Adopting Terraform Multiple / Repeatable environments: staging production
17. Adopting Terraform now iterate on and maintain this infrastructure … staging production
18. Adopting Terraform … and empower engineers to launch additional databases / caches / … staging production
19. Reproduce environments using Workspaces
20. Terraform config defines cloud resources (for example): Terraform Workspaces resource "aws_rds_cluster" "mysvc" { cluster_identifier = "mysvc-staging-db" ... } resource "aws_s3_bucket" "mysvc" { bucket = "mysvc-staging-bucket" acl = "private" ... } resource "aws_elasticache_cluster" "mysvc" { cluster_id = "mysvc-staging-cache" ... }
21. Terraform apply instantiates environments with state tracking changes: default Terraform Workspaces resource "aws_rds_cluster" "mysvc" { cluster_identifier = "mysvc-staging-db" ... } resource "aws_s3_bucket" "mysvc" { bucket = "mysvc-staging-bucket" acl = "private" ... } resource "aws_elasticache_cluster" "mysvc" { cluster_id = "mysvc-staging-cache" ... } state
22. Terraform workspaces allow us to instantiate multiple environments, keeping separate state: Terraform Workspaces resource "aws_rds_cluster" "mysvc" { cluster_identifier = "mysvc- ${terraform.workspace}-db" ... } resource "aws_s3_bucket" "mysvc" { bucket = "mysvc- ${terraform.workspace}-bucket" acl = "private" ... } resource "aws_elasticache_cluster" "mysvc" { cluster_id = "mysvc- ${terraform.workspace}-cache" ... } staging state prod state
23. Empower engineers using Modules
24. Modules Portable Terraform configurations (packages) Allow separation of concerns and responsibilities among teams Why? - Provide off-the-shelf functionality for Engineers - Enforce best practices and conventions for cloud infrastructure
25. Modules Spoiler Alert! Modules are just Terraform configurations inside a folder - there's nothing special about them.
26. $ tree my-module my-module └── main.tf Terminal
27. variable "ami_id" {} variable "instance_type" {} resource "aws_instance" "db" { # ... } output "address" { value = "${aws_instance.db.private_dns}" } my-instance-module/main.tf
28. module "special_instance" { source = "./my-instance-module" ami_id = "value" instance_type = "value" } main.tf
29. $ terraform init Downloading modules... Get: file:///home/training/terraform-workshop/example-module Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.2.0)... - Downloading plugin for provider "null" (1.0.0)... ... Terminal
30. Modules: Source Terraform supports module import from: • Local file paths • Git (GitHub / BitBucket / Generic Git ... ) • HTTP URLs • S3 buckets • Terraform Registry
31. Module Source: Git Using git (ssh): source = "git@github.com:hashicorp/example.git//subdir" GitHub source URLs require that Git is installed on your system and that you have access to the repository
32. Module Source: Git Initial adoption: source = "git@github.com:hashicorp/example.git//subdir" Pros: - Get started using shared modules fast (fetch directly from git) Cons: - workflow becomes slow (full mono-repo is cloned per module import) - versioning requires the use of tags and careful release management
33. Module Source: HTTP Fetching modules over http: source = "https://modules.mycompany.com/module.tar.gz" Terraform will download and uncompress archives automatically
34. Module Source: HTTP Fetching modules over http: source = "https://modules.mycompany.com/module.tar.gz" Pros: - Modules are fetched much faster and take up less space Cons: - Requires proper CI/CD set up (linting / builds / file hosting) - Requires private network & VPN set up …
35. Module Source: HTTP Example CI / CD set up - using Drone.io & https://github.com/mbtproject/mbt
36. Module Source: HTTP Serving from s3 using jessfraz/s3server - https://github.com/honestbee/s3server
37. Module Source: GIT Fetching modules over HTTP vs GIT 60s
38. Module Source: HTTP Fetching modules over HTTP vs GIT 20s
39. Writing Modules: Balancing - big / generic modules VS small single purpose modules Note: HCL Conditionals are a HACK! this module provisions: - postgres instance - s3 bucket - iam policy - dns entries - ... postgres instance s3 bucket iam policy simple dns
40. Writing Modules: Balancing - individual variables VS config "maps" Note: HCL map support can be icky! variable "db_host" { } variable "db_user" { } variable "db_password" { } db_host = "" db_user = "" db_password = "" variable "db" { type = "map" } db = { host = "", user = "", password = "", }
41. Writing Modules: Balancing - Beware of lists / count(list) Note: Cascading state changes
42. Adopt Git-Ops with Atlantis
43. Atlantis Open source project originally out of Hootsuite https://www.runatlantis.io/ Enable engineers to run terraform directly from pull requests Why? - Better collaboration and visibility - Review and Approval Workflow - Engineers don't need full AWS Access
44. Demo Add cross cluster load balancing to http://apse1a-devops-sg-demo.honestbee.com http://apse1b-devops-sg-demo.honestbee.com
45. Demo We want http://devops-sg-demo.honestbee.com to use: - Weighted Route53 Records across apse1a and apse1b AZs - Route53 HealthChecks for automated failover on AZ failure https://github.com/honestbee/devops-singapore-example/pull/4/files
46. Conclusion Today: Terraform configurations essential for hosted services not managed by Kubernetes Long term: migrate towards control loops and declarative resource manifests: See https://kubernetes.io/docs/concepts/service-catalog/ Terraform will always remain as a way to define our core cloud infrastructure.
47. We are hiring … https://careers.honestbee.com/departments/engineering/
48. Thank you