CloudFormation For Fun & Profit (But Mostly Sanity)

The Platonic ideal of ops (and computing tasks generally) is to be able to describe an expected state, and have the computer figure out the details. Tools like Chef, Puppet, and Ansible get us closer to this state, but often our work ends up being little more than abstractions over commands to manually provision infrastructure.

Enter AWS CloudFormation. CloudFormation allows us to provision & maintain complete application infrastructures from a single template file, and solves many problems for us along the way, including dependency resolution, data passing, and sharing or eliminating secrets. Come learn about these techniques, as well as the many ways to author & maintain CloudFormation stacks.

Transcript

1. CLOUDFORMATION PHILLY DEVOPS MEETUP 2017-01-17
2. CLOUDFORMATION FOR FUN& PROFIT BUT MOSTLY SANITY
3. YOUR HOST: NATE ABELE ▸Some PHP frameworks (Li3, CakePHP) ▸AngularUI Router ▸Architect @ Radify ▸nate@radify.io ▸@nateabele
5. GROWING AN APP
15. - name: Create launch config for worker nodes ec2_lc: name: "lc_{{ app_environment }}_workers_{{ ami }}_{{ stamp }}" image_id: "{{ ami }}" instance_monitoring: 'yes' instance_profile_name: "cloudwatch-write" key_name: "{{ key_name }}" region: "{{ asg_region }}" security_groups: "{{ asg_ec2_instance_sg }}" instance_type: "{{ asg_instance_type }}" assign_public_ip: yes volumes: - device_name: /dev/sda1 volume_size: 8 delete_on_termination: true service remote_syslog restart # Inject env vars into workers perl -pi -e "s/{{ sqs_queue_notifications }}/g" /etc/init/app-worker-noti… perl -pi -e "s/{{ sqs_queue_emails }}/g" /etc/init/app-worker-emails.conf perl -pi -e "s/{{ sqs_queue_kpis }}/g" /etc/init/app-worker-gather-kpis… if [[ "{{ app_environment }}" == "production" ]] then update-rc.d datadog-agent enable service datadog-agent start fi if [[ "{{ app_environment }}" == "demo" ]] then # remove the gather KPI worker on demo rm -f /etc/init/app-worker-gather-kpis.conf else # Symlink newrelic PHP extension and PHP config ln -s /usr/lib/newrelic-php5/agent/x64/newrelic-20131226.so … ln -s /etc/php/5.6/mods-available/newrelic.ini /etc/php/5.6/… ln -s /etc/php/5.6/mods-available/newrelic.ini /etc/php/5.6/… # Start all monitoring services service newrelic-sysmond restart fi # Kill FPM and Nginx service nginx stop service php5.6-fpm stop # Restart services /usr/local/bin/restart_services tags: launch_config_workers - name: Upsert autoscale group for workers [in foreground] ec2_asg: name: "{{ app_environment }}-workers-group" launch_config_name: "lc_{{ app_environment }}_workers_{{ ami }}_{{ stamp }}" replace_all_instances: yes region: "{{ asg_region }}" health_check_period: 300 health_check_type: EC2 wait_timeout: 600 state: present default_cooldown: 150 vpc_zone_identifier: "{{ asg_subnets }}" availability_zones: "{{ asg_availability_zones }}" min_size: "{{ asg_min_size_workers }}" max_size: "{{ asg_max_size_workers }}" desired_capacity: "{{ asg_desired_capacity_workers }}" tags: - Environment: "{{ app_environment }}" - Class: "worker" - Type: "autoscaling" - Name: "{{ app_environment }}-worker-autoscaling" user_data: | #!/bin/bash # Install env vars to /etc/environment cat > /etc/environment <<EOF PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/bin" export app_domain="{{ app_domain }}" export app_environment="{{ app_environment }}" export app_mongo_server="{{ app_mongo_server }}" export app_mongo_database="{{ app_mongo_database }}" export app_mongo_login="{{ app_mongo_login }}" export app_mongo_password="{{ app_mongo_password }}" export app_mongo_timeout={{ app_mongo_timeout }} export app_mongo_replica_set={{ app_mongo_replica_set }} export app_translations_url="{{ app_translations_url }}" export app_elasticsearch_scheme="{{ app_elasticsearch_scheme }}" export app_elasticsearch_host="{{ app_elasticsearch_host }}" export app_elasticsearch_port="{{ app_elasticsearch_port }}" export app_elasticsearch_index="{{ app_elasticsearch_index }}" export app_elasticsearch_timeout="{{ app_elasticsearch_timeout }}" export app_elasticsearch_username="{{ app_elasticsearch_username }}" export app_elasticsearch_password="{{ app_elasticsearch_password }}" export aspera_root="{{ aspera_root }}" export aspera_username="{{ aspera_username }}" export smtp_username="{{ smtp_username }}" export smtp_password="{{ smtp_password }}" export smtp_port="{{ smtp_port }}" export smtp_host="{{ smtp_host }}" export app_monitor_api="{{ app_monitor_api }}" export app_monitor_protocol="{{ app_monitor_protocol }}" export app_monitor_username="{{ app_monitor_username }}" export app_monitor_password="{{ app_monitor_password }}" export xhprof_domain="{{ xhprof_domain }}" export cloudfront_domain="{{ cloudfront_domain }}" export cloudfront_key="{{ cloudfront_key }}" export cloudfront_pem="{{ cloudfront_pem }}" export cloudfront_pem_dir="{{ cloudfront_pem_dir }}" export cloudfront_distribution="{{ cloudfront_distribution }}" export sqs_region="{{ sqs_region }}" export sqs_queue_notifications="{{ sqs_queue_notifications }}" export sqs_queue_emails="{{ sqs_queue_emails }}" export sqs_queue_feedback="{{ sqs_queue_feedback }}" export sqs_disable_ssl="{{ sqs_disable_ssl }}" export aws_key="{{ aws_key }}" export aws_secret="{{ aws_secret }}" export redis_host="{{ redis_host }}" EOF # Set papertrail hostname /etc/log_files.yml cat > /etc/log_files.yml <<EOF --- files: - /var/log/php-fpm.log - /var/log/php-slow.log - /var/log/nginx/*.log - /var/log/syslog hostname: "`hostname`_{{ app_environment }}" destination: host: logs.papertrailapp.com port: 37801 protocol: tls EOF # Restart papertrail logging service pm.status_path = /status chdir = / env[app_faye_endpoint] = "{{ app_faye_endpoint }}" env[app_domain] = "{{ app_domain }}" env[app_environment] = "{{ app_environment }}" env[app_mongo_server] = "{{ app_mongo_server }}" env[app_mongo_database] = "{{ app_mongo_database }}" env[app_mongo_login] = "{{ app_mongo_login }}" env[app_mongo_password] = "{{ app_mongo_password }}" env[app_mongo_timeout] = {{ app_mongo_timeout }} env[app_mongo_replica_set] = "{{ app_mongo_replica_set }}" env[app_translations_url] = "{{ app_translations_url }}" env[app_elasticsearch_scheme] = "{{ app_elasticsearch_scheme env[app_elasticsearch_host] = "{{ app_elasticsearch_host }}" env[app_elasticsearch_port] = "{{ app_elasticsearch_port }}" env[app_elasticsearch_timeout] = "{{ app_elasticsearch_timeou env[app_elasticsearch_index] = "{{ app_elasticsearch_index }} env[app_elasticsearch_username] = "{{ app_elasticsearch_usern env[app_elasticsearch_password] = "{{ app_elasticsearch_passw env[smtp_username] = "{{ smtp_username }}" env[smtp_password] = "{{ smtp_password }}" env[smtp_port] = "{{ smtp_port }}" env[smtp_host] = "{{ smtp_host }}" env[xhprof_domain] = "{{ xhprof_domain }}" env[app_s3_endpoint] = "{{ app_s3_endpoint }}" env[cloudfront_domain] = "{{ cloudfront_domain }}" env[cloudfront_key] = "{{ cloudfront_key }}" env[cloudfront_pem] = "{{ cloudfront_pem }}" env[cloudfront_pem_dir] = "{{ cloudfront_pem_dir }}" env[cloudfront_distribution] = "{{ cloudfront_distribution }} env[sqs_region] = "{{ sqs_region }}" env[sqs_queue_notifications] = "{{ sqs_queue_notifications }} env[sqs_queue_emails] = "{{ sqs_queue_emails }}" env[sqs_queue_feedback] = "{{ sqs_queue_feedback }}" env[sqs_disable_ssl] = "{{ sqs_disable_ssl }}" env[bucket_attachments] = "{{ bucket_attachments }}" env[bucket_previews] = "{{ bucket_previews }}" env[aws_key] = "{{ aws_key }}" env[aws_secret] = "{{ aws_secret }}" env[redis_host] = "{{ redis_host }}" EOF # Configure /opt/app/app-ui/build/release/js/config.js with e perl -pi -e "s/^.*MESSAGE_ENDPOINT.*$/\t.constant('MESSAGE_EN perl -pi -e "s/^.*S3_ENDPOINT.*$/\t.constant('S3_ENDPOINT', '{ perl -pi -e "s/^.*JWPLAYER_KEY.*$/\t.constant('JWPLAYER_KEY', perl -pi -e "s/^.*IntercomAPI.*$/\t.constant('IntercomAPI', ' perl -pi -e "s/^.*appname.*$/newrelic.appname = \"app API {{ # Set papertrail hostname /etc/log_files.yml cat > /etc/log_files.yml <<EOF files: - /var/log/php-fpm.log - /var/log/php-slow.log - /var/log/nginx/*.log - /var/log/syslog hostname: "`hostname`_{{ app_environment }}" destination: host: logs.papertrailapp.com port: 37801 protocol: tls EOF # Restart papertrail logging service service remote_syslog restart
16. PR-BASED DEMO ENVIRONMENTS
19. “The Oven”
20. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates { "AWSTemplateFormatVersion": "2010-09-09", "Description": "...", "Parameters": { ... }, "Resources": { ... }, "Outputs": { ... } }
21. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks aws:cloudformation:logical-id: WebServerFleet aws:cloudformation:stack-id: arn:aws:cloudformation:us-east-1:022261289334:stack/ prod/458dc900-d0a5-11e6-b7e8-50d501eed2b3 aws:cloudformation:stack-name: prod
22. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks ▸ Parameters "SSHLocation": { "Description": "Lockdown SSH access", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\ \d{1,3})\\.(\\d{1,3})/(\\d{1,2})", "ConstraintDescription": "must be a valid CIDR range of the form x.x.x.x/x." }
23. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks ▸ Parameters "KeyName": { "Description": "Name of an existing EC2 KeyPair", "Type": "AWS::EC2::KeyPair::KeyName", "ConstraintDescription": "must be the name of an existing EC2 KeyPair." }
24. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks ▸ Parameters ▸ Resources ▸ Refs "VPC": { "Type": "AWS::EC2::VPC", "Properties": { ... } }, "DatabaseSecurityGroup": { "Type": "AWS::EC2::SecurityGroup", "Properties": { "VpcId": { "Ref": "VPC" } } }
25. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks ▸ Parameters ▸ Resources ▸ Refs ▸ Functions "DBName": { "Fn::Join": [ "", [ “appname", { "Ref": "AWS::StackName" } ]] }
26. PHILLY DEVOPS SO, HOW DOES ONE CLOUDFORMATION? ▸ Templates ▸ Stacks ▸ Parameters ▸ Resources ▸ Refs ▸ Functions ▸ Outputs "URL": { "Description": "URL of new stack", "Value": { "Fn::Join": [ "", [ "http://", { "Fn::GetAtt": [ "PublicElasticLoadBalancer", "DNSName" ] ]] } }
27. USER DATA, SIGNALS, & SECRETS
28. "UserData": { "Fn::Base64": { "Fn::Join": ["", [ "#!/bin/bash -xe\n", "/opt/aws/bin/cfn-init --stack ", {"Ref": "AWS::StackId"}, " --resource WebServerFleet ", " --region ", { "Ref": "AWS::Region" }, "\n\n", " db_username=", { "Ref": "DBUserName" }, " db_password=", { "Ref": "DBPassword" }, “; export APP_ENV=production", "\n\n" ...Other bootstrappy stuff "# Signal completion\n", "/opt/aws/bin/cfn-signal -e $? ", " --stack ", { "Ref": "AWS::StackId" }, " --resource WebServerFleet ", " --region ", { "Ref": "AWS::Region" } ]]}}
29. LAYERS; FOR ONIONS, FOR OGRES, FOR STACKS
30. PHILLY DEVOPS QUANTIFYING THE WIN ▸ Declarative: No process logic ▸ Fully self-contained: no one-off or bespoke resources ▸ Easy name-spacing for global objects ▸ Eliminates / isolates secrets
31. PHILLY DEVOPS TRY THIS AT HOME
32. PHILLY DEVOPS TRY THIS AT HOME
33. PHILLY DEVOPS TRY THIS AT HOME ▸ CloudFormation Designer ▸ Do it by hand! ▸ Bootstrap via AWS Console + CloudFormer
34. PHILLY DEVOPS TRY THIS AT HOME ▸ CloudFormation Designer ▸ Do it by hand! ▸ Bootstrap via AWS Console + CloudFormer
35. PHILLY DEVOPS TRY THIS AT HOME ▸ CloudFormation Designer ▸ Do it by hand! ▸ Bootstrap via AWS Console + CloudFormer ▸ VisualOps
36. THANKS
37. QUESTIONS?
38. NOTES The Platonic ideal of ops (and computing tasks generally) is to be able to describe an expected state, and have the computer figure out the details. Tools like Chef, Puppet, and Ansible get us closer to this state, but often our work ends up being little more than abstractions over commands to manually provision infrastructure. Enter AWS CloudFormation. CloudFormation allows us to provision & maintain complete application infrastructures from a single template file, and solves many problems for us along the way, including dependency resolution, data passing, and sharing or eliminating secrets. Come learn about these techniques, as well as the many ways to author & maintain CloudFormation stacks. - Brief intro to AWS - Progression of stack complexity: EC2 instance -> EC2 + S3 -> + DB -> + ELB -> + ASG -> + VPC - Staging environment? Good luck refactoring all your playbooks / recipes, sucker! -- More importantly, unless you're really experienced, the definition of your infrastructure is now tightly coupled to your playbooks, and probably to some extent to your build process - Example: demo nodes (flowchart) - Parameters - App / stack name (& naming things like S3 buckets) - Roles (vs. secrets) - Digression into user data - Building and maintaining - Do it by hand! Great way to learn and understand the inner workings - Bootstrap via AWS Console + CloudFormer - AWS CloudFormation Designer (It's terrible) - VisualOps