Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWS CloudFormation

ProdOps
October 16, 2018

AWS CloudFormation

Amazing and powerful abilities of AWS CloudFormation that mostly pass unnoticed. How to decouple stacks using exported values, how to utilize `cfn-init` and update code on existing servers, how to create completely custom resources for CloudFormation to manage and more!

This was presented at the Devops in Israel meetup event https://www.meetup.com/devops-in-israel/events/254851565/

ProdOps

October 16, 2018
Tweet

More Decks by ProdOps

Other Decks in Technology

Transcript

  1. { "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "EC2 Instance and a

    Security Group", "Parameters" : { "KeyName": { "Description" : "An existing EC2 KeyPair to enable SSH access", "Type": "AWS::EC2::KeyPair::KeyName" }, "InstanceType" : { "Description" : "EC2 instance type", "Type" : "String", "Default" : "t2.small", "AllowedValues" : [ "t2.small", "t2.medium", "t2.large"] }, "SSHLocation" : { "Description" : "CIDR from where SSH access is allowed", "Type": "String", "MinLength": "9", "MaxLength": "18", "Default": "0.0.0.0/0", "AllowedPattern": "(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})\\.(\\d{1,3})/(\\d{1,2})" } ...
  2. "Resources" : { "EC2Instance" : { "Type" : "AWS::EC2::Instance", "Properties"

    : { "InstanceType" : { "Ref" : "InstanceType" }, "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], "KeyName" : { "Ref" : "KeyName" }, "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] } } }, "InstanceSecurityGroup" : { "Type" : "AWS::EC2::SecurityGroup", "Properties" : { "GroupDescription" : "Enable SSH access via port 22", "SecurityGroupIngress" : [ { "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : { "Ref" : "SSHLocation"} } ] } }
  3. /prəˈviZHən/ supply with equipment the IT manager was responsible to

    provision servers for the production release pro·vi·sion
  4. /ˈôdəˌmāt/ convert a process to reduce human intervention to a

    minimum jobs that used to be done by people are now able to be done through automation au·to·mate
  5. /prəˈdiktəb(ə)l/ behaving or occurring in a way that is expected.

    unsurprising. Murphy's unpredictable rule predictably hits you pre·dict·a·ble
  6. Create a VPC aws --region us-east-1 \ cloudformation create-stack \

    --stack-name VPC \ --template-url https://aws-quickstart.s3.amazonaws.com/quickstart-aws-vpc/templates/aws-vpc.template \ --parameters \ ParameterKey=KeyPairName,ParameterValue=evgeny-ssh \ ParameterKey=AvailabilityZones,ParameterValue="'us-east-1a,us-east-1b'" { "StackId": "arn:aws:cloudformation:us-east-1:675783095585:stack/VPC/76271620-d0f8-11e8-b3a0-500c28b04cd1" } aws.amazon.com/quickstart/architecture/vpc
  7. Using Nested Stacks Resources: NestedStack: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Ref

    NestedStackS3File InstanceType: !Ref InstanceType Outputs: NestedStackBucketName: Value: !GetAtt NestedStack.InstanceName
  8. Never, ever, should you actually use Nested Stacks because Nested

    Stacks are Stuck in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS, UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS, or UPDATE_ROLLBACK_IN_PROGRESS A nested stack failed to roll back. Because of potential resource dependencies between nested stacks, AWS CloudFormation doesn't start cleaning up nested stack resources until all nested stacks have been updated or have rolled back. When a nested stack fails to roll back, AWS CloudFormation cancels all operations, regardless of the state that the other nested stacks are in. A nested stack that completed updating or rolling back but did not receive a signal from AWS CloudFormation to start cleaning up because another nested failed to roll back is in an UPDATE_COMPLETE_CLEANUP_IN_PROGRESS or UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state. A nested stack that failed to update but did not receive a signal to start rolling back is in an UPDATE_ROLLBACK_IN_PROGRESS state. A nested stack might fail to roll back because of changes that were made outside of AWS CloudFormation, when the stack template doesn't accurately reflect the state of the stack. A nested stack might also fail if an Auto Scaling group in a nested stack had an insufficient resource signal timeout period when the group was created or updated. To fix the stack, contact AWS customer support. WTF? docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html
  9. Stack B: Resources: EC2Instance: Type: AWS::EC2::Instance Properties: NetworkInterfaces: - AssociatePublicIpAddress:

    true DeviceIndex: 0 GroupSet: - Fn::ImportValue: Global-SecurityGroup Stack A: Outputs: SecurityGroup: Value: !Ref SecurityGroup Description: ID of security group Export: Name: Global-SecurityGroup Exported Output and Imported Value
  10. Custom Resources - Definition and usage Resources: CustomCWEventsLambda: Type: AWS::Lambda::Function

    Properties: Runtime: nodejs8.10 Handler: index.handler Role: !GetAtt LambdaExecutionRole.Arn Code: ZipFile: > const Response = require('cfn-response'); exports.handler = function handler(event, context) { Response.send(event, context, Response.FAILED, {}); throw new Error('Not Implemented'); } CloudWatchEventsRule: Type: Custom::Events::Rule Parameters: ServiceToken: !GetAtt CustomCWEventsLambda.Arn Name: 'STRING_VALUE' - A name for this rule. *optional* Description: 'STRING_VALUE' - A description of the rule. State: 'ENABLED' - Indicates whether the rule is enabled or disabled. from github.com/devops-israel/aws-cf-custom-events
  11. Custom Resources - Implementation const Response = require('cfn-response'); const log

    = { info(...args) { console.log(...args) }, error(...args) { console.error(...args) }, } exports.handler = function handler(event, context) { log.info(event, context); switch (event.RequestType) { case 'Create': return createResource(event, context); case 'Delete': return deleteResource(event, context); case 'Update': return updateResource(event, context); default: log.error(new Error(`ERROR: Unknown event.RequestType provided: ${event.RequestType}`)); Response.send(event, context, Response.FAILED, {}); } } from github.com/devops-israel/aws-cf-custom-events
  12. Resources: AutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: LaunchConfigurationName: !Ref LaunchConfiguration CreationPolicy: ResourceSignal:

    Timeout: PT15M LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: UserData: Fn::Base64: !Sub | #cloud-config packages: - aws-cfn-bootstrap bootcmd: - "while ! ping -i 10 -I eth0 -n -w 60 -W 2 -c 1 -q s3.amazonaws.com; do sleep 1; done" runcmd: - /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource LaunchConfiguration - /opt/aws/bin/cfn-signal -e $? --region ${AWS::Region} --stack ${AWS::StackName} --resource AutoScalingGrou Updating EC2 Instances in an Auto Scaling Group
  13. LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: ... Metadata: AWS::CloudFormation::Init: config: commands: ecs_task_sysctl1:

    command: "sysctl -w net.ipv4.conf.all.route_localnet=1" ecs_task_sysctl2: command: "sysctl -w net.ipv4.ip_local_port_range='1024 65000'" ecs_task_roles1: command: "iptables -t nat -A PREROUTING -p tcp -d 169.254.170.2 --dport 80 -j DNAT --to-destination 127.0.0.1:51679 ecs_task_roles2: command: "iptables -t nat -A OUTPUT -d 169.254.170.2 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 51679" save_iptables: command: "service iptables save" yum: ecs-banner: [] ecs-init: [] Updating EC2 Instances in an Auto Scaling Group
  14. LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: ... Metadata: AWS::CloudFormation::Init: config: ... files:

    "/etc/sysctl.d/10-route_localnet.conf": mode: "0444" owner: root group: root content: "net.ipv4.conf.all.route_localnet = 1" "/etc/sysctl.d/10-more_ports.conf": mode: "0444" owner: root group: root content: "net.ipv4.ip_local_port_range = 1024 65000" Updating EC2 Instances in an Auto Scaling Group
  15. "/etc/ecs/ecs.config": mode: 000644 owner: root group: root # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html #

    and https://github.com/aws/amazon-ecs-agent#environment-variables content: !Sub | # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html ECS_CLUSTER=${Cluster} ECS_ENGINE_AUTH_TYPE=${ECSEngineAuthType} ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs","syslog","journald","gelf","fluentd","splunk"] ECS_ENABLE_TASK_IAM_ROLE=true ECS_DISABLE_PRIVILEGED=true ECS_INSTANCE_ATTRIBUTES={"env":"${EnvironmentName}"} ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m # http://docs.aws.amazon.com/AmazonECS/latest/developerguide/automated_image_cleanup.html ECS_IMAGE_CLEANUP_INTERVAL=10m ECS_IMAGE_MINIMUM_CLEANUP_AGE=15m ECS_NUM_IMAGES_DELETE_PER_CYCLE=10 AWS_DEFAULT_REGION=${AWS::Region} Updating EC2 Instances in an Auto Scaling Group
  16. "/etc/cfn/cfn-hup.conf": mode: 000400 owner: root group: root content: !Sub |

    [main] stack=${AWS::StackId} region=${AWS::Region} "/etc/cfn/hooks.d/cfn-auto-reloader.conf": content: !Sub | [cfn-auto-reloader-hook] triggers=post.update path=Resources.LaunchConfiguration.Metadata.AWS::CloudFormation::Init action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource LaunchCon Updating EC2 Instances in an Auto Scaling Group The cfn-hup helper is a daemon that detects changes in resource metadata and runs user-specified actions when a change is detected. This allows you to make configuration updates on your running Amazon EC2 instances through the UpdateStack API action.
  17. Updating EC2 Instances in an Auto Scaling Group services: upstart:

    ecs: enabled: true ensureRunning: true files: - /etc/init/ecs.conf - /etc/ecs/ecs.config sysvinit: cfn-hup: enabled: true ensureRunning: true files: - /etc/cfn/cfn-hup.conf - /etc/cfn/hooks.d/cfn-auto-reloader.conf
  18. We invite you to join Operations Israel Facebook group on

    on.fb.me/Ops-IL Thank you! www.prodops.io