Upgrade to Pro — share decks privately, control downloads, hide ads and more …

And you thought you knew EC2, GOT edition

Ben Whaley
November 28, 2017

And you thought you knew EC2, GOT edition

Presented at AWS re:Invent 2017

Ben Whaley

November 28, 2017
Tweet

More Decks by Ben Whaley

Other Decks in Technology

Transcript

  1. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. And you thought you knew EC2… Ben Whaley D i r e c t o r , S e c u r i t y a n d O p e r a t i o n s K o u n t a b l e
  2. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Pertinent announcements since June 2017 Docker Device and Init Flags in Container Task Definitions Amazon ECS Allows Containers to Directly Access Environmental Metadata Announcing New AWS Deep Learning AMI for Amazon EC2 P3 Instances
 Amazon EC2 Systems Manager Parameter Store Adds Versioning Support Amazon EC2 Systems Manager Now Integrates With GitHub Application Load Balancers Now Support Multiple TLS Certificates With Smart Selection Using SNI Introducing Amazon EC2 P3 Instances Introducing Lifecycle Policies for Amazon EC2 Container Registry Application Load Balancers now support multiple SSL certificates EC2 Per second billing ECS Adds Support for Adding or Dropping Linux Capabilities to Containers Network Load Balancer now supports load balancing to IP addresses as targets Amazon EC2 Spot Can Now Stop and Start Your Spot Instances Amazon EC2 Systems Manager Run Command Adds Tag-Based Permissions and Multi- Tag Support Auto Scaling Lifecycle Hooks Enhancements A new addition to the Amazon EC2 memory-optimized X1 Instance family – x1e.32xlarge Amazon EC2 Container Service Now Integrated with Network Load Balancer Application Load Balancer Adds Support for New RequestCountPerTarget CloudWatch Metric EC2 Systems Manager Now Supports Linux Patching Sync Amazon EC2 Systems Manager Inventory Data to Amazon S3 Buckets ECS RunTask and StartTask APIs now support additional override parameters EC2 Systems Manager Adds Hierarchy, Tagging, and Notification Support for Parameter Store Announcing Network Load Balancer for Elastic Load Balancing Announcing improved networking performance for Amazon EC2 instances Application Load Balancer now supports load balancing to IP addresses as targets Amazon EC2 Systems Manager Adds Configuration Compliance Reporting and Auto- Remediation Amazon ECS is now HIPAA Eligible Amazon EC2 Systems Manager now HIPAA eligible Tag Your Spot Fleet EC2 Instances Introducing Amazon EC2 G3 Instances, the next-generation of GPU-powered instances for graphics-intensive applications Support for LCU metrics on Classic Load Balancer Amazon EC2 Systems Manager Adds Cross-Platform and Multi-Step Document Support Amazon EC2 Systems Manager Adds Raspbian OS and Raspberry Pi Support Introducing Target Tracking Scaling Policies for Auto Scaling
  3. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. C5 Compute-optimized with 3.0GHz Intel Skylake P3 Next-gen GPU instances suitable for ML, HPC R4 Memory optimized (up to 488GiB) I3 High I/O with NVMe SSD 10k-300k IOPS X1e In-memory databases - up to 3.8TiB, 128vCPU D2 Up to 43TiB HDD EC2 Instance Types
  4. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. EBS Volume Types Type Description I/O Throughput Cost io1 Provisioned IOPS SSD Highest High Highest gp2 General purpose SSD High Low(ish) Medium st1 Throughput-optimized HDD Low Highest Low sc1 Non-optimized HDD Low Medium Lowest
  5. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Miscellaneous changes, features ▪ IPv6 ▪ KVM hypervisor for new instance types ▪ Elastic Network Adapter - 25 Gbps ▪ Per second billing ▪ Elastic GPUs ▪ Target Tracking Scaling Policies for Auto Scaling ▪ New regions 2016 - Ohio, Canada, London, Mumbia, Seoul Soon - China (Ningxia), Paris, Stockholm, Hong Kong, Bahrain
 #awswishlist - Kenya
  6. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. CloudWatch Integration Scheduled tasks Event handling Metrics Logs
  7. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. CloudWatch Events ▪ Scheduled Events ▪ Instance state changes ▪ Systems manager ▪ Maintenance window ▪ ECS event stream ▪ EBS snapshots, encryption/decryption ▪ Cross-account event delivery
  8. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. CloudWatch Telemetry ▪ High resolution (1s/3H) custom metrics ▪ High resolution alarms (10s) ▪ collectd CloudWatch plugin ▪ Dashboards (GUI, CLI) ▪ Percentile statistics (p50, p90, p99…) ▪ Logs ▪ Metric filters for log parsing and alarms
  9. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. EC2 Container Service ▪ Elastic Container Registry ▪ Docker Device and Init Flags in Container Task Definitions ▪ CloudWatch metrics for CPU and memory utilization across the cluster ▪ IAM roles for ECS tasks ▪ github.com/blox/blox marching towards v1.0 ▪ 3rd party tooling (Convox, Empire) ▪ Integration with Application Load Balancer ▪ Run tasks on a schedule ▪ Execute tasks in response to CloudWatch events
  10. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. EC2 Systems Manager Superpowers for EC2 instances and on-premises systems. ▪ Remote command execution with Run Commands ▪ Controlled secrets and configuration data with the Parameter Store ▪ Periodic tasks with the State Manager and Maintenance Windows ▪ Stepwise Automation workflows for initializing nodes ▪ Collect and query Inventory and Patch status
  11. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Key Systems Manager Benefits ✓ All actions recorded in CloudTrail (e.g. immutable audit trail) ✓ Trigger SNS, Lambda from Systems Manager events ✓ Store command history and output to S3 ✓ Fine-grained access control to Run Commands ✓ Integration with Config to track changes over time
  12. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. The SSM Agent ▪ Open source (Golang) executable for Linux and Windows ▪ Available for cloud and on-premises systems ▪ Assign IAM role with permissions to interface with SSM API ▪ Install at boot or on existing systems ▪ Polls for commands to execute
  13. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. { "schemaVersion":"2.0", "description":"Run a script", "parameters":{ "commands":{ "type":"String", “description”:"Commands to run" } }, "mainSteps":[ { "action":"aws:runShellScript", "name":"runShellScript", "inputs":{ "runCommand":"{{ commands }}" } } ] } Systems Manager Documents
  14. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. github.com/kountable/pssh A s h e l l f o r t h e E C 2 P a r a m e t e r S t o r e
  15. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. λ pssh CloudWatch Event EC2 Parameter Store Game of Thrones Transfer of Power CloudWatch Metrics
  16. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Scoring System -40 Illegitimate -30 Handicapped -20 Exile 0 Deceased 30 Distinction 50 Noble 50 Royalty 100 Magical
  17. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Community Curated Pro Tips
  18. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Automated infrastructure is table stakes.
  19. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Ansible for provisioning and fleet control. Lambdas for glue. Packer for building images.
  20. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Single node ASGs for self-healing and resilience.
  21. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use Config Rules to enforce compliance with tagging schemes, EBS snapshots, security group rules, or other site preferences.
  22. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Many EC2 IAM actions do not support resource-level permissions. Exercise caution. { "Statement": [ { "Action": [ "ec2:AllocateAddress", "ec2:AssociateAddress", "ec2:DescribeAddresses", "ec2:DisassociateAddress", "ec2:CreateKeyPair", "ec2:DeleteKeyPair" ], "Effect": "Allow", "Resource": [ "*" ] } ] }
  23. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use the BurstBalance CloudWatch metric to monitor I/O credit balance for gp2, st1, sc1 EBS volumes.
  24. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use the CPUCreditUsage and CPUCreditBalance metrics in CloudWatch to track CPU burst usage on T2 instance types.
  25. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. github.com/awslabs/aws-shell T h e r e a l h e r o
  26. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Network throughput increases substantially with instance type. (Don't forget to enable enhanced networking)
  27. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. In CloudFormation, explicitly request SSD ephemeral disks or you may not get them.
  28. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Require MFA for SSH access and enable Fail2ban to block IPs with failed login attempts.
  29. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Improve your SSH experience with ControlPersist.
  30. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use ssh -D and the SwitchyOmega Chrome extension for convenient access to services in a private network.
  31. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Running multiple apps per instance? Use AssumeRole to assign granular permissions to each app.
  32. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Protect the EC2 metadata and userdata.
  33. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use the instance identity document to validate the authenticity of EC2 instances.
  34. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Use Linux >= 4.4 for best results.
  35. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. checkip.amazonaws.com is HTTP only hence it cannot be trusted. Use icanhazip.com instead.
  36. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. #!/bin/bash -x # Save userdata cmd output to a log exec > /var/log/userdata.log 2>&1 # Initialize instance …
  37. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Pace yourself tonight. It’s going to be a busy week.
  38. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Recently released!
  39. © 2017, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. ` Ben Whaley D i r e c t o r , S e c u r i t y a n d O p e r a t i o n s K o u n t a b l e