Bringing a Machete to the Amazon - Blackhat EU 2014

Transcript

1. Bringing a Machete to the Amazon Erik Peterson - @silvexis - epeterson@veracode.com
2. Hi, I’m Erik • AppSec guy for 17 years • Director of Technology Strategy at Veracode • Have been researching the Cloud and building products on top of AWS since 2009 @silvexis
3. AGENDA I II III IV INTRO GETTING ACCESS V VI IV PROBLEMS MACHETE FULL STACK
 HACKS THE API IS KING EMERGENT INSECURITY
4. CLOUD
 IS AN OPERATING SYSTEM
5. CLOUD INFRASTRUCTURE 
 IS CODE
6. A Fully Realized AWS Application
7. Traditional App represents small % of system • Java, .Net, etc… including 3rd party code in red
8. Majority of the system provided by AWS Using it all securely however is your responsibility
9. What you get from AWS • The digital equivalent of infinite empty rack space • A friendly looking web interface • A mile long list of compliance certifications • The rest is up to you
10. …and yet after a few months this might be you
11. Forklifting also Dangerous Forklifting is the process of taking legacy data center applications and loading them into a cloud provider with little or no changes Often this is the most expensive way to use AWS It is also dangerous IDS and network access controls are unaware of AWS API activity Previously ignored or de-prioritized vulnerabilities can become critical The meaning of availability changes
12. EMERGENT INSECURITY The individual components of your system may be secure but when deployed into the Cloud, the system becomes insecure AND IT GETS WORSE AT SCALE
13. EMERGENT INSECURITY FACTORS “Internet Weather” - AWS systems and API calls are subject to unpredictable, non-persistent, network latency, system performance and connection interruption Guaranteed Failure - System availability is a factor of redundancy and automation, not the stability and performance of monolithic systems Software defined everything (SDe) - Virtualized networks, network interfaces, file systems, computing power and more can change independent of the underlying system Out-of-band management - AWS API’s operate outside of traditional security controls, can make all existing controls irrelevant
14. EMBRACE AN EVENTUALLY CONSISTENT
 SECURITY MODEL Instead of trying to enforce change control which creates brittle systems that are insecure and not survivable, design your systems to be eventually consistent with your security and operational goals If your system requires strict change control to maintain order, in the Cloud, you will eventually have chaos.
15. ALL HAIL THE KING In the cloud, the king of the jungle is the API
16. What’s your Real Attack Surface? “If your security sucks now, you’ll be pleasantly surprised by the lack of change when you move to Cloud.”* In reality is it gets worse AWS API endpoints are open by design, trumps all existing controls Private IP’s might be public, data leaks through 3rd party services Only thing standing between total compromise of your _entire_ datacenter is the secrecy of your API keys *Chris Hoff, @beaker
17. API ACCESS
    
    
    
    
    PHYSICAL ACCESS IN THE CLOUD
18. API Credential Exposure Impact The attacker can: Sets up bitcoin mining operations within your cloud environment Alter your applications to spread malware Uses your environment as a means to launch additional attacks Downloads all your customers data
19. Destroy your entire virtual datacenter
20. Bypassing traditional controls via the API Start with an EC2 system in AWS-VPC, block all network traffic, throw away SSH keys/ passwords, place IDS on VPC Gateway to detect intrusion attempts, log all network traffic to/from system In a traditional data center, system is now inaccessible, any attempt to access would be detected In AWS, use API to snapshot AMI/EBS vol’s, mount snapshots on different system, extract whatever I need/want. Zero indication from traditional controls that any access has taken place Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB
21. Getting Access to the API Checking your API keys into Github Exploiting vulnerabilities to access Metadata
22. Oops
23. API Honeypot? Curious how fast an API key can be stolen from GitHub? Create a key, lock it to only one IP address and watch your logs You should see failed access attempts in 60 minutes or less
25. What is Cloud Metadata? Based on RFC 3927 - Dynamic Configuration of IPv4 Link-Local Addresses Metadata contains all kinds of awesome things, like startup scripts and your AWS access credentials It’s not just AWS, all cloud providers have it (except for MS Azure!) AWS: http://169.254.169.254/latest/user-data Google: http://169.254.169.254/computeMetadata/v1/ OpenStack/RackSpace: http://169.254.169.254/openstack HP Helion: http://169.254.169.254/2009-04-04/meta-data/ On Azure, metadata is not dynamic but is copied to /var/lib/waagent (linux) or %SYSTEMDRIVE%\AzureData\CustomData.bin (Windows) There is nothing wrong with any of this, as long as you are aware and protect it
26. $ wget -q -O - http://169.254.169.254/latest/meta-data/ ami-id ami-launch-index ami-manifest-path block-device-mapping/ hostname iam/ instance-action instance-id instance-type kernel-id local-hostname local-ipv4 mac metrics/ network/ placement/ profile public-hostname public-ipv4 public-keys/ reservation-id security-groups Accessing AWS Metadata AWS command line tool ec2-metadata will extract some (but not all!) of the metadata To get everything use wget or curl Each one of these represents meta data you can access $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id i-0496132e
27. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security- credentials/<YOUR PROFILE HERE> { "Code" : "Success", "LastUpdated" : "2014-08-13T16:43:54Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA", "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV", "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/ Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==", "Expiration" : "2014-08-13T23:05:19Z" }
28. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security- credentials/<YOUR PROFILE HERE> { "Code" : "Success", "LastUpdated" : "2014-08-13T16:43:54Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA", "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV", "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/ Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==", "Expiration" : "2014-08-13T23:05:19Z" }
29. OLD VULNS NEW LIFE These vulnerabilities can result in a
 total data center compromise
 in AWS CWE-918: SSRF CWE-611: XXE CWE-441: Unintended Proxy or Intermediary CWE-77: Command Injection CWE-200: Information Exposure Why? All of these can lead to unintended exposure of metadata
30. Remote Access to Metadata Real world Examples Prezi experienced this first hand http://engineering.prezi.com/blog/2014/03/24/prezi-got- pwned-a-tale-of-responsible-disclosure/ Andres Riancho demonstrated this in his BlackHat talk “Pivoting in Amazon Clouds” Exploited SSRF vuln to inject malicious AWS SQS messages to then exploit a celery/python vuln and take control
31. Controlling API Access IP Restrictions, they are not just for security groups! Keys locked down to specific IP’s (like your VPC gateway) have less chance of damage Doesn’t fully eliminate the problem however { "Version": "2012-10-17", "Statement":[{ "Effect":"Deny", "Action":"*", "Resource":"*", "Condition":{ "NotIpAddress":{ “aws:SourceIp”:[“VPC GATEWAY IP/32”] } } }] }
32. Your bill is not an IDS Many AWS customers today use their bill as their IDS So far, most of these people have been pretty lucky
33. Implement API Logging…please! Logging is off by default in AWS In AWS, turn on CloudTrail for all regions Use LogStash, develop your own or buy a commercial solution to make API activity logs accessible
34. EXCESSIVE LACK OF
 ACCESS CONTROL { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }
35. DevOps Culture tends to “Fail Open” I love DevOps but…. Be wary of Developers building systems who are new to the Cloud because they might now be the ones configuring the firewall rules and API Access Policies as well Developers and Operations are often pre-disposed to “Just make it work” This is one of the reasons DevOps is such a powerful movement It’s on us as security people to adapt and keep DevOps powerful
36. Leaking Tags Tags in AWS are one of the primary means of keeping track of things Tags however often can contain private data that does not have strong access protections I’ve seen API keys and passwords in tags!!! If you use any 3rd party AWS management system like StackDriver, CloudHealth or others, your tag data is replicated to those systems
37. PRIVATE DNS IS PUBLIC DNS AWS DNS from within AWS will resolve EC2 Classic public DNS entries to RFC 1918 private address Good chance of Information Exposure (CWE-200) from web servers listening on private IP’s A casual poke around the EC2 10.0.0.0/8 address space reveals hundreds of web servers Remediation: Migrate EC2 Classic environments to VPC [ec2-user@heyzeus ~]$ host example.veracodelabs.com example.veracodelabs.com is an alias for ec2-107-22-197-187.compute-1.amazonaws.com. ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67
38. Data and File Leakage AWS S3 1 in 7 buckets left open to anyone attempting to access them This might very well be intentional but security through obscurity is a real problem AWS EBS If I have API access, I can clone your running systems and EBS volumes into an external account You can prevent this by encrypting your EBS volumes (you can’t encrypt root volumes however)
39. The Full Stack Hack 3. SSRF or XXE vulnerability exposes EC2 Metadata, revealing AWS API Keys 1. AWS DNS Reveals Private IP of web server 2. Private IP of web server reveals detailed errors or admin interface 4. AWS API Key lets you create new instances where your clone and mount any existing EBS vol. 5. Cloned system gets you SSH keys to app servers and API key with IAM:* giving you access to everything 6. With new root credentials create trust relationship with external account and clone DB for quiet extraction
40. CUTTING THROUGH THE JUNGLE MACHETE
41. Capabilities Analyzes IAM users and permissions Requires only readonly API access Records all services and objects in use Track changes, know what is in your account quickly Image Cracker Crack open AMI and EBS snapshots and extract access keys, user accounts, data Coming Soon Not quite ready yet, sorry!
42. IAM Permissions Analysis epeterson@epeterson-mac:~/work/machete$ bin/machete -e 67d54268-dab1-4c73-a424-dbe75d466bb5 refresh Machete - v0.1.5 - Connecting... ...connected Starting scan of environment 67d54268-dab1-4c73-a424-dbe75d466bb5 ================================================================================ id: 67d54268-dab1-4c73-a424-dbe75d466bb5|AROAIEMNXXXX3FXXXXXHM6 cust uuid: f4cec92f-9184-4394-a4d3-13b7d8708a13 env uuid: 67d54268-dab1-4c73-a424-dbe75d466bb5 First seen: 2014-10-11 13:38:20.335481+00:00 Last seen: 2014-10-17 10:04:00.754355+00:00 Parent: None Type: IAM_ROLE Tags: {} Metadata: -------------------------------------------------------------------------------- {u'arn': u'arn:aws:iam::034817543207:role/cloudatlas_node', u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole', u'Effect': u'Allow', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Sid': u''}], u'Version': u'2012-10-17'}, u'create_date': u'2014-06-11T20:25:43Z', u'id': u'AXXXXXXNKYAIXXXXXXXX', u'name': u'cloudatlas_node', u'path': u'/', u'role_id': u'AXXXXXXNKYAIXXXXXXXX', u'role_name': u'cloudatlas_node'} -------------------------------------------------------------------------------- Versions: 1 -------------------------------------------------------------------------------- Change detected on: 2014-10-17 12:03:48 {u'added': {}, u'changed': {u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole', u'Effect': u'Allow', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Sid': u''}], u'Version': u'2012-10-17'}}, u'removed': {}} --------------------------------------------------------------------------------
43. Resource tracking (machete)epeterson@epeterson-mac:~/work/machete/bin$ ./machete list -e 67d54268-dab1-4c73-a424-dbe75d466bb5 # : ID Type State First Seen Last Seen Versions -------------------------------------------------------------------------------------------------------------------------------------------- 1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes 2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes 3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00 {u'1408649760.41495': {u'added': {}, u'changed': {u'dns_name': u'', u'ip_address': None, u'public_dns_name': u'', u'reason': u'User initiated (2014-08-21 19:34:02 GMT)', u'state': [u'stopped', 80], u'state_reason': {u'code': u'Client.UserInitiatedShutdown', u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}}, u'removed': {}}} 4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes 5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes 6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes 7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes 8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes 9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes 10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes 11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes 12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes 13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes 14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes 15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes 16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes 17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes 18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes 19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes
44. Other Tools Netflix Security Monkey http://techblog.netflix.com/2014/06/announcing-security-monkey-aws-security.html …and a whole ton of other OSS stuff you should check out AWS Trusted Advisor https://aws.amazon.com/premiumsupport/trustedadvisor/ NimboStratus https://andresriancho.github.io/nimbostratus/
45. @silvexis epeterson@veracode.com THANK YOU May all your Clouds come with a Luck Dragon