Intel Theater Presentation @ SC11

Presented at the Intel Theater at SC11

Transcript

1. HPC with Amazon EC2 Deepak Singh @mndoci P r i n c i p a l P r o d u c t M a n a g e r
2. Amazon Web Services
5. 1. Infrastructure
7. ec2-run-instances
9. secure global on demand
10. programmable
14. elastic
16. instance types
17. standard (m1) high memory (m2) high CPU (c1) t1.micro
18. high performance
19. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1 billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
20. cluster computing
21. MPI
22. bandwidth intensive
23. Cluster Compute Instance
24. 2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7 TB disk HVM cc1.4xlarge
25. 10 gig E
26. Placement Group
27. Placement group full- bisection
28. linpack
29. Cores 7040 Rmax 41.82 Rpeak 82.51
30. 231 November 2010
31. 451 June 2011
32. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
33. New Cluster Compute Instance
34. 2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB disk HVM cc2.8xlarge
35. linpack
36. Cores 17024 Rmax 240.09 Rpeak 354.12
37. 42 November 2011
38. optimizing costs
39. on-demand
40. reserved
41. spot
45. 30,472 cores
46. $1279/hr
47. 2. Orchestration
49. AWS CloudFormation
50. bootstrap
51. Cloud Init
52. #cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
53. #!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
54. chef/puppet
55. familiar tools
56. LSF
57. Grid Engine
58. Bright Cluster Manager
59. combining worlds
60. MIT Starcluster
61. $ starcluster start mycluster $ starcluster listclusters
62. http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
64. Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB Public Dataset Provisioned Stack = Submit jobs to Grid Engine
68. Image: Chris Dagdigian