Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
180
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
100
FGED Keynote
mndoci
3
93
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Platforms for Data Science
mndoci
3
200
Other Decks in Technology
See All in Technology
ブラウザのAPIで Nintendo Switch用の特殊なゲーム用コントローラーを体験型コンテンツに / IoTLT @ストラタシス・ジャパン
you
PRO
0
130
Open Table Format (OTF) が必要になった背景とその機能 (2025.10.28)
simosako
1
180
プロファイルとAIエージェントによる効率的なデバッグ / Effective debugging with profiler and AI assistant
ymotongpoo
1
120
AI-Readyを目指した非構造化データのメダリオンアーキテクチャ
r_miura
1
320
AIとともに歩んでいくデザイナーの役割の変化
lycorptech_jp
PRO
0
880
ゼロコード計装導入後のカスタム計装でさらに可観測性を高めよう
sansantech
PRO
1
400
個人でデジタル庁の デザインシステムをVue.jsで 作っている話
nishiharatsubasa
3
5.1k
Building a cloud native business on open source
lizrice
0
180
「最速」で Gemini CLI を使いこなそう! 〜Cloud Shell/Cloud Run の活用〜 / The Fastest Way to Master the Gemini CLI — with Cloud Shell and Cloud Run
aoto
PRO
1
180
NLPコロキウム20251022_超効率化への挑戦: LLM 1bit量子化のロードマップ
yumaichikawa
3
460
生成AI時代のPythonセキュリティとガバナンス
abenben
0
140
JSConf JPのwebsiteをGatsbyからNext.jsに移行した話 - Next.jsの多言語静的サイトと課題
leko
2
190
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
KATA
mclloyd
PRO
32
15k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Reflections from 52 weeks, 52 projects
jeffersonlam
353
21k
How to Think Like a Performance Engineer
csswizardry
27
2.1k
Build your cross-platform service in a week with App Engine
jlugia
233
18k
Being A Developer After 40
akosma
91
590k
Why Our Code Smells
bkeepers
PRO
340
57k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.5k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Optimizing for Happiness
mojombo
379
70k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian