Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
170
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
92
FGED Keynote
mndoci
3
89
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Platforms for Data Science
mndoci
3
190
Other Decks in Technology
See All in Technology
猫でもわかるS3 Tables【Apache Iceberg編】
kentapapa
2
260
AIフレンドリーなプロダクト開発を目指して 〜MCPを橋渡しにした環境移行〜
shinpr
0
130
【Gen-AX】20250514開催_Findyオンラインイベント_技術選定を突き詰める
genax
0
100
ソフトウェアテスト 最初の一歩 〜テスト設計技法をワークで体験しながら学ぶ〜 #JaSSTTokyo / SoftwareTestingFirstStep
nihonbuson
PRO
4
300
激動の一年を通じて見えてきた「技術でリードする」ということ
ktr_0731
8
8.4k
木を見て森も見る-モジュールが織りなすプロダクトの森
kworkdev
PRO
0
280
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
720
テストコードにはテストの意図を込めよう(2025年版) #retechtalk / Put the intent of the test 2025
nihonbuson
PRO
11
2.1k
Docker Compose で手軽に手元環境を実現する / Simplifying Local Environments with Docker Compose #CinemaDeLT
nabeo
0
270
MagicPodが描くAIエージェント戦略とソフトウェアテストの未来
magicpod
0
310
10年もののアプリケーションを運用・開発するアプリケーションエンジニアのDatadog活用術
miyamu
0
120
Опыт использования Nessie в Азбуке Вкуса
emeremyanina1234
0
360
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
81
9k
Building a Modern Day E-commerce SEO Strategy
aleyda
40
7.3k
We Have a Design System, Now What?
morganepeng
52
7.6k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
How to Think Like a Performance Engineer
csswizardry
23
1.6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
KATA
mclloyd
29
14k
Building an army of robots
kneath
305
45k
A better future with KSS
kneath
239
17k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian