Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Intel Theater Presentation @ SC11
Search
Deepak Singh
November 19, 2011
Technology
6
180
Intel Theater Presentation @ SC11
Presented at the Intel Theater at SC11
Deepak Singh
November 19, 2011
Tweet
Share
More Decks by Deepak Singh
See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
mndoci
2
100
Platforms for scientific data analysis
mndoci
3
100
FGED Keynote
mndoci
3
93
Open Mic Science - May 7, 2012
mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
mndoci
1
250
A Platform for Data Science
mndoci
6
14k
Talk at West Coast Association of Shared Directors meeting
mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
mndoci
3
110
Platforms for Data Science
mndoci
3
200
Other Decks in Technology
See All in Technology
OCI Network Firewall 概要
oracle4engineer
PRO
1
7.7k
FastAPIの魔法をgRPC/Connect RPCへ
monotaro
PRO
0
480
Green Tea Garbage Collector の今
zchee
PRO
2
350
Why React!?? Next.jsそしてReactを改めてイチから選ぶ
ypresto
9
3.6k
DEFCON CHV CTF 2025 Write-up
bata_24
0
190
API提供者のためのMCPサーバー設計ガイド / MCP Server Design Guide for API Providers
yokawasa
0
240
Goのビルドシステムの変遷 / The history of Go's build system
ymotongpoo
12
3.4k
Azure SynapseからAzure Databricksへ 移行してわかった新時代のコスト問題!?
databricksjapan
0
110
「技術負債にならない・間違えない」 権限管理の設計と実装
naro143
31
9.5k
AIを導⼊しても、 開発⽣産性は"爆増"していない なぜ?
kinosuke01
4
3.6k
コンテキストエンジニアリングとは? 考え方と応用方法
findy_eventslides
4
820
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
339
57k
Speed Design
sergeychernyshev
32
1.1k
Unsuck your backbone
ammeep
671
58k
4 Signs Your Business is Dying
shpigford
185
22k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
32
2.2k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
35
3.2k
Gamification - CAS2011
davidbonilla
81
5.4k
GitHub's CSS Performance
jonrohan
1032
460k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
The Cult of Friendly URLs
andyhume
79
6.6k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
30
2.9k
Building Applications with DynamoDB
mza
96
6.6k
Transcript
HPC with Amazon EC2 Deepak Singh @mndoci P r i
n c i p a l P r o d u c t M a n a g e r
Amazon Web Services
4
2
1. Infrastructure
None
ec2-run-instances
None
secure global on demand
programmable
None
None
None
elastic
None
instance types
standard (m1) high memory (m2) high CPU (c1) t1.micro
high performance
“Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1
billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
cluster computing
MPI
bandwidth intensive
Cluster Compute Instance
2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7
TB disk HVM cc1.4xlarge
10 gig E
Placement Group
Placement group full- bisection
linpack
Cores 7040 Rmax 41.82 Rpeak 82.51
231 November 2010
451 June 2011
WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)
Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)
New Cluster Compute Instance
2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB
disk HVM cc2.8xlarge
linpack
Cores 17024 Rmax 240.09 Rpeak 354.12
42 November 2011
optimizing costs
on-demand
reserved
spot
None
None
None
30,472 cores
$1279/hr
2. Orchestration
None
AWS CloudFormation
bootstrap
Cloud Init
#cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d
http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html
#!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t
m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt
chef/puppet
familiar tools
LSF
Grid Engine
Bright Cluster Manager
combining worlds
MIT Starcluster
$ starcluster start mycluster $ starcluster listclusters
http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/
None
Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB
Public Dataset Provisioned Stack = Submit jobs to Grid Engine
None
None
None
Image: Chris Dagdigian