Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intel Theater Presentation @ SC11

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Avatar for Deepak Singh Deepak Singh
November 19, 2011
Technology
210
6

Intel Theater Presentation @ SC11

Presented at the Intel Theater at SC11

Avatar for Deepak Singh

Deepak Singh

November 19, 2011

More Decks by Deepak Singh

See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
Avatar for Deepak Singh mndoci
2
130
Platforms for scientific data analysis
Avatar for Deepak Singh mndoci
3
120
FGED Keynote
Avatar for Deepak Singh mndoci
3
110
Open Mic Science - May 7, 2012
Avatar for Deepak Singh mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
Avatar for Deepak Singh mndoci
1
270
A Platform for Data Science
Avatar for Deepak Singh mndoci
6
15k
Talk at West Coast Association of Shared Directors meeting
Avatar for Deepak Singh mndoci
3
160
A platform for data science - Systems Bioinformatics Workshop
Avatar for Deepak Singh mndoci
3
130
Platforms for Data Science
Avatar for Deepak Singh mndoci
3
210

Other Decks in Technology

See All in Technology
実装だけじゃない！ CCA-F取得エンジニアが教えるClaude Code開発プロセス活用術
Avatar for morimorikochan diggymo
1
110
初めてのDatabricks勉強会
Avatar for Takaaki Yayoi taka_aki
2
240
プライバシー保護の理論と実践
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
240
Zoom2Youtube.Claude
Avatar for Yasunobu Kawaguchi kawaguti
PRO
2
400
知見・人・API・DB・予算 ─ ナイナイ尽くしだった人事データ整備 with dbt、5年間の学び
Avatar for ken ken6377
1
150
Kotlin 開発のツラミを爆破した話！ / Explode the difficulty of Kotlin dev!
Avatar for Kengo TODA eller86
0
150
從觀望到全公司落地：AI Agentic Coding 導入實戰 — 流程整合與安全治理
Avatar for Bo-Yi Wu appleboy
1
940
AI時代のエンジニアキャリアについて今一度考える
Avatar for Sakamoto sakamoto_582
1
1.2k
AI Agentをシステムに組み込む前にゆるく向き合ってみる
Avatar for hayama hayama17
0
200
Empower GenAI with Agile - あなたのアジャイルが生成AIのバフになる仕組み
Avatar for Hiroyuki Ito (TheHiro) hageyahhoo
0
110
SRE Lounge Hiroshimaへの招待
Avatar for gr1m0h grimoh
0
230
記録をかんたんに、提案をパーソナルに ── AIであすけんが目指すもの
Avatar for oprst oprstchn
0
140

Featured

See All Featured
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
Avatar for Tomoya Matsuura tomoyanonymous
3
890
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.7k
How to audit for AI Accessibility on your Front & Back End
Avatar for davetheseo davetheseo
0
460
Evolving SEO for Evolving Search Engines
Avatar for Ryan Jones ryanjones
0
230
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
Avatar for Szymon Słowik szymonslowik
1
1.2k
Bridging the Design Gap: How Collaborative Modelling removes blockers to flow between stakeholders and teams @FastFlow conf
Avatar for Kenny Baas-Schwegler baasie
0
610
From π to Pie charts
Avatar for Rasagy Sharma rasagy
0
230
How to Grow Your eCommerce with AI & Automation
Avatar for Katarina Dahlin katarinadahlin
PRO
1
220
First, design no harm
Avatar for Per Axbom axbom
PRO
2
1.2k
Building Experiences: Design Systems, User Experience, and Full Site Editing
Avatar for Michelle Schulp Hunt marktimemedia
0
550
Tell your own story through comics
Avatar for letsgokoyo letsgokoyo
1
980
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
160

Transcript

  1. HPC with Amazon EC2 Deepak Singh @mndoci P r i

    n c i p a l P r o d u c t M a n a g e r

  2. Amazon Web Services

  3. 4

  4. 2

  5. 1. Infrastructure

  6. None

  7. ec2-run-instances

  8. None

  9. secure global on demand

  10. programmable

  11. None
  12. None
  13. None

  14. elastic

  15. None

  16. instance types

  17. standard (m1) high memory (m2) high CPU (c1) t1.micro

  18. high performance

  19. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1

    billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets

  20. cluster computing

  21. MPI

  22. bandwidth intensive

  23. Cluster Compute Instance

  24. 2*Intel Xeon 5570 8 cores w/HT 23 GB RAM 1.7

    TB disk HVM cc1.4xlarge

  25. 10 gig E

  26. Placement Group

  27. Placement group full- bisection

  28. linpack

  29. Cores 7040 Rmax 41.82 Rpeak 82.51

  30. 231 November 2010

  31. 451 June 2011

  32. WIEN2K Parallel Performance H size 56,000 (25GB) Runtime (16x8 processors)

    Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) 1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington)

  33. New Cluster Compute Instance

  34. 2*Intel Xeon 16 cores w/HT 60.5 GB RAM 3.4 TB

    disk HVM cc2.8xlarge

  35. linpack

  36. Cores 17024 Rmax 240.09 Rpeak 354.12

  37. 42 November 2011

  38. optimizing costs

  39. on-demand

  40. reserved

  41. spot

  42. None
  43. None
  44. None

  45. 30,472 cores

  46. $1279/hr

  47. 2. Orchestration

  48. None

  49. AWS CloudFormation

  50. bootstrap

  51. Cloud Init

  52. #cloud-config packages: ! - httpd ! runcmd: ! - /etc/init.d

    http start ! - echo "<h1>hello, world"</h1> \ ! ! > /var/www/html/ index.html

  53. #!/bin/sh ec2-run-instances ami-8c1fece5 \ ! -n 1 \ ! -t

    m1.small \ ! -g deesinghdemo-SG \ ! -k deesinghdemo-keypair \ ! --user-data-file \ .\cloudconfig.txt

  54. chef/puppet

  55. familiar tools

  56. LSF

  57. Grid Engine

  58. Bright Cluster Manager

  59. combining worlds

  60. MIT Starcluster

  61. $ starcluster start mycluster $ starcluster listclusters

  62. http://www.bioteam.net/2011/03/dude-you-got-some-chef-in-my-starcluster/

  63. None

  64. Provisions Cluster Shared Storage Monitoring Bootstraps StarCluster Includes 200 GB

    Public Dataset Provisioned Stack = Submit jobs to Grid Engine
  65. None
  66. None
  67. None

  68. Image: Chris Dagdigian