A platform for data science - Systems Bioinformatics Workshop

Talk given at Systems Bioinformatics Workshop at ISB

Transcript

1. There is no magic There is only awesome D e e p a k S i n g h A platform for data science
2. bioinformatics image: Ethan Hein
4. collection
5. curation
6. analysis
7. what’s the big deal?
9. Source: http://www.nature.com/news/specials/bigdata/index.html
10. Image: Yael Fitzpatrick (AAAS)
11. Image: Yael Fitzpatrick (AAAS)
12. lots of data
13. lots of people
14. lots of places
15. to make data effective
16. versioning
17. provenance
18. filter
19. aggregate
20. extend
21. mashup
22. human interfaces
24. hard problem
25. really hard problem
27. change how we think about compute
28. change how we think about data
29. change how we think about science
30. information platforms
31. Image: Drew Conway
32. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data
33. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)
34. accept all data formats
35. evolve APIs
36. beyond the database and the data warehouse
37. data as a programmable resource
38. data as a royal garden
39. compute as a fungible commodity
40. which brings us to ...
41. amazon web services
44. common characteristics
45. on demand
46. pay as you go
47. secure
48. elastic
49. 3000 CPU’s for one firm’s risk management application !"#$%&'()'*+,'-./01.2%/' 344'+567/'(.' 8%%9%.:/' ;<"&/:1=' >?,3?,44@' A&B:1=' >?,>?,44@' C".:1=' >?,D?,44@' E(.:1=' >?,F?,44@' ;"%/:1=' >?,G?,44@' C10"&:1=' >?,H?,44@' I%:.%/:1=' >?,,?,44@' 3444JJ' 344'JJ'
50. programmable
51. “infrastructure as code”
53. include_recipe "packages" include_recipe "ruby" include_recipe "apache2" if platform?("centos","redhat") if dist_only? # just the gem, we'll install the apache module within apache2 package "rubygem-passenger" return else package "httpd-devel" end else %w{ apache2-prefork-dev libapr1-dev }.each do |pkg| package pkg do action :upgrade end end end gem_package "passenger" do version node[:passenger][:version] end execute "passenger_module" do command 'echo -en "\n\n\n\n" | passenger-install-apache2-module' creates node[:passenger][:module_path] end
54. import boto import boto.emr from boto.emr.step import StreamingStep from boto.emr.bootstrap_action import BootstrapAction import time # set your aws keys and S3 bucket, e.g. from environment or .boto AWSKEY= SECRETKEY= S3_BUCKET= NUM_INSTANCES = 1 conn = boto.connect_emr(AWSKEY,SECRETKEY) bootstrap_step = BootstrapAction("download.tst", "s3://elasticmapreduce/bootstrap-actions/download.sh",None) step = StreamingStep(name='Wordcount', mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py', cache_files = ["s3n://" + S3_BUCKET + "/boto.mod#boto.mod"], reducer='aggregate', input='s3n://elasticmapreduce/samples/wordcount/input', output='s3n://' + S3_BUCKET + '/output/wordcount_output') jobid = conn.run_jobflow( name="testbootstrap", log_uri="s3://" + S3_BUCKET + "/logs", steps = [step], bootstrap_actions=[bootstrap_step], num_instances=NUM_INSTANCES) print "finished spawning job (note: starting still takes time)" state = conn.describe_jobflow(jobid).state print "job state = ", state print "job id = ", jobid while state != u'COMPLETED': print time.localtime() time.sleep(30) state = conn.describe_jobflow(jobid).state print "job state = ", state print "job id = ", jobid print "final output can be found in s3://" + S3_BUCKET + "/output" + TIMESTAMP print "try: $ s3cmd sync s3://" + S3_BUCKET + "/output" + TIMESTAMP + " ." Connect to Elastic MapReduce Install packages Set up mappers & reduces job state
56. “I terminate the instance and relaunch it. Thats my error handling” Source: @jtimberman on Twitter
57. compute is a fungible commodity
58. emphasis on productivity
59. you can get a lot of awesome
60. dive in
61. just a little
62. Simple Storage Service S3
63. highly durable
64. 99.999999999%
65. Highly scalable
66. Elastic Compute Cloud EC2
67. dynamic
68. autoscaling
69. EC2 instance types
70. standard “m1” high cpu “c1” high memory “m2” http://aws.amazon.com/ec2/instance-types/ EC2 instance types
71. text cluster compute instances
72. cluster GPU instances
73. http://aws.amazon.com/ec2/instance-types/ cluster compute “cc1” EC2 instance types cluster GPU “cg1”
74. 10gbps
75. Placement Group
76. full bisection bandwidth
77. Linpack benchmark 880-instance CC1 cluster Performance: 41.82 TFlops* *#231 in Nov 2010 Top 500 rankings
78. KS for huge system at 1 k-point VERY DEMANDING network performance H size 56,000 (25GB) Runtime (16x8 processors) Local (Infiniband) 3h:48 Cloud (10Gbps) 1h:30 ($40) WIEN2k Parallel Performance Credit: K. Jorissen, F. D. Villa, and J. J. Rehr (U. Washington) •1200 atom unit cell; SCALAPACK+MPI diagonalization, matrix size 50k-100k
79. cost and use models
80. % Utilization time Ideal Effective Utilization Reserved Utilization On Demand Utilization Spot Utilization
81. making things easier
84. Elastic Beanstalk
86. Heroku
87. % Utilization time Ideal Effective Utilization Reserved Utilization On Demand Utilization Spot Utilization
88. data at scale
89. some practical considerations
90. everything fails all the time
91. compute needs vary
92. new data/compute paradigms
94. Amazon Elastic MapReduce
96. doing stuff
100. Customer Case Study: cyclopic energy http://aws.amazon.com/solutions/case-studies/cyclopic-energy/ OpenFOAM®
101. NASA JPL
102. Credit: Angel Pizzaro, U. Penn
103. http://aws.amazon.com/solutions/case-studies/numerate/
104. Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/
106. http://usegalaxy.org/cloud
107. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
108. http://cloudbiolinux.org/
109. in summary
110. large scale data requires a rethink
111. data architecture
112. compute architecture
113. in infrastructure
114. the cloud
115. distributed, programmable infrastructure
116. rapid, massive, scaling
117. architecture evolved with the internet
118. can we build data science platforms?
119. there is no magic there is only awesome
120. two more things
121. 10 minutes
122. http://aws.amazon.com/about-aws/build-a-cluster-in-under-10/
123. http://aws.amazon.com/education
124. deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood& Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license