A Platform for Data Science

Data, life science, productivity and cloud computing

Transcript

1. A Platform for Deepak Singh P r i n c i p a l P r o d u c t M a n a g e r Data Science
3. bioinformatics
4. collection
5. curation
6. analysis
7. so what?
8. Image: Yael Fitzpatrick (AAAS)
9. lots of data
10. lots of people
11. lots of places
12. lots of data
13. lots of data really fast
14. constant change
15. we want to make our data more effective
16. versioning
17. provenance
18. filter Via asklar under a CC-BY license
19. aggregate Image: Chris Heiler
20. extend Image: Bethan
21. human interfaces Image: Sebastian Anthony
22. share
23. image: Leo Reynolds communicate
24. hard problem
25. really hard problem
26. so how do you get there?
27. information platforms
28. Image: Drew Conway
29. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data
30. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)
31. accept all data formats
32. evolve APIs
33. data as a programmable resource
34. Heroku Data Clips
35. Data Clips allow the results of SQL queries on a Heroku Postgres database to be easily shared. Simply create a query on postgres.heroku.com, and then share the resulting URL with co- workers, colleagues, or the world. Data clips can be shared through e-mail, Twitter, irc, or any other medium - they are just URLs. The recipients of a data clip are able to view the data in their browser or download it in JSON, CSV, XML, or Microsoft Excel formats.
36. https://postgres.heroku.com/dataclips/psywmdixgtrxkorpzcueiohkwona
37. data is a royal garden
38. compute is a fungible commodity
39. constraints everywhere
40. Hardware CPU, storage, memory Data management Collections, datasets, provenance Software parallelization, optimization Availability Backup, redundant, replicated Cost Small
41. remove constraints Credit: Pieter Musterd a CC-BY-NC-ND license
43. your infrastructure
45. ec2-run-instances
48. secure global on demand
49. programmable
51. CloudFormation Templates "ClusterUserKeys" : { "Type" : "AWS::IAM::AccessKey", "Properties" : { "UserName" : { "Ref": "ClusterUser" } } }, ! "Ec2Instance" : { ! "Type" : "AWS::EC2::Instance", ! "Properties" : { ! "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ], ! ! ! "InstanceType" : "t1.micro", ! "ImageId" : "ami-7341831a", ! ! "KeyName" : { "Ref" : "KeyName" }, ! "Tags" : [{ ! "Key" : "Role", ! "Value" : "Controller" ! }], ! ! ! "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", ! ! ! [ ! ! ! ! ! ! "#!/bin/sh\n", ! ! ! "/opt/aws/bin/cfn-init ", " -s ", { "Ref" : "AWS::StackName" }, " -r Ec2Instance ", ! ! ! "--access-key=", { "Ref" : "ClusterUserKeys" }, " ", "--secret-key=", { "Fn::GetAtt" : ["ClusterUserKeys", "SecretAccessKey"]}, "\n", ! ! ! ! ! ! "cd /usr/src/pycrypto/pycrypto-2.4; /usr/bin/python setup.py build\n", ! ! ! ! ! ! "cd /usr/src/pycrypto/pycrypto-2.4; /usr/bin/python setup.py install\n", ! ! ! ! ! ! "cd /home/ec2-user/starcluster; /usr/bin/python distribute_setup.py\n", ! ! ! ! ! ! "cd /home/ec2-user/starcluster; /usr/bin/python setup.py install\n", ! ! ! ! ! "/bin/mkdir /home/ec2-user/.starcluster\n", ! ! ! ! ! "/bin/chown ec2-user:ec2-user -R /home/ec2-user/.starcluster\n", ! ! ! ! ! "/usr/bin/ruby /home/ec2-user/parser.rb /home/ec2-user/cc2-template.erb /home/ec2-user/values.yml > /home/ec2- user/.starcluster/config\n", ! ! ! ! ! "/usr/bin/starcluster -c /home/ec2-user/.starcluster/config createkey ", { "Ref" : "ClusterKeypair" }, " -o / home/ec2-user/.ssh/rsa-", { "Ref" : "ClusterKeypair" }, "\n", ! ! ! ! ! "/bin/chown ec2-user:ec2-user -R /home/ec2-user/.ssh/rsa-", { "Ref" : "ClusterKeypair" }, "\n", ! ! ! ! ! "cd /home/ec2-user/; /usr/bin/starcluster -c /home/ec2-user/.starcluster/config start ec2-cluster \n"! ! ! ! ! ! ! ! ! ! ! ! ]]}}
52. elastic
53. Netflix needed to transcode 17,000 titles (80TB of data) to support the launch of Sony PS3. They provisioned 1200 Amazon EC2 instances and completed the transcoding process in just days. Source: Adrian Cockroft (Netflix)
54. Source: Adrian Cockroft (Netflix)
55. scalable
56. Storage: Scale of Amazon S3 Total  Number  of  Objects  Stored  in  Amazon  S3 Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011 Peak  Requests: 500,000+ per  second 2.9  Billion 14  Billion 40  Billion 102  Billion 762  Billion 262  Billion
57. durable
58. 99.999999999%
59. I did say data was a royal garden
60. performance
61. “Our 40-instance (m2.2xlarge) cluster can scan, filter, and aggregate 1 billion rows in 950 milliseconds.” Mike Driscoll - Metamarkets
62. Druid: 26 billion rows/second
63. 200 Elastic MapReduce Clusters per day Asynchronous personalization requirements for 39M unique users 300 TB of data per day
64. Cores 17024 Rmax 240.09 Rpeak 354.12 Linpack
65. November 2011
66. “Our tests have shown more than 90 percent scaling efficiency on clusters of up to 128 GPUs each”
67. consumption models
68. on-demand
69. Reserved Instances Light Utilization Medium Utilization Heavy Utilization
70. $0# $5,000# $10,000# $15,000# $20,000# $25,000# 5%# 10%# 15%# 20%# 25%# 30%# 35%# 40%# 45%# 50%# 55%# 60%# 65%# 70%# 75%# 80%# 85%# 90%# 95%# 100%# On#Demand# Light#U<liza<on# Medium#U<liza<on# High#U<liza<on# 40% 80% 10%
71. what is the value of your data?
74. 30,472 cores
75. $1279/hr
76. the clouds biggest value
77. remove constraints
79. Image: Chris Dagdigian
80. http://www.bio-itworld.com/2012/02/14/two-guys-credit-card-metrum-amazon-cloud-makeover.html
81. “Now clients don’t have to wait in a queue… All projects get priority.” -- Adam Kraut
82. queues suck!!!
83. AWS for life sciences
84. services Amazon Elastic Compute Cloud (EC2) Amazon Elastic MapReduce Amazon Simple Storage Service AWS Import/Export Amazon Elastic Block Store Amazon RDS Amazon DynamoDB Amazon VPC AWS CloudFormation
85. services Zero  Administra.on Low  Latency  SSD’s Reserved  Capacity Unlimited  Poten.al  Storage   and  Throughput DynamoDB is a fully managed NoSQL database service that provides extremely fast and predictable performance with seamless scalability
86. Public Data Sets
87. AWS$Public$Data$ EBS$Snapshot$ S3$Bucket$ EC2$Instance$$ (Orchestrator)$ $ $ $ $ $ $ $ CloudForma=on$ Template$ Route 53 Zone Apex (optional) Chef$Cookbook$ Public$AMIs$ Autoscale (optional) Web$Server$ Chef$ Pre/Post$ processing$ End User Developer Console API/ CLI HTTP/HTTPS Client
88. EC2$Instance$$ (Orchestrator)$ $ $ $ $ $ $ $ Route 53 Zone Apex (optional) Autoscale (optional) Web$Server$ Chef$ Pre/Post$ processing$ End User HTTP/HTTPS Client EC2$ $ EC2$ $ EC2$ $ EC2$ $ EC2$ $ Star Cluster Blastz
89. Community http://cloudbiolinux.org/ http://clovr.org/ http://sourceforge.net/projects/seqware
90. Community http://bowtie-bio.sourceforge.net/myrna/index.shtml http://bowtie-bio.sourceforge.net/crossbow/index.shtml
91. Community http://sagebase.org/research/Synapse1.php http://www.genomespace.org/
92. Community http://galaxy.psu.edu/
93. Commercial
94. Getting Stuff Done
95. AMBER Contact Me
96. in summary
97. large scale data requires a rethink
98. data architecture
99. compute architecture
100. distributed, programmable infrastructure
102. remove constraints
103. can we build data science platforms?
104. there is no magic there is only awesome
105. deesingh@amazon.com Twitter:@mndoci http://speakerdeck.com/u/mndoci http://deepaksingh.net Inspiration and ideas from Matt Wood& Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license