Platforms for Data Science

Talk given at the "Computing on the Brink" series

Transcript

1. There is no magic There is only awesome D e e p a k S i n g h Platforms for data science
2. bioinformatics image: Ethan Hein
4. collection
5. curation
6. analysis
7. what’s the big deal?
9. Source: http://www.nature.com/news/specials/bigdata/index.html
10. Image: Yael Fitzpatrick (AAAS)
11. Image: Yael Fitzpatrick (AAAS)
12. lots of data
13. lots of people
14. lots of places
15. constant change
16. we want to make our data more effective
17. versioning
18. provenance
19. filter
20. aggregate
21. extend
22. mashup
23. human interfaces
25. image: Leo Reynolds
26. hard problem
27. really hard problem
28. so how do get there?
29. information platforms
30. Image: Drew Conway
31. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise of the data scientist, Beautiful Data
32. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent Systems, 24, 8-12 (2009)
33. accept all data formats
34. evolve APIs
35. beyond databases and the data warehouse
36. data as a programmable resource
37. data is a royal garden
38. compute is a fungible commodity
39. optimizing the most valuable resource
40. compute, storage, workflows, memory, transmission, algorithms, cost, …
41. people Credit: Pieter Musterd a CC-BY-NC-ND license
42. Image: Chris Dagdigian
43. my bias
44. cloud services
45. distributed systems
46. scale
47. global
48. consumption models
49. on-demand
50. what is the value of your data?
53. Credit: Angel Pizzaro, U. Penn
54. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml
56. Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/
59. 30,472 cores
60. $1279/hr
61. http://cloudbiolinux.org/
62. http://usegalaxy.org/cloud
63. in summary
64. large scale data requires a rethink
65. data architecture
66. compute architecture
67. distributed, programmable infrastructure
68. cloud services
69. remove constraints
70. can we build data science platforms?
71. there is no magic there is only awesome
72. deesingh@amazon.com Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood& Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license