Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Platforms for Data Science

Avatar for Deepak Singh Deepak Singh
October 01, 2011
Technology
3
200

Platforms for Data Science

Talk given at the "Computing on the Brink" series

Avatar for Deepak Singh

Deepak Singh

October 01, 2011
Tweet

More Decks by Deepak Singh

See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
Avatar for Deepak Singh mndoci
2
110
Platforms for scientific data analysis
Avatar for Deepak Singh mndoci
3
100
FGED Keynote
Avatar for Deepak Singh mndoci
3
93
Open Mic Science - May 7, 2012
Avatar for Deepak Singh mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
Avatar for Deepak Singh mndoci
1
260
A Platform for Data Science
Avatar for Deepak Singh mndoci
6
14k
Intel Theater Presentation @ SC11
Avatar for Deepak Singh mndoci
6
180
Talk at West Coast Association of Shared Directors meeting
Avatar for Deepak Singh mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
Avatar for Deepak Singh mndoci
3
110

Other Decks in Technology

See All in Technology
Bakuraku Engineering Team Deck
Avatar for LayerX layerx
PRO
11
5.7k
Design System Documentation Tooling 2025
Avatar for takanorip takanorip
1
930
あなたの知らないDateのひみつ / The Secret of "Date" You Haven't known #tqrk16
Avatar for Shu OGAWARA expajp
0
110
Eight Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
5.7k
Symfony AI in Action
Avatar for Christopher Hertel el_stoffel
2
370
こがヘンだよ！Snowflake?サービス名称へのこだわり
Avatar for タロウ tarotaro0129
0
110
手動から自動へ、そしてその先へ
Avatar for Sammy(MoritaMasami) moritamasami
0
180
All About Sansan – for New Global Engineers
Avatar for Sansan, Inc. sansan33
PRO
1
1.3k
Playwrightのソースコードに見る、自動テストを自動で書く技術
Avatar for Yusuke Iwaki yusukeiwaki
2
710
プラットフォームエンジニアリングとは何であり、なぜプラットフォームエンジニアリングなのか
Avatar for doublemarket doublemarket
1
550
セキュリティAIエージェントの現在と未来 / PSS #2 Takumi Session
Avatar for GMO Flatt Security flatt_security
3
1.4k
段階的に進める、 挫折しない自宅サーバ入門
Avatar for yut yu_kod
5
2.2k

Featured

See All Featured
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
333
22k
XXLCSS - How to scale CSS and keep your sanity
Avatar for Zaharenia Atzitzikaki sugarenia
249
1.3M
The Art of Programming - Codeland 2020
Avatar for Erika Heidi erikaheidi
56
14k
It's Worth the Effort
Avatar for 3n 3n
187
29k
Designing Experiences People Love
Avatar for Jonathan Moore moore
142
24k
Principles of Awesome APIs and How to Build Them.
Avatar for Keavy McMinn keavy
127
17k
Done Done
Avatar for chrislema chrislema
186
16k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
35
2.3k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
10 Git Anti Patterns You Should be Aware of
Avatar for Lemi Orhan Ergin lemiorhan
PRO
659
61k
Into the Great Unknown - MozCon
Avatar for Noah Learner thekraken
40
2.2k

Transcript

  1. There is no magic There is only awesome D e

    e p a k S i n g h Platforms for data science

  2. bioinformatics image: Ethan Hein

  3. 3

  4. collection

  5. curation

  6. analysis

  7. what’s the big deal?

  8. None

  9. Source: http://www.nature.com/news/specials/bigdata/index.html

  10. Image: Yael Fitzpatrick (AAAS)

  11. Image: Yael Fitzpatrick (AAAS)

  12. lots of data

  13. lots of people

  14. lots of places

  15. constant change

  16. we want to make our data more effective

  17. versioning

  18. provenance

  19. filter

  20. aggregate

  21. extend

  22. mashup

  23. human interfaces

  24. None

  25. image: Leo Reynolds

  26. hard problem

  27. really hard problem

  28. so how do get there?

  29. information platforms

  30. Image: Drew Conway

  31. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise

    of the data scientist, Beautiful Data

  32. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent

    Systems, 24, 8-12 (2009)

  33. accept all data formats

  34. evolve APIs

  35. beyond databases and the data warehouse

  36. data as a programmable resource

  37. data is a royal garden

  38. compute is a fungible commodity

  39. optimizing the most valuable resource

  40. compute, storage, workflows, memory, transmission, algorithms, cost, …

  41. people Credit: Pieter Musterd a CC-BY-NC-ND license

  42. Image: Chris Dagdigian

  43. my bias

  44. cloud services

  45. distributed systems

  46. scale

  47. global

  48. consumption models

  49. on-demand

  50. what is the value of your data?

  51. None
  52. None

  53. Credit: Angel Pizzaro, U. Penn

  54. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml

  55. None

  56. Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/

  57. None
  58. None

  59. 30,472 cores

  60. $1279/hr

  61. http://cloudbiolinux.org/

  62. http://usegalaxy.org/cloud

  63. in summary

  64. large scale data requires a rethink

  65. data architecture

  66. compute architecture

  67. distributed, programmable infrastructure

  68. cloud services

  69. remove constraints

  70. can we build data science platforms?

  71. there is no magic there is only awesome

  72. [email protected] Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood&

    Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license