Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Platforms for Data Science

Avatar for Deepak Singh Deepak Singh
October 01, 2011
Technology
3
190

Platforms for Data Science

Talk given at the "Computing on the Brink" series
Avatar for Deepak Singh

Deepak Singh

October 01, 2011
Tweet

More Decks by Deepak Singh

See All by Deepak Singh
Changing the Calculus of Containers (Datadog Dash)
Avatar for Deepak Singh mndoci
2
100
Platforms for scientific data analysis
Avatar for Deepak Singh mndoci
3
92
FGED Keynote
Avatar for Deepak Singh mndoci
3
89
Open Mic Science - May 7, 2012
Avatar for Deepak Singh mndoci
4
1.3k
Talk at "Genome Informatics Alliance 2012" meeting
Avatar for Deepak Singh mndoci
1
250
A Platform for Data Science
Avatar for Deepak Singh mndoci
6
14k
Intel Theater Presentation @ SC11
Avatar for Deepak Singh mndoci
6
170
Talk at West Coast Association of Shared Directors meeting
Avatar for Deepak Singh mndoci
3
150
A platform for data science - Systems Bioinformatics Workshop
Avatar for Deepak Singh mndoci
3
110

Other Decks in Technology

See All in Technology
CloudBruteによる外部からのS3バケットの探索・公開の発見について / 20250605 Kumiko Hennmi
Avatar for SHIFT EVOLVE shift_evolve
3
160
うちの会社の評判は？SNSの投稿分析にAIを使ってみた
Avatar for Kiyotaka Doumae doumae
0
160
Cursor Meetup Tokyo
Avatar for Shunta Komatsu iamshunta
2
470
データプレーンプログラミングとは？ DPU＆スイッチASICの開発経験から語る
Avatar for ebiken ebiken
PRO
1
260
OSMnx Galleryの紹介
Avatar for mopinfish mopinfish
0
150
令和最新版TypeScriptでのnpmパッケージ開発
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
0
110
Oracle Base Database Service 技術詳細
Avatar for oracle4engineer oracle4engineer
PRO
8
65k
Devin＆Cursor、それぞれの「本質」から導く最適ユースケース戦略
Avatar for empitsu empitsu
8
2.4k
いまさら聞けない Git 超入門 〜Gitって結局なに？から始める第一歩〜
Avatar for とことんDevOps devops_vtj
0
160
What's Next in OpenShift Q2 CY2025
Avatar for Red Hat Livestreaming redhatlivestreaming
1
810
プラットフォームとしての Datadog / Datadog as Platforms
Avatar for Kento Kimura aoto
PRO
1
340
mnt_data_とは？ChatGPTコード実行環境を深堀りしてみた
Avatar for Issei.Komori icck
0
210

Featured

See All Featured
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
49
8.1k
Rails Girls Zürich Keynote
Avatar for Gregor Martynus gr2m
94
13k
Faster Mobile Websites
Avatar for Dean Hume deanohume
307
31k
Raft: Consensus for Rubyists
Avatar for Patrick Van Stee vanstee
137
7k
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
12k
The Web Performance Landscape in 2024 [PerfNow 2024]
Avatar for Tammy Everts tammyeverts
5
620
Distributed Sagas: A Protocol for Coordinating Microservices
Avatar for Caitie McCaffrey caitiem20
331
21k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
81
9k
It's Worth the Effort
Avatar for 3n 3n
184
28k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
45
9.6k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
50k
Code Review Best Practice
Avatar for Trisha Gee trishagee
68
18k

Transcript

  1. There is no magic There is only awesome D e

    e p a k S i n g h Platforms for data science

  2. bioinformatics image: Ethan Hein

  3. 3

  4. collection

  5. curation

  6. analysis

  7. what’s the big deal?

  8. None

  9. Source: http://www.nature.com/news/specials/bigdata/index.html

  10. Image: Yael Fitzpatrick (AAAS)

  11. Image: Yael Fitzpatrick (AAAS)

  12. lots of data

  13. lots of people

  14. lots of places

  15. constant change

  16. we want to make our data more effective

  17. versioning

  18. provenance

  19. filter

  20. aggregate

  21. extend

  22. mashup

  23. human interfaces

  24. None

  25. image: Leo Reynolds

  26. hard problem

  27. really hard problem

  28. so how do get there?

  29. information platforms

  30. Image: Drew Conway

  31. dataspaces Further reading: Jeff Hammerbacher, Information Platforms and the rise

    of the data scientist, Beautiful Data

  32. the unreasonable effectiveness of data Halevy, et al. IEEE Intelligent

    Systems, 24, 8-12 (2009)

  33. accept all data formats

  34. evolve APIs

  35. beyond databases and the data warehouse

  36. data as a programmable resource

  37. data is a royal garden

  38. compute is a fungible commodity

  39. optimizing the most valuable resource

  40. compute, storage, workflows, memory, transmission, algorithms, cost, …

  41. people Credit: Pieter Musterd a CC-BY-NC-ND license

  42. Image: Chris Dagdigian

  43. my bias

  44. cloud services

  45. distributed systems

  46. scale

  47. global

  48. consumption models

  49. on-demand

  50. what is the value of your data?

  51. None
  52. None

  53. Credit: Angel Pizzaro, U. Penn

  54. mapreduce for genomics http://bowtie-bio.sourceforge.net/crossbow/index.shtml http://contrail-bio.sourceforge.net http://bowtie-bio.sourceforge.net/myrna/index.shtml

  55. None

  56. Bioproximity http://aws.amazon.com/solutions/case-studies/bioproximity/

  57. None
  58. None

  59. 30,472 cores

  60. $1279/hr

  61. http://cloudbiolinux.org/

  62. http://usegalaxy.org/cloud

  63. in summary

  64. large scale data requires a rethink

  65. data architecture

  66. compute architecture

  67. distributed, programmable infrastructure

  68. cloud services

  69. remove constraints

  70. can we build data science platforms?

  71. there is no magic there is only awesome

  72. [email protected] Twitter:@mndoci http://slideshare.net/mndoci http://mndoci.com Inspiration and ideas from Matt Wood&

    Larry Lessig Credit” Oberazzi under a CC-BY-NC-SA license