Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Open Source Data Tooling Landscape
Search
Carol Willing
PRO
August 24, 2021
Technology
1
95
The Open Source Data Tooling Landscape
Given for Coiled webinar on August 24, 2021.
Carol Willing
PRO
August 24, 2021
Tweet
Share
More Decks by Carol Willing
See All by Carol Willing
CPython: Foundation for Scientific Python
willingc
PRO
0
18
Be a SLQAR. Micromentoring for all.
willingc
PRO
0
27
Lessons in Leadership: Python, AI, and Heuristics
willingc
PRO
0
120
Embracing Python, AI, and Heuristics: Optimal Paths for Impactful Software
willingc
PRO
0
900
Thriving with Python: Navigate the pitfalls in a polyglot world
willingc
PRO
1
180
Pragmatic Python: Python 3.12 and beyond
willingc
PRO
0
200
The Future is Notebooks
willingc
PRO
0
120
PyCon 2023 Keynote
willingc
PRO
0
210
Python: The People's Programming Language
willingc
PRO
0
100
Other Decks in Technology
See All in Technology
「AI駆動開発」のボトルネック『言語化』を効率化するには
taniiicom
1
210
ビジネス文書に特化した基盤モデル開発 / SaaSxML_Session_2
sansan_randd
0
120
Recoil脱却の現状と挑戦
kirik
3
460
MCPと認可まわりの話 / mcp_and_authorization
convto
2
290
FAST導入1年間のふりかえり〜現実を直視し、さらなる進化を求めて〜 / Review of the first year of FAST implementation
wooootack
1
180
AIエージェントを支える設計
tkikuchi1002
11
2.3k
隙間時間で爆速開発! Claude Code × Vibe Coding で作るマニュアル自動生成サービス
akitomonam
2
200
経理出身PdMがAIプロダクト開発を_ハンズオンで学んだ話.pdf
shunsukenarita
1
230
P2P ではじめる WebRTC のつまづきどころ
tnoho
1
270
Datasets for Critical Operations by Dataform
kimujun
0
120
AI時代の知識創造 ─GeminiとSECIモデルで読み解く “暗黙知”と創造の境界線
nyagasan
0
160
AI駆動開発 with MixLeap Study【大阪支部 #3】
lycorptech_jp
PRO
0
270
Featured
See All Featured
How to Ace a Technical Interview
jacobian
278
23k
Bash Introduction
62gerente
613
210k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
A Modern Web Designer's Workflow
chriscoyier
695
190k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
126
53k
Being A Developer After 40
akosma
90
590k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.9k
Transcript
The Open Source Data Tooling Landscape Carol Willing VP of
Learning Noteable web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Headline Slide Sub-headline The 10 Best Practices for Remote Software
Engineering Focusing on the human element of remote software engineer productivity Vanessa Sochat DOI:10.1145/3459613 Attribution: xkcd 1 Today
Common Data Challenges Exploring Solutions with Open Source Data Tools
2 Data
SCALE
SPEED
CONNECTIONS
CHOICES
The Data Pipeline Perspectives Attribution: Red Bull 3 People
The Data Pipeline Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Executives Opportunity and
Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear Users Productivity and Needs
Attribution: Red Bull Start small...
@WillingCarol 14 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l ...and scale.
Open Source Data Tooling Landscape 4 Ecosystem
Python R Julia Fortran SQL C++ Go Rust Java Scala
4 Ecosystem Programming Languages JavaScript TypeScript Data Analysis Workflows Interactivity
4 Ecosystem Data Work fl ow Project Definition Data Collection
Computation and Modeling Evaluation Deploy at Scale Monitoring Data Preparation Exploratory Analysis Share Results Revisit Goals
Challenges ‣ Foundation (existing infrastructure to cloud) ‣ Variability (DIY
to Hosted/Managed Service) ‣ Complexity ‣ Language ecosystems ‣ Growth
Challenges (cont.) ‣ Best practices / de facto standards ‣
Jargon ‣ Abstractions ‣ Hype CRISP-DM Attribution: IBM Cross-industry standard process for data mining 1996
4 Ecosystem Taxonomy Business Goals People Ethics Model creation Training
Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical
4 Ecosystem Julia Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical DrWatson.jl ParameterSchedulers.jl Pluto.jl IJulia JupyterLab nteract VSCode Plots.jl (Viz) Gadfly.jl (Viz) Makie.jl (Viz - GPU) Flux.jl (ML) Knet.jl (ML/BL) MLJ.jl (ML) Mocha.jl (ML/DL) Tensorflow.jl (ML/DL wrapper) JuMP (optimization) Dataframes.jl ProgressMeters.jl
4 Ecosystem Python Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical Dask JupyterHub Binder Kubernetes papermill Dagster Airflow prefect scipy statsmodel JupyterLab nteract VSCode matplotlib seaborn altair plotly numpy scikit-learn pytorch tensorflow pandas PyJanitor dask datasette evidently bokeh panel voila dash python scripts napari geopandas feast keras fastai fairlearn
4 Ecosystem R Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical RStudio JupyterLab IRkernel ggplot tidyverse dplyr tidyr lubridate readr readxl googlesheets4 ggplot2 rmarkdown Shiny plumber purrr reticulate Keras Tensorflow sparklyr ropensci.org knitr forcats mlr3 CNTK theanos
Algorithmic Business Thinking (ABT) 5 Management Paul McDonagh-Smith MIT Sloan
School of Management https://mitsloan.mit.edu/faculty/directory/paul-mcdonagh-smith https://www.youtube.com/watch?v=bqtn2tYg-kw
@WillingCarol 25 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l Got data at scale? Use open source tools.
web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Thank you The Open Source Data Tooling Landscape Carol Willing VP of Learning Noteable
6 Additional Resources https://krzjoa.github.io/awesome-python-data-science/#/ https://github.com/FavioVazquez/ds-cheatsheets https://www.the-modeling-agency.com/crisp-dm.pdf https://github.com/academic/awesome-datascience