Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Open Source Data Tooling Landscape
Search
Carol Willing
PRO
August 24, 2021
Technology
1
95
The Open Source Data Tooling Landscape
Given for Coiled webinar on August 24, 2021.
Carol Willing
PRO
August 24, 2021
Tweet
Share
More Decks by Carol Willing
See All by Carol Willing
Be a SLQAR. Micromentoring for all.
willingc
PRO
0
25
Lessons in Leadership: Python, AI, and Heuristics
willingc
PRO
0
110
Embracing Python, AI, and Heuristics: Optimal Paths for Impactful Software
willingc
PRO
0
890
Thriving with Python: Navigate the pitfalls in a polyglot world
willingc
PRO
1
170
Pragmatic Python: Python 3.12 and beyond
willingc
PRO
0
190
The Future is Notebooks
willingc
PRO
0
110
PyCon 2023 Keynote
willingc
PRO
0
200
Python: The People's Programming Language
willingc
PRO
0
100
A Random Walk with Snakes and Friends
willingc
PRO
0
60
Other Decks in Technology
See All in Technology
強化されたAmazon Location Serviceによる新機能と開発者体験
dayjournal
3
250
20250625 Snowflake Summit 2025活用事例 レポート / Nowcast Snowflake Summit 2025 Case Study Report
kkuv
1
370
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
2
230
製造業からパッケージ製品まで、あらゆる領域をカバー!生成AIを利用したテストシナリオ生成 / 20250627 Suguru Ishii
shift_evolve
PRO
1
160
生成AI時代の開発組織・技術・プロセス 〜 ログラスの挑戦と考察 〜
itohiro73
1
370
Tech-Verse 2025 Keynote
lycorptech_jp
PRO
0
1.3k
急成長を支える基盤作り〜地道な改善からコツコツと〜 #cre_meetup
stefafafan
0
150
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
500
Understanding_Thread_Tuning_for_Inference_Servers_of_Deep_Models.pdf
lycorptech_jp
PRO
0
150
Oracle Cloud Infrastructure:2025年6月度サービス・アップデート
oracle4engineer
PRO
2
310
Model Mondays S2E03: SLMs & Reasoning
nitya
0
240
Yamla: Rustでつくるリアルタイム性を追求した機械学習基盤 / Yamla: A Rust-Based Machine Learning Platform Pursuing Real-Time Capabilities
lycorptech_jp
PRO
4
170
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Faster Mobile Websites
deanohume
307
31k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
A Tale of Four Properties
chriscoyier
160
23k
Thoughts on Productivity
jonyablonski
69
4.7k
A designer walks into a library…
pauljervisheath
207
24k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
Transcript
The Open Source Data Tooling Landscape Carol Willing VP of
Learning Noteable web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Headline Slide Sub-headline The 10 Best Practices for Remote Software
Engineering Focusing on the human element of remote software engineer productivity Vanessa Sochat DOI:10.1145/3459613 Attribution: xkcd 1 Today
Common Data Challenges Exploring Solutions with Open Source Data Tools
2 Data
SCALE
SPEED
CONNECTIONS
CHOICES
The Data Pipeline Perspectives Attribution: Red Bull 3 People
The Data Pipeline Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Executives Opportunity and
Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear
The Data Pipeline Engineers Infrastructure and Process Data Scientists Algorithms
and Models Executives Opportunity and Fear Users Productivity and Needs
Attribution: Red Bull Start small...
@WillingCarol 14 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l ...and scale.
Open Source Data Tooling Landscape 4 Ecosystem
Python R Julia Fortran SQL C++ Go Rust Java Scala
4 Ecosystem Programming Languages JavaScript TypeScript Data Analysis Workflows Interactivity
4 Ecosystem Data Work fl ow Project Definition Data Collection
Computation and Modeling Evaluation Deploy at Scale Monitoring Data Preparation Exploratory Analysis Share Results Revisit Goals
Challenges ‣ Foundation (existing infrastructure to cloud) ‣ Variability (DIY
to Hosted/Managed Service) ‣ Complexity ‣ Language ecosystems ‣ Growth
Challenges (cont.) ‣ Best practices / de facto standards ‣
Jargon ‣ Abstractions ‣ Hype CRISP-DM Attribution: IBM Cross-industry standard process for data mining 1996
4 Ecosystem Taxonomy Business Goals People Ethics Model creation Training
Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical
4 Ecosystem Julia Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical DrWatson.jl ParameterSchedulers.jl Pluto.jl IJulia JupyterLab nteract VSCode Plots.jl (Viz) Gadfly.jl (Viz) Makie.jl (Viz - GPU) Flux.jl (ML) Knet.jl (ML/BL) MLJ.jl (ML) Mocha.jl (ML/DL) Tensorflow.jl (ML/DL wrapper) JuMP (optimization) Dataframes.jl ProgressMeters.jl
4 Ecosystem Python Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Workflow Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical Dask JupyterHub Binder Kubernetes papermill Dagster Airflow prefect scipy statsmodel JupyterLab nteract VSCode matplotlib seaborn altair plotly numpy scikit-learn pytorch tensorflow pandas PyJanitor dask datasette evidently bokeh panel voila dash python scripts napari geopandas feast keras fastai fairlearn
4 Ecosystem R Taxonomy Business Goals People Ethics Model creation
Training Testing Project Definition Data Collection Computation and Modeling Cleaning Labeling Validating Data Preparation Ingest Exploratory Analysis Descriptive statistics Visualization Evaluation Deploy at Scale Monitoring Share Results Revisit Goals Charts Reports Dashboard Web app Scheduling CI/CD Platform Metrics Comparison Satisfy goals Automation Infrastructure Model Observability Technical Business Ethical RStudio JupyterLab IRkernel ggplot tidyverse dplyr tidyr lubridate readr readxl googlesheets4 ggplot2 rmarkdown Shiny plumber purrr reticulate Keras Tensorflow sparklyr ropensci.org knitr forcats mlr3 CNTK theanos
Algorithmic Business Thinking (ABT) 5 Management Paul McDonagh-Smith MIT Sloan
School of Management https://mitsloan.mit.edu/faculty/directory/paul-mcdonagh-smith https://www.youtube.com/watch?v=bqtn2tYg-kw
@WillingCarol 25 Justine Dupont surfs the greatest wave of her
life in Nazaré, Portuga l © Rafael G. Riancho / Red Bull Content Poo l Got data at scale? Use open source tools.
web: noteable.io email: carol AT noteable.io twitter: @WillingCarol github: willingc
Thank you The Open Source Data Tooling Landscape Carol Willing VP of Learning Noteable
6 Additional Resources https://krzjoa.github.io/awesome-python-data-science/#/ https://github.com/FavioVazquez/ds-cheatsheets https://www.the-modeling-agency.com/crisp-dm.pdf https://github.com/academic/awesome-datascience