e n c e V i s i o n 2 https://earthdata.nasa.gov/esds/open-science for 👩🔬 in everyone_in_the_world: for 📄 in all_scientific_knowledge: 👩🔬.verify(📄) discovery = 👩🔬.extend(📄) This would transform the 🌎 by allowing all of humanity to participate in the scienti fi c process. What are the barriers to realizing this vision?
e n c e V i s i o n 3 https://earthdata.nasa.gov/esds/open-science for 👩🔬 in everyone_in_the_world: for 👨🔬 in everyone_else_in_the_world: 📄 = (👩🔬 + 👨🔬).collaborate() This would transform the 🌎 by allowing all of humanity to participate in the scienti fi c process. What are the barriers to realizing this vision?
Findable, Accessible, Interoperable, Reusable FAIR is great. Nobody disagrees with FAIR. But making data-intensive scienti fi c work fl ows FAIR is easier said than done. FAIR does not specify the protocols, technologies, or infrastructure. FAIR is not a platform.
gure/Earth_Data_Cube/4822930/2 Equations Big Data Big Data 💡 Insights 💡 Understanding 💡 Predictions • known computational problem • optimized, scalable algorithm • standard architecture (HPC / MPI) • open-ended problem • exploratory analysis • “human in the loop” • visualization needed • highly varied computational patterns / algorithms • no standard architecture
is something you can build on—speci fi cally, new scienti fi c discoveries and new translational applications. Let’s call these projects. 📄 • For open science to take off at a global scale, everyone in the world needs access to the platform (like Facebook) • This is why we are excited about cloud, but cloud as-is (e.g. AWS) is not itself an open-science platform. • Does the open science platform need to be open? 🤔 C l a i m : O p e n S c i e n c e n e e d s a P l at f o r m 6 Infrastructure Platform Open- Science Project Platform Open- Science Project Open- Science Project Open- Science Project
The status quo of data-intensive scienti fi c infrastructure 10 mins What elements of an open science platform exist today? 10 mins Pangeo Forge: an ETL service for Open Science 10 mins Where are things headed?
v e S c i e n c e I n f r a s t r u c t u r E : T h e S tat u s Q U O * 9 Personal Laptop Group Server Department Cluster Agency Supercomputer more storage, more CPU, more security, more constraints
at I n f r a s t r u c t u r e C a n W e R e ly o n ? 10 ✅ UNIX operating system 💾 ✅ Files / POSIX fi lesystems 🗂 ✅ Programming languages: C, FORTRAN, Python, R, Julia ✅ Terminal access ⚠ Batch queuing system ⏬ HPC only ⚠ The internet 🌐 Not on HPC nodes! ⚠ Globus for fi le transfer 🔄 Not supported everywhere ❌ High level data services, APIs, etc. Virtually unknown in my world
a d ” M o d e l 11 a) file-based approach step 1 : dow nload step 2: clean / organize ` file file file b) database / api approach record file file file local disk query files step 3: analyze
n l o a d ” M o d e l a) file-based approach step 1 : dow nload step 2: clean / organize ` file file file b) database / api approach record file file file local disk query files step 3: analyze
n l o a d ” M o d e l a) file-based approach step 1 : dow nload step 2: clean / organize ` file file file b) database / api approach record file file file local disk query files step 3: analyze
n l o a d ” M o d e l a) file-based approach step 1 : dow nload step 2: clean / organize ` file file file b) database / api approach record file file file local disk query files step 3: analyze
n l o a d ” M o d e l a) file-based approach step 1 : dow nload step 2: clean / organize ` file file file b) database / api approach record file file file local disk query files step 3: analyze
I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 16 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons *Coined by Chelle Gentemann
I n s t i t u t i o n s c r e at e “ D ata F o r t r e s s e s * ” 17 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons Data *Coined by Chelle Gentemann # step 1: open data open(“/some/random/files/on/my/cluster”) # step 2: do mind-blowing AI!
exchange creates lots of work for individual scientists (downloading, organizing, cleaning). Most fi le-based datasets are a mess— even simulation output. 😫 Yet the hard work of data wrangling is rarely collaborative. Outputs are not reusable and can’t be shared. 💰 Doing data-intensive science requires either expensive local infrastructure or access to a big agency supercomputer. This really limits participation. 🏰 Data intensive science is locked inside data fortresses. Limiting access to outsiders is a feature, not a bug. Restricts collaboration and reproducibility! ❄ Each fortress is a special snow fl ake. Code developed in one will not run inside another. p r o b l e m s w i t h t h e S tat u s Q u o 18
Overhead policies, purchasing challenges, lack of IT support, etc. • Cloud is too complicated! The services offered are not useful to scientists: An extra layer of science-oriented services must be developed • Europe basically forbids scientists from using US-based cloud providers • Not much has changed for university scientists since 2017 W h at A b o u t C l o u d ? 19
l l a b o r at i o n 21 Some of the most impactful services used in open science…. These are all proprietary SaaS (Software as a Service) applications. They may use open standards, but they are not open source. We (or our institutions) have no problem paying for them.
cloud infrastructure • Define science questions • Use software / infrastructure • Identify bugs / bottlenecks • Provide feedback to developers • Contribute widely the the open source scientific python ecosystem • Maintain / extend existing libraries, start new ones reluctantly • Solve integration challenges • Deploy interactive analysis environments • Curate analysis-ready datasets • Platform agnostic Agile development 👩💻 T h e Pa n g e o C o m m u n i t y P r o c e s s 22
o u d - N a i v e S ta c k 24 Cloud-optimized storage for multidimensional arrays. Flexible, general-purpose parallel computing framework. High-level API for analysis of multidimensional labelled arrays. Kubernetes Object Storage Rich interactive computing environment in the web browser. xgcm xr f xhistogram gcm-filters climpred Cloud Infrastructure Domain specific packages Etc.
JupyterHub in the cloud with Xarray, Dask, etc., connected to ARCO data sources • But there are many distinct deployments of this platform - dozens of similar yet distinct JupyterHubs with different con fi gurations, environments, capabilities, etc ➡ Sharing projects between these hubs is still very hard • Deploying hubs generally requires DevOps work (billed a developer time or contractor services). There is no “Pangeo as a Service” • Getting data into the cloud in ARCO format is hard and full of toil L i m i tat i o n s o f t h e Pa n g e o A p p r o a c h 26
ata S ta c k 28 https://future.com/emerging-architectures-modern-data-infrastructure/ • In the past 5 years, a platform has emerged for enterprise data science called the Modern Data Stack • The MDS is centered around a “data lake” or “data warehouse” • Different platform elements are provided by different SaaS companies; integration through standards and APIs • No one in science uses any of this stuff
r s 31 • Charles Stern (Columbia / LDEO) • Joe Hamman (CarbonPlan) • Anderson Banihirwe (CarbonPlan) • Rachel Wegener (U. Maryland) • Chiara Lepore (GRO Intelligence) • Sean Harkins (Development Seed) • Aimee Barciauskas (Development Seed) • Alex Merose (Google Research) • Tom Augspurger (Microsoft) • Martin Durant (Anaconda) • Many recipe contributors Funding: NSF Earthcube Program ($1.5M for 3 years)
need for tedious homogenizing / cleaning steps • Curated and cataloged A R C O D ata 32 Analysis Ready, Cloud Optimzed ZRUN DERXW ZHUHêVDWLVĆHGëRUEHWWHU How a Data Scientist Spends Their Day +HUHèVZKHUHWKHSRSXODUYLHZRIGDWDVFLHQWLVWVGLYHUJHVSUHWW\VLJQLĆFDQWO\IURP ZHWKLQNRIGDWDVFLHQWLVWVEXLOGLQJDOJRULWKPVH[SORULQJGDWDDQGGRLQJSUHGLFWLY actually not what they spend most of their time doing, however. $V\RXFDQVHHIURPWKHFKDUWDERYHRXWRIHYHU\GDWDVFLHQWLVWVZHVXUYH\HG PRVWWLPHFOHDQLQJDQGRUJDQL]LQJGDWD<RXPD\KDYHKHDUGWKLVUHIHUUHGWRDVêG FRPSDUHGWRGLJLWDOMDQLWRUZRUN(YHU\WKLQJIURPOLVWYHULĆFDWLRQWRUHPRYLQJFRP Data scientist job satisfaction 60% 19% 9% 4% 5% 3% Building training sets: 3% Cleaning and organizing data: 60% Collecting data sets; 19% Mining data for patterns: 9% 5HĆQLQJDOJRULWKPV Other: 5% ,!;&!;!9$-'2ধ9;996'2&;,'1 How do data scientists spend their time? Crowd fl ower Data Science Report (2016) What is “Analysis Ready”?
Supports lazy access and intelligent subsetting • Integrates with high-level analysis libraries and distributed frameworks A R C O D ata 33 Analysis Ready, Cloud Optimzed What is “Cloud Optimized”?
ARCO Data is Hard! Domain Expertise: How to fi nd, clean, and homogenize data Tech Knowledge: How to ef fi ciently produce cloud-optimized formats Compute Resources: A place where to stage and upload the ARCO data Analysis Skills: To validate and make use of the ARCO data. To produce useful ARCO data, you must have: Data Scientist 😩
i t t o M a k e A R C O D ata? 36 Data providers are concerned with preservation and archival quality. Scientists users know what they need to make the data analysis-ready.
package for describing and running data pipelines (“recipes”) Cloud platform for automatically executing recipes stored in GitHub repos. https://github.com/pangeo-forge/pangeo-forge-recipes https://pangeo-forge.org/
R e c i p e s 40 FilePattern StorageConfig Recipe Executor Describes where to fi nd the source fi les which are the inputs to the recipe Describes where to store the outputs of our recipe A complete, self- contained representation of the pipeline Knows how to run the recipe. https://pangeo-forge.readthedocs.io/
C l o u d 41 Feedstock Contains the code and metadata for one or more Recipes Bakery https://pangeo-forge.org/ Storage Runs the recipes in the cloud using elastic scaling clusters GCS
l a b o r at i v e D ata C u r at i o n 42 Feedstock 🤓 Data User 🤓 Data Producer 🤓 Data Manager These data look weird… …Oh the metadata need an update. Ok I’ll make a PR to the recipe.
pangeo-forge.org 💣 Our recipes often contain 10000+ tasks. We are hitting the limits on Prefect as a work fl ow engine. Currently refactoring to move to Apache Beam. 😫 Data has lots of edge cases! This is really hard. 🌎 But we remain very excited about the potential of Pangeo Forge to transform how scientists interact with data. C u r r e n t s tAT U S 44
f C l o u d N at i v e S c i e n t i f i c D ata A n a ly t i c s 46 1. Analysis-Ready, Cloud-Optimized Data 2. Data-Proximate Computing 3. Elastic Distributed Processing compute node compute node compute node compute node compute node compute node compute node compute node compute node Compute Environment
S t o r a g e a n d C o m p u t e 47 Storage costs are steady. Data provider pays for storage costs. May be subsidized by cloud provider. (Thanks AWS, GCP, Azure! Or can live outside the cloud (e.g. Wasabi, OSN) Compute costs for interactive data analysis are bursty. Can take advantage of spot pricing Multi-tenancy: We can all use the same stack, but each institution pays for its own users. This is completely di ff erent from the status quo infrastructure!
e P l at f o r m 48 “Analysis Ready, Cloud Optimized Data” Cleaned, curated open-access datasets available via high-performance globally available strorage system “Elastic Scaling” Automatically provision many computers on demand to accelerate big data processing. “Data Proximate Computing” Bring analysis to the data using any open-source data science language. Generic cloud object storage Generic cloud computing Data Library Compute Environment Expert Analyst Direct Access via Jupyter Non-Technical User Access via apps / dashboards / etc. Runs on any modern cloud-like platform or on premises data center Web front ends Downstream third-party services / applications https://doi.org/10.1109/MCSE.2021.3059437
x t e n s i b l e M o d e l 49 Compute Environment Data Library Data Library Data Library Data Library Data Library Compute Environment Compute Environment Front-end Services
• Cultivate community-operated SaaS: e.g. Wikipedia, Conda Forge, Binder, 2i2c Hubs, Pangeo Forge • We probably need a mix of both C h a l l e n g e : h o w c a n w e d e l i v e r a n o p e n s c i e n c e p l at f o r m i n a s c a l a b l e , s u s ta i n a b l e w ay ? 50
gravity is the ability of a body of data to attract applications, services and other data." - Dave McCrory NASA (200 PB) NOAA BDP ASDI (incl. CMIP6) NCAR Datasets etc… Planetary Computer NOAA BDP Earth Engine NOAA BDP Descartes Pangeo SentinelHub Climate Change Atmosphere Marine ECMWF DOE XSEDE HECC NCAR
rabernat
bob3bob3
rmaruy
.jpg){kind=avatar}
matlantis
brainpadpr
wemeet
miura55
aronwalsh
hurutoriya
gpeyre
konakalab
02hattori11sat03
astronomyhouston
lauravandoore
eileencodes
bermonpainter
marcelosomers
aleyda
stephaniewalter
danielanewman
dougneiner
kevinliebholz
mraible
jrom
