Climate Scientist Ocean transport of Heat, Carbon Oxygen Impact of small scale processes on global climate variability. 🤓Developer/Data Nerd Pangeo CMIP6 Cloud Data xMIP/xGCM 🤝 Open Science Advocate Manager for Data and Computation - NSF- LEAP Lead of Open Research - m2lines 🙂↔💀 Screw you, Elon!
- Trying to integrate the lessons learned in the past years into the new ESGF infrastructure - Representative of many di ff erent CMIP6 Users and their struggles - Many people fi nd it incredibly hard to work with CMIP6 data, and I believe we should do as much as we can to make it as easy as possible for users. - If work is shifted downstream to each user, toil is ampli fi ed, and results become harder to reproduce. This hurts science. - This can often be overlooked when we have privileged access to the data and are focused on deeply technical details - I learned a lot about CMIP data, ESGF and how users work with that data in the past years! - Thank you for all your work! WHAT HAT AM I WEARING TODAY?
"Traditional Users" at large orgs like universities and labs - "New Users" are increasingly interested in accessing CMIP data - Industry (insurance, climate service providers, ...) - Local Gov, Non-Pro fi t, Defense ... - The "Random person" not belonging to any of the above INTRO “Universal Declaration of Human Rights.” United Nations. Accessed August 19, 2024. https://www.un.org/en/about-us/universal-declaration-of-human-rights.
I n s t i t u t i o n s c r e a t e “ D a t a F o r t r e s s e s * ” 8 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons *Coined by Chelle Gentemann ❌ Results not reproducible outside fortress ❌ Barrier to collaboration ❌ Inefficient / duplicative ❌ Can’t scale to future data needs ❌ Limits inclusion and knowledge transfer
I n s t i t u t i o n s c r e a t e “ D a t a F o r t r e s s e s * ” 8 Image credit: Moahim, CC BY-SA 4.0, via Wikimedia Commons *Coined by Chelle Gentemann ❌ Results not reproducible outside fortress ❌ Barrier to collaboration ❌ Inefficient / duplicative ❌ Can’t scale to future data needs ❌ Limits inclusion and knowledge transfer
Experiment with di ff erent approaches: - Convert netcdfs to zarr (LDEO) - Google Cloud Storage - Rewrites data but enhances user experience and performance - Replicate netcdfs on cloud storage (GFDL) - AWS - Works as a ESGF replica node THE PANGEO / ESGF CLOUD DATA WORKING GROUP Centralize this cache in the cloud!
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) • Cataloging and Testing inline Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) • Cataloging and Testing inline • Recipe Execution Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) • Cataloging and Testing inline • Recipe Execution • Orchestrating thousands of single recipes in dictionary Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) • Cataloging and Testing inline • Recipe Execution • Orchestrating thousands of single recipes in dictionary Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
identi f iers of a dataset (made up of .nc f iles) • dynamically request download urls • dynamic chunking (based on the size of each dataset) • Cataloging and Testing inline • Recipe Execution • Orchestrating thousands of single recipes in dictionary • Currently runs on Google Data f low but could run on any beam runner! Zarr Ingestion via Pangeo-Forge The anatomy of the CMIP6 recipe
A single data repository in the cloud serves all use cases Everybody rolls their own Custom Code Custom Code Custom Code University Lab Industry ❌ ✋🚫 Storage Provided by Google as Public Dataset
serves all use cases Collaborative and agile Research Inclusive Education on real climate data Fast Iteration - Lower Barrier of Entry Portable Methods and Results not just for Academia
serves all use cases Collaborative and agile Research Inclusive Education on real climate data Portable Methods and Results not just for Academia Fast Iteration - Lower Barrier of Entry
serves all use cases Collaborative and agile Research Portable Methods and Results not just for Academia Inclusive Education on real climate data Fast Iteration - Lower Barrier of Entry
serves all use cases Portable Methods and Results not just for Academia Collaborative and agile Research Inclusive Education on real climate data Fast Iteration - Lower Barrier of Entry
serves all use cases Collaborative and agile Research Inclusive Education on real climate data Fast Iteration - Lower Barrier of Entry Portable Methods and Results not just for Academia
- Simple CSV fi le with facets powering intake catalog - BUT this is workable and cataloging demands are very community speci fi c - Focusing on maximum performance and fl exibility could enable many di ff erent downstream cataloging approaches! - Usage Statistics - We have no usage statistics from Google - Could likely be remedied with a di ff erent agreement in the future - "Yet another way to access CMIP" - Users are already confused by the many options, +1 adds to the burden if docs are fragmented - Consolidated Docs and/or exposing ARCO cache data in the ESGF catalog would improve this!
point to same level as fi le download in the base API and instructions/documentation! - Rewriting the data is really hard! - REST API docs unclear + synchronous clients slow. - Lots of QC issues to work around. - So lets not do it! - Creating virtual (zarr) references preserves a lot of the user experience that people love - Will make it easier to enable high performance (cloud storage based) and/or specialized (rechunked) 'caches' of the o ffi cial data! WHAT WE LEARNED AND WHERE TO GO NEXT
idea: - Expose multiple logically connected fi les (e.g. timesteps) as a single dataset by combining metadata and referencing data chunks from each fi le - > User accesses the smallest scienti fi cally useful unit of data! - Kerchunk (requires fsspec/python) - References can be stored as json or parquet fi les - Works right now with s3 and http served CMIP6 fi les - Zarr V3 (e.g. implemented via icechunk) - Works with development branches - Would allow access - Aggregated NetCDF - ... - Each one of these approaches requires creating and hosting a few small additional fi les per dataset and pointing to a single fi le/url FILE AGGREGATION AND VIRTUAL ZARR REFS Kerchunk JSON example Xarray usage example
CEDA for ESGF-NG and will likely be a part of the new STAC catalog 🎉 - We can reference netcdfs over http and s3 - Open Questions: - When to generate/update references? - As part of the publishing or separate event? - Separate event could enable rebuilding di ff erent refs after publishing! - Can only publishers generate refs to 'their' fi les, or can I build references only? - Some caveats! - Additional QC/QA requirements: - Required: Consistent compression codecs and chunk size between fi les - Nice to have: Chunk size range checks FILE AGGREGATION AND VIRTUAL ZARR REFS Publish Node Reference Publish Node Reference Node ? Reference
icechunk) - Demonstrate work fl ow and test performance - Fully working examples of virtual zarr references from netcdf fi les on s3 and http - Using VirtualiZarr to combine fi les using xarray - Currently producing kerchunk references, but experimenting with icechunk - Please raise issues, reach out if you are interested. ONGOING WORK
of our lessons into the new ESGF Infrastructure - We can reproduce much of the UX from the Pangeo CMIP6 Cloud Data without rewriting to Zarr 🎉 - Requires: - ✅ Support for dataset level references/aggregations in the base data access API - ❓Support multiple references/aggregation methods? - ✅ Support replication to S3 storage - ❓QC on compression and chunking - Producing references using Virtualizarr can work as part of the publishing - This will signi fi cantly enhance the user experience and enable downstream e ff orts - Orgs can run cloud enabled replica nodes to enhance performance for parallel access (no commitment from ESGF) - Makes e ff orts to create specialized caches (rechunked native zarr, icechunk, ...) easier - Depending on QC/QA procedure these could be fully veri fi ed against published fi les too - Consolidated Docs 🙏! - Aside from the data access, having a single website to get info for new users, users who want to build downstream tools, maintainers of core infrastructure in one place would go a looooong way towards making access to climate data more inclusive and reduce time demand on ESGF maintainers. SUMMARY
jbusecke
kuri8ive
masatoto
hponka
sansan_randd
moda0
dmattsun
sgk
hiromu1996
udonda
eumesy
bermonpainter
panda_program
erikaheidi
paulrobertlloyd
honnibal
jmmastey
shpigford
chriscoyier
jrom
productmarketing
sonatard
palkan
![WHO AM I? M²LInES jbusecke juliusbusecke.com @JuliusBusecke @[email protected] @codeandcurrents.bsky.social 🌊](https://files.speakerdeck.com/presentations/d3fb9ec80fe043c29bab53607d998c5e/slide_1.jpg){kind=link}
