chunked, compressed, N-dimensional arrays. zarr.readthedocs.io • Zarr began in 2016 as a storage library for the scientific python ecosystem. • Integrates closely with Numpy, Xarray, and Dask. zarr-developers/zarr-python
chunked, compressed, N-dimensional arrays. zarr-developers/zarr-python constantinpape/z5 bcdev/jzarr meggart/Zarr.jl gzuidhof/zarr.js/ Open spec (V2) has led to implementations in several language.
chunked, compressed, N-dimensional arrays. Zarr Group: .zgroup .zattrs .zarray .zattrs Zarr Array: foo 0.0 0.1 2.0 1.0 1.1 2.1 Zarr can be stored in any key-value store. • Directory Store • ZipFile Store • Cloud object storage (e.g. S3, GCS) • Database (e.g. Redis, MongoDB) • Get creative—Zarr is designed to be hackable!
Model Intercomparison Project: latest climate model projections from modeling centers around the world • Pangeo project worked with Google Cloud to mirror data from ESGF • Zarr was chosen as format because of its interoperability with cloud object storage • 600 TB and growing https://cloud.google.com/blog/products/data-analytics/new-climate-model-data-now-google-public-datasets
of Zarr V2 spec as an OGC Community Standard: “Community Standard serves to bring de facto standards from the larger geospatial community to be a stable reference point that can normatively referenced by governments and other organizations.” • Zarr is generic (comparable to HDF5) but can serve as a base format for other OGC standards, e.g. web coverage service. • Unidata will soon release a version of NetCDF with support for Zarr as an underlying storage container. • Zarr project received and EOSS Grant from the Chan Zuckerberg Institute to support development of V3 spec. Get involved @ zarr-developers/zarr-specs/