Earthcube 2020 - CMIP6 without the interpolation: Grid-native analysis with Pangeo in the cloud

The Pangeo project recently introduced large parts of the CMIP6 data archive into the cloud. This enables, for the first time, centralized, reproducible science of state-of-the-art climate simulations without the need to own large storage or a supercomputer as a user. The data itself however, still presents significant challenges for analysis, one of which is applying operations across many models. Two of the major hurdles are different naming/metadata between modeling centers and complex grid layouts, particularly for the ocean components of climate models. Common workflows in the past often included interpolating/remapping desired variables and creating new files, creating organizational burden, and increasing storage requirements. We will demonstrate two Pangeo tools which enable seamless calculation of common operations like vector calculus operators (grad, curl, div) and weighted averages/integrals across a wide range of CMIP6 models directly on the data stored in the cloud. cmip6_preprocessing provides numerous tools to unify naming conventions and parse grid information and metrics (like cell area). This information is used by xgcm to enable finite volume analysis on the native model grids. The combination of both tools facilitates fast analysis while ensuring a reproducible and accurate workflow.

Transcript

1. CMIP6 WITHOUT THE INTERPOLATION GRID-NATIVE ANALYSIS WITH PANGEO IN THE CLOUD JULIUS BUSECKE | RYAN ABERNATHEY
2. EYRING ET AL., BOM 2003. … Each “MIP” consists of many numerical models
3. SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰
4. SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset (~600TB)
5. SLOW AND STORAGE INTENSIVE TYPICAL WORKFLOW Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset (~600TB)
6. CURVILINEAR GRIDS DELANDMETER AND VAN SEBILLE, 2019 X0 , Y0 X1 , Y1 X2 , Y2 X3 , Y3 u0 u1 v0 v1 T0 F0 F1 F2 F3 x,y x y ⇠ ⌘ 0, 0 1, 0 1, 1 0, 1 F0 F1 F2 F3 U0 U1 V0 V1 T0 ⇠,⌘ (a) (b) ENES.ORG
7. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
8. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
9. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
10. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
11. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
12. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X Not too complicated but tedious and error prone
13. COMPUTING A DERIVATIVE XGCM DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
14. COMPUTING A DERIVATIVE XGCM Starting from xarray dataset DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
15. COMPUTING A DERIVATIVE XGCM Starting from xarray dataset Create a `grid` object DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X
16. DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X COMPUTING A DERIVATIVE XGCM
17. DELANDMETER AND VAN SEBILLE, 2019 u, v, T u v T & ` ' A ep g b j 1 j 1 j j j + 1 j + 1 j + 2 i 1 i 1 i i i + 1 i + 1 Axis:Y axis:X COMPUTING A DERIVATIVE XGCM Currently supported operations: difference interpolation cumulative sum min/max average integral cumulative integral derivative
18. CMIP6_PREPROCESSING
19. CMIP6_PREPROCESSING consistent renaming
20. CMIP6_PREPROCESSING consistent renaming
21. CMIP6_PREPROCESSING consistent renaming
22. CMIP6_PREPROCESSING consistent renaming
23. CMIP6_PREPROCESSING consistent renaming homogenize units
24. CMIP6_PREPROCESSING consistent renaming homogenize units consistent longitude convention
25. CMIP6_PREPROCESSING consistent renaming homogenize units xgcm grids consistent longitude convention
26. Sea Surface Temperature Gradient Magnitude
27. Sea Surface Temperature Gradient Magnitude
28. SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset
29. SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset
30. SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset cmip6_preprocessing
31. SPENDING TIME ON DISCOVERY NOT PREPROCESSING Download Data ⏰ Apply Analysis Discovery Regrid Data ⏰ Homogenize Data ⏰ Pangeo’s CMIP6 Google Cloud Public Dataset cmip6_preprocessing ❤
32. @JULIUSBUSECKE JBUSECKE@PRINCETON.EDU INTERESTED? JOIN US ON GITHUB! HTTPS://XGCM.READTHEDOCS.IO/ HTTPS://GITHUB.COM/JBUSECKE/CMIP6_PREPROCESSING HTTPS://GITHUB.COM/XGCM/XGCM Check out the full tutorial in the Earthcube notebook: HTTPS://GITHUB.COM/EARTHCUBE2020/EC20_BUSECKE_ETAL