Encoding and decoding

cf_xarray aims to support encoding and decoding variables using CF conventions not yet implemented by Xarray.

Compression by gathering

The “compression by gathering” convention could be used for either pandas.MultiIndex objects or pydata/sparse arrays.

MultiIndex

cf_xarray provides encode_multi_index_as_compress() and decode_compress_to_multi_index() to encode MultiIndex-ed dimensions using “compression by gethering”.

Here’s a test dataset

ds = xr.Dataset(
    {"landsoilt": ("landpoint", np.random.randn(4), {"foo": "bar"})},
    {
        "landpoint": pd.MultiIndex.from_product(
            [["a", "b"], [1, 2]], names=("lat", "lon")
        )
    },
)
ds

/tmp/ipykernel_2885/746089171.py:1: FutureWarning: the `pandas.MultiIndex` object(s) passed as 'landpoint' coordinate(s) or data variable(s) will no longer be implicitly promoted and wrapped into multiple indexed coordinates in the future (i.e., one coordinate for each multi-index level + one dimension coordinate). If you want to keep this behavior, you need to first wrap it explicitly using `mindex_coords = xarray.Coordinates.from_pandas_multiindex(mindex_obj, 'dim')` and pass it as coordinates, e.g., `xarray.Dataset(coords=mindex_coords)`, `dataset.assign_coords(mindex_coords)` or `dataarray.assign_coords(mindex_coords)`.
  ds = xr.Dataset(

<xarray.Dataset> Size: 128B
Dimensions:    (landpoint: 4)
Coordinates:
  * landpoint  (landpoint) object 32B MultiIndex
  * lat        (landpoint) object 32B 'a' 'a' 'b' 'b'
  * lon        (landpoint) int64 32B 1 2 1 2
Data variables:
    landsoilt  (landpoint) float64 32B -0.9892 0.3492 0.553 0.2199

xarray.Dataset

• Dimensions:
  • landpoint: 4
• Coordinates: (3)
  • landpoint
    (landpoint)
    object
    MultiIndex

    array([('a', 1), ('a', 2), ('b', 1), ('b', 2)], dtype=object)

  • lat
    (landpoint)
    object
    'a' 'a' 'b' 'b'

    array(['a', 'a', 'b', 'b'], dtype=object)

  • lon
    (landpoint)
    int64
    1 2 1 2

    array([1, 2, 1, 2])

• Data variables: (1)
  • landsoilt
    (landpoint)
    float64
    -0.9892 0.3492 0.553 0.2199

    foo :

    bar

    array([-0.98916643,  0.34923079,  0.55302866,  0.21993237])

• Indexes: (1)
  • landpoint
    lat
    lon
    PandasMultiIndex

    PandasIndex(MultiIndex([('a', 1),
                ('a', 2),
                ('b', 1),
                ('b', 2)],
               name='landpoint'))

• Attributes: (0)

First encode (note the "compress" attribute on the landpoint variable)

encoded = cfxr.encode_multi_index_as_compress(ds, "landpoint")
encoded

<xarray.Dataset> Size: 96B
Dimensions:    (landpoint: 4, lat: 2, lon: 2)
Coordinates:
  * lat        (lat) object 16B 'a' 'b'
  * lon        (lon) int64 16B 1 2
  * landpoint  (landpoint) int64 32B 0 1 2 3
Data variables:
    landsoilt  (landpoint) float64 32B -0.9892 0.3492 0.553 0.2199

xarray.Dataset

• Dimensions:
  • landpoint: 4
  • lat: 2
  • lon: 2
• Coordinates: (3)
  • lat
    (lat)
    object
    'a' 'b'

    array(['a', 'b'], dtype=object)

  • lon
    (lon)
    int64
    1 2

    array([1, 2])

  • landpoint
    (landpoint)
    int64
    0 1 2 3

    compress :

    lat lon

    array([0, 1, 2, 3])

• Data variables: (1)
  • landsoilt
    (landpoint)
    float64
    -0.9892 0.3492 0.553 0.2199

    foo :

    bar

    array([-0.98916643,  0.34923079,  0.55302866,  0.21993237])

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index(['a', 'b'], dtype='object', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([1, 2], dtype='int64', name='lon'))

  • landpoint
    PandasIndex

    PandasIndex(Index([0, 1, 2, 3], dtype='int64', name='landpoint'))

• Attributes: (0)

At this point, we can write encoded to a CF-compliant dataset using xarray.Dataset.to_netcdf() for example. After reading that file, decode using

decoded = cfxr.decode_compress_to_multi_index(encoded, "landpoint")
decoded

<xarray.Dataset> Size: 128B
Dimensions:    (landpoint: 4)
Coordinates:
  * landpoint  (landpoint) object 32B MultiIndex
  * lat        (landpoint) object 32B 'a' 'a' 'b' 'b'
  * lon        (landpoint) int64 32B 1 2 1 2
Data variables:
    landsoilt  (landpoint) float64 32B -0.9892 0.3492 0.553 0.2199

xarray.Dataset

• Dimensions:
  • landpoint: 4
• Coordinates: (3)
  • landpoint
    (landpoint)
    object
    MultiIndex

    array([('a', 1), ('a', 2), ('b', 1), ('b', 2)], dtype=object)

  • lat
    (landpoint)
    object
    'a' 'a' 'b' 'b'

    array(['a', 'a', 'b', 'b'], dtype=object)

  • lon
    (landpoint)
    int64
    1 2 1 2

    array([1, 2, 1, 2])

• Data variables: (1)
  • landsoilt
    (landpoint)
    float64
    -0.9892 0.3492 0.553 0.2199

    foo :

    bar

    array([-0.98916643,  0.34923079,  0.55302866,  0.21993237])

• Indexes: (1)
  • landpoint
    lat
    lon
    PandasMultiIndex

    PandasIndex(MultiIndex([('a', 1),
                ('a', 2),
                ('b', 1),
                ('b', 2)],
               name='landpoint'))

• Attributes: (0)

We roundtrip perfectly

ds.identical(decoded)

True

Sparse arrays

This is unsupported currently but a pull request is welcome!