Welcome to Data Cache Management Service documentation!

Cache-A

Service Overview

The Data Cache Management Service is a core service of the DestinE platform providing the following functionalities:

  1. It allows to routinely retrieve data from the source, configure the ingestion and publication in chosen specific formats, and store it on the proper archive on the platform infrastructure.

  2. It allows DestinE platform users to search, access, and download data via S3 and STAC protocols.

CACHE-A Service context

DestinE platform users can exploit cache data via STAC API. The SpatioTemporal Asset Catalog (STAC) specification provides a common structure for describing and cataloguing spatiotemporal assets. This specification is adopted in the context of Destination Earth as main interface for data searching, in conformity with the SpatioTemporal Asset Catalog (STAC) specification v1.0.0.

CACHE-A High Level Architecture

More information about STAC API are available here

Collections

The following data collections are included in this service:

  • Copernicus Sentinel-1 SLC over Europe

  • Copernicus Sentinel-2 L2A (global)

  • Copernicus Sentinel 3 Level 2 (global)

  • Copernicus Sentinel-5P Level 2 (global)

  • Copernicus Copernicus Sentinel-2 Global Quarterly Mosaics

Data providers are:

  • Copernicus Data Space Ecosystem (CDSE)

  • ONDA DIAS (ONDA)

Cache-B

Credentials and tools configuration

In order to access the datasets in CacheB cache you need to obtain CacheB credentials and instruct your tools to use them when downloading the data.

With your DestinE platform credentials at hand, download the cacheb-authentication.py script.

You first need a Python interpreter (>= 3.9) and the installation of some dependencies:

pip install requests conflator lxml

Then run the script:

python cacheb-authentication.py
Username: <Enter your DestinE username here>
Password: <Type your DestinE password here>

The output will be a configuration snippet which follows the netrc format.

It will be similar to this:

machine cacheb.dcms.destine.eu
    login anonymous
    password <…a-very-long-string…>

Copy the configuration in the .netrc file inside your home directory (~/.netrc on Linux/MacOS, on Windows the file must be named _netrc). Create the file if doesn’t exists yet.

Now you should be able to access CacheB datasets with Xarray / Zarr.

Warning

CacheB credentials are valid for a limited period of time and need to be regenerated and reconfigured periodically by running the cacheb-authentication.py script.

Access the Data

After setting up your CacheB credentials, you can access CacheB datasets with Xarray / Zarr.

Here is a list of available datasets:

datasets = [
 "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr",
 "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-o2d-v0.zarr",
 "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-pl-v0.zarr",
 "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-ICON-0001-high-sfc-v0.zarr",
 "https://cacheb.dcms.destine.eu/era5/reanalysis-era5-land-no-antartica-v0.zarr",
 "https://cacheb.dcms.destine.eu/era5/reanalysis-era5-single-levels-v0.zarr",
]

Each dataset can be reached via its dedicated URL.

You can use the following code snippet to access the data:

# pip install xarray zarr fsspec requests aiohttp dask
import xarray as xr
data = xr.open_dataset(
    "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr",
    engine="zarr",
    storage_options={"client_kwargs": {"trust_env": "true"}},
    chunks={},
)

Be careful, the amount of data you can download is limited. Do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close.