Welcome to Data Cache Management Service documentation!
Cache-A
Service Overview
The Data Cache Management Service is a core service of the DestinE platform providing the following functionalities:
It allows to routinely retrieve data from the source, configure the ingestion and publication in chosen specific formats, and store it on the proper archive on the platform infrastructure.
It allows DestinE platform users to search, access, and download data via S3 and STAC protocols. Please note that S3 access is available only for DestinE service providers.
DestinE platform users can exploit cache data via STAC API. The SpatioTemporal Asset Catalog (STAC) specification provides a common structure for describing and cataloguing spatiotemporal assets. This specification is adopted in the context of Destination Earth as main interface for data searching, in conformity with the SpatioTemporal Asset Catalog (STAC) specification v1.0.0.
More information about STAC API are available here
Collections
The following data collections are included in this service:
Copernicus Sentinel-1 SLC over Europe
Copernicus Sentinel-2 L2A (global)
Copernicus Sentinel 3 Level 2 (global)
Copernicus Sentinel-5P Level 2 (global)
Copernicus Copernicus Sentinel-2 Global Quarterly Mosaics
Data providers are:
Note
The collections’ items can be subjected to a rolling policy according to the data portfolio evolution and definition.
Collections attributes
Please find below the list of attributes provided for the STAC collections:
Name |
Description |
Collections |
|---|---|---|
productIdentifier |
Identifier of the collection item |
all |
instrument |
Sensor name |
SENTINEL-1, SENTINEL-2, SENTINEL-3, SENTINEL-5P |
platformName |
Platform Name |
all |
productType |
Product Type |
all |
timeCoverageStart |
Item Sensing Start |
all |
timeCoverageStop |
Item Sensing Stop |
all |
provider |
Data Source Provider |
all |
providerDescription |
Data Source Provider Description |
all |
publicationDate |
Publication Date on Cache-A |
all |
format |
Product Format |
all |
geometry |
Product Footprint |
all |
eo:cloud_cover |
Cloud Cover Percentage |
SENTINEL-2, SENTINEL-3 |
Search for items by attributes
Currently, those attributes are supported:
bbox
datetime
item id (corresponding to the productIdentifier attribute)
eo:cloud_cover
Search Items by bbox
Attribute bbox will list all products from a given collection within the Area of Interest (AOI). This attribute requires between 4 and 6 values (coordinates) where a comma separates each coordinate.
To search for items by bbox:
Search Items by datetime
Attribute datetime will list all products within a specified time interval.
To search for items within specified datetime:
Search Items by item id
To search for products by item id:
Search Items by eo:cloud_cover
The filter eo:cloud_cover will list all products with cloud cover percentage equal, greater, greater or equal, less, less or equal than the specified value:
https://cachea.destine.eu/stac/api/search?filter=eo:cloud_cover<20
This filter is applicable only to SENTINEL-2 and SENTINEL-3 collections
User Quota Management
Each registered user has a quota limitation for data that can be downloaded via STAC API. In particular, 2 parallel downloads are allowed per user.
A dedicated S3 access to data is allowed for Service Providers with no quota restriction.
Cache-B
Credentials and tools configuration
In order to access the datasets in CacheB cache you need to obtain CacheB credentials and instruct your tools to use them when downloading the data.
With your DestinE platform credentials at hand, download the cacheb-authentication.py script.
You first need a Python interpreter (>= 3.9) and the installation of some dependencies:
pip install requests conflator lxml
Then run the script:
python cacheb-authentication.py
Username: <Enter your DestinE username here>
Password: <Type your DestinE password here>
The output will be a configuration snippet which follows the netrc format.
It will be similar to this:
machine cacheb.dcms.destine.eu
login anonymous
password <…a-very-long-string…>
Copy the configuration in the .netrc file inside your home directory (~/.netrc on Linux/MacOS, on Windows the file must be named _netrc). Create the file if doesn’t exists yet.
Now you should be able to access CacheB datasets with Xarray / Zarr.
Warning
CacheB credentials are valid for a limited period of time and need to be regenerated and reconfigured periodically by running the cacheb-authentication.py script.
Access the Data
After setting up your CacheB credentials, you can access CacheB datasets with Xarray / Zarr.
Here is a list of available datasets:
datasets = [
"https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr",
"https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-o2d-v0.zarr",
"https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-pl-v0.zarr",
"https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-ICON-0001-high-sfc-v0.zarr",
"https://cacheb.dcms.destine.eu/era5/reanalysis-era5-land-no-antartica-v0.zarr",
"https://cacheb.dcms.destine.eu/era5/reanalysis-era5-single-levels-v0.zarr",
]
Each dataset can be reached via its dedicated URL.
You can use the following code snippet to access the data:
# pip install xarray zarr fsspec requests aiohttp dask
import xarray as xr
data = xr.open_dataset(
"https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr",
engine="zarr",
storage_options={"client_kwargs": {"trust_env": "true"}},
chunks={},
)
Be careful, the amount of data you can download is limited. Do not try to download an entire dataset, chances are that you will exceed your quota well before you come even close.