.. CACHE-A documentation master file, created by sphinx-quickstart on Mon Jul 15 21:35:50 2024. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Welcome to Data Cache Management Service documentation! ======================================================= .. toctree:: :maxdepth: 2 :caption: Contents: Cache-A ======= Service Overview ----------------- The Data Cache Management Service is a core service of the DestinE platform providing the following functionalities: 1. It allows to routinely retrieve data from the source, configure the ingestion and publication in chosen specific formats, and store it on the proper archive on the platform infrastructure. 2. It allows DestinE platform users to search, access, and download data via S3 and STAC protocols. Please note that S3 access is available only for DestinE service providers. .. image:: assets/context.png :alt: CACHE-A Service context :align: center | DestinE platform users can exploit cache data via STAC API. The SpatioTemporal Asset Catalog (STAC) specification provides a common structure for describing and cataloguing spatiotemporal assets. This specification is adopted in the context of Destination Earth as main interface for data searching, in conformity with the SpatioTemporal Asset Catalog (STAC) specification v1.0.0. .. image:: assets/stac.png :alt: CACHE-A High Level Architecture :align: center :width: 50% | More information about STAC API are available |here| | Collections ----------------- The following data collections are included in this service: * Copernicus Sentinel-1 SLC over Europe * Copernicus Sentinel-2 L2A (global) * Copernicus Sentinel 3 Level 2 (global) * Copernicus Sentinel-5P Level 2 (global) * Copernicus Copernicus Sentinel-2 Global Quarterly Mosaics Data providers are: * Copernicus Data Space Ecosystem (|CDSE|) * ONDA DIAS (|ONDA|) .. note:: The collections' items can be subjected to a rolling policy according to the data portfolio evolution and definition. Collections attributes ---------------------- Please find below the list of attributes provided for the STAC collections: .. list-table:: Collections' attributes :widths: 25 25 50 :header-rows: 1 * - Name - Description - Collections * - productIdentifier - Identifier of the collection item - all * - instrument - Sensor name - SENTINEL-1, SENTINEL-2, SENTINEL-3, SENTINEL-5P * - platformName - Platform Name - all * - productType - Product Type - all * - timeCoverageStart - Item Sensing Start - all * - timeCoverageStop - Item Sensing Stop - all * - provider - Data Source Provider - all * - providerDescription - Data Source Provider Description - all * - publicationDate - Publication Date on Cache-A - all * - format - Product Format - all * - geometry - Product Footprint - all * - eo:cloud_cover - Cloud Cover Percentage - SENTINEL-2, SENTINEL-3 Search for items by attributes ------------------------------ Currently, those attributes are supported: - bbox - datetime - item id (corresponding to the productIdentifier attribute) - eo:cloud_cover Search Items by bbox -------------------- Attribute bbox will list all products from a given collection within the Area of Interest (AOI). This attribute requires between 4 and 6 values (coordinates) where a comma separates each coordinate. To search for items by bbox: |https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?bbox=-80.673805,-0.52849,-78.060341,1.689651| Search Items by datetime ------------------------ Attribute datetime will list all products within a specified time interval. To search for items within specified datetime: |https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?datetime=2024-10-28T00:00:00Z/2024-10-31T00:00:00Z| Search Items by item id ----------------------- To search for products by item id: |https://cachea.destine.eu/stac/api/collections/GLOBAL-MOSAICS/items/Sentinel-2_mosaic_2024_Q1_13VEG_0_0| Search Items by eo:cloud_cover ------------------------------ The filter eo:cloud_cover will list all products with cloud cover percentage equal, greater, greater or equal, less, less or equal than the specified value: |https://cachea.destine.eu/stac/api/search?filter=eo:cloud_cover<20| This filter is applicable only to SENTINEL-2 and SENTINEL-3 collections User Quota Management ---------------------- Each registered user has a quota limitation for data that can be downloaded via STAC API. In particular, 2 parallel downloads are allowed per user. A dedicated S3 access to data is allowed for Service Providers with no quota restriction. .. |here| raw:: html here .. |CDSE| raw:: html CDSE .. |ONDA| raw:: html ONDA .. |https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?bbox=-80.673805,-0.52849,-78.060341,1.689651| raw:: html https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?bbox=-80.673805,-0.52849,-78.060341,1.689651 .. |https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?datetime=2024-10-28T00:00:00Z/2024-10-31T00:00:00Z| raw:: html https://cachea.destine.eu/stac/api/collections/SENTINEL-2/items?datetime=2024-10-28T00:00:00Z/2024-10-31T00:00:00Z .. |https://cachea.destine.eu/stac/api/collections/GLOBAL-MOSAICS/items/Sentinel-2_mosaic_2024_Q1_13VEG_0_0| raw:: html https://cachea.destine.eu/stac/api/collections/GLOBAL-MOSAICS/items/Sentinel-2_mosaic_2024_Q1_13VEG_0_0 .. |https://cachea.destine.eu/stac/api/search?filter=eo:cloud_cover<20| raw:: html https://cachea.destine.eu/stac/api/search?filter=eo:cloud_cover<20 Cache-B ======= Credentials and tools configuration ----------------------------------- In order to access the datasets in CacheB cache you need to obtain CacheB credentials and instruct your tools to use them when downloading the data. With your DestinE platform credentials at hand, download the `cacheb-authentication.py script `_. You first need a Python interpreter (>= 3.9) and the installation of some dependencies: .. code-block:: bash pip install requests conflator lxml Then run the script: .. code-block:: bash python cacheb-authentication.py Username: Password: The output will be a configuration snippet which follows the `netrc format `_. It will be similar to this: .. code-block:: none machine cacheb.dcms.destine.eu login anonymous password <…a-very-long-string…> Copy the configuration in the `.netrc` file inside your home directory (`~/.netrc` on Linux/MacOS, on Windows the file must be named `_netrc`). Create the file if doesn't exists yet. Now you should be able to access CacheB datasets with Xarray / Zarr. .. note:: CacheB credentials are valid for a limited period of time and need to be regenerated and reconfigured periodically by running the cacheb-authentication.py script. Access the Data --------------- After setting up your CacheB credentials, you can access CacheB datasets with Xarray / Zarr. Here is a list of available datasets: .. code-block:: python datasets = [ "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr", "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-o2d-v0.zarr", "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-pl-v0.zarr", "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-ICON-0001-high-sfc-v0.zarr", "https://cacheb.dcms.destine.eu/era5/reanalysis-era5-land-no-antartica-v0.zarr", "https://cacheb.dcms.destine.eu/era5/reanalysis-era5-single-levels-v0.zarr", ] Each dataset can be reached via its dedicated URL. You can use the following code snippet to access the data: .. code-block:: python # pip install xarray zarr fsspec requests aiohttp dask import xarray as xr data = xr.open_dataset( "https://cacheb.dcms.destine.eu/d1-climate-dt/ScenarioMIP-SSP3-7.0-IFS-NEMO-0001-high-sfc-v0.zarr", engine="zarr", storage_options={"client_kwargs": {"trust_env": "true"}}, chunks={}, ) Quota limits --------------- **Request Limit**: 500000 requests per user per month **Authentication**: Required for all data access **Recommended Approach**: we recommend carefully planning your data retrieval strategy, as downloading entire datasets can quickly consume your quota. Our tutorials provide guidance on how to efficiently access and work with large datasets while managing your request limits effectively.