.. _model_data:

Data and Access
===============

Climate DT simulation data is made available via the DestinE Service
Platform. To learn how to access the impact-sector application
data, visit :ref:`impact_sector_apps`.

.. note::

   This guide covers Generation 2 data. More information about the
   Generation 1 data can be found
   `here <https://destine.ecmwf.int/climate-change-adaptation-digital-twin-climate-dt/>`_.

.. admonition:: Prerequisite to access the data

   To access Climate DT data, you first need to create an account in the  
   `Destination Earth Platform <https://platform.destine.eu/>`_. 
   Once your account is created, you will then need to apply for upgraded access.
   You can find instructions on how to apply for upgraded access, processing time, terms and conditions, and `user access policy
   <https://platform.destine.eu/support-pages/access-policy/>`_.

There are three ways to access Climate DT data on the HEALPix grid:

- :ref:`polytope_access` — retrieve data programmatically from any local
  or HPC environment using a Python client and `Jupyter notebooks <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/climate-dt>`_.
- :ref:`insula_access` — a browser-based Jupyter environment hosted within
  the DestinE Platform, with Polytope pre-configured and no local installation required.
- :ref:`hda_access` — a single API providing access to the full DestinE
  Data Portfolio, including Climate DT output.

The data structure and request keys are the same across all three methods.
See :ref:`data_structure` for a full description.

.. toctree::
   :maxdepth: 1
   :hidden:

   polytope
   insula
   hda
   data_structure
   data_catalogue
   ocean_model_levels
.. Polytope works by constructing a ``request`` that defines the required fields,
.. as shown in the examples below. Example notebooks are available in the
.. `Polytope examples repository <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/>`_.

.. There are two ways to run Polytope:

.. - **Your own machine** — Polytope can be run on any local or HPC machine,
..   requiring a Python environment and DESP authentication.
.. - **Insula Code** — a Jupyter environment hosted within DESP, with Polytope
..   pre-configured and no local setup required.

.. .. _data_access_any_machine:

.. Running Polytope on your own machine
.. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. To install ``polytope``, follow the installation instructions in the `Polytope examples repository <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/>`_.

.. Authentication with DESP credentials is required, which can be achieved by either:

.. 1. Downloading and running
..    `desp-authentication.py <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/desp-authentication.py>`_.
..    This creates a ``.polytopeapirc`` file in the home directory, which Polytope uses for authentication.

.. 2. Calling the Polytope client directly with a user email and key. Take care never
..    to commit these credentials to a public repository:

.. .. code::

..   from polytope.api import Client
..   client = Client(
..           address=<address where data is located>,
..           user_email='<YOUR EMAIL>',
..           user_key='<YOUR ECMWF API KEY>' or '<YOUR_DESP_KEY>'
..   )

..   # Optionally revoke previous requests
..   client.revoke("all")

..   # The data will be saved in the current working directory
..   files = client.retrieve("destination-earth", request, output_path)


.. Using Insula Code
.. ~~~~~~~~~~~~~~~~~

..    1. Log in to DESP: `Destination Earth Service Platform <https://platform.destine.eu/>`_.

..    2. Launch Insula: `Insula Code <https://code.insula.destine.eu/>`_.

..    3. Click "Go to service".

..    4. Insula Code will spawn a Jupyter session with example notebooks for data
..       exploration.

.. Select "polytope-lab" from the navigation menu on the left, then "climate-dt". If
.. "polytope-lab" is not visible, navigate back to the starting folder. The climate-dt
.. folder contains example Jupyter notebooks for use as a starting point.

.. As an example, open ``climate-dt-earthkit-fe-story-nudging.ipynb`` and select the
.. ``Polytope`` kernel from the upper right corner.

.. **Creating a new environment:**

.. If you require a fully custom Python environment, you can create one by
.. following the setup instructions described above.
.. This will create a new environment and register it as a Jupyter kernel,
.. which will then be available for selection within Insula Code.

.. **Updating an existing environment:**

.. Additional packages can be installed by running pip install in one of the first
.. cells and restarting the kernel. For example:

.. .. code-block::

..    pip install --upgrade --user earthkit
..    pip install --user kaleido


.. .. Using AQUA
.. .. ~~~~~~~~~~

.. .. The Application for Quality Assessment (`AQUA <https://github.com/DestinE-Climate-DT/AQUA>`__) is a Python package developed within Climate DT for real-time evaluation of climate model data. Based on dask and xarray, it provides lazy access to Climate DT data through Polytope, performs basic analysis (e.g. interpolation), and assesses model performance using a suite of diagnostic tools. AQUA can also be used to access Climate DT simulation data directly.

.. .. It supports GRIB, NetCDF and Zarr formats, and is available on PyPI in two components: `AQUA-core <https://pypi.org/project/aqua-core/>`_ (basic data access) and `AQUA-diagnostics <https://pypi.org/project/aqua-diagnostics/>`_ (diagnostics developed within the project).

.. .. A step-by-step setup guide and example notebooks demonstrating AQUA capabilities are available in the `Climate DT community resources <https://github.com/DestinE-Climate-DT/climatedt-community-resources/blob/main/example_aqua/aqua_installation_and_setup.md>`__.
.. .. .. _data_keys:

.. Data location and structure
.. ---------------------------

.. Both retrieval methods (local Polytope installation and Insula Code) use the same dictionary-based ``request`` syntax for retrieving data.
.. A comprehensive description of all relevant ``request`` keys and values is provided in :ref:`data_keys`, and an :ref:`example request <data_example_request>` is provided directly below.

.. .. _data_example_request:

.. **Example request**::

..    #Request for 1 month of 1 hourly data, 2m temperature
..    request = {
..         'activity': 'baseline',
..         'class': 'd1',
..         'dataset': 'climate-dt',
..         'experiment': 'hist',
..         'expver': '0001',
..         'model': 'ifs-fesom',
..         'generation': '2',
..         'realization': '1',
..         'resolution': 'high',
..         'date': '20140101/to/20140131',
..         'time': '0000/to/2300',
..         'stream': 'clte',
..         'type': 'fc',
..         'levtype': 'sfc',
..         'param': '167'
..    }

..    data_1hourly = earthkit.data.from_source(
..       "polytope",
..       "destination-earth",
..       request,
..       stream=False,
..       address="polytope.lumi.apps.dte.destination-earth.eu")

..    #Request for 1 month of monthly data, 2m temperature
..    request = {
..         'activity': 'baseline',
..         'class': 'd1',
..         'dataset': 'climate-dt',
..         'experiment': 'hist',
..         'expver': '0001',
..         'model': 'ifs-fesom',
..         'generation': '2',
..         'realization': '1',
..         'resolution': 'high',
..         'year' : '2014',
..         'month': '01',
..         'time' : '0000',
..         'stream': 'clmn',
..         'type': 'fc',
..         'levtype': 'sfc',
..         'param': '167'
..    }

..    data_monthly = earthkit.data.from_source(
..       "polytope",
..       "destination-earth",
..       request,
..       stream=False,
..       address="polytope.lumi.apps.dte.destination-earth.eu")


.. Two useful starting points are the `Climate DT community resources <https://github.com/DestinE-Climate-DT/climatedt-community-resources>`_ and the `Polytope examples repository <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/>`_. The examples in the community resources include a short introduction to Climate DT data processing and analysis with the :ref:`AQUA <aqua>` software, data access tools, and a cyclone tracking application. The community resources act as a collective point where you can find further information and links to other resources besides this user guide.

.. The table below summarises, for each group of simulations, the ``activity``
.. metadata used during the data query, the data bridge hosting the data, and the
.. available output resolutions on the HEALPix grid.

.. See :ref:`Data Keys <finding_the_data>` for a detailed explanation of the different keys.

.. +--------------------------------------------+-----------------------+------------------+---------------------------+
.. | Simulations                                | Activity              | Data Bridge      | HEALPix output resolution |
.. +============================================+=======================+==================+===========================+
.. | ICON – control, historical, SSP3-7.0 (r1)  | baseline / projections| LUMI             | high/standard             |
.. +--------------------------------------------+-----------------------+------------------+---------------------------+
.. | IFS-FESOM – control, historical,           | baseline / projections| LUMI             | high/standard             |
.. | SSP3-7.0 (r1)                              |                       |                  |                           |
.. +--------------------------------------------+-----------------------+------------------+---------------------------+
.. | IFS-NEMO – control, historical (r1),       | baseline / projections| MN5              | high/standard             |
.. | SSP3-7.0 (r1)                              |                       |                  |                           |
.. +--------------------------------------------+-----------------------+------------------+---------------------------+
.. | IFS-FESOM – storyline simulations (r1–r5)  | story-nudging         | MN5              | high/standard             |
.. |                                            |                       |                  |                           |
.. +--------------------------------------------+-----------------------+------------------+---------------------------+

.. For a full list of available simulations, including model details, temporal extent, and experiment IDs, see :ref:`models_simulations_intro`. The data portfolio, including available variables and output frequencies, is described in :ref:`data_catalogue`.

.. The **Generic State Vector (GSV)** ensures that all three coupled models produce output with consistent variable names and output frequencies. Beyond the model name, the primary differences between models when retrieving data are the vertical coordinate systems used for ocean variables, which remain heterogeneous across models. As the three models use different native grids, output is interpolated onto a common `HEALPix <https://healpix.sourceforge.io/>`_ mesh to provide a unified spatial representation. Output is provided at two spatial resolutions: ``high`` (H1024 for the 5 km runs, H512 for the 10 km run) and ``standard`` (H128, around 50 km). Atmospheric variables are available at hourly and monthly frequency, while ocean and sea-ice variables are available as daily and monthly means. Further details on the GSV concept and the Climate DT data infrastructure are described in `Doblas-Reyes et al. (2025) <https://doi.org/10.5194/egusphere-2025-2198>`_.

.. .. _finding_the_data:

.. Data Keys
.. ---------

.. Climate DT data stored in the DestinE data bridges and served via the DESP can be
.. requested using a combination of keys and values. The table below lists the
.. available keys and their meaning. Keys are case-insensitive. Details of which keys
.. correspond to each simulation are available in :ref:`data_keys`.

.. .. list-table::
..    :header-rows: 1
..    :widths: 15 35 50

..    * - Key
..      - Relevant values
..      - Description

..    * - activity
..      - story-nudging, baseline, projections
..      - The type of activity the simulation corresponds to

..    * - date
..      - YYYYMMDD (YYYYMMDD/to/YYYYMMDD)
..      - Date or dates for which the data is required (Valid only for clte stream)

..    * - experiment
..      - cont, hist (baseline or story-nudging), ssp3-7.0 (projections)
..      - Experiment name (in parenthesis, the associated activity related to each experiment)

..    * - levtype
..      - sfc, pl, o2d, o3d, hl, sol
..      - Level type depends on variable (sfc=surface, pl=pressure levels, o2d=ocean 2D,
..        o3d=ocean 3D, hl=height levels in the atmosphere, sol=soil levels)

..    * - level
..      - lev1/lev2/lev3
..      - Present only in the case of levtype=pl, o3d, hl or sol

..    * - model
..      - ifs-fesom, ifs-nemo, icon
..      - Available models in Climate DT

..    * - param
..      - paramid (167/235/...)
..      - `ParamID <https://codes.ecmwf.int/grib/param-db/>`_. The correspondence
..        between the param, the long name and the levtype is available in the
..        portfolio documentation

..    * - realization
..      - 1 or ensemble number
..      - Number of the ensemble member. See :ref:`data_keys` for options for each
..        of the simulations

..    * - resolution
..      - high or standard
..      - HEALPix resolution: high=closest to the model native resolution (H1024
..        for 5 km, or H512 for 10 km depending on simulation),
..        standard=interpolated to H128 (~50 km)

..    * - stream
..      - clte, clmn
..      - clte: Hourly instantaneous or daily mean values, clmn: monthly mean values

..    * - time
..      - HHMM or HHMM/HHMM/HHMM
..      - Select a specific time (Valid only for clte stream)

..    * - year
..      - YYYY
..      - To select a specific year (Valid only for the clmn stream)

..    * - month
..      - MM
..      - To select a specific month (Valid only for the clmn stream)

.. The following keys are fixed for all simulations in this documentation:

.. .. list-table::
..    :header-rows: 1
..    :widths: 15 35 50

..    * - Key
..      - Relevant values
..      - Description

..    * - class
..      - d1
..      - The data originates from Destination Earth

..    * - dataset
..      - climate-dt
..      - Selects the Climate DT data from Destination Earth

..    * - expver
..      - 0001
..      - Experiment version for operational simulations

..    * - type
..      - fc
..      - All Climate DT data uses type forecast

..    * - generation
..      - 2
..      - Climate DT generation

.. Check out the data catalogue:

.. .. toctree::
..    :maxdepth: 2

..    data_catalogue
..    ocean_model_levels