.. _polytope_access:

Polytope
========

Polytope is a data extraction service for retrieving Climate DT data from the DestinE data
bridges. It supports both full global field retrieval on the HEALPix grid and
feature-based extraction, allowing users to request n-dimensional subsets
(polytopes) from a datacube without downloading entire fields.

Rather than reading full fields and post-processing them (as with bounding-box
extraction), Polytope retrieves only the precise data requested — for example, a
flight path, a time series, or a geographic region — significantly reducing I/O
and retrieval time.

Polytope follows a client–server architecture. The client defines and submits data
extraction requests via a REST API; the Polytope server, which has direct access to
the Field Database (FDB), handles the computational geometry and retrieves only the
required data subsets.

More detailed information on the polytope service is available in the `polytope documentation <https://polytope.readthedocs.io/en/latest/>`_.

.. hint::

   A comprehensive collection of ready-to-run examples is available in the
   `polytope-examples <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/climate-dt>`_
   repository.

   In particular, the Climate DT Explorer allows to lazily browse the Climate DT data in Python, using ``earthkit-data`` under the hood:

   - Lazily browse the full `monthly catalogue <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/climate-dt/explorer/03_lazy_browse_portfolio.ipynb>`_
   - Lazily browse the full `hourly catalogue <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/climate-dt/explorer/04_lazy_browse_portfolio_hourly.ipynb>`_
   - Search for variables using the `variable lookup tool <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/climate-dt/explorer/05_variable_lookup.ipynb>`_

   See the :ref:`data access page <model_data>` for how to perform analyses from any local or HPC environment.

   For the data request syntax, see :ref:`data_structure`.



.. Polytope vs. Harmonised Data Access (HDA)
.. ------------------------------------------

.. Polytope is dedicated to Digital Twin data, whereas the
.. :ref:`HDA <hda_access>` provides access to the full DestinE Data Portfolio.
.. Although Digital Twin data is also available through the HDA, it does not
.. support feature-based extraction. 


Prerequisites and access methods
---------------------------------

.. admonition:: Prerequisite

   A DestinE Platform account with upgraded access is required. `Apply for upgraded
   access here <https://platform.destine.eu/access-policy-upgrade/>`_.

There are two ways to access Polytope:

- **Your own machine** — any local or HPC environment with Python3 (>= 3.10) and a DestinE Platform
  authentication token. Requires installation of additional Python libraries.
  See :ref:`polytope_installation`.
- **Insula Code** — a browser-based Jupyter environment hosted within the DestinE Platform, with
  all required libraries pre-installed. See :ref:`insula_access`.


.. _polytope_installation:

Installation
------------

For a minimal setup, install ``earthkit-data``, ``polytope-client`` and ``covjsonkit``:

.. code-block:: bash

   python3 -m pip install earthkit-data polytope-client covjsonkit

For a full installation that includes additional dependencies for working with the
retrieved data, follow the setup instructions in the README of the
`polytope-examples repository <https://github.com/destination-earth-digital-twins/polytope-examples/tree/main/climate-dt>`_.


Authentication
--------------

A DestinE Platform authentication token is required to use Polytope. There are two ways to
provide credentials:

1. **Recommended — configuration file:** run the
   `desp-authentication.py <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/desp-authentication.py>`_
   script from the polytope-examples repository. This writes a ``.polytopeapirc``
   file to your home directory, which both ``earthkit-data`` and ``polytope-client``
   will pick up automatically.

2. **Direct credentials (polytope-client only):** pass your email and API key
   directly when constructing the client (see :ref:`polytope_client_example`).
   This option is not available with the ``earthkit-data`` client.

   .. warning::

      Never commit API keys or credentials to a public repository.


How to extract data
-------------------

The simplest approach is to use the
`earthkit-data <https://github.com/ecmwf/earthkit-data>`_ library (part of the
`earthkit <https://github.com/ecmwf/earthkit>`_ meta-package), which simplifies
both authentication and request syntax via its ``from_source()`` method.
See :ref:`earthkit_data_example` for a sample request.

Under the hood, ``earthkit-data`` uses the lower-level ``polytope-client`` library
to interact with the Polytope service. For advanced use cases or fine-grained
control, the
`polytope-client <https://github.com/ecmwf/polytope-client>`_ package can be used
directly. See :ref:`polytope_client_example` for a sample request.


.. _earthkit_data_example:

Example: earthkit-data
----------------------

The ``from_source()`` method accepts a Polytope source, a collection name, and a
request dictionary. Full method documentation is available in the
`earthkit-data polytope reference <https://earthkit-data.readthedocs.io/en/stable/guide/sources.html#polytope>`_.
The `Climate DT Explorer notebooks <https://github.com/destination-earth-digital-twins/polytope-examples/blob/main/climate-dt/README.md#using-the-explorer-as-a-request-generator>`_ can also be used to generate working requests.

.. code-block:: python

   import earthkit.data

   request = {
       'activity': 'projections',
       'class': 'd1',
       'dataset': 'climate-dt',
       'date': '20200102',
       'experiment': 'SSP3-7.0',
       'expver': '0001',
       'generation': '2',
       'levtype': 'sfc',
       'model': 'IFS-NEMO',
       'param': '134/165/166',
       'realization': '1',
       'resolution': 'standard',
       'stream': 'clte',
       'time': '0100',
       'type': 'fc'
   }

   data = earthkit.data.from_source(
       "polytope",           # use the Polytope web service backend
       "destination-earth",  # Polytope collection name
       request,
       address="polytope.mn5.apps.dte.destination-earth.eu",
       stream=False  # if True, data is read as a stream (GRIB/CoverageJSON only);
                     # if False, data is retrieved to a local cache file
   )


.. _polytope_client_example:

Example: polytope-client
------------------------

For direct use of the lower-level client:

.. code-block:: python

   from polytope.api import Client

   client = Client(
       address="<address where data is located>",
       user_email="<YOUR EMAIL>",
         user_key="<YOUR DestinE Platform KEY>"
   )

   # Optionally cancel any pending requests from a previous session
   client.revoke("all")

   # Retrieve data; files are saved to output_path
   files = client.retrieve("destination-earth", request, output_path)


Quota limits
------------


The following limits apply to all users to ensure fair access and system stability:

.. list-table::
   :widths: 40 60
   :header-rows: 1

   * - Limit
     - Value
   * - API rate limit
     - Up to 50 requests per second (subject to adjustment based on load)
   * - Concurrent downloads
     - No more than 2 active download requests at a time (subject to adjustment
       under high load)