.. _data_access_apps:

Accessing Impact-Sector Applications Data
============================================

Data from the applications can be accessed via the Harmonised
Data Access (HDA). The following are required:

- DestinE Service Platform account with upgraded access
- the ``destinelab`` Python package
- the unique identifier of the desired dataset (see the table below)

+-------------------+------------------------------------------+
| Application       | Identifier                               |
+===================+==========================================+
| Energy indicators | EO.BSC.DAT.ENERGY_INDICATORS             |
+-------------------+------------------------------------------+
| HydroLand         | EO.UFZ.DAT.HYDROLAND                     |
+-------------------+------------------------------------------+
| HydroMet          | EO.DWD.STAT.HYDROMET_EXTREMES            |
+-------------------+------------------------------------------+
| Wildfires: FWI    | EO.FMI.DAT.DESTINE_CLIMATE_WILDFIRE_FWI  |
+-------------------+------------------------------------------+

To register for a DestinE Service Platform account, go to the `DestinE Platform <https://platform.destine.eu/>`_  and create an user account.
Afterwards, you can apply for `upgraded access <https://platform.destine.eu/access-policy-upgrade/>`_ ,
which grants you access to the original Climate DT model output and the derived use case output.  
An overview of all available datasets and their description is available at the 
`Data Lake portfolio <https://data.destination-earth.eu/data-portfolio>`_. Use the search field to find any keyword or use
one of the filters on the left side. To view all available datasets, a DestinE Service Platform account login is required. 

Once a dataset is selected, users can use their own environment on any machine with internet access or use the 
preconfigured JupyterHub `Insula Code <https://platform.destine.eu/services/service/insula-code/>`_.
To download the data, the ``destinelab`` package is required. It manages the authentication to the Destination Earth Data Lake. It is available on PyPI and can be installed with 

.. code::

      pip install destinelab


The key steps are summarised below. For a complete walkthrough, see the
`DestinE DataLake Lab example notebook <https://github.com/destination-earth/DestinE-DataLake-Lab/blob/main/HDA/REST/HDA-REST-quick-start.ipynb>`_
for a generic example, or the
`App-specific example notebook <https://github.com/destination-earth-digital-twins/destine-climate-dt-use-cases/blob/main/README.md>`_
for an application-tailored walkthrough.

Furthermore, data retrieval via HTTP requests is described concisely below. There are more methods that use additional packages but offer more possibilities.
An introduction can be found in the `Quick Start Guide <https://destine-data-lake-docs.data.destination-earth.eu/en/latest/dedl-discovery-and-data-access/Harmonized-Data-Access/Quick-Start/Quick-Start.html>`_
of the HDA documentation.

**1. Authentication**

The first step is to retrieve an authentication header that identifies the user to the service. The easiest solution is to use the ``destinelab`` package.
The code below asks the user for their credentials.

.. code::
      
      import destinelab as deauth
      from getpass import getpass

      DESP_USERNAME = input("Please input your DestinE Service Platform username or email: ")
      DESP_PASSWORD = getpass("Please input your DestinE Service Platform password: ")

      auth = deauth.AuthHandler(DESP_USERNAME, DESP_PASSWORD)
      access_token = auth.get_token()
      auth_headers = {"Authorization": f"Bearer {access_token}"}



**2. Search**

After successful authentication, the user can search for the dataset by sending an HTTP request using the Python ``requests`` package. The results are written out in the JSON format.

.. code::

      import requests
      import json

      BODY = {
      "collections": [
            "EO.DWD.STAT.HYDROMET_EXTREMES",
      ]
      }

      r=requests.post("https://hda.data.destination-earth.eu/stac/v2/search",
                json=BODY, headers=auth_headers)
      if(r.status_code!= 200):
            (print(r.text))
      r.raise_for_status()

      print(json.dumps(r.json(), indent=4))

**3. Download**

The JSON contains download links that can be clicked for direct use or incorporated in a Python workflow.
The following code downloads the first entry of the JSON file as a zip archive.

.. code::

      #select the first item in the result to download
      product = r.json()["features"][0]

      # DownloadLink is an asset representing the whole product
      download_url = product["assets"]["downloadLink"]["href"]
      ITEM_ID = product["id"]

      response = requests.get(download_url,stream=True,headers=auth_headers)

      # If the request was successful, download the file
      if (response.status_code == 200):
            print("Downloading ...")
            filename = ITEM_ID + ".zip"
            with open(filename, 'wb') as f:
                  for chunk in response.iter_content(chunk_size=1024): 
                     if chunk:
                        f.write(chunk)
                        f.flush()
            print("The dataset has been downloaded to: {}".format(filename))
      else: print("Request Unsuccessful! Error-Code: {}".format(response.status_code))