DestinEStreamer

Introduction

DestinEStreamer provides state-of-the-art compression and streaming technologies tailored for climate and earth observation data. It achieves exceptionally high compression rates while maintaining data quality. With DestinEStreamer, you can effortlessly access hundreds of terabytes of datasets via a browser on both desktop and mobile devices. DestinEStreamer enhances the functionality of DestinE Digital Twins by streamlining processing and simplifying access. Additionally, our comprehensive Python Toolbox allows you to connect to data streams and create data cubes on-the-fly for your analysis. Experience the fastest access to Digital Twins with DestinEStreamer.

Documentation Overview

The documentation consists of two main parts: a responsive Web-Application and an API.

DestineStramer Web-Application

The DestinEStreamer web application offers a interactive interface for accessing and visualizing climate and earth observation data. Below is an overview of its key features:

Visual Streams

Temporal Resolution: In the web application overview movies for climate data variables are displayed for daily intervals. Note that datastreams, accessable via the API or Jupyter notebook provide access full temporal resolution.
Spatial Resolution: On mobile devices, visual streams are presented at a lower resolution to ensure optimal performance. Full resolution is accessable via the desktop browser.

Streams

The data streams available on the platform are categorized into two main groups:

ERA5
ERA5 Non Antarctica
Climate DT

These streams can be selected using the Movie button located on the right side of the interface.

Info Button 

An Info Button is available below the Media button. This button provides an overview of the DestinEStreamer, explaining its purpose it offers to users.

Opacity Control

The Opacity Control button, allows users to adjust the opacity of the displayed data streams, providing more control over the visualization experience.

Legend

In the bottom right corner of the Map View, users will find the Legend of the selected stream. The Legend provides contextual information about the data being visualized.

Stream Controls

At the bottom center of the Map View, users can control the stream as they would a video. This includes pausing, playing, and seeking through the data.

Advanced Features

For users with the data access privileges on the DestinE platform. The web-application offers an option to display a code snippet to configure the JupyterLab notebook to access the exact data shown in the web-application in the DestinE platform.

StatsOnEdge Feature

The DestinEStreamer application includes a built-in tool called StatOnEdge, which enables users to explore and analyze climate data streams directly in the browser. Statistical calculations and visualizations are performed securely and locally using edge processing.

StatsOnEdge - Video Tutorial

The following video summarizes the most important aspects of the StatsOnEdge feature.

StatsOnEdge - Use Case

You can use this feature to calculate the ERA5 Surface Air Temperature Monthly Average for 2023 across all months on a selected geographic scale.

To access the StatsOnEdge feature, press the designated button located on the right-hand side of the user interface.

Select Geographic Scope: Begin by selecting the desired geographic scope. Options may include a global view (worldwide) or more detailed regional levels (e.g., NUTS level).
Define Input Parameters: Next, configure the input parameters needed to calculate the statistics. You can specify multiple parameters based on your requirements. For quick exploration, pre-defined presets are provided in a dropdown menu.
Start the Calculation: Once the input is set, press the Start button to initiate the real-time calculation. The line graph will display the results immediately.
Visualizing and Downloading Results:
When the calculation is complete, the results can be downloaded by clicking the Download CSV button.

API Endpoints

This documentation provides an overview of the available API endpoints for the streaming data and visual services.

Endpoints

Get Streaming Visual Overview
- URL: /streaming/visual/overview
- Method: GET
- Response Model: ResponseVisualOverview
- Status Code: 200
- Summary: Returns an overview of visual data streams.
- Description: This endpoint provides an overview of existing visual data streams.
- Parameters: - user_info (Security): User authentication information.
Example Request: .. code-block:: bash

curl -X ‘GET’

‘http://streamer.destine.com/api/v1/streaming/visual/overview’ -H ‘accept: application/json’ -H ‘Authorization: Bearer <your_token>’

Example Response: .. code-block:: json

{

“category_dict”: {

“category1”: [

{

“id”: 1, “title”: “Ocean Temperature”, “subtitle”: “Climate DT daily Ocean and Sea Ice”, “stream_file_desktop”: “path/to/desktop_file”, “stream_file_mobile”: “path/to/mobile_file”, “description”: “Time-mean sea surface temperature [K]”, “legend”: “sea-surface-temperature_560”, “period”: “2020/01/01 - 2029/09/08”, “metadata”: “str”, “short_name”: “2d”

}

]

}

}
Get Streaming Visual Metadata
- URL: /streaming/visual/metadata/{id}
- Method: GET
- Response Model: ResponseVisualMetadata
- Status Code: 200
- Summary: Returns metadata for a specific visual data stream.
- Description: This endpoint provides metadata for a visual data stream identified by its ID.
- Parameters: - id (Path): The ID of the visual data stream. - user_info (Security): User authentication information.
Example Request: .. code-block:: bash

curl -X ‘GET’

‘http://streamer.destine.com/api/v1/streaming/visual/metadata/1’ -H ‘accept: application/json’ -H ‘Authorization: Bearer <your_token>’

Example Response: .. code-block:: json

{

“id”: 1, “name”: “2 metre temperature”, “unit”: “K”, “frame_count”: 12418, “min_date”: “1949-01-01T00:00:00”, “max_date”: “2023-12-31T23:59:00”, “increment”: 1

}
Get Streaming Data Overview
- URL: /streaming/data/overview
- Method: GET
- Response Model: ResponseDataOverview
- Status Code: 200
- Summary: Returns an overview of data streams.
- Description: This endpoint provides an overview of existing data streams.
- Parameters: - user_info (Security): User authentication information.
Example Request: .. code-block:: bash

curl -X ‘GET’

‘http://streamer.destine.com/api/v1/streaming/data/overview’ -H ‘accept: application/json’ -H ‘Authorization: Bearer <your_token>’

Example Response: .. code-block:: json

{

“category_dict”: {

“category1”: [

{

“title”: “Ocean Temperature”, “period”: “2020/01/01 - 2029/09/08”, “short_name”: “2d”, “compression_ratio”: “1:13”, “ssim”: “0.998”, “mre”: “0.3%”

}

]

}

}
Get Streaming Data Metadata
- URL: /streaming/data/metadata/{category_name}/{short_name}/{start_date}/{end_date}
- Method: GET
- Response Model: ResponseDataMetadata
- Status Code: 200
- Summary: Returns metadata for a specific data stream.
- Description: This endpoint provides metadata for a data stream identified by category name, short name, and date range.
- Parameters: - category_name (Path): The category name of the data stream. - short_name (Path): The short name of the data stream. - start_date (Path): The start date of the data stream. - end_date (Path): The end date of the data stream. - user_info (Security): User authentication information.
Example Request: .. code-block:: bash

curl -X ‘GET’

‘http://streamer.destine.com/api/v1/streaming/data/metadata/category1/2d/2020-01-01/2029-09-08’ -H ‘accept: application/json’ -H ‘Authorization: Bearer <your_token>’

Example Response: .. code-block:: json

{

“parameter_name”: “2 metre temperature”, “parameter_short_name”: “t2m”, “type_of_level”: “surface”, “unit”: “K”, “stream_path”: “…”, “nx”: 1440, “ny”: 721, “images”: [

{

“img_date”: “1940-01-01T00:00:00”, “img_number”: 1, “img_min_value”: -123.8, “img_max_value”: 287.4

}

]

}