Building Cloud-Based Services to Enable Earth Science Workflows across HPC Centres

In this video from FOSDEM 2020, John Hanley from ECMWF presents: Building Cloud-Based Services to Enable Earth Science Workflows across HPC Centres.

Weather forecasts produced by ECMWF and environment services by the Copernicus programme act as a vital input for many downstream simulations and applications. A variety of products, such as ECMWF reanalyses and archived forecasts, are additionally available to users via the MARS archive and the Copernicus data portal. Transferring, storing and locally modifying large volumes of such data prior to integration currently presents a significant challenge to users. The key aim for ECMWF within the H2020 HiDALGO project is to migrate some of these tasks to the cloud, thereby facilitating fast and seamless application integration by enabling precise and efficient data delivery to the end-user. The required cloud infrastructure development will also feed into ECMWF’s contribution to the European Weather Cloud pilot which is a collaborative cloud development project between ECMWF and EUMETSAT.

ECMWF and its HiDALGO partners aim to implement a set of services that enable the simulation of complex global challenges which require massive high performance computing resources alongside state-of-the-art data analytics and visualization. ECMWF’s role in the project will be to enable seamless integration of two pilot applications with its meteorological data and services delivered via ECMWF’s Cloud and orchestrated by bespoke HiDALGO workflows. The demonstrated workflows show the increased value of weather forecasts, but also derived forecasts for air quality as provided by the Copernicus Atmospheric Monitoring Service (CAMS).

The HiDALGO use-case workflows are comprised of four main components: pre-processing, numerical simulation, post-processing and visualization. The core simulations are ideally suited to running in a dedicated HPC environment, due to their large computational demands, coupled with the heavy communication overhead between parallel processes. However, the pre-/post-processing and visualization tasks generally do not demand more than a few cores to compute and do not require message passing between instances, hence they are good candidates to run in a cloud environment. Enabling, managing and orchestrating the integration of both HPC and cloud environments to improve overall performance is the key goal of HiDALGO.

This talk will give a general overview of HiDALGO project and its main aims and objectives. It will present the two test pilot applications which will be used for integration, and an overview of the general workflows and services within HiDALGO. In particular, it will focus on how ECMWF’s cloud data and services will couple with the test pilot applications to improve overall workflow performance and enable access to new data for the pilot users.

John Hanley has a professional background in meteorology, software engineering and high-performance computing. Since March 2019, he has been working at the European Centre for Medium-Range Weather Forecasts (ECMWF) as a HPC and Big Data analyst as part of the HiDALGO H2020 project (https://hidalgo-project.eu/). The core aim of HiDALGO is to implement a set of services and functionality to enable the simulation of complex global challenges which require massive high performance computing resources alongside state-of-the-art data analytics and visualization.

Check out our insideHPC Events Calendar