[Sponsored Post]
For decades, compute resources used for weather forecasting have tracked with advances in state-of-the-art supercomputing. Which is to say that the weather segment demands systems with the greatest data ingest and storage capacity combined with the most powerful processing capabilities. As the accuracy of daily weather forecasts and warnings of severe weather depend on high-performance computing combined, increasingly, with artificial intelligence, it is perhaps not surprising that weather segment IT spending has not been affected by the COVID-19 pandemic. Hyperion Research predicts that it will in fact grow by an astonishing 33 percent between 2021–20241, significantly outpacing expected growth for the overall HPC industry.
HPE’s SmartSim enabled a scientist to run climate-scale oceanographic model simulation with prediction of movement of energy in the ocean (Eddy Kinetic Energy – EKE) at a resolution which beats current state of the art for global model (left pic). Here’s a video simulation.
Spurring this growth is not only the need to predict increasingly volatile, climate change-driven weather events but[1] also the emergence of advanced technologies and technology strategies enabling more effective weather modeling.
As we approach the exascale supercomputing era, characterized by extreme compute power and extreme heterogeneity, it seemed a good time to sit down with a leading weather technologist, Ilene Carpenter, earth sciences segment manager at Hewlett Packard Enterprise, to discuss HPC trends in weather forecasting – particularly the rapid adoption of AI techniques by major weather centers.
“AI has been used in weather forecasting for a long time, but now there’s a resurgence because of advances in machine learning, driven by the availability of massive amounts of data and the power of GPUs,” Carpenter told us. “Weather forecasting centers were among the first supercomputer users. Now, they are combining physical modeling on supercomputers with AI and data-driven approaches to enable better predictions.”
Carpenter explained that HPE’s acquisitions of SGI in 2016 and of Cray two years ago, and the subsequent melding of the three companies’ HPC technologies, has resulted in supercomputing capabilities uniquely capable of supporting combinations of AI with traditional physics-based simulations, creating a paradigm opening a new world of forecasting possibilities.
HPE Cray EX
For example, the National Center for Atmospheric Research (NCAR) next year will deploy a new HPE Cray EX supercomputer, named Derecho. It will be nearly 3.5x faster and 6x more energy efficient than the current HPE SGI 8600 Cheyenne system.
“NCAR needs a supercomputer powerful enough to support both complex physical modeling and machine learning algorithms,” she said, “to improve the understanding of weather and climate, as well as things like seasonal water supply and drought risk. NCAR’s research is also empowering emergency responders with better predictive models of wildfire behavior, so they can save lives and millions of dollars in property damage.”
The system’s capacity to combine HPC and AI, in part via a tool called SmartSim, enables it to support NCAR’s highly demanding needs.
“Each technology, HPC and AI, is dependent on the other because alone, they simply couldn’t get the job done that well,” she said. “We wanted to make the use of this new paradigm easier so Cray, now HPE, developed SmartSim, which allows researchers to add AI to their traditional simulations by adding just a few lines of code.”
Ilene Carpenter of HPE
SmartSim is an open source framework, which enables weather and climate applications written in Fortran or C/C++ to interface with AI models and modern data analytics methods. The SmartSim tool is instrumental to supporting the convergence of HPC simulations, data analytics and artificial intelligence.
In HPE’s recent cooperation with the National Oceanic and Atmospheric Administration (NOAA) and the NCAR, SmartSim enabled the agencies to run MOM6, a climate-scale oceanographic model simulation showing energy movement with a level of detail that previously was computationally too expensive for use in production decadal-to-century climate models.
“Using AI and SmartSim, they were able to replace a time-consuming eddy kinetic energy calculation with an AI model,” Carpenter said. “As over 80 percent of the excess heat in global warming is captured and stored in the ocean, this simulation is a crucial piece of a puzzle helping scientists understand impact of the climate change.”
Increasing diversity, specialization and heterogeneity in the post Dennard scaling era is a particular challenge for weather forecasting centers. “We need to support lots of choices for processors and accelerators,” Carpenter said, “in fact, we could foresee a day when there’s heterogeneity within a processor. Customers’ choices will be all over the map, so applications need to run on many architectures in a rapidly changing hardware and software landscape with huge amounts of data to manage.”
Carpenter told us that just like commercial organizations, weather centers are increasingly looking at cloud-first or cloud-smart strategies to fit their economic goals, in order to make their services available to a wider user-base, as well as to cope with an unprecedented explosion of data generated and processed to offer more precise, more frequent and more detailed weather forecasts.
HPE, starting with Cray, has been servicing the UK Met Office, one of the world’s largest weather forecasting agencies, for generations. Now Microsoft has partnered with HPE to take the Met Office’s new HPE Cray EX supercomputer to the cloud.
HPE and Microsoft have had a supercomputing alliance through Cray in place for years, working towards this supercomputing-as-a-service model. It’s now coming to fruition – and at scale – in the Azure cloud, enabling the Met Office to work and act on the insights of their data using AI, modeling, and simulation.
“An overwhelming majority of national weather forecasting centers around the world currently use our systems,” Carpenter said. “While some customers, such as NCAR run their own data centers, other customers, such as the U.S. Air Force has Oak Ridge National Laboratory, procure and host their system. HPE Cray EX supercomputers are quickly becoming the HPC system of choice for leading weather centers the world over deploying these systems in a variety of consumption models.”
[1] [1]Hyperion Research HPC Forecast, March 31st, 2021.
