Oak Ridge, TN — Oak Ridge National Laboratory has launched a new, more powerful computer system to support climate data research conducted by the DOE’s Atmospheric Radiation Measurement (ARM) user facility.
Procured and managed by the National Center for Computational Sciences (NCCS) at ORNL, the Cumulus-2 cluster is a Dell HPC system with 16,384 processing cores — surpassing ARM’s previous Stratus (1,080 cores) and Cumulus‑1 (4,032 cores) systems. The new system is designed to enable ARM to offer the HPC infrastructure needed to support next-generation atmospheric model simulations, petascale data storage, big-data analytics, and machine learning for atmospheric and climate science research, according to the lab.
“This new cluster will greatly accelerate processing speeds for simulations and boost capabilities to interpret ARM’s storehouse of data,” said Giri Prakash, director of the ARM Data Center at ORNL. “Cumulus-2 will offer roughly 4 times the power of Cumulus-1.”
Combined with its 4-petabyte parallel file system for data storage, Cumulus-2’s increased computational power is intended to support ARM’s mission “to advance a robust predictive understanding of earth’s climate and environmental systems and to inform the development of sustainable solutions to the nation’s energy and environmental challenges.”
At right: Giri Prakash, data center director for DOE’s Atmospheric Radiation Measurement facility, and Ryan Prout, HPC data analytics engineer, NCCS, with Cumulus-2 system at ORNL. credit: Carlos Jones/ORNL
Formed in 1986 under DOE’s Biological and Environmental Research program, ARM’s primary task was to establish a series of instrument platforms to record atmospheric data for studying the earth’s climate. ARM continues to collect this data from three atmospheric observatories across North America along with data gathered by mobile laboratories around the world in a partnership that spans nine national laboratories, including ORNL. With Cumulus-2, the user facility now provides modern HPC resources to atmospheric and climate researchers whose approved projects use ARM’s observational data.
“The majority of ARM users are from other labs, universities, and global research organizations. Last year, there were around 1,000 unique scientific users who produced over 200 journal articles. What the ARM Data Center team does at ORNL is the key for those scientists to access the data,” said Prakash, who also serves as section head for Earth System Informatics and Data Discovery in ORNL’s Environmental Sciences Division.
Prakash manages the data generated from ARM’s 450+ instruments that collect atmospheric phenomena. With 3,000–4,000 data streams coming in from facilities each day, Prakash’s team collects, processes, distributes, and archives the information—usually making it available to users within 24 hours. When he started his position in 2001, there were 20 terabytes of data collected from the instruments. By 2010, it grew to 200 terabytes, and from 2010 to 2016, it reached its first petabyte. Since December of 2016, they have amassed 3 petabytes of data, mostly from high-resolution instruments and 11,000+ data products.
“As part of the premier atmospheric facility at ORNL, we’re providing cutting-edge data and computing capabilities to support science use cases. We continue to improve with the mission to give a better user experience to our scientists,” Prakash said.
In addition to Prakash’s data management, the NCCS provides extensive support from its Operations and Systems Section to keep Cumulus-2 up and running—from hardware to software. Ryan Prout, an HPC data analytics engineer, serves as program manager; Elton Cranfill, group lead for HPC Clusters, manages the compute infrastructure; Dustin Leverman, group lead for HPC Storage Systems, manages the data infrastructure; and Brian Zachary, group lead for HPC Networking and Infrastructure, manages the network infrastructure.
“The NCCS team led by Ryan Prout was key to the design and deployment of the Cumulus-2 architecture within the NCCS computing ecosystem. I would like to extend my sincere appreciation for their hard work and technical expertise in bringing this cluster online,” Prakash said.
A significant ARM project that uses Cumulus-2 is the Large-Eddy Simulation (LES) ARM Symbiotic Simulation and Observation (LASSO) effort. LASSO enhances ARM observations by using LES modeling to provide context for the data and a self-consistent representation of the atmosphere surrounding ARM’s observations. Current LASSO LES libraries include simulations at ARM’s Southern Great Plains atmospheric observatory and the Cloud, Aerosol, and Complex Terrain Interactions field campaign in Argentina.
“We’re integrating high-resolution modeling with the ARM observations so we can add value to the ARM products and provide users with new ways to conceptualize and use the data,” said William Gustafson, an atmospheric scientist at Pacific Northwest National Laboratory and principal investigator for LASSO.
For example, the models could help resolve bubbles of air near the ground surface to see how they turn into shallow convection clouds.
“These high-resolution simulations can provide insight into connections between different observations,” Gustafson said. “We don’t always know how they’re connected. Modeling gives us a plausible realization of what’s happening between those observations.”
source: Coury Turczyn, ORNL
