HPC and AI Workloads Drive Storage System Design

How Dell HPC storage meets these design needs

High performance computing (HPC) and artificial intelligence (AI) are increasingly used across industries and are reshaping the future. HPC and AI workloads are used by organizations including academia, scientific research, and in industries including telecommunication, healthcare and life sciences, manufacturing, financial services, electronic design automation (EDA), media and entertainment, retail, government as well as oil and gas. These workloads create vast amounts of data which must be computed and stored. Data storage resources are a critical part of HPC and AI systems. The exponential growth of new workloads using HPC/AI systems puts extraordinary pressure on data storage systems for innovative solutions that can scale as data files grow and are easy to manage and update. 

HPC and AI storage challenges

Many organizations have outdated HPC system designs that impact the ability to easily configure and manage HPC storage. Organizations need a way to quickly make difficult decisions involved in choosing memory types, locations, and configurations in planning an upgrade for their HPC systems. Organizations need to find the right mixture of the storage types that are configured into pluggable solutions, which can balance capacity, performance, and cost requirements.

HPC storage overview

Data storage needs can vary greatly depending on the type of workload, the volume of data, and the complexity involved in processing the data. HPC systems can run on-premises, in the cloud, or in hybrid mode. It is important that data storage resources can scale seamlessly across all locations, and ideally, present data as though it were sourced locally.

HPC storage resources are scaled by adding and removing specialized servers in clusters. Based on their configuration, storage servers can add petabytes of new storage in the form of HDDs, SSDs, NVMe, and network attached storage (NAS). Each storage type operates at different speeds, fills different needs, and performs in a different manner:

The type, size, and speed: HPC workloads generate and consume large amounts of structured and unstructured data, depending on the type of workload (e.g., simulations, modeling, imaging, or video editing). The design of storage resources is critical in allowing HPC systems to reach top efficiency and throughput performance levels.

The nature of the data: High throughput or burst data needs different storage resources than I/O intensive or scalable data. HPC systems can use parallel, concurrent, clustered, and distributed computing resources to process data from HPC workloads. 

Dell validated storage solutions meet HPC storage needs

Dell Validated Designs for HPC Storage solutions are delivered with hardware, software, and support from Dell Technologies. The Dell HPC Validated HPC storage solution uses the Dell PowerScale platform for system design. Dell’s validated storage solution provides these advantages

• Simplicity — Simplify monitoring and management without specialized training or expertise required.
• Reliability — Reduce configuration and interoperability issues while improving quality with storage that’s highly available, with no single point of failure.
• Efficiency — Lower the cost of planning, deploying HPC storage, and managing data center hardware from anywhere, at any time with the power of Dell OpenManage.

Dell PowerScale storage 

The validated Dell PowerScale storage solution combines the latest server technology with NAS resources, the OneFS operating system, and management software to create a state-of-the-art storage solution. PowerScale solutions can include important software add-ons including pixstor™ and the BeeGFS® filesystem. 

PowerScale storage solutions provide modular upgrades for HPC systems and can manage data and scale storage to almost any size, while satisfying performance and business requirements. A PowerScale server can add up to 256 nodes in a cluster to an HPC environment, allowing HPC systems to scale in minutes. Validated server designs are tested in the most demanding storage workload environments, including AI, machine learning and deep learning. 

Pixstor™

For workloads that require unstructured data growth, Dell offers the pixstor™ file system (developed by Kalray, a Dell software partner). pixstor creates and manages a multitiered storage environment that is scalable in terms of both throughput and capacity. 

BeeGFS

For workloads that require I/O intensive data storage, Dell offers the BeeGFS developed by Dell software partner ThinkparQ. BeeGFS is a proven open-source, parallel cluster file system designed specifically to manage scalable on-premises or cloud-based resources in performance-critical environments. 

Summary

Many organizations are tied to outdated storage systems that cannot meet HPC and AI workload needs. Designing high‑throughput, highly scalable HPC storage systems require expert planning and configuration. The Dell Validated Designs for HPC Storage solution offers customers a way to quickly upgrade antiquated storage systems while simplify monitoring and overall management of HPC storage without requiring specialized training or expertise to operate, making storage simpler to maintain. The Dell HPC storage solution can also shorten deployment timelines and reduce the costs involved in designing, testing, and integrating HPC system storage upgrades.  

Watch a recent CUBE interview with Dell HPC Storage experts on the State and Future of HPC Storage: