In this Let’s Talk Exascale podcast, Andrew Younge from Sandia National Laboratories describes the new SuperContainers project, which aims to deliver containers and virtualization technologies for productivity, portability, and performance on the first exascale computing machines are planned for 2021.
Container technology has revolutionized software development and deployment for many industries and enterprises because it provides greater software flexibility, reliability, ease of deployment, and portability for users. But several challenges must be addressed to get containers ready for exascale computing. As the Exascale Computing Project has evolved since its inception in 2016, containers have been an increasing area of interest.
Essentially, containers allow you to encompass your entire environment in a simple and reproducible way,” says Younge. “So not only do I have my container image that has my application and my entire software stack with it, I also have a manifest for how I got there. That’s a really important notion for many people.”
The SuperContainers project is uniquely positioned by ECP’s collaborative framework to deliver first on the container runtimes research and development while helping colleagues across ECP leverage the technology, Younge said. And, he noted, the benefits will ripple across DOE labs.
I have collaborators, and I’m able to leverage not only some of the great work that’s being done, say, at Berkeley Lab, but I’m also able to provide that in a centralized way to application developers across the entire DOE, from Brookhaven to Argonne, to any of the major labs,” he said.
One of the major objectives of the SuperContainers project is to provide training and outreach services to various ECP teams and to the DOE facilities. “It is getting our users familiar with and being able to leverage some of the modern tools with containers, working in sort of a DevOps model, which is a little bit different than how we’ve built HPC applications historically,” Younge said. The DevOps approach combines software development and information technology operations for efficiency in delivering the best quality software.
Younge shared some of the accomplishments of the SuperContainers team to date: It has developed a set of container images that can be deployed on a number of systems. And at least one container runtime is available on the vast majority of the pre-exascale machines. Performance and scalability numbers for the container runtimes have been very impressive. Also, some first images have been created with the Extreme-scale Software Stack (E4S).
The good news is we already have this technology available today in some capacity,” he said. “We’re improving the scalability, and I think that’s one of the things that we’re actually really proud of. I’m surprised by how quickly we were able to demonstrate that containers don’t add fundamental overhead or performance impact.”
The team is optimizing the images and ensuring that the improvements reach the consumers of the container technologies—the ECP Application Development teams—immediately, and that the developers don’t have to fine-tune the solutions to meet their requirements, Younge said.
They can take this cookie-cutter recipe or set of recipes and be able to quickly extend it to their needs without having to sacrifice performance,” he said. “There’s this continual balance between portability and performance.”
Open standards, interoperability, and the creation of tools that will, by design, outlast ECP, hold promise to be the central aspects of the enduring legacy of the Supercontainers project, Younge said.
Source: Scott Gibson at the Exascale Computing Project
