Today Quobyte announced that the company’s Data Center File System is the first distributed file system to offer a TensorFlow plug-in, providing increased throughput performance and linear scalability for ML-powered applications to enable faster training across larger data sets while achieving higher-accuracy results.
By providing the first distributed file system with a TensorFlow plug-in, we are ensuring as much as a 30 percent faster throughput performance improvement for ML training workflows, helping companies better meet their business objectives through improved operational efficiency,” said Bjorn Kolbeck, Quobyte CEO. “With the higher accuracy of results, scalability to handle bigger data sets and flexibility to run on-prem to the cloud, and edge, we believe we are providing an optimal experience that allows customers to fully leverage the value of their Machine Learning infrastructure investments.”
TensorFlow is an open source library for numerical computation and large-scale machine learning used across industries such as autonomous vehicles, robotics, financial services, healthcare, government, aerospace, defense, and many others. Using Quobyte storage with TensorFlow helps to simplify and streamline the operation of machine learning.
Quobyte’s TensorFlow File System Plug-in allows TensorFlow applications to talk directly to Quobyte, bypassing the operating system kernel to significantly reduce kernel mode context switches and lower CPU usage. While Quobyte storage can be used with all stages of ML, the resulting increased GPU utilization from the TensorFlow plug-in speeds up model training of ML workflows.
Quobyte provides users the flexibility to train anywhere and seamlessly move models into production to better support ML workloads from the data center to the cloud to the edge. The TensorFlow plug-in can be used to train models locally on sample data sets and use the Google Cloud Platform for training at scale because Quobyte runs on-prem and in the cloud. Additionally, because it bypasses the kernel entirely, Quobyte’s TensorFlow plug-in works with both current and older versions of Linux, providing a full range of flexible deployment options for use in ML. Using the Quobyte TensorFlow-Plugin is seamless since there are no application modifications required.
As more and more businesses look to leverage ML to increase innovation, achieve a faster time to market and provide a more positive customer experience, there is an increasing need for storage infrastructures that offer higher performance and increased flexibility that these workloads need,” said Frederic Van Haren, Lead Analyst HPC and AI Systems of analyst firm Evaluator Group. “Vendors, like Quobyte, that offer high performance, broad platform support and flexibility of deployment options are well positioned to help companies handle bigger data sets, achieve more accurate results and run ML workloads in any environment.”
With Quobyte, there is no need for specialized storage systems to get the most out of ML. Quobyte is a single storage system that addresses many different performance profiles, including the high-throughput, low-latency requirement of ML’s model training stage, as well as large block sequential, small block random or mixed general workloads. Quobyte supports the broadest set of access protocols and clients, such as S3, Linux, Hadoop, Windows and NFS for greater platform flexibility and more complete data ingest and preparation. Data is readily available at any stage all within a single global namespace and all managed through Quobyte’s intuitive management console.
Additional benefits of Quobyte’s TensorFlow File System Plug-in include:
- The ability to leverage HDD and SSD to get the best price-performance ratio without cumbersome tiering
- Prefetching of training data can deliver substantial performance improvement. Much machine-generated data uses a sequential naming convention that makes it ideal for prefetching.
- Infinite scalability that allows users to grow storage in terms of throughput and capacity when they need it. As ML project requirements change – oftentimes more quickly than anticipated – the Quobyte installation will adapt. Disks or servers can be quickly and easily added when needed to provide more capacity or performance without any interruption to applications or services.
- Multi-tenancy that provides additional security by allowing users to define isolated namespaces and physical separation of data/workloads inside the same cluster. Administrators can further isolate tenants by controlling to which physical hardware they have access in order to ensure performance and that data is not accessible to any unauthorized users on the network.
Sign up for our insideHPC Newsletter
