Today, we’re proud to announce that Google Cloud is a founding member of the DAOS Foundation, launched today by the Linux Foundation. Created in collaboration with partners, the DAOS Foundation will accelerate investment and development of the world-record-setting open-source DAOS storage stack, which supports next-generation HPC and AI/ML workloads.
Intel Corporation initiated investment in DAOS in 2012 to drive a new paradigm in storage for high performance computing (HPC). The combination of fully-distributed metadata management, a key-value store architecture, and the lack of dependency on kernel modifications has led to notable breakthroughs, such as:
impressive gains in throughput, I/O, and metadata operations performanceeasier container/Kubernetes integrationincreased architectural flexibility and decreased infrastructure overhead (eliminating the need for metadata servers)
The upcoming deployment at Argonne National Laboratories, which is targeted to deliver >25TB/s of throughput for HPC, demonstrates the significant impact DAOS has had on the world of scientific computing.
Further, we recently launched Parallelstore, a scratch storage service based on DAOS that delivers up to 6.3x read-throughput performance compared to competitive Lustre scratch offerings. Both large language models (LLMs) and HPC have demanding IOPS, throughput, MDOPS, and ultra-low latency requirements. Customer demand for AI/ML training based on large data sets with small files that require low-latency operations aligns extremely well with the performance characteristics and architecture of DAOS.
“Google is delighted to be a founding member of the DAOS Foundation,” said Sameet Agarwal, Google Storage VP/GM. “This industry-leading architecture underpins our new storage service, Parallelstore, and we are excited to accelerate the growth of HPC and AI/ML workloads in the cloud.”
The DAOS Foundation will be established as a Linux Foundation project, which provides a structure for shared governance and continued investment in the open source community. The collaboration across companies has been strong and Google Cloud is excited to continue this investment.
Cloud BlogRead More
