Supermicro (Nasdaq ticker: SMCI) today announced the launch of a new KV caching storage solution in collaboration with IBM and NVIDIA. This joint solution integrates Supermicro Petascale all-flash storage servers equipped with AMD EPYC 9005 series processors (“Turin”), IBM Storage Scale software, as well as NVIDIA Dynamo inference framework software and Spectrum-X Ethernet networking.

Supermicro indicates that by combining these technologies, the time required for the AI model to start generating responses can be significantly reduced. This is particularly crucial as companies adopt agent-based and long-context AI workflows. Expanding the KV cache capacity beyond local GPU memory helps to improve GPU utilization and inference throughput, while also supporting high availability of the cached content and uninterrupted capacity expansion. The company claims that this will result in a more scalable and resilient inference architecture, thereby providing a faster and more responsive AI experience.

Inference workloads typically consist of two phases: prefill and decode. During the prefill phase, the input text is converted into token and processed by a large language model to generate the intermediate key-value states required for inference (KV). By storing these previously computed KV states in a persistent KV cache, the system can reuse them in subsequent requests without having to recalculate the same context. Agent-based workflows, code assistants, long-context RAG, multimodal AI, and multi-round applications, when dealing with large codebases, documents, conversations, images, and other contexts, may generate a large number of KV states, which can quickly exceed the memory capacity of GPU and CPU. Therefore, it is necessary to offload the KV cache to high-capacity, shared network storage.

The KV caching solutions for Supermicro, IBM, and NVIDIA provide this persistent storage layer, which helps to improve inference efficiency, throughput, and scalability, while also reducing redundant calculations.

The main advantages of this solution for customers include:

• Joint Solution Verification – Supermicro, IBM, and NVIDIA have undergone extensive cluster-level testing to measure performance, including high-concurrency and “noisy neighbor” network traffic scenarios.
• Significantly faster and more stable reasoning performance – in the joint tests of Supermicro and NVIDIA, when using a large language model with 120 billion parameters, persistent caching with KV increased TTFT by up to 56 times under 130,000 token prompts.[1] When the input length increased from 30,000 token to 130,000 token, enabling KV caching only increased TTFT by 0.377 seconds, whereas without using KV caching, it increased by 29.55 seconds, thus providing more consistent performance for long-context workloads.
• Higher GPU utilization and throughput – By keeping frequently reused contexts outside of GPU memory, this approach helps to reduce GPU idle time, increase inference throughput, and support higher concurrency without proportionally increasing GPU resources.
• Proven turnkey AI infrastructure – Supermicro, IBM and NVIDIA have jointly verified this integrated architecture under high-concurrency and “noisy neighbor” scenarios, combining NVIDIA Dynamo, IBM Storage Scale software, Supermicro Petascale servers, and the NVIDIA Spectrum-X network.
• Accelerated deployment services – The factory rack integration and on-site deployment services of Supermicro can reduce the deployment time of this solution from several weeks to just a few days.

IBM Storage General Manager Sam Werner stated: "Enterprises need a fundamentally better way to expand AI reasoning. IBM, Supermicro, and NVIDIA are jointly providing a high-performance, resilient KV caching solution that helps customers maximize their GPU investments, enable expansion independent of computing resources, and significantly improve infrastructure efficiency. This collaboration represents an important step towards making enterprise AI reasoning more scalable, reliable, and cost-effective."

NVIDIA AI The product lead, Shruti Koparkar, stated: "Agent-based AI and long-context applications rely on efficient reuse of context across multiple steps, while NVIDIA Dynamo helps to orchestrate distributed inference, enabling enterprises to obtain more available throughput from each GPU. By combining NVIDIA Dynamo and Spectrum-X Ethernet with Supermicro Petascale storage servers and IBM Storage Scale, this KV caching solution creates a high-performance shared storage layer that can reduce redundant calculations, improve the time to first output a token, and assist customers in scaling their resilient AI inference services."

For more information on the solutions for Supermicro and IBM, click here; to read IBM Redbook " Context without Limits : A High - Performance KV Cache Platform for Large - Scale AI Inference ", click here.

Regarding Super Micro Computer, Inc: Supermicro (Nasdaq ticker: SMCI) is a global leader in providing application-optimized, end-to-end IT solutions. Headquartered in San Jose, California, the company is dedicated to offering market-leading innovations for enterprises, cloud, AI, and 5G telecommunications/edge IT infrastructure. The company provides comprehensive IT solutions that include servers, AI, storage, Internet of Things, switching systems, software, and support services. Its expertise in motherboard, power supply, and chassis design further enhances its development and manufacturing capabilities, enabling it to deliver next-generation innovations to customers worldwide, from the cloud to the edge. The company's products are designed and manufactured in the United States, Taiwan, and the Netherlands, leveraging global operations to achieve scale and efficiency, while optimizing total cost of ownership (TCO) and reducing environmental impact (green computing). Its award-winning Server Building Block Solutions ® product portfolio allows customers to choose from a wide range of systems optimized for specific workloads and applications. These systems are built using flexible and reusable modules, supporting a full spectrum of form factors, processors, GPU, storage, networking, power supply, and cooling solutions (air cooling, free-air cooling, or liquid cooling).

Supermicro, Server Building Block Solutions and We Keep IT Green are trademarks and/or registered trademarks of Super Micro Computer, Inc.

All other brands, names, and trademarks are the property of their respective owners.

[1] For details of the results, see IBM Redbook : https :// www.redbooks.ibm.com / docs / MD260021 / MD260021.html