Vera Rubin NVL72 Makes MLPerf Debut: 3.7x Throughput Leap, NVFP4 Quantization, and the Rack-Scale Inference Era

The release of the MLCommons MLPerf Inference v6.1 benchmark suite marks an inflection point in datacenter artificial intelligence infrastructure. For the first time, peer-reviewed engineering benchmarks have captured measured performance from NVIDIA's next-generation Vera Rubin architecture, debuting within a full rack-scale Vera Rubin NVL72 configuration.

The benchmark data confirms a formidable generational leap over current-generation hardware: Vera Rubin NVL72 delivers up to 3.7x higher throughput on multimodal frontier models like Qwen3-VL and up to 2.5x higher throughput on complex reasoning architectures such as DeepSeek-R1 compared to the NVIDIA GB300 NVL72 baseline.

As frontier AI laboratories transition from pre-training monolithic models to orchestrating massive test-time compute, reasoning loops, and multi-agent workflows, inference economics have become the central battleground of AI infrastructure. The MLPerf v6.1 results demonstrate that raw compute acceleration is no longer a standalone metric; modern serving throughput depends on extreme full-stack codesign spanning silicon, quantization math, high-radix interconnects, and disaggregated runtime orchestration.

The core architectural thesis of the NVL72 platform is treating the entire 72-GPU rack as a single, monolithic super-accelerator rather than a collection of discrete servers connected by network adapters. In the Vera Rubin NVL72, this scale-up paradigm reaches unprecedented density.

Connecting 72 Rubin GPUs alongside NVIDIA's new Vera CPUs—specifically architected to accelerate agentic orchestration, data ingest, and sequential reasoning tasks—the system relies on sixth-generation NVLink and NVLink Switch silicon. The interconnect delivers 10x higher packet routing rates and 3x lower communication latency than commercial top-of-rack Ethernet fabrics.

This scale-up fabric directly resolves the primary bottleneck plaguing modern Mixture-of-Experts (MoE) models. In architectures such as DeepSeek-R1, every generated token requires dynamic routing across hundreds of specialized expert feed-forward networks distributed across the compute cluster. When expert routing traverses conventional Ethernet or InfiniBand switches, inter-GPU communication latency stalls the pipeline. By maintaining a coherent, ultra-low-latency 72-GPU memory pool, Vera Rubin executes all-to-all expert dispatch operations across MoE layers with virtually zero synchronization overhead.

A pivotal factor behind Vera Rubin's 3.7x throughput multiple is native hardware support for NVFP4 precision. While the industry spent the past two years transitioning production workloads from FP16 to microscopic FP8 formats, the Vera Rubin Transformer Engine extends native floating-point math down to 4-bit precision across model weights, activations, and Key-Value (KV) cache tensors.

Operating at 4-bit precision cuts memory capacity requirements by half compared to FP8 and by fourfold compared to FP16, while effectively doubling arithmetic throughput per Tensor Core cycle. Historically, post-training 4-bit quantization resulted in unacceptable perplexity degradation and numerical instability during long reasoning chains. However, Vera Rubin utilizes fine-grained block-level dynamic scaling vectors, allowing mathematical representations to retain high dynamic range across sensitive attention outlier tokens.

Crucially, compressing the KV cache into NVFP4 directly attacks the memory wall that constrains interactive serving. In reasoning models generating tens of thousands of intermediate chain-of-thought tokens, KV cache allocation typically exhausts high-bandwidth memory (HBM) long before GPU compute cores reach peak saturation. By halving the byte footprint per token state, Vera Rubin NVL72 accommodates drastically higher batch concurrency per rack without triggering memory paging or model offloading.

In addition to hardware density, the MLPerf v6.1 results reflect a structural overhaul of inference runtime orchestration. NVIDIA's submissions relied extensively on disaggregated serving architectures powered by vLLM and the open-source NVIDIA Dynamo inference framework.

Traditional LLM serving collocated prompt prefill (ingesting input tokens) and token decode (generating subsequent tokens autoregressively) on the identical GPU hardware. This created an inefficient compromise: prefill operations are computationally dense and saturate matrix math units, whereas decode operations are severely memory-bandwidth bound and underutilize Tensor Cores.

Under the disaggregated serving model deployed in the Vera Rubin submissions, incoming prompt contexts are routed to dedicated prefill nodes that run at full mathematical saturation. Once the initial context is encoded, the generated KV state is transmitted across the high-speed NVLink backplane to dedicated decode nodes optimized for streaming token emission. This physical separation prevents long prompt ingest operations from blocking ongoing user generation streams, maintaining sub-15-millisecond time-to-first-token (TTFT) metrics even under extreme enterprise load.

Alongside the Vera Rubin preview figures, NVIDIA submitted multi-rack scaling benchmarks demonstrating near-linear scaling across production hardware. In the DeepSeek-R1 offline scenario, scaling from a single GB300 NVL72 rack (72 GPUs) to a four-rack cluster (288 GPUs) achieved 99% scaling efficiency. Throughput scaled in direct mathematical proportion to the hardware added, proving that rack-scale clusters do not suffer diminishing returns when managed by cohesive software orchestration.

Furthermore, on generative video workloads—tested via the WAN 2.2 text-to-video benchmark—a full-rack GB300 NVL72 generated 0.65 720p video streams per second at an average generation time of 5.7 seconds per video, representing a 9x throughput increase and 7.5x latency reduction over standalone compute nodes.

For hyperscale cloud providers—including CoreWeave, Nebius, Microsoft Azure, Google Cloud, and Oracle Cloud Infrastructure, all of which participated in the v6.1 evaluation cycle—these figures redefine datacenter economics. As global electricity grids struggle to supply the gigawatts demanded by planned mega-datacenters, the primary constraint on cloud revenue has shifted from capital expenditure to available power capacity. By delivering up to 3.7x higher token throughput within the physical boundaries of a single liquid-cooled rack footprint, Vera Rubin NVL72 dramatically expands the revenue generation capacity of every provisioned megawatt.

• NVIDIA Official Blog - Official Release & MLPerf Inference v6.1 Overview
• StorageReview - MLPerf Inference v6.1 Full Benchmark Analysis & Peer Review
• Unite.AI - NVIDIA Vera Rubin NVL72 Posts First MLPerf Inference Preview Results
• Business Wire / CoreWeave - CoreWeave Brings Up Multi-Rack NVIDIA Vera Rubin NVL72 Cluster