Epoch AI’s new inference index shows GB300 NVL72 far outpaces the H100

Epoch AI has launched a new inference index that benchmarks AI hardware on real-world inference workloads rather than raw compute, with Nvidia's GB300 NVL72 system substantially outperforming the older H100. The index emphasizes memory capacity and inter-chip communication as the dominant factors in inference performance, reflecting a shift in how integrated AI systems are evaluated. The findings matter because inference—not training—is increasingly the bulk of AI compute demand, and system-level design choices now drive performance gaps between hardware generations. What to watch next is whether rival accelerators and chipmakers adopt comparable system-level benchmarks, and how quickly operators migrate inference workloads to newer integrated platforms.