AI inference capacity metrics unveiled… covering memory and chip communication

Epoch AI on October 19 added version 1 of its Inference Performance Index to its changelog, a metric that models how many concurrent inference requests a single AI accelerator can handle, normalized so Nvidia's ($NVDA) GB300 scores 1. The index incorporates not only compute but also memory, chip-to-chip communication, and the prefill chips needed to process inputs before answer generation, averaging equally weighted results across four models. The metric matters because it shifts accelerator comparison beyond raw FLOPS toward real-world serving capacity, where memory and interconnect can bottleneck throughput. Performance gaps between accelerators will vary by model and output speed, so the single composite score may mask workload-specific differences. Watch for Epoch AI to expand the model set and for chipmakers to cite index scores in competitive positioning against the GB300 baseline.