Together AI unveils GPU inference code optimization tool

Together AI unveiled Together Compiler on June 4, an AI-agent tool that iteratively optimizes GPU inference code rather than model weights, targeting the code that executes trained models during user request handling. The system automates re-optimization of inference code when model architectures, request traffic patterns, or GPU hardware change, adapting code execution to new models and hardware. Together AI said multiple agents repeatedly test code changes, running 252 times more experiments than its existing agent-based inference optimization system. The launch matters because inference—not training—drives the bulk of AI compute costs, and automated code-level tuning could let providers and enterprises extract more throughput from existing GPU fleets as model and hardware turnover accelerates. What to watch next is whether Together AI discloses benchmark gains in latency or tokens per second, and whether the compiler is offered to its cloud customers or as standalone software.