NVIDIA Run:ai Revolutionizes Large Language Model Inference with GPU Fractioning

NVIDIA Run:ai achieves high throughput and efficient resource usage through intelligent scheduling and dynamic GPU fractioning. A joint benchmarking effort with Nebius AI Cloud shows that fractional GPUs can improve large language model inference performance, increasing effective capacity by 77% and concurrent user capacity by 86%, while maintaining latency SLAs. Fractional GPUs also enable near-linear throughput scaling and production-ready autoscaling with no latency cliffs or error spikes during scale-out.