The Shift to AI Inference Is a Software Story Now. Datadog Is the Bellwether

When cloud infrastructure groups add members, the instinct is to glance at the press release and move on. But the announcement this June from the Cloud Native Computing Foundation — the organization that manages Kubernetes and has quietly become the referee for how enterprises run AI — is worth a second look, because it carries the industry's own read on where the next wave of AI spending is going: out of training and into inference.

The foundation's data makes the shift concrete. Sixty-six percent of organizations running generative AI now use Kubernetes to manage at least some of their inference workloads, and 82% of container users run Kubernetes in production. That is the institutional version of a product transition: enterprises have stopped building AI and started operating it.

What most investors miss is what that transition does to where the value lands. Training is a months-long, batch job — a giant pile of raw compute that runs until the model is finished. It is the phase where Nvidia's CUDA moat is near-absolute and the chip is the whole story. Inference is the opposite: always-on, latency-sensitive, cost-sensitive, and spread across thousands of machines whose job is to keep a single query cheap and reliable. In training, the hardware sets the ceiling. In inference, the software that makes that hardware predictable becomes the contested layer — an efficiency problem no single chip architect gets to own.

That is the mechanism the CNCF has been quietly documenting, and it is early. Only 7% of organizations deploy generative AI daily in continuous production, and 44% still run no AI on Kubernetes at all. The buildout is a real but unfolding one — which is exactly the kind of setup where the software layer can compound before the market finishes pricing the shift.

Datadog is the purest public test of the inference-software trade

The clearest public embodiment of that software layer is Datadog, the observability company that sells the tools to watch all this machinery. It built GPU monitoring specifically to keep track of the "AI factory" — GPU utilization, heat, power, interconnect — the operational layer of inference running in production. And its management said outright in the last earnings call that the company's center of gravity is production and inference, not the training phase it is also probing.

The operating results have started to show it. In the quarter ended June 30, Datadog reported revenue of $1.12 billion, up 36% from a year earlier — and, more tellingly, a sequential acceleration of 11.4%, its fastest quarter-over-quarter growth since early 2022. Management describes the acceleration as broad-based: growth excluding AI-native customers accelerated to the high-20s, and AI-native customers sit on top of that. The customer base includes more than 750 AI customers, eight of them above $10 million in annual recurring revenue, and all ten of the top AI leaders are paying customers. This is not a headline about selling shovels to one hyperscaler; it is a widening base of enterprises paying to keep their AI reliable, which is what the training-to-inference shift actually demands.

Why a real thesis still needs a check on price

Here is the part of the analysis that matters more than the growth rate. Execution like this is real, but the stock has already absorbed a large part of the story. Datadog is up roughly 66% year-to-date and has run about 68% higher over the last four months, lifting its valuation to roughly 19 times trailing sales. A thesis the market has largely digested has to evolve rather than be defended, and the near-term return curve is the live question.

There are also signs the second half is more tempered than the headline growth suggests. Datadog's own guidance for the next quarter implies a year-over-year growth rate in the high-20s — an actual deceleration quarter over quarter — and the company baked into that outlook a reduction in usage by its largest customer. That is the double-edged nature of usage-based billing: the same AI complexity that drives monitoring billings can be optimized away when a big spender decides it is paying too much for logs and data.

So the honest framing is a conditional one. The thesis — inference shifts value to the software layer, and Datadog sits squarely on it — is intact and supported by the operating evidence. The open question is whether, at 19 times sales after a two-thirds run in price, that shift is still priced like a discovery or a conclusion. The buildout is real and early enough to reward patience, but the multiple has pulled a meaningful share of that return forward. Demand is not the issue. Whether today's price is still the best place to put new capital — against everything else in the AI trade — is.