Groq's Valuation Triples to $6.9B After $750M Funding Round

Groq, a startup focused on AI chips, has confirmed that its valuation has surged to approximately 6.9 billion dollars following a new round of funding. The company, which is one of the major competitors to NVIDIA in the AI chip market, raised 750 million dollars in this latest funding round. This valuation is significantly higher than the 6 billion dollars estimated in July, when rumors of the funding round first surfaced.

Groq's focus is on selling core AI computing infrastructure—AI chip clusters—to global data centers and enterprise platforms, similar to NVIDIA's data center business, which is its largest revenue and profit contributor. In August 2024, Groq raised 640 million dollars at a valuation of 2.8 billion dollars. The latest funding round has nearly tripled the company's valuation in just one year.

Groq's LPU (Language Processing Unit) is a specialized accelerator designed for inference, particularly for large language models (LLMs). The core architecture of the LPU is Groq's proprietary TSP (Tensor Streaming Processor), which uses a static, deterministic data flow path to replace the traditional AI GPU paradigm of "threads/kernels/cache." This design emphasizes low latency, stable latency, and high throughput for small batch sizes.

The LPU chip developed by Groq features a large on-chip SRAM capacity of approximately 220MB and ultra-high on-chip bandwidth, with official documentation citing up to 80TB/s. The chip uses a compiler to explicitly schedule operators and data flows in both time and space, minimizing reliance on reactive hardware components such as caches, arbiters, and replay mechanisms.

Through the use of TSP's streaming, static, and deterministic compile-time scheduling, combined with high-bandwidth on-chip SRAM, the LPU offers lower latency, more stable throughput, and potentially higher efficiency in low/zero batch LLM inference compared to AI GPU clusters. However, for large model training, dynamic workloads, and ecosystem completeness, GPU clusters still have systematic advantages.

For comprehensive AI workloads that focus on "AI large model training" and "super large batch throughput," NVIDIA's AI GPU ecosystem, which includes CUDA, high-bandwidth memory, and NVLink, remains dominant. The LPU's advantages lie in interactive/real-time inference and low-latency LLM inference workloads.

In scenarios where the batch size is very small (even batch=1), the LPU does not require "stacking batches to run efficiently." The tokens/s per chip is higher, and the scheduling overhead is lower, meeting the requirement for interactive products to have "fast responses." Groq's LPU uses large on-chip SRAM for direct computation, with official documentation showing on-chip bandwidth up to 80TB/s, while GPUs frequently access off-chip HBM. This reduces the "compute-storage" round trips, improving the efficiency of running AI models and achieving an extremely high power efficiency ratio. The deterministic execution of the LPU results in a smoother power curve, and with a simplified data path, the power consumption per token for inference is lower. Reports indicate that the LPU's power consumption for equivalent inference is approximately one-third that of common GPUs.

While AI ASICs cannot completely replace NVIDIA on a large scale, their market share is expected to continue to grow. For standardized mainstream inference and some training tasks (especially continuous long-tail training/fine-tuning), the "unit throughput cost/power consumption" of customized AI ASICs is significantly better than pure GPU solutions. For rapid exploration, cutting-edge large model training, and multi-modal new operator trials, NVIDIA's AI GPUs remain the primary choice. Therefore, in current AI engineering practices, tech giants are increasingly adopting a hybrid architecture of "ASIC for routine tasks, GPU for exploration peaks/new model development" to minimize TCO.