AI Agent UpdateDeepSeek just open-sourced the engine behind its absurdly cheap AI
DeepInfer is the serving stack that lets DeepSeek sell tokens at $0.60 per million. Now anyone can run it. Inference margins just got a lot thinner.
The News
DeepSeek open-sourced DeepInfer on Oct 20 under the permissive Apache 2.0 licence — anyone can use it, including commercially, for free. DeepInfer is the inference framework behind DeepSeek's own production service: the software layer that takes a trained AI model and serves it to millions of users quickly and cheaply. It is built primarily on the open-source vLLM project, with ideas borrowed from SGLang and Nvidia's TensorRT-LLM.
Two details carry the story. First, the hardware: DeepInfer supports Nvidia's SM100 (Blackwell) GPUs and Huawei's Ascend 950 NPUs — meaning a frontier AI lab is running its production service, in part, on Chinese-designed chips, a first-party endorsement of Huawei silicon that will land heavily in Washington. Second, the performance claims: DeepSeek says one cluster serves 155 trillion tokens per day with a 96% input cache hit rate (nearly all repeated work is reused rather than recomputed), earning roughly $38 per GPU-hour at off-peak pricing.
DeepSeek also released deepseek-recipe under the MIT licence, its token encoding/decoding layer — the plumbing that turns text into tokens and back. The timing is pointed: the release came one day after the US government ordered federal agencies to remove DeepSeek technology from their systems.
DeepSeek just gave away the recipe for cheap AI — and everyone charging premium prices for inference felt the floor move.
Why It Matters
Inference is where AI companies make their money back — and DeepSeek just published the machinery behind the industry's lowest prices ($0.60 per million tokens) for anyone to copy. Every company whose business is selling access to AI models at a comfortable markup now competes against rivals running the same efficiency stack. Serving software was a moat; it is becoming a commodity.
For Nvidia, the signal is double-edged. DeepInfer runs best-in-class on Blackwell, which keeps $NVDA at the center of the serving stack — but a 96% cache hit rate is also a reminder that smarter software means fewer GPU-hours burned per token. Efficiency gains historically grow total usage rather than shrink it, but the margin per token compresses first and the volume argument has to catch up.
Analyst's Take
Plain talk on the ticker first: DeepSeek is a private Chinese lab — there is no DeepSeek stock. $NVDA is the listed angle here, both as the hardware DeepInfer runs on and as the company whose chips face a new, lab-endorsed Huawei alternative. Huawei is the competitive foil in this story, and a frontier lab choosing Ascend 950s for production work is the strongest signal yet that China's fallback silicon is usable at scale.
What the market is missing is who this hurts most. Open weights attacked the model layer; DeepInfer attacks the serving layer, where the actual unit economics live. The winners are the labs with distribution and products; the losers are undifferentiated inference resellers whose entire pitch was efficient serving. Watch whether US labs and clouds adopt DeepInfer techniques (the licence lets them) — if the efficiency playbook spreads in weeks, token prices fall again, usage explodes again, and the whole cycle repeats one level cheaper.
By The Numbers
FAQ
What is inference, in plain terms?
Inference is running a trained AI model to answer real requests — every chat reply, summary or image a user gets. Training builds the model; inference is the serving work that costs money on every single query.
What exactly did DeepSeek release?
DeepInfer, the inference framework behind its production service (built mainly on vLLM, with ideas from SGLang and TensorRT-LLM), under the Apache 2.0 licence — plus deepseek-recipe, its token encoding/decoding layer, under MIT.
Why would DeepSeek give away its cost advantage?
Open-sourcing the serving stack grows the ecosystem around DeepSeek's models, pressures rivals' prices, and — with support for Huawei's Ascend 950 alongside Nvidia Blackwell — builds a serving standard that works on Chinese hardware too.
What does this mean for Nvidia?
Mixed. DeepInfer runs on Nvidia's Blackwell chips, keeping Nvidia central — but its efficiency (96% cache hits, ~$38/GPU-hour earnings) shows software squeezing more out of each chip, and its Huawei support endorses the main non-Nvidia alternative.
Source: SiliconAngle / DeepSeek