DeepSeek Open-Sources Its DeepInfer AI Engine
DeepSeek open-sourced DeepInfer, the inference engine behind its production AI services, with support for Nvidia Blackwell and Huawei Ascend 950 chips.
DeepSeek has released DeepInfer, the inference framework that powers its own production AI services, as open-source software on GitHub. The release, published on October 20, 2026 under the Apache 2.0 license, gives developers a rare look at the real serving stack of a frontier AI lab. It currently supports a single model, DeepSeek-V4.1-Flash, and runs on Nvidia's newest Blackwell chips as well as Huawei's Ascend 950.
What DeepInfer is
An inference engine is the software that serves a trained AI model to users. It splits the model across chips, batches incoming requests, and controls how fast answers come back. The cost of serving is a big part of what an AI company spends for every answer. DeepSeek calls serving a massive user base at a reasonable price "the core problem an inference framework has to solve". According to its GitHub README, DeepInfer was originally built on top of the open-source vLLM engine, with ideas taken from SGLang and TensorRT-LLM. It bundles optimized kernels from DeepSeek's earlier open projects, including FlashMLA, DeepGEMM and DeepEP, plus TileKernels, DeepSelect and custom TileLang kernels.
What it can do
- Supports DeepSeek-V4.1-Flash, which has a 1M-token context window, for now the only model in this release. - Runs on Nvidia SM100 (Blackwell) GPUs and Huawei Ascend 950 NPUs. Older Nvidia architectures such as Hopper are not supported. - DSpark speculative decoding: a draft model proposes five tokens at a time, and a confidence head predicts which ones the main model will accept. Verification compute goes only to the likeliest candidates. DeepSeek says this cuts verified draft tokens by about 30% and adds 4% to 7% throughput at production load. - Sliding-window attention bounded replay: on a cache hit, only the last 128 tokens are recomputed, which DeepSeek says nearly halves total prefill computation. - Batch-invariant inference and deterministic sampling, available on Nvidia hardware only, which keep results identical across runs with the same inputs. - Structured outputs constrained to a regex, JSON schema or EBNF grammar. - Performance figures published by DeepSeek, not independently tested: one cluster serves 155 trillion tokens per day with a 96% input cache hit rate, and decode tops 25,000 output tokens per second per GPU. DeepSeek also published deepseek-recipe, a separate MIT-licensed component that converts text into tokens and back using the same logic as its official API. The company says inference code for future models will land in the same repository.
What it does not do yet
The release is narrower than the name suggests. It covers offline inference only. The KV cache management system, prefill and decode disaggregation across separate machines, and load balancing across replicas are not included. DeepSeek says those components are tightly coupled to its internal infrastructure. Operators who want to run a full live service must add those parts themselves.
Why it matters
Frontier AI labs rarely publish the software that serves their models in production, so this is a rare look inside a real serving stack. Cloud providers, chip makers and independent developers can study exactly how DeepSeek keeps its serving costs low. The Ascend 950 support is notable. This is the first time a frontier lab has given Huawei's chips first-class support alongside Nvidia's newest GPUs, and it comes as DeepSeek reportedly plans to pre-train future models on Ascend hardware. DeepSeek also plans to deploy at least 160,000 Ascend 950DT accelerators at a new data center in Inner Mongolia, according to Bloomberg reporting. The timing is striking too. DeepInfer arrived one day after the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, including third-party services built on its technology. The open-source release drew thousands of GitHub stars within hours and became one of the most discussed AI infrastructure stories of the day.
Sources
- SiliconANGLE - https://siliconangle.com/2026/10/20/deepseek-open-sources-deepinfer-the-inference-framework-behind-its-production-services/ - MarkTechPost - https://www.marktechpost.com/2026/10/20/deepseek-deepinfer-open-source-inference-framework/