DeepInfer Is Apache-2.0, but Check the Hardware Line First

On 16 October 2026, DeepSeek published DeepInfer on GitHub under the Apache-2.0 license. The repository describes it as a "high-performance, large-scale inference framework for DeepSeek models", and the README says it is the framework that powers DeepSeek's production services. Within days it had about 15,000 stars.

Most of the coverage was about cost and about pressure on closed-source rivals. Bloomberg described it as DeepSeek opening up the software it uses to run its models at scale, and SCMP called it a framework designed to make AI models cheaper and more efficient to deploy. The release is real, and the license is genuine. If you run models on your own hardware, the useful question is not whether it is open source. It is what hardware it needs, and what DeepSeek left out.

What this open-source inference engine actually is

Start with where it came from. The first line of setup.py reads "Derived from vLLM; modified by DeepSeek". The README's acknowledgments say DeepInfer "was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM". It is a vLLM derivative, and the project says so in its own files.

That is good news if you already run vLLM. The general shape of the code, with a scheduler, model code and a layer of custom kernels underneath, will feel familiar. Your mental model transfers, even where the details differ.

Next, look at the scope. The README says: "This open-source release provides offline inference with the core features needed for serving at scale." The supported models section lists one entry, DeepSeek-V4.1-Flash, with deepseek-ai/DeepSeek-V4.1-Flash-1017 on Hugging Face as the default checkpoint. This is an engine built to run one model family as efficiently as possible. It is not a general engine you point at whatever model you downloaded last week.

"Offline inference" deserves a plain explanation. It means you hand the engine a batch of prompts from a script, it processes them, and you collect the results. It is not a server that sits on a port and answers chat requests from many users at once, with a shared cache and a router in front. If you were picturing something like the OpenAI-compatible server you start with vLLM, that is not what shipped.

The hardware line is the real headline

The README lists exactly two supported targets: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Nothing else appears on the list. SM100 refers to NVIDIA's data-center Blackwell generation, and consumer GeForce cards report a different compute capability.

The prerequisites follow from that. You need Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 on NVIDIA or CANN 9.2 on Ascend. The README describes a source build with a script and an environment variable that selects the backend:

export DEEPINFER_ACCELERATOR=nvidia   # or ascend
./build.sh

Then there is the benchmark rig. The benchmark section says its numbers were "Measured on 4 SM100 GPUs for prefill and 64 for decode". That is 68 data-center accelerators behind the published figures. Those numbers describe a cluster, not a workstation.

If you own a 24 GB consumer card, here is the blunt version: this release is not for running on your machine. You can check what your card reports in a few seconds:

nvidia-smi --query-gpu=name,compute_cap --format=csv

An RTX 3090 reports 8.6 and an RTX 4090 reports 8.9. Neither is on the list. There is a community attempt to port the engine to SM80, but that is an outside experiment, not official support. Your weekend is better spent on the engine you already run.

This is the same question I raised in how to avoid GPU lock-in. A permissive license does not help you if the code only runs on hardware you will never buy. The license tells you what you are allowed to do. The hardware list tells you what you can actually do.

What is excluded, and why it matters

The README is honest about the gaps. It lists three components that are required for a complete inference service and are not included in this release:

A KV cache system. This stores the attention state for text the model has already processed, so shared prompts and long conversations are not recomputed on every request.
VE/Prefill/Decode disaggregation. This runs different phases of a request on separate pools of hardware, so the heavy prompt-reading work and the token-by-token generation work do not compete for the same GPUs.
Load balancing across data-parallel replicas. When you run several copies of the model, something has to decide which copy gets each request.

Look at that list next to the benchmark line again. The published numbers came from a split setup, with 4 GPUs for prefill and 64 for decode, but the component that manages the split is not in the repository. You can read how the engine behaves inside that setup. You cannot rebuild the setup from what was released.

This is the difference between a benchmark and a service. A benchmark measures an engine under controlled load. A service also has to handle repeated prompts, uneven traffic, replicas that fill up, and requests that arrive while others are halfway done. The excluded pieces are exactly the parts that turn a fast engine into something users can depend on.

What a self-hoster can still take from it

None of this makes the release useless. It makes it reading material instead of something to install.

Start with the kernels it builds on: FlashMLA, DeepGEMM, DeepEP, TileKernels and DeepSelect. Each one lives in its own repository under the same deepseek-ai organization, so you can study them one at a time. Reading how a production team writes attention, matrix-multiply and expert-communication kernels teaches you more than most blog posts will.

Then there are the techniques worth copying into your own thinking:

Batch-invariant inference. Normally the same prompt can produce slightly different output depending on what else shares its batch. DeepInfer exposes DEEPINFER_BATCH_INVARIANT=1 to remove that effect. It is NVIDIA only.
Deterministic sampling. A fixed seq_seed lets you reproduce a sampled result, which matters for testing and for comparing prompt changes fairly. This is also NVIDIA only.
Grammar-constrained structured output. Through so_args, you can constrain output with a regex, a JSON schema or an EBNF grammar. If you parse model output in scripts, this is the habit to adopt in whatever engine you use.
Speculative decoding with confidence-scheduled verification. The general idea is that cheap guesses are checked by the main model, and how much checking happens depends on how confident the guesses are.
Bounded replay of a cached prefix. The idea is to reuse work already done on a shared prompt prefix while putting a limit on how much gets replayed.

The community has also started producing study material. nano-deepinfer is a small, readable engine under the MIT license, and core-deepinfer is an inference loop written in NumPy. I have not checked how faithful either one is to the original, so treat them as study notes rather than references. Still, the fact that they appeared within days tells you how many people want to read this code.

A five-minute checklist for any "we open-sourced our engine" announcement

DeepInfer will not be the last release of this kind. Before you clear a weekend for the next one, check these five things in the repository itself:

License: read the LICENSE file, not the press release.
Supported hardware: find the exact list and compare it with your compute_cap output.
Supported models: check whether your model is named or only one family is listed.
What is excluded: search the README for "not included" and read that section twice.
Can it build on my machine: compare the compiler, Python and CUDA versions with what you have installed.

For DeepInfer, the answers come out as Apache-2.0, SM100 and Ascend 950, DeepSeek-V4.1-Flash only, no KV cache system, no disaggregation, no load balancing, and GCC 13, Python 3.12 and CUDA 13 or CANN 9.2. That takes five minutes to find, and it settles the weekend question.

Where this leaves you

DeepInfer is a genuine open release and a useful window into how one team runs its own model at scale. For most people reading this blog, it is something to study rather than something to deploy. The engine you already run on the card you already own is still the right place to put your time.

If you want your inference stack chosen and tuned for the hardware you actually have, take a look at my services page.

Fiverr: hiteshsaini459 · Upwork: hiteshsaini25