On 20 October 2026, llm-d project contributors announced llm-d 1.0 at the PyTorch Conference North America. It is a Kubernetes-native distributed inference serving network: it serves whatever model you point it at, and is not a model itself. The llm-d/llm-d repository is Apache-2.0 licensed, and IBM Research says the project is hosted by the CNCF as a sandbox project.

The next day, on IBM's third-quarter 2026 earnings call, CEO Arvind Krishna described an average large enterprise starting with “maybe two or three racks” to run open-weight models. Those models, he said, cost “anywhere from 25% to 10% of the token cost of the frontier models” and are “now 90% as effective”. Then he asked:

“Would you like something that's 10% of the cost if it's 90% as good as the best, or do you want to pay that 10 times higher cost? Then the question is: how many tasks require that?”

He was describing his customers, and his range is 4x to 10x cheaper per token, not a flat 10x. But that last question is the one I care about, because I pay my own model bills. It can be answered with a test rather than an opinion, and this article is that test.

Open weight models: the test that decides which jobs can move

I run a small content engine on my own server. Before a job moves from the premium model to a cheaper route, it goes through the same six steps.

1. Write the acceptance check before the run. A script that answers pass or fail, using objective criteria only: exact file names, exact dimensions, non-blank output, structural invariants, and a manifest that matches what is on disk. If a criterion needs taste, it does not belong here.
2. Run the check against an empty directory and confirm it fails, with a non-zero exit code.
3. Run the identical job on both routes from separate working directories. Same prompt, same inputs, one directory per route, so neither run can pick up the other's files. This is only easy if your pipeline can switch models without a rewrite, which is why the integration, not the model, is the real lock-in.
4. Record output tokens, total tokens, turns and wall time for each side. Written down per session, not remembered.
5. Price each side at its own rates. Never price both runs with one rate card.
6. Look at the artefact, not just the checker. The check proves the floor. Your eyes catch what it did not think to test.

On disk, it looks like this:

mkdir run-empty run-premium run-cheap
./verify run-empty      # must exit non-zero
# same prompt, same inputs, one route per directory
./verify run-premium    # note tokens in/out, turns, wall time
./verify run-cheap      # same fields, priced at its own rates

Step 2 is the one people skip. A checker that loops over output files and reports errors will report no errors when there are no files. It exits cleanly, like a pass. That mistake is cheap to make and expensive to trust: if your check passes an empty directory, a pass from the cheap model tells you nothing.

Why a price list is not a bill

A price list describes tokens. A job is made of turns, retries, verbose answers and your own time, and each of those moves the real number.

• Turns. Each extra turn usually re-sends the conversation so far. A cheap model that needs three turns is not cheap.
• Retries. A run that fails the check gets paid for twice.
• Verbosity. On most APIs output tokens cost more than input tokens, so compare output tokens on their own, not just totals.
• Wall time and your time. A slow route holds up the pipeline, and every failure you inspect by hand costs you, even though no invoice shows it.

So our rule is blunt: a claimed saving from a price list is not a saving. We have seen the same job cost more on the “cheaper” route once retries were counted.

The honest limitation is that the cheap route can win on paper and still lose on a job with many small steps, because a small error rate per step adds up across a long chain. Krishna's 90% describes an average large enterprise's workloads, not your specific job.

What llm-d 1.0 actually changes if you run your own inference

The techniques llm-d builds on are not new in 1.0: prefix-cache-aware routing, KV-cache offloading, batch optimisation, prefill/decode disaggregation and wide expert parallelism all existed before. According to IBM Research, 1.0 adds the day-2 operational work around them:

• Multi-tenant quality of service and flow control, so one bursty batch agent does not starve a latency-sensitive service on the same GPUs.
• Zero-downtime operations, meaning upgrades that do not interrupt service.
• Agent-aware serving, for agents that send long, repeated context turn after turn.

The cost argument rests on reuse. On the earnings call, Krishna said llm-d reuses previously processed context and helps clients run large AI models at 5 to 10 times lower cost by making more efficient use of existing GPU infrastructure. IBM Research reports that in a recent agentic coding workload test, llm-d's prefix-aware router captured 97% of the available cache reuse, and that IBM and Red Hat ran 3,000 software-engineering agents concurrently on a self-hosted deployment for 5 to 10 times less than the actual price of serving the same open model as a service. These are IBM's own reported results, not independent benchmarks.

The project was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA, with AMD, Intel and others now contributing. Having several hardware vendors behind one serving layer makes it less likely to tie you to any one of them, which is the problem I covered in what TorchTPU teaches about GPU lock-in.

Now the caveat. This is Kubernetes-shaped infrastructure. It solves problems you only have once inference is a service with tenants, a latency target and an on-call rota. If you are running your first model on one box at home, llm-d is not your next step, and Krishna's “two or three racks” is not a homelab shopping list. Hardware, power and your own time are real costs that an efficient serving layer does not remove.

The split we actually use

Our routing is a written rule, not a feeling about which model seems smarter this week.

• Content work stays on the premium model. Anything that gets published: scripts, narration, captions and design judgement. Cheap models lose on judgement, and a bad published artefact costs more than the tokens saved.
• Mechanical work goes to the cheap route. Batch image and file prep, bulk formatting, data shuffling.

Neither route gets a job without an acceptance check that was written first and failed on an empty directory. The dividing line is whether a script can tell me the output is right. Image dimensions are checkable. Whether a caption is good is not.

The limitation is real: the check takes time to write. For a one-off job, writing and proving a verifier can cost more time than the move would save, so one-offs run on whatever route is already set up. The test pays off on jobs that repeat.

When to stay on the API

Stay on a hosted API, and often on the premium model, when:

• Your volume is small. The token gap is then a small number, and an API is often cheaper than hardware you would have to buy, power and maintain.
• The work is latency-sensitive and the hosted route is already fast enough.
• You do not already own a GPU. Buying one to trim a small bill rarely adds up.
• You cannot write an objective check. If pass or fail needs a human's taste, you cannot prove the cheaper route is good enough, so do not move it.

Moving a job to an open-weight model also does not require hosting it yourself. Many providers serve open models through an API, so you can run the test, and switch if it passes, without buying any hardware.

Krishna's question about how many tasks require the frontier tier is a good one. The answer for your setup is in a verifier that fails on an empty directory, two working directories, one prompt, and a bill priced at each side's own rates.

If you want someone to work out which of your workloads can move, and prove it before you commit, that is the kind of work I do. The details are on the services page. Fiverr: hiteshsaini459 · Upwork: hiteshsaini25