How to Avoid GPU Lock-In: What TorchTPU Teaches Self-Hosters

On 20 October 2026, at PyTorch Conference 2026, Google announced that TorchTPU, its native PyTorch stack for Google TPUs, is now open source on GitHub. Google says it was co-developed with the PyTorch team at Meta, and that existing PyTorch workloads can run on TPUs "with minimal code changes". The code is at github.com/google-pytorch/torch_tpu, and the README says it is licensed under Apache 2.0. Google also lists integrations with vLLM, SGLang, llm-d, TorchTitan, Ray, TorchRec and Hugging Face.

Google's blog and the repo are the right places to read about the launch itself. I want to use it for something more useful if you run AI at home: a check on how hard it would be to move your own stack to different hardware.

To be clear up front: I have not run TorchTPU. I have no TPU, and TPUs are cloud hardware you cannot put in a homelab. I run local models on my own hardware and self-host the stack around them, and that is where this article comes from. There are no benchmarks here, because I have not run any.

The lesson is simple. The cost of switching hardware is decided in the software layer, and this week that layer got more portable. So the question to ask about your own setup is this: would your models, your serving engine and your data survive a change of accelerator?

The checklist: how to avoid GPU lock-in in your own AI stack

You can work through these six points today, on the hardware you already own. None of them needs a new purchase.

1. Keep the weights portable

Store your models in open formats: safetensors for the original checkpoints, GGUF for llama.cpp style quantised builds. Download them from the source, usually the model author's own page, and keep a copy on your own disk.

Avoid setups where the only copy of a model lives inside one vendor's container image, or in a format that only one runtime can read. If that runtime goes away or changes its licence, your model goes with it.

Can you list every model you serve and point to the file on disk?
Do you know where each one came from and which licence it carries?
If you only have a quantised GGUF, do you also have the original safetensors, or know exactly where to get them? Converting down is easy. Converting back up is not possible.

2. Keep the serving layer swappable

If vLLM, SGLang and llama.cpp can all load your model, the accelerator becomes a price decision instead of a rewrite.

The way to get there is to make your apps talk to the model server through one common interface. vLLM, SGLang and llama.cpp's llama-server can all expose an OpenAI-compatible HTTP API. If your chat UI, your scripts and your agents only know a base URL and a model name, then swapping the engine underneath is a config change.

What to avoid: application code that imports one engine's Python internals everywhere, or depends on a feature only one engine has, with no fallback.

A good test: start a second engine against the same model on a different port, point one app at it, and write down everything that breaks. That list is your lock-in, in plain text.

3. Run the inference stack in containers you control

Put the model server, the UI and anything around them in a Docker Compose file you wrote and understand. Pin image tags. Mount models as volumes instead of baking them into images. Keep settings in an env file.

Done this way, moving the stack to another machine or another provider is a copy of the model folder, the compose file and the env file, not a rebuild from memory.

The hardware-specific parts, such as GPU device reservations or an image built for one accelerator, should live in one place. I keep them in a small override file per machine, so the main file stays the same everywhere:

docker compose -f compose.yml -f compose.gpu.yml up -d

When the hardware changes, you write a new override file. Everything else stays.

4. Measure the number that decides migrations

The sticker price per GPU-hour tells you very little. The number that decides whether a move is worth it is cost per token or per request on your real workload.

A cheaper hour that serves fewer requests is not cheaper. A faster card that sits idle most of the day may not be either. At home, the cost includes power, the hardware spread over its useful life, and your own time.

Most OpenAI-compatible servers return token counts in each response. Log them, count your requests over a week, and divide what you spent by what you served. Once you have that number for your current setup, any new option, cloud or local, can be compared against it honestly.

5. Keep the data and the eval set portable too

Your prompts, system prompts and fine-tuning data should sit in plain files, such as JSONL or Markdown, in a git repo you own. They should outlive any vendor.

Just as important is a small eval set: real prompts from your own use, with notes on what a good answer looks like. Run it before and after every change of model, engine or hardware. Without it, you cannot tell whether a move worked or quietly made things worse.

If you fine-tune, keep the training data and the exact recipe, not only the adapter. An adapter is tied to its base model, and you may need to train it again after a move.

6. One honest limit

All of this reduces the cost of switching. It does not make hardware interchangeable.

Silicon still differs in memory, in interconnects, and in what actually runs on it. Quantisation formats, kernels and supported features vary between backends. A self-hoster with one GPU is not going to fail over to a TPU this weekend, and nobody should pretend otherwise.

The realistic goal is smaller: when you do decide to move, it should be a weekend of testing, not a month of rewriting. I go through the same idea for the rest of a self-hosted setup in my vendor lock-in exit plan.

What the TorchTPU launch actually shows

Here is how Google, in a Google Cloud Blog post by Mark Lohmeyer and Bill Jia, described the launch:

"Today at PyTorch Conference 2026, we announced a range of new products and integrations designed to help you run PyTorch effectively on Google Cloud or in your own infrastructure. Notably, TorchTPU, our native PyTorch stack for Google TPUs, is now available as open source on GitHub. With TorchTPU, TPUs join GPUs and CPUs as a native option for PyTorch."

A few things stand out to me.

It was built with the PyTorch side, not around it. Google says TorchTPU was co-developed with the PyTorch team at Meta. Google also says it supports eager mode for debugging, then torch.compile for production performance, which is the normal PyTorch workflow.
It is open. The repo is under Apache 2.0, so anyone can read it, fork it and build on it.
It plugs into tools people already use. Google lists vLLM (a native PyTorch backend developed with Inferact), SGLang (with RadixArk), llm-d for distributed inference, TorchTitan, Ray, TorchRec, Hugging Face and LiteRT.
The Hugging Face path is short, by Google's account. Google says that in Hugging Face Transformers v5, deploying models to TPUs is "as simple as passing device_map="tpu"", and that Diffusers runs natively on TPUs.

These are Google's claims about its own work. I cannot check them, and I am not going to guess at performance. People with TPU access will test that.

What I can say is what the launch tells us about how lock-in works. CUDA's advantage was never only the silicon. It was that the software layer made leaving expensive. Your training code, your serving engine and your tutorials all assumed one vendor, so moving meant rewriting.

When the framework and the serving engines people already use treat another accelerator as a native option, that cost of leaving goes down. That is the part that matters, whatever you think of TPUs.

The self-hoster's version of the same question

The point here is not "switch to TPUs". Most readers of this blog will never touch one, and neither will I.

The point is that the vendor's software layer decides how much your stack costs to leave. Google is spending real effort to lower that cost at the framework level. A self-hoster can make the same decision for their own stack, cheaply, by keeping the model, the server and the data independent of the hardware underneath.

Think about the moves you might actually make in the next year or two:

Your current GPU at home, then a bigger rented cloud GPU for a model that does not fit.
A card from one vendor now, a card from a different vendor at your next upgrade.
A local model for private work, with a hosted model as a fallback when you need more.

If your weights are open files, your apps talk to a standard API, your stack is a compose file and your eval set lives in git, each of these is an experiment you can run and undo. If not, each one is a project.

Here is a five-minute audit you can do tonight:

Where are my model files, and are they in safetensors or GGUF?
Could a second serving engine load my main model without changes to my apps?
Could I rebuild this stack on a fresh machine from files I have in one folder?
Do I know my cost per request today?
Do I have a set of test prompts that would tell me if quality dropped?

Every "no" is a small job. None of them needs new hardware.

The lesson: lock-in is decided before you buy the hardware, in the formats, the serving engine and the data. The cheapest time to make those portable is while everything still works.

If you want a second pair of eyes on your self-hosted AI stack, or help moving it between machines or providers, take a look at my self-hosting services.

Fiverr: hiteshsaini459 · Upwork: hiteshsaini25