Sign in Subscribe

Buying a GPU for Local AI: The Memory Is the Machine

Buying a GPU for Local AI: The Memory Is the Machine

Wccftech reported on 23 October 2026 that GeForce RTX prices in China's channel market have gone up again, citing price tracking from Board Channels and the @harukaze5719 account. The RTX 5080 16 GB moved from 12,700–13,500 yuan to 14,500–15,000 yuan, a rise of more than 2,000 yuan that the outlet puts at about $300. The RTX 5070 Ti 16 GB went from 9,000–9,150 yuan to 9,650–9,850 yuan, about 600 yuan or roughly $100. These are partner and retail prices in China, not a new price announced by NVIDIA.

On 22 October, ASUS added 1,000 yuan to all its RTX 5080 16 GB models and 200 yuan to its 5070 Ti, 5070 and 5060 Ti 16 GB models. Gigabyte added 1,200 yuan to its RTX 5080 models and 200 yuan to its 5070 Ti and 5070 models. If you are buying a card to run models at home, though, the headline price is the wrong number to watch. The number that matters is what you pay for each GB of VRAM.

What VRAM buys you when you run AI locally

When you run AI locally, the model has to sit in your GPU's memory to run at a useful speed. If it does not fit, tools like Ollama and llama.cpp push part of it into system RAM and run those layers on the CPU. That still works, but it is much slower.

Here is a rough rule for the 4-bit quantized models most people download, such as the Q4_K_M files that are the usual default. Budget a bit over half a GB per billion parameters, plus extra room for the context window. Longer chats and bigger documents need more of that extra room.

Here is what that looks like in practice:

  • 8 GB: 7B to 8B models such as Llama 3.1 8B or Qwen 2.5 7B, at about 5 GB, with a modest context. Fine for quick chat and autocomplete, and not much else at the same time.
  • 16 GB: 14B models such as Qwen 2.5 14B or Phi-4, at around 9 GB, with room for a long context. You can also squeeze in a 24B model like Mistral Small (about 14 GB) if you keep the context short, or keep an 8B chat model and an embedding model loaded together.
  • 24 GB: 32B models such as Qwen 2.5 32B, at around 20 GB, with some context to spare. This is the tier where local models get good enough for everyday coding and writing help.
  • 70B models: roughly 40 GB or more at 4-bit. That means two 24 GB cards or heavy offloading to the CPU.

An 8 GB card will run a model, but it leaves you no room to grow. Feed it a long PDF or a big chunk of code and the context pushes you off the GPU. The step from 8B to 14B is also where summarising documents, answering questions over your own files and code help start to feel dependable. That is why I treat 16 GB as the honest floor for a useful local model.

Sixteen GB also gives you room to share. I wrote up how to do that in this guide to running several models on one GPU. To check whether your model really fits, load it and run:

ollama ps

The PROCESSOR column should read 100% GPU. If it shows a split like 48%/52% CPU/GPU, the model plus its context is too big for your card.

Why memory now sets the price of the card

Wccftech puts this round down to memory. GDDR7, the memory on RTX 50-series cards, is getting more expensive and harder to buy because memory makers are prioritising HBM, the stacked memory used on AI datacenter accelerators.

The background is simple. A few companies, mainly Samsung, SK hynix and Micron, make almost all of the world's DRAM, and that includes GDDR7, HBM and the DDR5 in your desktop. Their factory capacity is limited. When datacenter buyers pay more for HBM, more of that capacity goes to HBM, and consumer memory gets what is left. Card makers pay more for GDDR7 and pass the cost on.

I would not treat this as a one-quarter blip. Partner prices in China have gone up in several rounds since late July 2026. VideoCardz reported ASUS and Gigabyte increases in August and again in September, and now there is this one. New memory factories take years to build, and the AI demand pulling on supply has not eased. The outlet also expects price moves like these in China and Asia to reach US and EU retail in a week or two, so do not assume your local shop is safe.

There is also a rumour worth knowing about, and it is only a rumour. Reports say NVIDIA may stop making the 16 GB RTX 5080, and that a 24 GB replacement, expected around Q1 2027, could cost around 19,999 yuan (about $3,000). Nothing has been announced. If it happens, that tier would get more VRAM at a higher entry price, which fits the same pattern: memory is what you are paying for.

A buying framework built on cost per GB

An earlier post covered shopping for local AI hardware by memory rather than price tag. Here it is with today's numbers.

Divide the price by the VRAM

At the reported China prices, the RTX 5080 16 GB works out to roughly 906–938 yuan per GB. The RTX 5070 Ti 16 GB works out to roughly 603–616 yuan per GB. Both have 16 GB, so both run the same models. The 5080 generates tokens faster, but you pay about 50% more for every GB.

If the rumoured 24 GB card does arrive at 19,999 yuan, that comes to about 833 yuan per GB. That would be cheaper per GB than the 16 GB RTX 5080 is today, but still well above the 5070 Ti.

To decide what to pay, find the cheapest new 16 GB card with a proper warranty in your country and work out its price per GB. That is your baseline. Only pay more per GB when you can name the reason: either a card with more VRAM that moves you up a model tier, or extra speed for a workload you run all day. Speed is nice. Capacity decides what runs.

The used 24 GB route

If you want 24 GB without paying flagship prices, most people go for a used RTX 3090, and used RTX 4090s turn up too. It can be a good deal, but the risks are real:

  • No warranty, or only a short one from the seller.
  • Worn fans and dried-out thermal paste or pads, especially on cards that ran hot for years.
  • Unclear history. In a listing photo, a card that ran around the clock for mining or rendering looks the same as one that sat in a gaming PC.
  • Power. The RTX 3090 is a 350 W card with short spikes above that, and an older PSU may not cope.

My rule is that a used 24 GB card is worth it only when its price per GB is at or below your new 16 GB baseline. You are taking on the risk, so you should pay less per GB, not the same. If you can, test the card before paying. Load a model that fills most of the memory and watch it for half an hour:

nvidia-smi --query-gpu=name,memory.used,memory.total,temperature.gpu,power.draw --format=csv -l 1

You want steady temperatures, no crashes or driver resets, and fans that do not rattle.

Why waiting has not paid off

Through 2026, waiting for prices to drop has been a losing bet. Partner prices in China have risen in several rounds since late July, so anyone who held off over the summer now pays more for the same card. Prices can only fall if memory supply loosens, and nothing in the current reporting points that way. If you need the card for a real job, waiting means months without it.

When renting beats buying

A GPU is not always the right answer. A pay-per-token API or an hourly cloud GPU makes more sense if you use a model a few times a week, if the model you need is 70B or larger, or if you are not yet sure which model does your job. Buying wins when you use it daily, when your data cannot leave your network, or when you want something always on, like a voice assistant for Home Assistant or document tagging in Paperless-ngx.

There is a cheap middle step. Rent a 24 GB GPU for an hour or two and run your real workload on a 14B and a 32B model before you buy hardware.

What to do if you were about to build

  1. Pick the model first. Write down the job, such as code help, chat, or questions over your own documents. Then find the smallest model that does it well, tested on your own prompts.
  2. Size the VRAM to that model. Take the model file size and add a few GB for context and overhead. A 14B model at about 9 GB fits a 16 GB card. A 32B model at about 20 GB needs 24 GB. If you want two models loaded at once, add both sizes together.
  3. Budget the whole box. Check the card maker's recommended PSU and leave some headroom. Plan for at least 32 GB of system RAM so loading and any CPU offload do not crawl, and price it now, because it comes from the same memory makers. Model files are often 5 to 20 GB each, so keep them on a fast NVMe drive. Then work out the power cost: watts × hours × your price per kWh. A card drawing 300 W for four hours a day uses 1.2 kWh a day, or about 36 kWh a month.
  4. Buy the card that fits, not the one on sale. A discounted 12 GB card is no bargain if your model needs 16 GB. At the China prices above, pick a 5080 over a 5070 Ti only if the extra speed is worth paying about 50% more per GB.
  5. Check it after install. Run your model, then run ollama ps. If it does not say 100% GPU, cut the context length (for example with OLLAMA_CONTEXT_LENGTH=8192) or drop down a model size.

If you want a second pair of hands

Sizing a local AI box is mostly arithmetic. Getting it running cleanly takes longer: the right drivers, a locked-down model server, and nothing exposed to the internet by accident. If you would rather have someone size and harden this for you, see my services page.

Fiverr: hiteshsaini459 · Upwork: hiteshsaini25