Local LLMs for companies: what hardware do you actually need?
GPU memory decides which models you can run and how many people can use them at once. A practical guide to sizing an AI server without overbuying.
The most common question I get is also the one with the most bad answers online. Here is how I reason when sizing an AI server for a company, without product names that will be outdated in six months.
GPU memory is the only number that matters first
A language model has to fit in the graphics card's memory to be fast. Everything else, CPU, RAM, storage, is secondary. Roughly, this is how it lines up:
| Model size | Memory at 4-bit quantisation | Fits in |
|---|---|---|
| 7–9 billion parameters | 5–7 GB | A basic card with 12–16 GB |
| 27–35 billion | 18–22 GB | A card with 24–32 GB, with some headroom |
| 70 billion | 40–45 GB | Two cards, or one professional card with 48 GB or more |
| 100+ billion | 60–120 GB | Professional cards with 80–96 GB, or several cards |
Quantisation means the model is stored at lower precision. At 4 bits most models lose very little quality but need roughly half the memory of 8 bits. It is the standard for local deployment.
Then come concurrent users
The numbers above are for one user at a time. Every ongoing conversation also needs room for its context, known as the KV cache. The longer the documents users paste in and the more people working at once, the more memory goes there.
A rule of thumb: expect to need 30–50 percent more memory than the model itself if ten people should be able to use the system at once with long documents. A card that just barely fits the model is a card that only fits one user.
Three typical tiers
Pilot and small teams. A consumer card with 24–32 GB in an ordinary workstation. Runs 30-billion-class models for a handful of users. Costs about as much as a good laptop and goes surprisingly far in proving the value.
A department or a smaller company. One professional card with 48–96 GB, or two consumer cards. Runs 70-billion models, or 30-billion models with many concurrent users. This is the most common recommendation I make.
The whole organisation. Several professional cards in a proper server with redundancy. This is where runtimes such as vLLM, built to handle many requests in parallel, come in, along with real monitoring.
An alternative that is often forgotten is workstations with unified memory, where CPU and GPU share one large memory pool. They are slower per response but can hold very large models at a lower price. For some workflows that is the right choice.
What you do not need
You do not need a data centre. You do not need liquid cooling, although I like it. You do not need the fastest CPU. And you almost never need to buy for training models. Running finished models, and perhaps lightly fine-tuning them, is a completely different and far cheaper class of hardware than training from scratch.
How to decide
Do not start with the hardware. Start with three questions:
- Which tasks should the model solve, and how long are the documents it will see?
- How many people will use it, and how many at the same time on an ordinary Tuesday at ten?
- What response time is acceptable? An internal knowledge search tolerates a couple of seconds. A coding assistant does not.
The answers point to a model class. The model class points to a memory requirement. The memory requirement points to a handful of concrete builds. In that order you avoid both overinvesting and ending up with a server that is not enough.