Choosing hardware for local LLMs: memory first, bandwidth second

Two numbers decide how well a machine runs a local language model. Fast memory capacity decides whether a model fits, and memory bandwidth sets the ceiling on generation speed. Compute matters less than most people expect for single-user chat. Quantization helps on both fronts, because it shrinks the weights you have to store and the bytes you have to read for every token.

Memory is the gate

Inference runs at full speed only when everything sits in the fastest memory the processor can reach: VRAM on a discrete GPU, or the unified pool on Apple Silicon. Three things compete for it:

• Weights: the quantized parameters, which are by far the largest item for short-context use.
• KV cache: keys and values for every token in context. This grows with context length and with the number of parallel requests. See KV cache quantization for the formula.
• Overhead: compute buffers, runtime allocations and, on a desktop, the display and other apps.

If the weights do not fit, the runtime either fails to load or spills layers into much slower system RAM. So start with arithmetic.

The sizing method: params × bits ÷ 8, plus overhead

Use the effective bits per weight, including scales, rather than the nominal label. llama.cpp's quantize documentation reports Llama 3.1 8B at 4.89 bits per weight for Q4_K_M, 5.70 for Q5_K_M and 8.50 for Q8_0, because block scales and mixed tensor types add to the nominal width.

A quick check: llama.cpp lists the F16 file at 14.96 GiB (16.06 GB), which implies about 8.03 billion parameters. 8.03 × 4.89 ÷ 8 ≈ 4.91 GB, and the listed Q4_K_M file is 4.58 GiB (4.92 GB). Then add the KV cache for your context and a reserve for buffers. Keeping roughly 20% free is a reasonable starting assumption at 8K–16K context.

What fits at each VRAM tier

Inverting the formula: maximum parameters ≈ (0.8 × capacity) × 8 ÷ bits per weight, keeping about 20% free for cache and buffers. These are planning estimates, not guarantees.

Mixture-of-experts (MoE) models break the simple rule. Memory must hold all experts, so size by total parameters. Each token reads only the active experts, though, so speed follows active parameters.

Memory bandwidth sets tokens per second

Each token in single-user decoding reads essentially every weight once, plus the KV cache. The arithmetic per byte is light, so the processor mostly waits on memory. That gives an upper bound:

Bytes read per token ≈ weight size (active weights for MoE) + KV cache in use. Treat the result as a ceiling: real decode speed lands below it. Prompt processing (prefill) is compute-bound and follows different rules.

This is why quantization speeds up decoding. Going from Q8_0 to Q4_K_M cuts bytes per token by about 42%, which raises the ceiling by about 74% if the dequantization kernels keep up. In batched serving, one weight read serves many sequences, so throughput grows with batch size until compute or KV cache becomes the limit.

NVIDIA GPUs, Apple Silicon and CPU-only machines

NVIDIA consumer and workstation cards

Consumer GeForce cards pair high bandwidth with limited capacity. NVIDIA lists 12 GB at 672 GB/s (RTX 5070), 16 GB at 896–960 GB/s (RTX 5070 Ti, RTX 5080) and 32 GB at 1,792 GB/s (RTX 5090). The RTX 5060 Ti has 8 or 16 GB at 448 GB/s, and the older RTX 3090 and RTX 4090 have 24 GB. Workstation cards add capacity: the RTX PRO 6000 Blackwell lists 96 GB of GDDR7 with ECC at 1,792 GB/s.

NVIDIA data-center GPUs

Data-center parts use high-bandwidth HBM. NVIDIA lists 80 GB at 2,039 GB/s for A100 80GB SXM, 80 GB at 3.35 TB/s for H100 SXM and 94 GB at 3.9 TB/s for H100 NVL. The L40S sits between the two classes with 48 GB of GDDR6 at 864 GB/s. They suit batched serving stacks such as vLLM.

Format support also depends on GPU generation. vLLM's compatibility table lists FP8 W8A8 for Ada and Hopper but not Ampere, and its INT4 W4A16 path needs an Ampere-generation or newer GPU. bitsandbytes NF4 runs on Pascal and newer. Check before you choose a quantized checkpoint (see GPTQ vs AWQ vs GGUF).

Apple Silicon unified memory

On Apple Silicon, the CPU and GPU share one memory pool, so a machine with large RAM can hold models that no single consumer GPU could. Apple's specifications list 153 GB/s for M5, 307 GB/s for M5 Pro, 460 or 614 GB/s for M5 Max depending on GPU core count, and 1.2 TB/s for M5 Ultra. M5 Ultra configurations go up to 512 GB.

Treat the headline RAM figure with caution, because by default the GPU does not get all of it. Metal publishes a recommendedMaxWorkingSetSize for each GPU, which Apple describes as an approximation of how much memory the GPU can allocate without hurting performance. llama.cpp's Metal backend reads that value as the device's usable total, prints it at startup and can warn when allocations exceed it. On Apple Silicon Macs this figure is typically below installed RAM, so read it from your own startup log rather than assuming a fixed fraction. For MLX, whose large-model handling needs macOS 15 or later, the mlx-lm documentation describes raising the system wired-memory limit with sudo sysctl iogpu.wired_limit_mb=N, where N is larger than the model but smaller than total memory. A higher limit lets the GPU wire memory that macOS and your applications would otherwise use, so leave a margin.

CPU-only inference

llama.cpp runs well on CPUs with AVX2, AVX-512 or ARM NEON. Speed again follows memory bandwidth: a dual-channel DDR5 desktop tops out near 90 GB/s in theory, while multi-channel server platforms reach several times that. CPU-only suits small models, MoE models with few active parameters and latency-tolerant batch jobs. Long prompts are slow, because prefill needs compute.

Multi-GPU splits and offloading

Splitting across GPUs

llama.cpp's default --split-mode layer gives each GPU a contiguous slice of layers along with that slice's KV cache. Capacity adds up, so two 24 GB cards hold what one 48 GB card would. Single-stream speed does not add up, because each token passes through the GPUs one after another. The experimental --split-mode tensor splits every layer across GPUs so they read their shares at the same time, which can raise single-stream speed. It needs a fast interconnect and does not support a quantized KV cache. -ts 3,1 sets an uneven split for mismatched cards. In vLLM, --tensor-parallel-size sets the number of GPUs.

llama-server -m model-Q4_K_M.gguf -ngl all -ts 1,1 -c 16384

Offloading to system RAM

When a model does not fit, llama.cpp can keep some layers on the CPU (-ngl sets how many go to the GPU), and its default --fit logic tries to choose sensible values automatically. The cost follows from the bandwidth rule. Each token's time is the sum of the GPU part and the CPU part, and the slow part dominates.

MoE models offload more gracefully. llama.cpp's --cpu-moe and --n-cpu-moe N keep expert weights in system RAM and attention plus shared weights on the GPU. Each token touches only a few experts, so less data crosses the slow path.

A practical selection checklist

1. Pick the largest model you need and the quant you will accept. Compute weight size with params × bpw ÷ 8.
2. Add the KV cache for your real context length and number of concurrent users.
3. Choose memory capacity with about 20% headroom, or more for long contexts.
4. Divide bandwidth by bytes per token to check that the speed ceiling meets your needs, and expect real results below it.
5. Confirm that your runtime and quant format support the hardware (CUDA, ROCm, Metal or CPU).