LLM VRAM calculator
Pick a model, a quantization format and a context length. You get the memory for weights, KV cache and runtime overhead, a fit check across 22 GPUs and Macs, and the arithmetic behind every number.
Model and format
Context and serving
Hardware
Same model and context, other formats
Fit check across devices
Green fits with at least 5% headroom; amber is a tight fit; faded does not fit. Numbers show usable memory.
Show the working
How the calculator works
Inference memory has three parts. Weights are the model's parameters stored in the chosen format: parameters × bits per weight ÷ 8 bytes. KV cache holds the attention keys and values for every token in the context, for every layer: 2 × layers × KV heads × head dimension × tokens × concurrent sequences × bytes per element. Overhead covers the runtime, CUDA context, activations and scratch buffers; we use 1 GB plus 5% of the weights, which matches what llama.cpp and similar runtimes typically report for single-user chat.
Bits per weight for GGUF formats are the effective values the llama.cpp quantize tool reports for Llama 3.1 8B, so they include the higher-precision embedding and output tensors that k-quant mixes keep. Larger models come out very slightly smaller per parameter. For GPTQ, AWQ, MXFP4, NVFP4 and NF4 we add the cost of the per-group scales to the nominal 4 bits. Many GPTQ and AWQ checkpoints keep the embedding and output layers in 16-bit, so for small models the real file can be noticeably larger than this estimate; for 70B-class models the difference is small.
About the tokens-per-second ceiling
Generating each new token requires reading every weight once, plus the cached keys and values. Decoding is therefore usually limited by memory bandwidth, and bandwidth ÷ bytes read per token gives a theoretical ceiling. Real throughput is lower — often 50–80% of the ceiling — because of kernel efficiency, attention compute and software overhead. Treat it as an upper bound for comparing devices, not a promise.
Ways to make a model fit
- Drop one quant level. Q5_K_M to Q4_K_M saves about 14% of weight memory with a small quality cost.
- Quantize the KV cache. 8-bit KV cache halves context memory and is close to lossless for most chat use. See KV cache quantization.
- Shorten the context. Many apps never use more than 8K–16K tokens; a 128K window you do not use still reserves memory in some servers.
- Offload layers to CPU. llama.cpp can keep some layers in system RAM. It fits larger models at the cost of speed.
- Split across devices. Two 24 GB GPUs hold what one 48 GB card holds, minus some duplication.
Frequently asked questions
Why does my runtime report a slightly different number?
Runtimes allocate memory in different ways: some reserve the full context up front, some grow it, and some keep extra buffers for batching or graph capture. Expect a difference of a few percent, more for servers tuned for throughput such as vLLM, which pre-allocates a fraction of all GPU memory by default.
Is GB here 109 or 230 bytes?
230 bytes (a gibibyte). That matches how GPU memory is reported by drivers and how llama.cpp reports file sizes, so a "24 GB" card really offers about 24 of these units.
How much memory can a Mac's GPU use?
Less than the installed RAM. macOS reports a recommended working-set limit for the GPU, which runtimes such as llama.cpp print at startup. Apple does not publish a fixed fraction, so we assume about two-thirds on machines with up to 36 GB and three-quarters above that. Advanced users can raise the limit, but leave room for the operating system and apps.
Do I need extra memory for fine-tuning?
Yes, much more. Training needs gradients and optimizer states as well as weights. Parameter-efficient methods such as QLoRA reduce this by keeping the base model in 4-bit and training small adapters, but this calculator is for inference only.