Will it fit?

KV cache maths for self-hosted LLM inference. Pick your model, your GPUs and your context length — see how the memory actually divides, how many concurrent sessions you'll really serve, and the vLLM flags to run it.

Read these off the model's config.json on Hugging Face: num_hidden_layers, num_key_value_heads, and head_dim — falling back to hidden_size ÷ num_attention_heads only when head_dim is absent, since several models set it independently.

Memory map

Every GiB on the node, to scale.

Weights Activations & CUDA graphs KV cache Held back by utilisation cap
—

Where the memory goes

Per-token KV cost is 2 × layers × kv_heads × head_dim × bytes. The 2 is for K and V. Note it does not scale with the number of attention query heads — grouped-query attention is the reason long context is servable at all.

Precision trade-offs

Your current setting is highlighted. This table tends to surprise people: quantising the KV cache buys far more concurrency than quantising the weights, because weights are a fixed cost paid once while KV scales with every token of every live session.

Run it