How to Estimate GPU Memory for Self-Hosted LLM Inference

Why model size alone does not determine GPU memory needs

A model’s parameter count gives you a useful first estimate, but it is not a complete GPU memory budget. Inference also needs memory for the key-value (KV) cache and activations, while the runtime, model architecture and serving configuration can affect the amount actually used.

A model that fits when loaded may still exceed available memory when requests arrive. Longer sequences and more active requests increase KV-cache demand. Other GPU workloads and runtime overhead also compete for memory.

Treat any calculation as a planning estimate, not a hardware guarantee. The result depends on the particular model, inference runtime and serving configuration.

• Estimate weights and KV cache as separate components.
• Describe the workload, including sequence lengths and concurrency, before comparing estimates with available GPU memory.
• Validate the planned configuration on the intended deployment rather than relying on a model name or parameter count alone.

Estimate model-weight memory, including quantization

For a first-pass estimate, multiply the number of model parameters by the bytes used per parameter. A common rough mapping is FP32 at 4 bytes, FP16 at 2 bytes, INT8 at 1 byte and INT4 at 0.5 bytes per parameter.

For example, an illustrative 8-billion-parameter model would have an idealized weight estimate of 16 GB at 2 bytes per parameter, 8 GB at 1 byte, or 4 GB at 0.5 bytes. These are decimal arithmetic estimates for the weights only; they do not include the KV cache, activations or runtime needs.

Quantization can reduce weight memory by using fewer bits per parameter, but the estimate is not a complete description of the deployed model. Check the actual model format and runtime, and consider precision trade-offs when deciding whether a quantized configuration is suitable.

• Record the parameter count and the precision or quantization format you intend to serve.
• Calculate weights as parameter count × bytes per parameter.
• Keep units consistent: decimal GB and binary GiB are different units, and GPU tools may report memory differently.
• Do not treat the result as the total memory required to serve requests.

Account for context length and the KV cache

During inference, the KV cache stores key and value data for tokens in the active sequences. Its memory use depends on batch size, sequence length, number of layers, number of KV heads, head dimension and bytes per cache element.

A useful estimate is: KV-cache bytes = 2 × batch size × sequence length × number of layers × number of KV heads × head dimension × bytes per cache element. The factor of 2 accounts for keys and values.

For an explicitly hypothetical example, suppose a configuration has 32 layers, 8 KV heads, a head dimension of 128, a 2-byte cache element, a 4,096-token sequence and a batch size of 1. The formula gives 536,870,912 bytes, or about 0.5 GiB. At batch size 4, with all other assumptions unchanged, the estimate is about 2 GiB. These values illustrate the calculation only; they do not describe a particular model.

For planning, use the sequence length the runtime may hold at once, including both prompt tokens and generated tokens. A long context limit does not mean every request will use that entire limit, but peak-length requests can drive peak cache demand.

• Find the model’s layer count, KV-head count and head dimension from reliable model documentation or configuration.
• Use the cache data type and serving behavior that match the intended runtime.
• Estimate both typical sequence lengths and the longest sequences you expect to serve.
• If the model uses grouped-query attention, use its KV-head count rather than assuming the number of query heads.

Consider concurrent requests, batching and shared GPU workloads

The KV-cache formula scales with both batch size and sequence length. As a result, several active requests can require substantially more cache than one request, particularly when requests have long contexts.

Concurrency and batch size are related but should not be treated as interchangeable without checking the serving configuration. A server may combine requests into batches or manage active sequences in other ways. Base the estimate on the behavior you expect from the actual runtime.

Also account for anything else using the GPU. Another process or model can consume memory that would otherwise be available to inference. Memory fragmentation and runtime choices can further affect whether a configuration that appears to fit on paper works in practice.

• Record expected average and peak active requests, not just daily request volume.
• Include the sequence lengths associated with those active requests.
• Check whether batching is enabled and how the serving runtime handles requests with different sequence lengths.
• List other GPU processes and workloads that may overlap with inference.

Include runtime and hardware overhead without assuming a fixed allowance

There is no universally reliable fixed overhead to add to every estimate. Actual memory use can vary with the inference engine, runtime optimizations, batching strategy, model architecture and memory fragmentation. Framework and model-specific factors also matter.

Instead of adding an arbitrary percentage and treating the result as definitive, keep overhead as an explicit unknown to measure. Start with the weight and cache calculations, then compare them with observed use under the intended runtime and load.

Check the GPU’s usable memory as reported in the environment where the service will run. A nominal capacity does not establish how much will remain available after other workloads and the runtime have allocated memory.

• Document the inference engine, model configuration and serving settings used for the estimate.
• Identify GPU workloads that will share the device and whether they run at the same time.
• Do not present an untested safety margin as a universal requirement.
• Use test results to establish an operational margin appropriate to your own workload.

Build a workload profile before choosing infrastructure

A useful estimate starts with a workload description, not just a model label. Capture the architectural details needed for the cache calculation, the precision used for weights, and the lengths and concurrency your service must handle.

Separate typical use from peak use. If your system serves varied requests, a single average sequence length can hide the memory demand of long prompts or simultaneous long-running requests. Test a representative pattern as well as the largest credible case you need to support.

• Model: parameter count, architecture details needed for the KV-cache formula, and weight precision or quantization.
• Prompt: typical and maximum prompt length in tokens.
• Output: expected and maximum generated-token length.
• Sequence: the prompt plus output length the runtime may need to hold.
• Serving: expected typical and peak active requests, batch behavior and relevant runtime settings.
• Environment: GPU memory available to the process and any other workloads sharing the GPU.

Validate the estimate with a representative test

Run the actual model with the intended inference runtime and serving configuration before committing to the estimate. Start with a normal workload, then test the longer sequences and higher concurrency that represent your expected peak.

Observe GPU memory during model loading and while requests are active. Record the configuration alongside the measurements so that results are useful when you change the model, context limits, quantization, runtime or batching.

A single successful request is not evidence that the system can handle peak load. Repeat the test with representative concurrent requests, and investigate unexpected memory growth or failures before treating the configuration as ready.

• Test the same model format, precision, runtime and settings planned for deployment.
• Include realistic prompt and output lengths, then test the intended peak concurrency.
• Monitor actual memory use throughout loading and serving, not just at startup.
• Repeat measurements after material changes to the model or serving configuration.

When the workload exceeds available GPU memory

First identify which part of the workload drives the excess. If long sequences are the cause, reducing the context length or limiting unusually long requests can reduce KV-cache demand. If model weights dominate, quantization can reduce weight memory, with a precision trade-off to evaluate for the model and task.

Other options include reducing peak concurrency or changing the serving configuration. Each choice changes the service behavior, so check the resulting context limits, request handling and output quality against the application’s requirements.

If those changes do not meet the workload’s needs, reassess the GPU capacity or deployment design. Do not assume that offloading, a particular runtime setting or a specific GPU will solve the problem without testing: the supplied estimates do not establish a universal hardware recommendation.

After any change, repeat the representative load test. A configuration that fits at reduced concurrency or context length must still meet the application’s actual requirements.

• Reduce context length when the workload can tolerate shorter prompts or outputs.
• Evaluate weight quantization while checking its precision trade-off for your use case.
• Reduce peak concurrent requests or adjust batching, then test the effect on the service.
• If the required workload still does not fit, compare deployment options using measured results rather than a weight-only calculation.