AIGuide 8 min ·

On-Premise LLM Sizing Guide: GPU Requirements, Kubernetes Setup and Operating Costs

By Daniel Weise · Founder & Managing Director, MAKONIS

In short

Rule of thumb for the GPU memory of an on-premises LLM: parameters in billions × 2 GB at FP16/BF16, × 1 GB at FP8/INT8 and × 0.6 GB at INT4/AWQ, plus a 20 to 40 percent buffer for the KV cache. An 8B model runs comfortably at FP8 on one NVIDIA L40S with 48 GB; a 70B model at FP8 needs at least two L40S, better four, or one H200. For production with many users, vLLM on Kubernetes is the standard. On cost alone, your own hardware only pays off under heavy sustained load – the more common reason is that data must not leave the company.

Book a free call

01

VRAM rule of thumb: how much GPU memory does an LLM need?

GPU memory (VRAM) decides which model loads at all. It consists of three parts: the model weights, the KV cache for running requests and a fixed overhead of a few gigabytes for the CUDA context and intermediate results.

For the weights, a simple calculation is enough: parameters in billions times bytes per parameter. At FP16 or BF16 that is about 2 GB per billion parameters, at FP8 or INT8 about 1 GB and with 4-bit methods such as AWQ or GPTQ about 0.6 GB – slightly more than the theoretical 0.5 GB because scaling factors and some layers stay at higher precision.

The KV cache stores attention values for every token in the context: 2 × layers × KV heads × head dimension × bytes. For Llama 3.3 70B that is 2 × 80 × 8 × 128 × 2 bytes, about 320 KB per token. vLLM manages this memory in small blocks with PagedAttention and wastes hardly any of it. Plan 20 to 40 percent on top of the weights – more for long RAG contexts and many concurrent users.

ModelParametersFP16/BF16FP8/INT8INT4/AWQKV cache per request with 8,192 tokens
Llama 3.1 8B8.0 bn≈ 16 GB≈ 8 GB≈ 5–6 GB≈ 1.0 GB
Mistral NeMo 12B12.2 bn≈ 25 GB≈ 12 GB≈ 8 GB≈ 1.25 GB
Qwen 2.5 14B14.7 bn≈ 29 GB≈ 15 GB≈ 9–10 GB≈ 1.5 GB
Qwen 2.5 32B32.8 bn≈ 66 GB≈ 33 GB≈ 19 GB≈ 2.0 GB
Llama 3.1 / 3.3 70B70.6 bn≈ 141 GB≈ 71 GB≈ 42 GB≈ 2.5 GB

02

Real benchmarks: what the hardware actually delivers

Memory tells you whether a model loads – throughput tells you how many people can use it at the same time. The figures in the table come from publicly documented measurements with vLLM. They depend heavily on prompt length, output length and configuration and are reference values, not guarantees.

Rule of thumb from the measurements: on Llama 3.1 8B, an L40S reaches about a quarter of H100 throughput (1,207 vs. 5,103 tokens/s in the server scenario) – similar to the ratio of memory bandwidth, 864 GB/s vs. 3.35 TB/s. For the speed of a single response, roughly: memory bandwidth divided by model size in memory. An L40S therefore delivers at most around 100 tokens/s per request for an 8 GB model.

There is no independently reproducible measurement for Mistral NeMo 12B on the L40S. Based on size and architecture, it sits between Llama 3.1 8B and Qwen 2.5 14B.

Model & precisionHardwareResultSource
Llama 3.1 8B, FP81× H1005,777 tokens/s (offline), 5,103 tokens/s (server)MLPerf Inference v5.1, Red Hat, Oct 2025
Llama 3.1 8B, FP81× L40S1,642 tokens/s (offline), 1,207 tokens/s (server)MLPerf Inference v5.1, Red Hat, Oct 2025
Llama 3.3 70B, FP81× H1001,496 tokens/s per GPU at 50 tokens/s per user (8k input, 1k output); peak 2,568 tokens/sSemiAnalysis InferenceX, Oct 2025
Qwen 2.5 32B, AWQ1× RTX 3090 (24 GB)about 42 tokens/s for a single request; weights 18 GiBCaleb Coffie, vLLM 0.21

03

Choosing GPUs: L40S, H100/H200 or Mac Studio?

Companies usually choose between three classes. What matters is memory size, memory bandwidth, the link between GPUs and power draw in the data centre.

The NVIDIA L40S is passively cooled, fits standard Dell or HPE servers and computes in FP8. For models up to 32 billion parameters it is in practice the price-performance winner for inference; 70B models run on two to four cards.

H100 and H200 show their strength with 70B models, long contexts and many concurrent users: high-bandwidth HBM memory and NVLink between GPUs. One H200 holds a 70B model at FP8 entirely and still has about 60 GB free for the KV cache.

Mac Studio and other edge hardware are good for trying out use cases with Ollama or llama.cpp within days – thanks to large unified memory even with big models. For multi-user operation they lack throughput and the CUDA ecosystem.

NVIDIA L40SNVIDIA H100 / H200Mac Studio (M3 Ultra) / edge
Memory48 GB GDDR680 GB HBM3 / 141 GB HBM3eup to 512 GB unified memory
Bandwidth864 GB/s3.35 TB/s (H100 SXM) / 4.8 TB/s (H200)819 GB/s
InterconnectPCIe Gen4, no NVLinkNVLink at 900 GB/s (SXM)none, single device
Power350 W per GPUup to 700 W per GPU (SXM), PCIe variants 350–400 Wmax. 480 W for the whole device
Strengthinference from 8B to 32B, 70B on 2–4 GPUslarge models, long contexts, many users at low latencypilots, development, single workstations
Limittensor parallelism only over PCIeprice, power and cooling (HGX servers up to about 10 kW)no CUDA, few concurrent users

04

Inference engines: vLLM, Triton or Ollama?

vLLM is the de facto standard for running open models in production. Continuous batching admits new requests as soon as there is room in the running batch; PagedAttention manages the KV cache in blocks; prefix caching saves compute on recurring system prompts. Thanks to the OpenAI-compatible API, existing applications switch over by changing only the base URL.

NVIDIA Triton Inference Server is a general model server for many frameworks. With the TensorRT-LLM backend it reaches very low latency but requires compiling models for each GPU generation. Triton pays off when embedding, classification or vision models run on the same platform alongside LLMs, or when every millisecond counts.

Ollama builds on llama.cpp and starts a model with one command. That is ideal for development, pilots and single workstations; for many concurrent users it lacks throughput and operational features.

CriterionvLLMTriton + TensorRT-LLMOllama
Useproduction, many usersproduction, mixed models, minimal latencydevelopment, pilot, edge
Batchingcontinuous batchingin-flight batchinglimited parallelism
InterfaceOpenAI-compatibleHTTP/gRPC, OpenAI frontend availableown API plus OpenAI-compatible endpoint
QuantisationFP8, AWQ, GPTQ and moreFP8, INT8, INT4 after compilationGGUF formats
Effortlowhighvery low

05

Kubernetes architecture: GPU Operator, KEDA and a chat UI with SSO

Beyond a single server we recommend Kubernetes – on-premises for example with RKE2 or OpenShift, in a sovereign cloud for example with Scaleway Kapsule. The architecture consists of a few proven building blocks:

  1. 1NVIDIA GPU Operator: installs drivers, container toolkit, device plugin and DCGM exporter automatically on every GPU node.
  2. 2vLLM as a separate deployment per model; weights come from a local model cache (persistent volume or S3-compatible object storage) so restarts do not download from the internet.
  3. 3KEDA scales vLLM replicas on the Prometheus metric vllm:num_requests_waiting (queue) and KV cache usage – not on CPU, which says nothing under GPU load.
  4. 4An LLM gateway such as LLMrouter.eu puts all models behind one interface, enforces budgets and permissions and logs every call.
  5. 5Open WebUI or LibreChat as chat UI with single sign-on via OpenID Connect, for example with Microsoft Entra ID or Keycloak, and groups for model access.
  6. 6Monitoring with Prometheus and Grafana: time to first token, tokens per second, queue, GPU utilisation and temperature.

06

TCO: what does an on-premises LLM cost for 50, 100 and 500 users?

The calculation compares four options for a 70B-class model at FP8, such as Llama 3.3 70B. All values are monthly net costs over a 36-month term. They rest on openly stated assumptions and do not replace a quote.

Assumptions: at peak, 30 percent of users send a request at the same time. Based on the measurements above, a server with 4× L40S handles about 20 concurrent responses at fluent reading speed, an HGX server with 8× H100 about 200. Hardware: server with 4× L40S from Dell or HPE €75,000, HGX server with 8× H100 €300,000, each depreciated over 36 months. Electricity €0.25/kWh at a PUE of 1.4 and average load of 1.8 kW (L40S server) or 7 kW (HGX); colocation €350 or €1,000 per month.

Cloud GPU at Scaleway list prices running 24/7: L40S €1.47 per GPU hour, 8× H100 SXM €25.33 per hour. Token API: per user 2.5 million input and 0.4 million output tokens a month, about 40 RAG requests per working day. Prices of an EU API (Mistral Large 4: US$0.70 and US$2.10 per million tokens) and a premium model (assumption: US$3 and US$15), converted at US$1 = €0.90.

Platform operations – updates, monitoring, model changes, on-call – are not included. They apply equally to own hardware and cloud GPUs, but largely not to a token API.

Monthly50 users100 users500 users
Concurrent requests (peak)≈ 15≈ 30≈ 150
On-premises hardware1 server, 4× L40S2 servers, 4× L40S each1 HGX server, 8× H100
Token volume≈ 145 million≈ 290 million≈ 1.45 billion
On-premises: depreciation (CAPEX)≈ €2,080≈ €4,170≈ €8,330
On-premises: power, cooling, colocation (OPEX)≈ €810≈ €1,620≈ €2,790
On-premises total≈ €2,900≈ €5,800≈ €11,100
Cloud GPU (Scaleway, 24/7)≈ €4,300≈ €8,600≈ €18,500
Token API EU (Mistral Large 4)≈ €120≈ €230≈ €1,170
Token API premium model≈ €610≈ €1,220≈ €6,080

07

Break-even: when your own GPUs pay off

The result is clear: for pure chat use, a token API is cheaper in all three scenarios. Chat users create load peaks in the morning but little volume – the GPUs sit idle most of the time.

On cost alone, a server with 4× L40S (≈ €2,900 a month) breaks even against the premium model (≈ €4.20 per million tokens blended) at about 700 million tokens a month, and at about 1.6 billion including platform operations from €3,900. Against a low-cost EU API (≈ €0.80 per million tokens) break-even would be above 3.6 billion tokens – barely less than such a server can process at full load around the clock.

Own GPUs therefore pay off in three cases: under sustained 24/7 load, for example agents or document processing with billions of tokens a month; compared with premium models, when an open model solves the task equally well; and – the most common reason for mid-sized companies – when data must not leave your own data centre for legal or contractual reasons. Then on-premises is not a cost decision but a compliance decision.

A hybrid approach has proven itself: a gateway such as LLMrouter.eu sends confidential requests to the local model and non-critical ones to an EU API. Your hardware stays well utilised without being oversized for rare peaks.

≈ 0.7 bn

tokens per month: break-even of 4× L40S vs. premium API (infrastructure only)

≈ 1.6 bn

tokens per month: break-even including platform operations

≈ 3.9 bn

tokens per month: capacity of 4× L40S at full load (≈ 1,500 tokens/s)

08

Sources and assumptions

Measurements and list prices come from publicly available sources as of November 2026. Server prices are assumptions and vary widely with configuration and supply; request quotes for your planning.

  1. MLPerf Inference v5.1 – Red Hat results with vLLM: redhat.com
  2. SemiAnalysis InferenceX – Llama 3.3 70B on H100: inferencex.semianalysis.com
  3. Qwen 2.5 32B AWQ with vLLM: calebcoffie.com
  4. Scaleway – prices for L40S and H100 instances: L40S instance · GPU pricing
  5. Mistral – API pricing: docs.mistral.ai
  6. Layers, KV heads and parameter counts – official model configurations: Hugging Face

09

How MAKONIS helps

We size on-premises LLMs from a load profile of your real requests rather than rules of thumb alone: model choice, GPU type, number of servers, Kubernetes architecture and a TCO calculation with your own power and data centre costs.

We build the platform with vLLM, GPU Operator, KEDA, LLMrouter.eu and Open WebUI or LibreChat – on your hardware, in a European data centre or hybrid. Ongoing operations are available as Managed Sovereign AI from €3,900 per month, run from the EU.

Frequently asked questions

For interactive chat, the response should start in under one second and then flow at 20 to 30 tokens per second or more – well above human reading speed. Time to first token mainly depends on prompt length and the queue, because long RAG contexts have to be processed first. Prefix caching in vLLM, shorter contexts and enough replicas keep it low.

Innovation from Munich.Active worldwide.

Talk to us about your cloud, DevOps and AI initiatives — no obligation, directly with our engineers.