Why local?
Privacy and confidentiality: prompts and documents stay in your own environment.
Control: you decide on model version, updates and availability.
Predictable cost at high usage instead of per-token billing.
Insights
Running LLMs on-premises: a guide for companies
MAKONIS team · · 7 min read
In short: Running an LLM locally means executing a language model on your own or dedicated infrastructure so prompts and data never leave the company. It requires a suitable open-weight model, enough GPU memory, inference software, a connection to company knowledge (RAG) and access control with logging.
Book a free callThe building blocks
Model: open-weight models in various sizes; smaller models are often enough for well-defined tasks.
Inference: software such as vLLM or comparable servers exposes the model via an API.
Knowledge (RAG): documents are made searchable and passed to the model per question, respecting permissions.
Interface: chat front ends or integration into existing applications.
Operations: monitoring, logging, updates and access control.
GPU sizing
GPU memory determines the possible model size. Rules of thumb (depending on quantisation, context length and number of users):
| Model size | Use | Hardware guideline |
|---|---|---|
| small (up to ~10B parameters) | classification, summarisation | one GPU |
| medium (~20–70B) | assistants, RAG, agents | one to several high-memory GPUs |
| large (100B+) | demanding tasks | several GPUs or servers |
Common mistakes
Choosing a model that is too large when a smaller one with good RAG would do.
Ignoring source-document permissions in RAG.
No evaluation of answer quality before rollout.
Missing logging for traceability and compliance.
How MAKONIS helps
We design and run AI infrastructure on your own hardware, in European data centres or hybrid, and connect models through LLMrouter. We can also take over operations as Managed Sovereign AI.
Frequently asked questions
No, but it makes compliance much easier. Purpose limitation, access rights, deletion concepts and documentation are still required.
Innovation from Munich.Active worldwide.
Talk to us about your cloud, DevOps and AI initiatives — no obligation, directly with our engineers.