
Private LLM Development for Enterprises: Build vs Buy, Architecture, and Cost
Private deployment, self-hosted or fine-tuned: which private LLM option you actually need, the seven-part architecture around the model, what it costs to build and run, and where projects fail.
A private LLM is a large language model your organization runs under terms and in an environment where your data does not leave your control: a frontier model deployed in your own cloud account, an open-weight model on infrastructure you own, or a model fine-tuned on your data and served privately. Private LLM development is the work of choosing which of those you actually need, deploying it securely, and wiring retrieval, access control and evaluation around it.
This guide is for the CTO, head of data or compliance lead who cannot send customer, patient or financial data to a public API as-is and is being asked to "get our own model". It covers the three ways to get a private LLM, when retrieval over a private deployment is enough and when fine-tuning is worth it, the architecture of a private deployment, what it costs to build and to run, and where these projects fail.
The short version: most teams asking for a private LLM need private retrieval over a well-governed deployment, not a trained model. Decide the data boundary first, the model second, and training last, if at all.
Do you need a private LLM, or private retrieval?
You need a private deployment when data sensitivity, contracts or regulation forbid sending data to a shared public endpoint. You need fine-tuning only when the model must behave in a way prompting and retrieval cannot achieve — a specialized output format, a domain vocabulary it consistently gets wrong, or latency and cost targets that require a smaller model. Those are different questions, and conflating them is the most expensive mistake in this space.
| Option | What it is | Fits when | Trade-off |
|---|---|---|---|
| Public API, enterprise terms | A frontier model via the provider's API with zero data retention and no training on your data | Data is sensitive but contractual controls are acceptable; speed matters | Data still transits a shared service; some regulators or clients will not accept it |
| Private cloud deployment | A frontier model inside your own cloud account or region (for example through a hyperscaler's model service) | Residency, BAA or client contracts require your environment; you want frontier quality | Higher per-token cost than public API; provider dependency remains |
| Self-hosted open-weight model | An open model served on GPUs you rent or own | Strict isolation, air-gapped or on-premises requirements, very high volume | You own serving, scaling, security patches and model quality; GPU cost is fixed, not per use |
| Fine-tuned model | Any of the above, additionally trained on your examples | A behaviour prompting and retrieval cannot reach; a smaller model must match a larger one | Data labelling, retraining cadence, evaluation burden; knowledge still goes stale without retrieval |
Retrieval-augmented generation is what gives any of these options your company's knowledge: documents, tickets, policies and records are indexed and fed to the model at query time, so answers are grounded in current data without retraining. For most private LLM requests, a private cloud deployment plus retrieval delivers the result in weeks. Our RAG implementation guide covers that layer in depth; this article stays at the deployment decision.
What is the architecture of a private LLM deployment?
A private deployment has seven parts: the model serving layer, a retrieval layer over your data, identity and access control, guardrails, logging and audit, an evaluation harness, and cost controls. The model is the smallest engineering task of the seven; the other six are what make the deployment safe to put in front of employees or customers.
- Model serving. A hyperscaler model service in your account, or an inference server on your GPUs, behind a private endpoint. Version pinning, so an upstream model change does not silently alter behaviour.
- Retrieval layer. Document ingestion, chunking, a vector index with metadata, and permission-aware search so users only retrieve what they may see.
- Identity and access. Single sign-on, role-based access, per-user or per-team data scopes, and service accounts for applications.
- Guardrails. Input and output filters, topic boundaries, personal-data redaction where required, and refusal behaviour for what the system must never do. The OWASP Top 10 for LLM applications is a useful checklist of what these controls must cover, from prompt injection to sensitive-information disclosure.
- Logging and audit. Who asked what, which documents were retrieved, what the model answered, with retention rules that match your policies.
- Evaluation harness. A test set of real questions with expected answers, run on every prompt, model or index change.
- Cost controls. Token budgets per user or application, caching, model routing (a smaller model for simple queries), and dashboards for spend.
Where the deployment must act as well as answer — update a record, file a ticket, trigger a workflow — it becomes an agent with bounded permissions, which our AI agent development practice covers.
Which model should you choose?
Choose the model by three criteria: quality on your evaluation set, the contractual terms you can get, and cost at your expected volume. Frontier models under private deployment usually win on quality and time to value; open-weight models win on isolation and on cost at very high, steady volume; a fine-tuned small model wins only when you have measured that a larger model with retrieval does not meet the bar.
"Private" is a contract as much as an architecture. Read the terms: zero data retention, no training on your prompts or outputs, the region where inference runs, a business associate agreement if health data is involved, and what the provider logs. A model in your own cloud account under those terms is private in every sense a regulator or client will ask about; a model on your own servers is private in one more sense, and you pay for that difference in operations. For health data the contractual side is explicit: HHS treats a cloud provider that stores or processes ePHI as a business associate that needs a BAA, even if it only holds encrypted data — see its guidance on HIPAA and cloud computing.
How much does a private LLM cost to build and run?
At Empat, private LLM work follows a staged ladder: discovery from $5,000 over one to two weeks, a proof of concept from $15,000 over two to four weeks, an MVP deployment from $30,000 over six to twelve weeks, and a full product from $50,000 over three to twelve months. The run cost is separate and is driven by how you serve the model: per-token for managed deployments, per-GPU-hour for self-hosting.
| Stage | What you get | Empat pricing | Run-cost drivers to settle here |
|---|---|---|---|
| Discovery | Data boundary, use cases, model and deployment option, evaluation set design, cost model | From $5,000, 1–2 weeks | Expected query volume, context size, residency requirements |
| Proof of concept | Private deployment answering your real questions from your real documents, measured against the evaluation set | From $15,000, 2–4 weeks | Measured tokens per query; quality of a smaller vs larger model |
| MVP | One use case in production: SSO, permission-aware retrieval, guardrails, audit, dashboards | From $30,000, 6–12 weeks | Caching and routing policy; per-user budgets |
| Full product | Multiple use cases, agent actions, fine-tuning if justified, ongoing evaluation | From $50,000, 3–12+ months | Retraining cadence if fine-tuned; GPU reservation vs on-demand |
Two honest notes on run cost. Managed private deployments cost more per token than the public API of the same model, but nothing when idle; self-hosting costs the same whether anyone is asking or not, so it only wins at steady high volume. And context is the quiet cost driver: retrieving ten documents into every prompt can cost more than the model choice. Our project cost calculator is a quick way to size the build before a discovery call.
Where do private LLM projects fail?
Private LLM projects fail in four predictable ways: training a model when retrieval would have answered the question, launching without an evaluation set so nobody can say whether a change helped, ignoring run cost until the first invoice, and treating the model as the deliverable while the data it needs has no owner. All four are discovery decisions.
- Training first. Fine-tuning does not teach a model your current facts; retrieval does. Train for behaviour, retrieve for knowledge.
- No evaluation set. Without a few hundred real questions and expected answers, every model or prompt change is a guess. The NIST AI Risk Management Framework makes the same point at governance level: measure before you deploy, and keep measuring after.
- Run cost as an afterthought. Budget per thousand queries before the build, and design caching and routing in.
- No data owner. The deployment is only as current and as clean as the documents behind it. Assign an owner to the content, not just to the system.
How Empat approaches private LLMs
Empat treats a private LLM as a deployment and governance problem first and a modelling problem second: a paid discovery that fixes the data boundary and often ends in "private deployment plus retrieval, no training", a PoC measured against your own evaluation set, and an MVP with single sign-on, permission-aware retrieval, guardrails and audit from day one. The practice is led by Vladyslav Fedenko, a Claude Certified Architect who designs the deployment, retrieval and evaluation layers on these projects.
The retrieval method is the one in our RAG implementation guide; the service scope is on our custom AI development page. For adding a private model to systems you already run, start with AI integration services; for health data specifically, our guide to HIPAA-compliant app development covers the agreements and safeguards. If you want a concrete design before committing, the AI Agent Blueprint Workshop produces one in a fixed engagement.
The two most common first uses of a private deployment are an internal assistant over company knowledge — see AI chatbot development services — and extraction from sensitive documents, covered in our guide to automated document processing. For product-level scope beyond the deployment, see generative AI development services.
FAQ
What is a private LLM?
A private LLM is a large language model that runs where your data stays under your control: a frontier model deployed inside your own cloud account under zero-retention terms, an open-weight model on infrastructure you own, or a model fine-tuned on your data and served privately. The privacy comes from the deployment and the contract — no training on your data, a fixed region, audit logging — not from the model itself.
How much does it cost to build a private LLM?
At Empat, discovery starts from $5,000 over one to two weeks, a proof of concept from $15,000 over two to four weeks, an MVP deployment with single sign-on, permission-aware retrieval, guardrails and audit from $30,000 over six to twelve weeks, and a full product from $50,000. The run cost is separate: per token for a managed private deployment, per GPU-hour for self-hosting, and it is driven mostly by query volume and how much context each answer retrieves.
Do we need to fine-tune a model, or is RAG enough?
For most private LLM requests, retrieval-augmented generation over a private deployment is enough: it gives the model your current documents and records at query time without training, and it stays current as the documents change. Fine-tune only when you have measured, on your own evaluation set, that prompting and retrieval cannot reach a required behaviour — a specialized output format, a domain vocabulary, or a smaller model that must match a larger one on cost or latency.
Can a private LLM be HIPAA compliant?
Yes, if the deployment meets HIPAA's requirements: a business associate agreement with every vendor that touches protected health information, including the model provider, inference running in an approved region, encryption in transit and at rest, role-scoped access, and audit logs of who asked what and which records were retrieved. A frontier model inside your own cloud account under a BAA, or a self-hosted model, can both meet this; a public endpoint without a BAA cannot.



