Private AI: When the Data Cannot Leave

The decision usually gets made backwards

The conversation tends to go like this. Someone proposes an AI use case. Someone else says the data cannot leave the organisation. The architecture is then decided in that sentence — on-premise, GPUs, open-weight models — and the project becomes an infrastructure programme before anyone has written down what obligation is actually being satisfied.

Sometimes that is the right outcome. Often it is not, and the cost of getting it wrong is not just money. An on-premise deployment chosen for a constraint that a data processing agreement and a regional endpoint would have satisfied commits you to capacity planning, GPU procurement, patching, model upgrades and an on-call rotation for the lifetime of the workload. Teams that make that commitment for the wrong reason usually discover it eighteen months later, when the model they deployed is two generations behind and nobody has the budget to refresh the hardware.

The reverse error is also real and more dangerous. Sending customer financial records, patient data or citizen records to whichever API was easiest to call, on the assumption that a vendor's marketing page constitutes a contractual guarantee, is how organisations end up explaining an unmapped cross-border transfer to a regulator.

This paper is a sequence for making the decision in the right order: establish the obligation, test whether it is real, then price the options against it honestly — including the operating burden, which is the part consistently left out of the business case.

Separating a real constraint from an assumed one

Start by writing one sentence per workload: this data cannot leave X because of Y, evidenced by Z. If you cannot complete the sentence with a specific clause, rule or written policy in the Z position, you have an assumption, not a constraint. Assumptions are legitimate inputs to a decision, but they should be labelled, because unlike obligations they can be renegotiated.

In our experience, the constraints people state fall into five categories, and only two of them usually force private deployment.

Stated constraint	What it usually means	Does it force private deployment?
"Data must stay in India."	A residency requirement from a sector regulator or an internal policy. Residency is about geography, not about ownership of the infrastructure.	Usually no. A regional endpoint with contractual residency commitments can satisfy it. Verify the region covers processing, logging, abuse monitoring and support access, not only storage at rest.
"Data cannot go to a third party."	Frequently an internal policy about processors, not a legal prohibition. Most organisations already use third-party processors for payroll, email and backup.	Usually no. A processor agreement with purpose limitation, no-training terms, defined retention and sub-processor disclosure is the normal instrument here.
"A regulator or contract names the environment."	A specific obligation — a government empanelment requirement, a customer contract specifying infrastructure, a sector circular naming permitted hosting.	Often yes. This is a real constraint. Get the exact clause; the scope is often narrower than the summary.
"The data class is too sensitive for any external processing."	A board-level or clinical-governance position. Not legally compelled but genuinely binding inside the organisation.	Sometimes. Worth testing whether it applies to the whole corpus or to an identifiable subset — the answer is almost always a subset.
"Our CISO said no to cloud AI."	Usually a response to a specific unanswered question: prompt retention, training use, logging, or who at the provider can read support tickets.	No, by itself. Find the question. It normally has a documentable answer.

Three practical tests sharpen this quickly.

Test the clause, not the summary. Ask for the actual text of the obligation. Compliance summaries compress and over-generalise. A circular that restricts storage of a specific record type is routinely read as a prohibition on all external processing of anything related to that business line.

Test the data class, not the system. People describe constraints at the level of the system — "the core banking data" — when the obligation attaches to specific fields. A claims triage workflow may need the policy wording, the diagnosis category and the claimed amount, and not need the name, the identifier or the address. If the sensitive fields can be left out or tokenised before the model call, the constraint may simply not apply to the payload you are actually sending. This is the single highest-leverage question in the whole exercise and it is skipped most often.

Test what residency means to your provider. Where is inference executed? Where are prompts and outputs logged, and for how long? Is there abuse-monitoring retention, and does a human ever see flagged content? Where does support tooling run, and can a support engineer outside the region access a trace? Are there sub-processors, and where are they? Is there a contractual commitment that your data is not used to train or improve models, and does it cover fine-tuning and evaluation? These questions have answers, they belong in the DPA, and gathering them takes a week rather than a quarter.

The three options, compared against constraints that matter

There are three serious shapes, and a fourth that is usually the right answer and gets covered later. The table below compares them against the dimensions that actually decide the choice. It is deliberately written in terms of mechanism and effort rather than price, because any rupee or dollar figure printed here would be wrong within two quarters.

Dimension	Hosted API (regional endpoint)	VPC / own-tenancy deployment	On-premise
Data residency	Contractual. Satisfies geographic residency where the provider offers a regional endpoint and the DPA covers processing, logging and support access. Does not satisfy requirements that name the infrastructure.	Technical and contractual. Data stays in your cloud account, in your chosen region, under your IAM. Satisfies most residency and third-party-processing positions. The cloud provider remains a processor.	Absolute. Nothing leaves your network. The only option that satisfies obligations naming your own or an empanelled environment.
Latency	Network round trip to the region, plus provider-side queueing under load. Generally the best time-to-first-token for large models because the provider's serving stack is heavily optimised and warm.	No internet egress; same-region or same-VPC hop. Predictable because capacity is yours, which also means you absorb cold starts and queueing yourself.	Lowest achievable network latency. Whether end-to-end latency is actually better depends entirely on whether you have provisioned enough GPU for peak concurrency.
Cost at low utilisation	Strongest position. Per-token pricing means you pay for what you use, and an idle workload costs nothing. For pilots and spiky internal tools this is rarely beaten.	Poor. Reserved or on-demand accelerators bill whether or not requests arrive. A workload with a working day of traffic and an idle night pays for the night.	Worst. Capital is sunk regardless of use, plus power, cooling, rack space and the depreciation clock, which runs whether the hardware is busy or not.
Cost at high, steady utilisation	Weakest position at scale. Per-token cost is linear, so a high-volume steady workload eventually crosses over.	Can be strong. With batching, autoscaling and genuinely high average utilisation, marginal cost per token falls well below list API pricing for comparable open-weight models.	Can be the cheapest per token over a multi-year horizon if utilisation stays high and the hardware is kept busy by more than one workload.
Upgrade effort	Lowest. New model versions appear as new endpoints. Your work is re-running evaluations and managing deprecation timelines, which the provider sets.	Moderate to high. You choose when to move, which is the benefit, and you own weight distribution, serving-stack compatibility, quantisation choices, capacity re-sizing and a full evaluation re-run each time.	Highest. Same as VPC, plus hardware generation becomes a constraint. A newer, larger model may not fit the accelerators you bought, which turns a software upgrade into a procurement cycle.
Staffing to run it	Application engineering plus a quality owner. No infrastructure specialisation required.	Add platform engineering with GPU and inference-serving experience, and an on-call rotation that understands the serving stack. Realistically not one person.	All of the above plus data-centre operations: hardware failure handling, firmware and driver management, spares, capacity expansion lead times.
Audit evidence	Provider certifications, the DPA, your own gateway logs. Strong on paper, but you cannot evidence what happens inside the provider beyond what they attest.	Your cloud audit logs, your IAM records, your gateway logs, your network flow logs. Usually the strongest evidence position per unit of effort.	Complete and entirely yours, including physical access records. Also entirely your burden to produce, and the one most likely to be incomplete because nobody built the logging.
Model quality ceiling	Access to the strongest frontier models, updated without effort on your side.	Open-weight models, plus any commercial model the vendor licenses for self-hosting. A real gap remains on the hardest reasoning tasks.	Same as VPC, further constrained by what your hardware can serve at acceptable latency.
Best fit	Pilots, non-sensitive or de-identified payloads, workloads needing frontier reasoning, spiky volume.	Sensitive data under a residency or third-party-processing constraint; steady volume; the default private option for most regulated enterprises.	Obligations naming the environment; air-gapped or empanelled infrastructure; existing GPU capacity with an owner.

One note on the middle column, because it is widely misunderstood. "In our VPC" covers two materially different things: a managed model service running inside your cloud account's boundary, and model weights you deploy and serve yourself on your own accelerators. The first is close to the hosted column on operating burden and close to the private column on residency, and it is frequently the best available trade. The second is close to on-premise on everything except the hardware. Decide which one you mean before you compare costs.

What self-hosting actually costs to operate

The business case for self-hosting is usually built on a per-token comparison against API list pricing. That comparison is not wrong, it is incomplete, and the missing items are the ones that make projects fail.

Utilisation is the whole economics. When you self-host, you pay for accelerator time, not tokens. A model served at low average utilisation is dominated by idle GPU cost, which is why batching, request queuing and autoscaling decide whether the economics work at all. The relevant variables are parameter count and precision (which set how much memory the weights occupy and therefore how many accelerators a replica needs), context length and the resulting key-value cache growth (which sets how many concurrent requests a replica can hold), concurrency pattern, and the redundancy factor you need for availability. Those variables, not the hourly rate, determine whether your cost per token beats an API. Work them out for your traffic shape before anyone signs a hardware order.

Capacity planning is harder than for conventional services. You cannot scale a GPU fleet the way you scale stateless web servers. Cold start includes loading many gigabytes of weights. Scarce accelerator types may not be available on demand in your region, which pushes you toward reservations, which pushes you back toward paying for idle. Peak concurrency, not average, sets the floor — and if your workload is an internal copilot used by staff during business hours, your peak-to-average ratio is poor by nature.

Upgrades are recurring projects, not patches. Each model refresh brings new weights, often a new serving-stack version, possibly new quantisation behaviour, different memory characteristics, and a full evaluation re-run against your own test set because a newer model is not automatically better on your task. Budget this as a scheduled project several times a year, with a named owner. Teams that do not end up frozen on the model they launched with, which is the most common quiet failure of private AI.

Security patching does not pause for your release calendar. GPU drivers, CUDA-layer components, inference servers, tokeniser and serving libraries, container base images, orchestration. This is a conventional vulnerability-management obligation on a stack your security team has probably never patched before, with the added complication that inference serving frameworks move fast and sometimes break compatibility on a minor version.

Someone has to be awake. At 2 a.m. the failure modes are GPU memory exhaustion from a long-context request, a node with a failing accelerator that passes health checks but produces degraded throughput, a queue backing up behind one slow request, a model replica that loaded but is serving from a half-warmed cache. Diagnosing these needs someone who understands inference serving, not a generalist following a runbook. One person is not a rotation. If you cannot staff three, you are planning to be unavailable.

Evaluation re-runs are an operating cost, not a project cost. Every change — model, quantisation, serving version, prompt, retrieval index — needs the evaluation set re-run. Self-hosting gives you more things that change and more freedom to change them, which means more re-runs. Automate it or it will stop happening.

The honest summary: self-hosting trades a predictable per-token bill for a fixed cost in capital and people, and buys you residency, control and a utilisation-dependent cost advantage. It is a good trade when the constraint is real and the volume is steady. It is a poor trade when you are still discovering whether the use case works.

Where open-weight models are good enough, and where they are not

We are not going to quote benchmark scores. They move monthly, they are gamed, and they correlate loosely with performance on your documents. What is stable is the shape of the gap, and it is useful to reason about.

Open-weight models that can be practically self-hosted are generally strong enough for:

• Classification and routing. Categorising a grievance, a ticket, a claim type or a document class. These tasks have a bounded label set, you can measure them precisely, and a smaller model fine-tuned or well-prompted on your own data frequently beats a frontier model used generically.
• Structured extraction. Pulling fields out of invoices, KYC documents, lab reports and forms, with confidence thresholds and an exception queue. The hard part here is document quality and validation logic, not model intelligence.
• Retrieval-grounded answering over a defined corpus. When the answer is in a retrieved passage and the job is faithful summarisation with a citation, retrieval quality dominates model quality. Teams routinely blame the model for what is an indexing and chunking problem.
• Redaction, de-identification and sensitivity tagging. Narrow, verifiable, high-volume, and the task most worth keeping inside the boundary because it is what lets other work move outside it.
• Embeddings. Self-hosting an embedding model is dramatically simpler than self-hosting a generative one, and it keeps your whole corpus inside the boundary. If only your documents are sensitive and not your queries, this alone can resolve the constraint.

The gap is still real for:

• Long multi-step reasoning with tool use. Agentic workflows that chain many decisions compound small per-step weaknesses. A model that is slightly less reliable per step is substantially less reliable over twelve steps.
• Hard, ambiguous judgement over long context. Reconciling a contradictory set of long documents and explaining the conflict. Frontier models remain meaningfully better here.
• Instruction adherence under pressure. Holding a complex output format and a long list of constraints across a long conversation, and resisting instructions embedded in retrieved content.
• Code generation and complex transformation. Where it matters, the gap is still visible.
• Low-resource Indian language performance. Highly variable across models and scripts, and the only way to know is to test on your own content. Published language coverage claims are not a substitute for a run against your own documents.

The practical rule: decide this empirically, per workload, on your own evaluation set, before you commit to architecture. Build the set first — a few hundred real cases from your own traffic with human-labelled answers — then run the candidate options through it. This takes a couple of weeks and resolves arguments that otherwise consume quarters. It also reveals the common and happy case: the workload you assumed needed a frontier model is a classification task that a small self-hosted model does well, cheaply, inside your boundary.

Hybrid patterns that hold up

Very few organisations have a single answer across all workloads, and architectures that try to force one are the ones that get abandoned. The useful question is not "hosted or private" but "what is the boundary, and what crosses it in what form".

Classify and route inside, reason outside. A small self-hosted model inspects every incoming payload, assigns a sensitivity class, and decides the destination. Sensitive items are handled entirely by in-boundary models; everything else goes to a hosted endpoint. The gateway enforces the routing, logs the decision, and that log is your audit evidence. This is the most common pattern we deploy and it holds up well under review, because the control is a system property rather than a developer's discipline.

De-identify at the boundary. Strip or tokenise identifiers before the hosted call, re-identify on return using a mapping that never leaves your boundary. Powerful where the sensitive part is identity rather than content — plenty of claims, support and document workflows are exactly this. Two cautions: de-identification is not reliable for free text with rich context, since quasi-identifiers can re-identify a person even with names removed; and you need to prove the stripping worked, which means testing it like a security control, including on the long tail of document formats.

Private retrieval, hosted generation. Embeddings, the index and the whole corpus stay inside. Only the user's question and the specific retrieved passages are sent out. This sharply reduces what crosses the boundary — but be clear-eyed that the retrieved passages are the sensitive content, so this only helps where the passages themselves are not the restricted part, or where they are de-identified.

Tiered by data class, not by department. Write down your data classes, assign a permitted hosting mode to each, and enforce it at the gateway. Department-based rules collapse the moment a workflow spans two departments, which they all eventually do.

Hosted for discovery, private for production. Prove the use case with a hosted API on de-identified or synthetic data where that is lawful and useful, then move the proven workload inside. This avoids the most expensive failure mode in private AI: buying infrastructure for a use case that turns out not to work.

Everything here depends on one piece of infrastructure being built properly: a model gateway that every call goes through, performing authentication, routing by data class, rate limiting, content controls and complete request logging. Without it, hybrid is a convention, and conventions do not survive the fourth team that integrates directly because the gateway was inconvenient. The gateway is also what makes your audit story short, and what lets you change providers without changing applications.

What teams underestimate

In order of how often it causes trouble:

1. The ongoing cost of staying current. Self-hosting is a subscription paid in engineering time, several times a year, forever. Teams plan the deployment and not the second upgrade.
2. Peak-to-average ratio. Internal tools are used during business hours. The capacity you provision for 11 a.m. is idle at midnight, and that idle time is the dominant cost line.
3. How much of the quality problem is retrieval. Chunking, metadata, freshness, permission filtering and index hygiene determine answer quality more than model choice in most enterprise deployments. Private deployments get blamed for quality problems that have nothing to do with hosting.
4. Context length economics. Long context is the easiest way to improve answers and the fastest way to destroy throughput, because key-value cache memory grows with sequence length and squeezes out concurrency. The same hardware serves far fewer users at long context.
5. That private does not mean secure. A self-hosted model with broad retrieval permissions and no gateway logging is less safe than a hosted model behind a well-built gateway. Prompt injection, over-broad retrieval and data leakage through outputs are all orthogonal to where the weights run. We see private deployments where the residency box is ticked and nothing else is.
6. Audit evidence is work you have to do. On-premise gives you the best possible evidence position and the entire burden of producing it. Access logs, change records, model versioning and evaluation history do not appear because the hardware is yours.
7. Non-production environments. Development, testing and evaluation all need accelerators too. Teams budget for production and then either starve development or quietly test against a hosted API using real data, which recreates the exact exposure the project was meant to eliminate.
8. The gateway as a dependency. Once every AI call goes through one gateway, that gateway is a tier-one service and needs the availability engineering to match.

A sequence you can run in about two weeks

This decision does not need a quarter. It needs the steps taken in order.

1. Write the constraint sentences. One per workload, with the clause or policy named in the evidence position. Mark each as obligation or assumption. Have the DPO or compliance owner confirm the marking in writing.
2. Reduce the payload. For each workload, list the fields the model genuinely needs. Remove the rest. Decide what can be tokenised. Re-read the constraint sentence against the reduced payload — this step alone changes the answer surprisingly often.
3. Ask the provider the written questions. Processing region, logging region, retention, abuse-monitoring human review, support access paths, sub-processors, training and fine-tuning use, deletion commitments. Get it in the DPA, not in an email.
4. Build the evaluation set. A few hundred real cases, stratified, human-labelled, including cases where refusal is correct. This is the artefact that outlasts the decision and that you will re-run for years.
5. Run the candidates. A hosted frontier model, a hosted smaller model, and a self-hostable open-weight model of the size you could realistically serve. Measure task correctness, groundedness and latency on your traffic shape — not on a published benchmark.
6. Model the economics at your utilisation. Take your actual concurrency profile, including the idle hours. Compute cost per unit of work for the hosted and self-hosted paths, with redundancy included. Compare against per-token pricing on the same workload. Keep the model, because inputs change.
7. Design the boundary and the gateway. Data classes, permitted hosting mode per class, routing enforcement, logging. Then build it before the first workload goes live, not after the third.
8. Decide, and write down what would change the decision. Name the triggers: volume crossing a threshold, a regulator clarifying a position, a provider adding a regional endpoint, an open-weight model closing the gap on your evaluation set. Review on a date, not on a rumour.

The output of that sequence is often not the architecture people expected walking in. The most frequent outcome we see is a tiered design: a small set of genuinely restricted workloads served from inside the boundary, a larger set served by a regional hosted endpoint under a properly negotiated agreement, and a gateway between them that makes the split visible and auditable. Fewer GPUs than the first conversation assumed, and a stronger compliance position than the all-on-premise plan would have produced, because the controls were designed rather than inherited from a hardware decision.

We do not publish invented cost comparisons or vendor benchmark tables, because by the time you read them they are wrong. What we will do is run this sequence with your workloads, your constraint documents and your data, and give you the numbers for your situation. If the honest answer is that a data processing agreement and a regional endpoint satisfy your obligation and you do not need private infrastructure at all, we will tell you that on the call.