This article discusses general approaches to compliance-sensitive AI deployment and is not legal advice. Validate requirements with your compliance and legal teams.
Why regulated industries can't just use a public API
For a hospital, a law firm, a bank, or a government agency, the most important question about any AI feature is not "how smart is the model?" It is "where does the prompt go?"
When you call a public LLM API, the text you send — which may contain protected health information (PHI), personally identifiable information (PII), attorney-client privileged material, or controlled financial records — leaves your network and is processed on hardware you do not own, in a region you may not control, under terms you did not write. That single fact triggers a cascade of obligations:
- Business associate agreements (BAAs). Under HIPAA, any vendor that touches PHI on your behalf is a business associate and needs a signed BAA. Not every API tier offers one, and "we have an enterprise plan" is not the same as "we have a signed BAA covering this exact data flow."
- Data residency. GDPR and a growing list of national and sector rules constrain where personal data may be stored and processed. A request that silently routes to a data center in another jurisdiction can breach residency requirements on its own.
- Vendor and supply-chain risk. Every external processor is another party in your audit scope, another DPA to negotiate, another breach-notification chain, and another set of subprocessors you have to trust transitively.
- Retention and training. "We don't train on your data" and "we don't retain your data" are distinct promises, and both depend on the specific tier and contract. Default consumer endpoints rarely give you either in writing.
None of this means the cloud is forbidden. It means the default path — paste sensitive text into a public endpoint — is usually a non-starter until legal, compliance, and security have all signed off on a specific, contracted data flow. For the most sensitive workloads, the cleanest way to answer "where does the prompt go?" is: nowhere. It stays inside your walls.
The on-prem / self-hosted answer
Running inference on infrastructure you control inverts the risk model. The prompt is generated, processed, and answered on your hardware, on your network, in your region. The data never crosses a trust boundary, so most of the third-party obligations above simply do not apply — there is no business associate to agree with, no subprocessor chain, no foreign-region routing to police.
The reason this is practical in 2026 and was not a few years ago is that open-weight models have closed most of the quality gap for everyday business tasks. You do not need a frontier model to summarize a discharge note, extract fields from a contract, classify a support ticket, or redact a document. Open models like the Llama 4 and Mistral families handle those reliably, and they run on hardware you can buy and rack.
A realistic single-box baseline looks like this:
| Workload | Model class | Quant | Approx. VRAM (weights) | Fits on |
|---|---|---|---|---|
| Summarize / extract / classify | 8B | Q4_K_M | ~4.4–4.9 GB | One 24 GB GPU, comfortably |
| Higher-quality drafting | 8B | Q8_0 | ~8.5 GB | One 24 GB GPU |
| Strongest single-box quality | 70B | Q4_K_M | ~38–43 GB | 2× 24 GB or 1× 48 GB+ GPU |
Add KV cache and runtime overhead on top of the weight figures — a 70B Q4_K_M needs roughly 48 GB total at modest context, which is why it lands on a pair of 24 GB cards or a single 48 GB card rather than one consumer GPU. To check what a specific GPU can hold before you buy, run what LLM can I run; to estimate throughput, the self-hosted LLM cost calculator compares owning hardware against per-token cloud pricing.
For a concrete, fully-private workload you can try today without any of this hardware, the in-browser private AI summarizer runs entirely on your own machine via WebGPU — the document never leaves the browser. It is a useful illustration of the on-prem principle at the smallest scale.
Meeting common requirements
On-prem hardware answers the data-residency question, but compliance is never a single control. The table below maps the controls that regulated frameworks care about to how you implement each one in a self-hosted AI stack. The frameworks overlap heavily — a single well-built control usually satisfies the analogous clause in HIPAA, SOC 2, and GDPR at once.
| Control | What it means | How you implement it on-prem | Maps to |
|---|---|---|---|
| Data residency | Data is processed and stored only where policy allows | Inference runs on hardware in your facility/region; the prompt never crosses a trust boundary | GDPR (Ch. V transfers), HIPAA (BAA scope), data-sovereignty rules |
| Access control | Only authorized identities can call the model or read outputs | API keys or SSO at a gateway; per-key scopes; network segmentation; least privilege | HIPAA §164.312(a) access control; SOC 2 CC6 logical access |
| Audit logging | Every request/response is attributable and reviewable | Gateway logs caller, timestamp, model, and input/output (or a hash) in your region | HIPAA §164.312(b) audit controls; SOC 2 CC7 monitoring |
| Encryption | Data is protected in transit and at rest | TLS to the gateway; encrypted disks for model files, logs, and any cached prompts | HIPAA §164.312(a)(2)(iv)/(e); GDPR Art. 32; SOC 2 CC6.7 |
| Retention | Data is kept only as long as policy requires, then disposed | Defined retention + deletion for logs and any cached I/O; no silent persistence | HIPAA §164.316; GDPR Art. 5(1)(e) storage limitation |
The point of the table is that "we run it on-prem" only fills in the first row. The other four are work you still have to do — but they are ordinary security controls you almost certainly already operate for the rest of your stack, and a single gateway in front of inference lets you implement access control, audit logging, and encryption-in-transit in one place instead of bolting them onto every application separately.
The hybrid pattern: local baseline, controlled burst
Very few organizations want either extreme — all cloud or all local. The pragmatic architecture is edge-first, cloud when you choose: serve the steady-state, sensitive workload from your own hardware, and reserve a cloud path only for the specific cases policy permits (non-sensitive overflow, or the rare task that genuinely needs a frontier model's reasoning).
┌─────────────────────────────┐
app ──────────► │ one OpenAI-compatible │
│ endpoint (gateway) │
└──────────────┬──────────────┘
│ per-request routing
┌────────────────────┼────────────────────┐
▼ ▼ ▼
edge cache your hardware cloud burst
(repeat hits) (llama.cpp / vLLM, (only when policy
your GPUs, your data) allows; failover)
The principle is yours, not theirs: your GPUs, your models, your data. The data only leaves your boundary if and when you explicitly route a request to the cloud — and for regulated workloads, you simply do not. The default path stays local.
This is exactly the wall where an AI gateway earns its place. Once you have more than one consumer of the model, you want a single OpenAI-compatible endpoint, centralized access control and audit logging, and a deterministic routing policy — rather than every app embedding its own model URL and logging logic. WideAreaAI is an edge-first AI gateway built for this shape: it gives your apps one OpenAI-compatible endpoint and routes each request — edge cache, then your own hardware (a llama.cpp node reached over a Cloudflare Tunnel), then optional cloud burst as failover. The pitch is WAN to WAI: own your baseline, burst to the cloud when you decide.
Two honesty guardrails, because this is a compliance article and the distinction matters:
- A gateway does request-level routing, failover, and edge caching across whole nodes. That is not model-splitting or tensor-parallelism across machines (that is "clustering," a separate technique with separate trade-offs). A gateway picks which node serves a request; it does not split one model across several.
- A gateway is plumbing, not a certification. It makes routing, centralized logging, and access control much easier to implement and attest — three of the five controls in the table above — but it cannot make you HIPAA, SOC 2, or GDPR compliant on its own. Compliance is a property of your whole program, not of one component.
Used honestly, the gateway is the piece that turns "we have a GPU in a closet" into "we have a governed, logged, single-endpoint AI service with a documented routing policy."
Cost and capacity planning
The economic case for on-prem is a break-even calculation: hardware and power are largely fixed costs, while cloud APIs bill per token forever. On your own hardware there are no per-token fees on the baseline workload — you have already paid for the compute. The crossover depends on your volume, your model size, and your hardware.
Rather than guess, size it with real numbers. The self-hosted LLM cost calculator compares cloud API spend against owning hardware and shows you the break-even point; what LLM can I run detects your GPU and tells you which models actually fit. The chart below sketches the shape of the trade-off — fixed-ish on-prem cost versus per-token cloud cost that climbs with usage. The crossover point is exactly where owning your baseline starts paying for itself.
Below break-even, low or bursty volume favors paying per token in the cloud. Above it — steady, high-volume, sensitive workloads — owning the baseline wins on both cost and control. For regulated data the calculus is even more lopsided, because the cloud path may carry compliance costs (BAAs, DPAs, audit scope) that never show up on the per-token invoice.
How InventiveHQ can help
Standing up governed on-prem AI is a security-and-infrastructure project, not just a model download. InventiveHQ works with regulated organizations to:
- Assess the data flows and map AI workloads to your actual compliance obligations, so you know which tasks can be local-only and which (if any) can ever touch a cloud path.
- Deploy the inference stack — hardware sizing, model selection, an OpenAI-compatible gateway, access control, audit logging, and encryption — wired into your existing controls rather than bolted on.
- Manage it over time: patching, monitoring, log retention, model updates, and the evidence your auditors will ask for.
If you are weighing on-prem AI for PHI, privileged, or otherwise regulated data, get in touch and we will help you scope it against your obligations before you spend on hardware.
Conclusion
For regulated data, control beats convenience. The default path — sensitive text into a public endpoint — drags in BAAs, residency rules, and vendor risk that most healthcare, finance, legal, and government teams cannot accept by default. On-prem inference inverts that: the prompt never leaves your walls, which answers the data-residency question outright and shrinks your third-party exposure to near zero.
It is no longer a hard build, either. Open-weight models are good enough for the bulk of business tasks, the hardware fits in a rack, and a gateway gives you one governed endpoint with centralized access control and logging. Just keep the framing honest — on-prem and a gateway are controls that make compliance achievable, not a certificate that grants it. Build the rest of the program around them, and you get capable AI that stays yours, not theirs.