On-premises LLM deployment puts a large language model on servers in a building the organization controls. It is one of several ways to run a model privately, beside colocation space, dedicated GPU capacity and dedicated tenancy in a public cloud, with a managed model API as the baseline. Each option places the hardware, the facility, the data, the model weights, the logs and day-to-day operations with a different mix of parties.
NIST's definition shows why the word private settles little on its own. A private cloud is provisioned for exclusive use by one organization, and it "may be owned, managed, and operated by the organization, a third party, or some combination of them, and it may exist on or off premises" [1]. What separates the options is who owns the servers, who runs the building and who operates the stack.
What "private" means for an LLM
NIST's 2012 guidance splits private clouds into two variants because they "affect the security perimeter" [2]. In an on-site private cloud, the organization builds the perimeter, and the perimeter exists only if it does [2]. An outsourced private cloud has two perimeters, the consumer's and the provider's, joined by a protected link, and the provider accepts responsibility for keeping the private resources apart from other cloud resources [2]. NIST adds a test: an outsourced arrangement that separates customers only with the normal public cloud mechanisms, such as hardware virtualization and VLANs, "would essentially become the public cloud scenario" [2]. Dedicated Instances on Amazon EC2 run on hardware dedicated to one AWS account, though their EBS volumes do not run on single-tenant hardware [16].
The service model sets how far the customer's control reaches. An IaaS consumer "should assume full responsibility for the guest OS's, while the IaaS provider controls the host OS" [3]. AWS calls the same split shared responsibility: it protects the hardware, software, networking and facilities, and the customer's share depends on the services it selects [4].
Who controls what in each option
| Option | Hardware | Facility | Model weights | Logs | Operations | Sources |
|---|---|---|---|---|---|---|
| Managed model API | Provider | Provider | Provider | Provider's abuse logs, plus the customer's own if enabled | Provider | [6, 9, 10, 13, 14] |
| Dedicated tenancy in a public cloud | Provider's hosts, dedicated to one account | Provider | Customer, in its instances | Customer, in its instances | Provider below the guest OS, customer above | [3, 4, 16] |
| Provider hardware on your premises | Owned and managed by the provider | Customer | Set by the service | Set by the service | Provider | [17] |
| Your servers, your facility | Customer | Customer | Customer | Customer, as set in its serving software | Customer or its contractor | [2, 28, 29] |
| Your servers in colocation | Customer | Colocation operator | Customer | Customer | Customer or its contractor | [18] |
| Dedicated GPU capacity | GPU cloud, under contract | A facility the GPU cloud leases | Set by contract | Set by contract | GPU cloud for the hardware | [2, 19] |
The table is a reading of the documents cited.
LLM on-premises vs cloud: what a managed API keeps
On a managed model API, the provider holds the hardware, the facility and the model, and the customer holds settings. On Azure, prompts and completions on Foundry Models sold by Azure are not available to the model providers [6]. A Global or DataZone deployment changes only where prompts are processed, and data at rest, including the abuse monitoring store, stays in the customer-designated geography [6]. On Amazon Bedrock, each model provider has one deployment account in each Region where Bedrock is available, owned and operated by the Bedrock service team, and model providers have no access to those accounts [8].
Retention differs by service. One model developer's API keeps abuse monitoring logs for up to 30 days unless longer retention is required by law or needed to prevent harm, and modified abuse monitoring takes prior approval [14]. On Bedrock, the default mode lets AWS retain data for safety and abuse prevention, and a zero data retention mode writes no request or response data to durable storage [9]. Some model families there require human review by AWS, and prompts and completions for those are kept for up to 30 days [9]. On Azure, when abuse indicators are detected, a sample of prompts and completions may be reviewed by automated means and, as necessary, by authorized employees. Approved customers can opt out of storage and human review [6, 7]. Google Cloud's managed model platform may log prompts to detect abuse, and its default cache is deleted no later than 24 hours after it is written [13].
The customer's own logs are a switch. Bedrock's model invocation logging is off by default, and once on, it can hold full requests and responses [10]. The cost side of this choice is worked through in Self-hosted LLM vs API cost: a break-even method.
On-premises LLM deployment on your own servers
Running the model on your own servers moves every column of the table to you, along with NIST's on-site costs: equipment paid for up front and a fixed capacity sized to expected workloads [2]. It also moves the serving software's defaults. The documentation of vLLM, an open-source serving engine, says traffic between the nodes of a multi-node deployment is insecure by default and must be protected by placing the nodes on an isolated network, and its API key option covers only some endpoint paths [28]. Its request logging is off by default, and it records prompt inputs only at the DEBUG log level [29].
The weights arrive as files that need checks. Hugging Face warns that loading a pickle file can allow arbitrary code execution, and pickle is the default format for PyTorch weights [27]. Joint guidance from April 2024 asks deployers to confirm each artifact's origin and integrity with digital signatures and checksums, and to inspect imported pre-trained models in a secure development zone first [24]. The license comes with the weights: one open-weight model family's community license requires a licensee whose products had more than 700 million monthly active users in the month before the licensed version's 23 July 2024 release date to request a license [32]. Sizing the servers is a separate calculation, set out in How much GPU memory LLM inference needs.
On premises does not settle who owns the hardware. AWS Outposts racks sit in a customer-managed building. Their racks, servers, switches and cabling are owned and managed by AWS, which operates, monitors and manages the capacity as part of an AWS Region [17].
Colocation and dedicated GPU capacity
Colocation keeps the hardware yours and the building someone else's. One colocation operator's 2025 annual report says its offerings are typically billed on the space and power a customer consumes, under a fixed duration contract [18]. It also describes an around-the-clock on-site service in which its technicians install and troubleshoot customer equipment [18]. Who may touch the servers is a contract term.
Dedicated GPU capacity adds a third party. One GPU cloud's 2025 annual report says customers generally buy a specified amount of capacity on a take-or-pay basis under multi-year committed contracts, with a weighted-average duration of approximately five years at 31 December 2025 [19]. It leases or licenses its data center space and states that it does not control the operation of those facilities [19]. NIST's outsourced private cloud is the closest of its models: two perimeters, a protected link, and startup costs that include negotiating the service level agreement but no server-side equipment [2].
Rillor deploys and operates the same private environment on your servers, in your colocation space or on dedicated GPU capacity it sources and contracts for you, and in each option your data, model weights and logs stay in your environment.
Air-gapped options from cloud vendors
A disconnected install removes the network link and keeps the vendor. Google Cloud's documentation says Google Distributed Cloud air-gapped needs no connectivity to Google Cloud, arrives as integrated racks delivered to the customer's premises, and takes software on a portable storage device with a checksum to verify it [20]. In August 2025, Google Cloud announced a commercial model family as a fully managed endpoint on that platform, in a customer or partner data center [22]. Who carries each update across the gap, and who checks it, are questions to settle in the agreement.
Questions to settle before choosing
The joint guidance is written for organizations that deploy and operate systems developed by another entity, and it is most applicable on premises or in private cloud environments [24]. Its practices become questions to settle where the organization runs the model, and to put to the provider where it does not:
- Roles. Who is accountable for each part, and where do the environment's security boundaries sit [24]?
- Weights. Who can reach them? The guidance asks for access control under two-person control and two-person integrity, and for isolated weight storage [24].
- Keys. Are weights, outputs and logs encrypted at rest, with keys in a hardware security module [24]?
- Logs. Do they cover inputs, outputs, intermediate states and errors, and which party keeps them [24]?
- Updates. Is each new model version fully evaluated for accuracy, performance and security before it is redeployed [24]?
- Third parties. NIST AI 600-1 suggests an inventory of every third party with access to organizational content, and vendor contracts that assign responsibility for incidents [25].
When agents call tools inside the environment, the same questions reach their credentials and audit trails, which MCP access control: scoped credentials, allow-lists and audit logs covers.
Read the full report
The full report adds the matrix's data column, a retention table for five managed services, the air-gapped row, confidential computing and its measured cost in one setup, and all 33 sources.