> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.
# Air-gapped Enterprise Deployment
> Full on-premises deployment with no internet egress — local models, local scoring, local auth
Some data cannot leave the building. Defence, intelligence, government, and certain financial and healthcare workloads operate under data handling requirements that prohibit cloud egress entirely. This reference architecture deploys the complete Cognisafe stack — inference, scoring, observability, and identity — within an air-gapped enterprise perimeter with zero runtime internet dependency.
## When to use this pattern
This architecture is appropriate when your workload must satisfy any of the following:
* **UK/US government classification** — IL3, IL4, CUI, OFFICIAL-SENSITIVE, or SECRET handling requirements prohibit sending data to commercial cloud APIs
* **NHS DSPT** — patient data must remain within NHS-approved infrastructure; commercial LLM APIs do not hold DSP Toolkit accreditation
* **Financial services** — FCA SYSC 8.1 operational resilience requirements, or internal risk committees that prohibit sending trading or customer data to third-party AI providers
* **ISO 27001 with contractual data residency** — customer contracts specify that data must remain within a defined geographic or network boundary
If your organisation uses Azure Government, AWS GovCloud, or a commercial cloud at a compliant classification level, consider the [Azure OpenAI + APIM pattern](/reference-architectures/azure-openai-apim) instead — it may satisfy your requirements with less operational overhead.
## Infrastructure requirements
| Component | Minimum spec | Notes |
| -------------------------- | --------------------------------------- | --------------------------------------------------------------------- |
| GPU server (inference) | 2× NVIDIA A100 80 GB or 4× H100 80 GB | Llama 3.1 70B requires \~140 GB VRAM; A100s can shard across 2 GPUs |
| GPU server (scoring) | 1× NVIDIA A10G or A100 40 GB | Llama Guard 3 8B fits in 16 GB; use A10G for cost efficiency |
| Kubernetes cluster | 3-node control plane + GPU worker nodes | RKE2 or OpenShift; do not use managed K8s (EKS/AKS) in a true air-gap |
| PostgreSQL | 16-core, 128 GB RAM, 10 TB NVMe | TimescaleDB extension required |
| Redis | 8-core, 32 GB RAM | Sentinel configuration for HA |
| Private container registry | Harbor or Artifactory | Required before internet is disconnected |
| Identity provider | Keycloak 24+ | OIDC bridge to Active Directory / LDAP |
| Monitoring | Prometheus + Grafana (self-hosted) | No Datadog, no cloud-native APM |
## Architecture
```text theme={null}
╔═══════════════════════════════════════════════════════════════════════╗
║ ENTERPRISE PERIMETER (air-gap boundary) ║
║ ║
║ ┌──────────────────────────────────────────────────────────────┐ ║
║ │ Identity layer │ ║
║ │ Active Directory ←→ Keycloak (OIDC bridge) │ ║
║ └───────────────────────────┬──────────────────────────────────┘ ║
║ │ OIDC/JWT ║
║ ┌───────────────────────────▼──────────────────────────────────┐ ║
║ │ Cognisafe Web (Next.js) :3000 │ ║
║ │ Cognisafe API (FastAPI) :8000 │ ║
║ │ Cognisafe Proxy (Go) :8080 │ ║
║ │ Safety Worker (Python) │ ║
║ │ PostgreSQL (TimescaleDB) :5432 │ ║
║ │ Redis (Sentinel) :6379 │ ║
║ └───────────────────────────┬──────────────────────────────────┘ ║
║ │ OpenAI-compatible HTTP ║
║ ┌───────────────────────────▼──────────────────────────────────┐ ║
║ │ vLLM inference server :8001 │ ║
║ │ Model: Llama-3.1-70B-Instruct (or Mistral Large 2) │ ║
║ │ GPU: 2× A100 80 GB (NVLink) │ ║
║ └──────────────────────────────────────────────────────────────┘ ║
║ ║
║ ┌──────────────────────────────────────────────────────────────┐ ║
║ │ Llama Guard 3 inference :8002 │ ║
║ │ Model: Llama-Guard-3-8B │ ║
║ │ GPU: 1× A10G 24 GB │ ║
║ └──────────────────────────────────────────────────────────────┘ ║
║ ║
║ ┌──────────────────────────────────────────────────────────────┐ ║
║ │ Private container registry (Harbor) │ ║
║ │ Vulnerability scanner (Trivy) │ ║
║ └──────────────────────────────────────────────────────────────┘ ║
║ ║
╚═══════════════════════════════════════════════════════════════════════╝
```
## Implementation
This step requires temporary internet access on an internet-connected machine (not in the air-gap). Pull, scan, and push all required images before disconnecting.
```bash theme={null}
#!/usr/bin/env bash
# Run on an internet-connected build host, NOT inside the air-gap
set -euo pipefail
REGISTRY="registry.internal.example.com"
TRIVY_SEVERITY="CRITICAL,HIGH"
images=(
"cognisafe/proxy:latest"
"cognisafe/api:latest"
"cognisafe/web:latest"
"vllm/vllm-openai:v0.6.0"
"ghcr.io/meta-llama/llama-guard-3-8b:latest"
"postgres:16"
"redis:7-alpine"
"goharbor/harbor-core:v2.11.0"
)
for image in "${images[@]}"; do
echo "Pulling $image"
docker pull "$image"
echo "Scanning $image with Trivy"
trivy image --severity "$TRIVY_SEVERITY" --exit-code 1 "$image"
tag="${REGISTRY}/${image##*/}"
docker tag "$image" "$tag"
echo "Pushing $tag"
docker push "$tag"
done
```
Transfer model weights separately (they are too large for a container image). Use a secure transfer medium (encrypted USB, secure file transfer appliance):
```bash theme={null}
# Download model weights from HuggingFace on the internet-connected host
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct \
--local-dir ./models/llama-3.1-70b \
--token hf_...
huggingface-cli download meta-llama/Llama-Guard-3-8B \
--local-dir ./models/llama-guard-3 \
--token hf_...
# Checksum the weights before transfer
sha256sum ./models/**/*.safetensors > model-checksums.sha256
```
Verify checksums after transfer into the air-gap before loading into vLLM.
Deploy vLLM on the GPU inference nodes. Mount model weights from a local NFS share or hostPath volume.
```yaml theme={null}
# vllm-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama3
namespace: inference
spec:
replicas: 1
selector:
matchLabels:
app: vllm-llama3
template:
metadata:
labels:
app: vllm-llama3
spec:
nodeSelector:
nvidia.com/gpu.product: A100-SXM4-80GB
containers:
- name: vllm
image: registry.internal.example.com/vllm-openai:v0.6.0
args:
- "--model=/models/Llama-3.1-70B-Instruct"
- "--tensor-parallel-size=2"
- "--max-model-len=8192"
- "--port=8001"
- "--served-model-name=llama-3.1-70b-instruct"
resources:
limits:
nvidia.com/gpu: 2
volumeMounts:
- name: model-weights
mountPath: /models
ports:
- containerPort: 8001
volumes:
- name: model-weights
nfs:
server: nfs.internal.example.com
path: /exports/llm-models
---
apiVersion: v1
kind: Service
metadata:
name: vllm-llama3
namespace: inference
spec:
type: ClusterIP
selector:
app: vllm-llama3
ports:
- port: 8001
targetPort: 8001
```
vLLM's API is OpenAI-compatible. The Cognisafe proxy `UPSTREAM_URL` points at `http://vllm-llama3.inference.svc.cluster.local:8001`.
Run a separate vLLM instance for Llama Guard 3. The Cognisafe safety worker calls this endpoint instead of the commercial OpenAI API.
```yaml theme={null}
# llama-guard-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: llama-guard
namespace: inference
spec:
replicas: 1
selector:
matchLabels:
app: llama-guard
template:
metadata:
labels:
app: llama-guard
spec:
nodeSelector:
nvidia.com/gpu.product: NVIDIA-A10G
containers:
- name: vllm
image: registry.internal.example.com/vllm-openai:v0.6.0
args:
- "--model=/models/Llama-Guard-3-8B"
- "--port=8002"
- "--served-model-name=llama-guard-3-8b"
resources:
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: model-weights
mountPath: /models
ports:
- containerPort: 8002
volumes:
- name: model-weights
nfs:
server: nfs.internal.example.com
path: /exports/llm-models
```
Configure the Cognisafe safety worker to use Llama Guard as its scoring backend:
```bash theme={null}
# safety_worker environment variables
OPENAI_API_KEY=not-required-for-local
OPENAI_BASE_URL=http://llama-guard.inference.svc.cluster.local:8002/v1
SCORER_MODEL=llama-guard-3-8b
```
The LLM-as-judge scorer will query Llama Guard using the OpenAI-compatible chat completions API at the local endpoint. No internet egress occurs.
Set environment variables to point all Cognisafe components at the local infrastructure. Use Kubernetes Secrets for sensitive values.
```yaml theme={null}
# cognisafe-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: cognisafe-env
namespace: cognisafe
data:
UPSTREAM_URL: "http://vllm-llama3.inference.svc.cluster.local:8001"
API_BACKEND_URL: "http://cognisafe-api.cognisafe.svc.cluster.local:8000"
REDIS_URL: "redis://redis-sentinel.cognisafe.svc.cluster.local:26379"
POSTGRES_URL: "postgresql+asyncpg://cognisafe:$(POSTGRES_PASSWORD)@postgres.cognisafe.svc.cluster.local:5432/cognisafe"
OPENAI_BASE_URL: "http://llama-guard.inference.svc.cluster.local:8002/v1"
SCORER_MODEL: "llama-guard-3-8b"
# Disable Stripe — all subscriptions managed manually (see Step 6)
STRIPE_SECRET_KEY: ""
STRIPE_WEBHOOK_SECRET: ""
# Email — use internal SMTP relay (see Step 7)
SMTP_HOST: "smtp.internal.example.com"
SMTP_PORT: "587"
```
Apply the Cognisafe Helm chart with these overrides:
```bash theme={null}
helm upgrade --install cognisafe ./charts/cognisafe \
--namespace cognisafe \
--create-namespace \
--values values-airgap.yaml \
--set image.repository=registry.internal.example.com/cognisafe \
--set image.pullPolicy=IfNotPresent
```
Keycloak bridges your on-premises Active Directory to OIDC, which Cognisafe's web UI uses for authentication (via Clerk-compatible OIDC or direct OIDC integration).
```bash theme={null}
# Deploy Keycloak
helm upgrade --install keycloak bitnami/keycloak \
--namespace identity \
--set auth.adminUser=admin \
--set auth.adminPassword="$(openssl rand -base64 32)" \
--set externalDatabase.host=postgres.cognisafe.svc.cluster.local \
--set externalDatabase.database=keycloak \
--set image.registry=registry.internal.example.com
```
In the Keycloak admin console:
1. Create a realm: `cognisafe`
2. Add a User Federation: LDAP — point at your Active Directory domain controller
3. Sync users from AD
4. Create a Client: `cognisafe-web` — Client Protocol: openid-connect, Access Type: confidential
5. Set Valid Redirect URIs: `https://cognisafe.internal.example.com/*`
Configure Cognisafe to use Keycloak:
```bash theme={null}
# web/ environment variables
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY="" # Clerk not used in air-gap
OIDC_ISSUER=https://keycloak.internal.example.com/realms/cognisafe
OIDC_CLIENT_ID=cognisafe-web
OIDC_CLIENT_SECRET=
OIDC_REDIRECT_URI=https://cognisafe.internal.example.com/auth/callback
```
Set `STRIPE_SECRET_KEY` to an empty string. The Cognisafe API falls back to reading subscription tier directly from the `subscriptions` table when Stripe is not configured.
Insert a manual subscription row for each project:
```sql theme={null}
INSERT INTO subscriptions (
project_id,
tier,
stripe_customer_id,
stripe_subscription_id,
requests_this_period,
period_start,
period_end
) VALUES (
'my-project-id',
'enterprise',
NULL, -- no Stripe customer
NULL, -- no Stripe subscription
0,
NOW(),
NOW() + INTERVAL '1 year'
);
```
Enterprise tier has no request limit. All users on an air-gapped deployment should use the enterprise tier — billing is handled through your existing procurement process, not Stripe.
Disable Resend (the cloud email provider) and configure Cognisafe to deliver alerts via your internal SMTP relay or Splunk HTTP Event Collector.
**Internal SMTP:**
```bash theme={null}
# safety_worker + api environment variables
ALERT_BACKEND=smtp
SMTP_HOST=smtp.internal.example.com
SMTP_PORT=587
SMTP_FROM=cognisafe-alerts@example.com
SMTP_TO=security-ops@example.com
SMTP_TLS=true
SMTP_USERNAME=cognisafe-svc
SMTP_PASSWORD=
```
**Splunk HEC (preferred for SOC integration):**
```bash theme={null}
ALERT_BACKEND=webhook
ALERT_WEBHOOK_URL=https://splunk.internal.example.com:8088/services/collector/event
ALERT_WEBHOOK_HEADERS='{"Authorization": "Splunk "}'
```
The Cognisafe webhook payload is JSON and maps directly to a Splunk sourcetype. Create a Splunk saved search on `sourcetype=cognisafe threat_detected` to drive your SOC alerting rules.
## Egress requirements
| Phase | Egress required | What for |
| --------------------------------------- | ------------------------------------ | ------------------------------------------------------------------- |
| Initial setup (internet-connected host) | Yes | Pull container images, download model weights, download Helm charts |
| Model weight transfer | Physical media only | Encrypted USB or secure file transfer appliance |
| Runtime (steady state) | None | All inference, scoring, identity, and alerting are local |
| Updates | Physical media or one-way data diode | Periodic model updates, security patches (see operational runbook) |
"Zero-egress runtime" means no network connections leave the air-gap boundary during normal operation. This includes NTP (use an internal NTP server), DNS (use internal resolver), and certificate revocation (pre-load CRLs or use OCSP stapling with cached responses).
## Model selection
**Llama 3.1 70B Instruct** — Best open-weight general model at time of writing. Requires 2× A100 80 GB with tensor parallelism. Competitive with GPT-4o on instruction following and coding.
**Mistral Large 2** — Strong alternative; slightly lower VRAM requirement. Consider for cost-constrained deployments.
**Llama 3.1 8B Instruct** — For low-latency use cases or constrained GPU budgets. Fits on a single A10G.
**Llama Guard 3 8B** — Meta's purpose-built content safety classifier. Trained specifically for LLM input/output classification across OWASP LLM risk categories. Fits on a single A10G 24 GB.
**ShieldLM** — Alternative if Llama Guard 3 is not available in your approved software list.
## Compliance posture
| Framework | Relevant control | How this architecture satisfies it |
| ----------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| UK Government IL3 / IL4 | Data must remain within HMG-approved infrastructure | All inference and logging within the perimeter; no commercial LLM API calls |
| NHS DSPT | Standard 07 — processing personal data | Patient data never leaves the network boundary; Llama Guard 3 scores for PII in every request |
| ISO 27001 A.12.4 | Logging and monitoring | TimescaleDB audit log for every LLM and tool invocation; Keycloak provides identity audit trail |
| ISO 27001 A.13.1 | Network segregation | Air-gap enforced at the network layer; Kubernetes NetworkPolicy restricts pod-to-pod communication |
| SOC 2 CC6.1 | Logical access | Keycloak + AD enforces authentication and RBAC; Cognisafe API keys scoped per project |
| NCSC CAF (UK) | B1 — Service protection policies | Change management runbook for model updates; vulnerability scanning of all container images before admission |
For accreditation evidence packs, export the following from Cognisafe:
* `llm_requests` table (full, or time-bounded) — demonstrates audit trail
* `safety_scores` table — demonstrates content screening
* Keycloak audit log (realm events) — demonstrates access control
* Trivy scan reports for all running container images
## Operational runbook: model updates
Model updates require careful change management in an air-gapped environment. Follow this runbook for every update.
```bash theme={null}
# Download the new model weights
huggingface-cli download meta-llama/Llama-3.1-70B-Instruct \
--revision v2.0.0 \
--local-dir ./models/llama-3.1-70b-v2
# Verify checksums from the official HuggingFace page
sha256sum -c official-checksums.sha256
# Scan with Trivy (model weights should be scanned for embedded malware)
trivy fs --severity CRITICAL,HIGH ./models/llama-3.1-70b-v2
# Create a manifest for the change management ticket
find ./models/llama-3.1-70b-v2 -name "*.safetensors" \
-exec sha256sum {} \; > model-update-manifest.sha256
```
Transfer the model weights and container image tarballs via your approved secure transfer mechanism. Log the transfer in your change management system (ServiceNow, Jira, etc.) with the SHA256 manifest as evidence.
```bash theme={null}
# Inside the air-gap: verify checksums after transfer
sha256sum -c model-update-manifest.sha256
# Copy weights to the NFS share
rsync -av --checksum ./models/llama-3.1-70b-v2/ \
nfs.internal.example.com:/exports/llm-models/Llama-3.1-70B-Instruct-v2/
```
Update the vLLM Deployment to point at the new model path. Kubernetes performs a rolling update — the old model instance stays up until the new one is healthy.
```bash theme={null}
kubectl set env deployment/vllm-llama3 \
--namespace inference \
"MODEL_PATH=/models/Llama-3.1-70B-Instruct-v2"
# Monitor the rollout
kubectl rollout status deployment/vllm-llama3 --namespace inference --timeout=10m
```
Run a smoke test against the new model before marking the change as complete:
```bash theme={null}
curl -s http://vllm-llama3.inference.svc.cluster.local:8001/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.1-70b-instruct", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 10}' \
| jq '.choices[0].message.content'
```
Never run `kubectl set image` or patch a Deployment with a new model path unless you have verified the SHA256 checksums of the transferred weights against the official manifest. A supply chain compromise at the model weight level is indistinguishable from a legitimate update without this check.
Configure Prometheus alerts for GPU memory utilisation and model inference latency (p99). A sudden spike in p99 latency after a model update is an indicator that the new model has different compute characteristics and may require vLLM tuning (`--max-model-len`, `--gpu-memory-utilization`) before the rollout is fully complete.