> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt > Use this file to discover all available pages before exploring further. # Azure OpenAI + APIM + Cognisafe > Enterprise AI gateway pattern — OAuth2, rate limiting, and runtime security in a single request path This pattern assembles three Azure-native components into a hardened AI gateway. Azure API Management owns identity validation, rate limiting, and request routing. Cognisafe owns AI-specific runtime security — OWASP LLM scoring, jailbreak detection, PII scanning, cost attribution, and the immutable audit trail. Azure OpenAI sits behind a private endpoint and is unreachable from the public internet. No component duplicates another's responsibility. Use this pattern when your organisation needs to: satisfy Azure Security Benchmark controls on AI workloads, enforce per-team request quotas at the gateway layer, demonstrate an audit trail of every LLM exchange to a compliance team, or integrate AI API access with an existing Azure AD enterprise application model. ## Architecture overview ```text theme={null} ┌─────────────────────────────────────────────────────────────────────┐ │ Client (web app / service / developer) │ │ Authorization: Bearer │ └───────────────────────────┬─────────────────────────────────────────┘ │ HTTPS (public) ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Azure API Management (Standard / Premium tier) │ │ │ │ Inbound policy chain: │ │ 1. validate-jwt → verify AAD token, check audience + issuer │ │ 2. rate-limit-by-key → quota per subscription / team │ │ 3. set-header → X-Cognisafe-Project-ID from JWT claim │ │ 4. set-header → Authorization: Bearer │ │ 5. forward → Cognisafe proxy (private ClusterIP) │ └───────────────────────────┬─────────────────────────────────────────┘ │ HTTP (private VNet) ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ AKS cluster (private) │ │ │ │ ┌───────────────────────────────────────┐ │ │ │ Cognisafe Proxy (Go, :8080) │ │ │ │ - Authenticates PROXY_API_KEY │ │ │ │ - Forwards to Azure OpenAI PE │ │ │ │ - POSTs /internal/log (async) │ │ │ └──────────────┬────────────────────────┘ │ │ │ │ │ ┌──────────────▼────────────────────────┐ │ │ │ Cognisafe API (FastAPI, :8000) │ │ │ │ - Persists request/response │ │ │ │ - Pushes scoring job to Redis │ │ │ └──────────────┬────────────────────────┘ │ │ │ │ │ ┌──────────────▼────────────────────────┐ │ │ │ Safety Worker (Python) │ │ │ │ - Scores async; writes to Postgres │ │ │ └───────────────────────────────────────┘ │ │ │ │ PostgreSQL (TimescaleDB) Redis │ └───────────────────────────┬─────────────────────────────────────────┘ │ Private endpoint (RFC 1918) ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ Azure OpenAI (no public access; firewall = deny all public) │ │ Private endpoint in AKS subnet │ └─────────────────────────────────────────────────────────────────────┘ ``` Response flows back through the same path. APIM passes the LLM response payload through unchanged. Cognisafe scoring runs entirely asynchronously — the client receives the LLM response with no scoring latency. ## Component responsibilities | Component | Owns | Does NOT own | | ---------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------- | | Azure AD | Token issuance, user/group identity, MFA | Request content, LLM policy | | Azure APIM | JWT validation, rate limiting by subscription, API versioning, developer portal | AI-specific threat detection, cost per-token attribution | | Cognisafe Proxy | LLM request forwarding, async logging trigger, PROXY\_API\_KEY auth | Identity management, coarse rate limiting | | Cognisafe API + Worker | Persistence, OWASP LLM scoring, cost attribution, alerting, audit trail | Network-layer access control | | Azure OpenAI | LLM inference | Security, logging, rate limiting — all delegated upstream | ## Implementation Disable public network access on the Azure OpenAI resource. Create a private endpoint in the same VNet as your AKS cluster. ```bash theme={null} # Create the Azure OpenAI resource az cognitiveservices account create \ --name my-aoai \ --resource-group my-rg \ --kind OpenAI \ --sku S0 \ --location eastus \ --custom-domain my-aoai \ --public-network-access Disabled # Create a private endpoint in the AKS subnet az network private-endpoint create \ --name aoai-pe \ --resource-group my-rg \ --vnet-name my-vnet \ --subnet aks-subnet \ --private-connection-resource-id $(az cognitiveservices account show \ --name my-aoai --resource-group my-rg --query id -o tsv) \ --group-id account \ --connection-name aoai-pe-conn # DNS: create private DNS zone and link to VNet az network private-dns zone create \ --resource-group my-rg \ --name privatelink.openai.azure.com az network private-dns link vnet create \ --resource-group my-rg \ --zone-name privatelink.openai.azure.com \ --name aoai-dns-link \ --virtual-network my-vnet \ --registration-enabled false ``` Verify the private endpoint resolves from inside the AKS cluster: ```bash theme={null} kubectl run -it --rm dns-test --image=busybox --restart=Never -- \ nslookup my-aoai.openai.azure.com # Should resolve to a 10.x.x.x address, not a public IP ``` Deploy the Cognisafe stack into AKS. The proxy service should be a `ClusterIP` (not `LoadBalancer`) — APIM reaches it over the VNet, not the public internet. ```yaml theme={null} # cognisafe-proxy-service.yaml apiVersion: v1 kind: Service metadata: name: cognisafe-proxy namespace: cognisafe spec: type: ClusterIP selector: app: cognisafe-proxy ports: - port: 8080 targetPort: 8080 --- apiVersion: apps/v1 kind: Deployment metadata: name: cognisafe-proxy namespace: cognisafe spec: replicas: 2 selector: matchLabels: app: cognisafe-proxy template: metadata: labels: app: cognisafe-proxy spec: containers: - name: proxy image: cognisafe/proxy:latest env: - name: UPSTREAM_URL value: "https://my-aoai.openai.azure.com" - name: API_BACKEND_URL value: "http://cognisafe-api.cognisafe.svc.cluster.local:8000" - name: PROXY_API_KEY valueFrom: secretKeyRef: name: cognisafe-secrets key: proxy-api-key ports: - containerPort: 8080 ``` Set `UPSTREAM_URL` to the Azure OpenAI private endpoint hostname. The proxy resolves this over the private DNS zone configured in Step 1. APIM must be deployed in the same VNet (or a peered VNet) to reach the Cognisafe ClusterIP. Use **Internal** mode to keep APIM itself off the public internet, or **External** mode if you need the developer portal publicly accessible. ```bash theme={null} az apim create \ --name my-apim \ --resource-group my-rg \ --publisher-email ops@example.com \ --publisher-name "My Org" \ --sku-name Standard \ --virtual-network External \ --vnet-type External # Inject APIM into the VNet subnet az apim update \ --name my-apim \ --resource-group my-rg \ --set virtualNetworkConfiguration.subnetResourceId=$(az network vnet subnet show \ --resource-group my-rg \ --vnet-name my-vnet \ --name apim-subnet \ --query id -o tsv) ``` Add the Cognisafe proxy ClusterIP as a named backend. Use the AKS internal load balancer IP or the internal DNS name if APIM is in the same cluster VNet: ```bash theme={null} az apim backend create \ --resource-group my-rg \ --service-name my-apim \ --backend-id cognisafe-proxy \ --url "http://:8080" \ --protocol http \ --title "Cognisafe Proxy" ``` The inbound policy does five things in sequence: validate the AAD JWT, enforce a quota, extract the project ID from the token claims, inject the Cognisafe API key, and set the backend. Create an API in APIM (e.g. `ai-gateway`) and apply this policy at the API level: ```xml theme={null} api://{app-registration-client-id} https://login.microsoftonline.com/{tenant-id}/v2.0 AI.ReadWrite @(context.Request.Headers.GetValueOrDefault("X-Project-ID", context.User?.Id ?? "default")) @("Bearer " + context.Variables["cognisafe-proxy-key"]) ``` Store `cognisafe-proxy-key` as an APIM named value (type: secret). Retrieve it from Azure Key Vault rather than embedding it directly. Clients authenticate to APIM using an AAD access token scoped to your API registration. ```bash theme={null} # Create the app registration az ad app create \ --display-name "AI Gateway API" \ --identifier-uris "api://{your-client-id}" # Expose an API scope az ad app update \ --id {your-client-id} \ --set api.oauth2PermissionScopes='[{ "id": "'$(uuidgen)'", "adminConsentDescription": "Read and write AI API", "adminConsentDisplayName": "AI.ReadWrite", "isEnabled": true, "type": "User", "userConsentDescription": "Access AI API", "userConsentDisplayName": "AI.ReadWrite", "value": "AI.ReadWrite" }]' # Grant admin consent az ad app permission admin-consent --id {your-client-id} ``` Client applications acquire a token with: ```python theme={null} from azure.identity import ClientSecretCredential credential = ClientSecretCredential( tenant_id="{tenant-id}", client_id="{app-client-id}", client_secret="{app-client-secret}", ) token = credential.get_token("api://{your-client-id}/.default") # Use token.token as the Bearer value in requests to APIM ``` Map your internal team/business unit structure to APIM products. Each product gets its own quota, and each team gets a subscription key tied to that product. ```bash theme={null} # Create a product for a high-volume team az apim product create \ --resource-group my-rg \ --service-name my-apim \ --product-id team-premium \ --product-name "Team Premium" \ --description "5M tokens/month — data science teams" \ --subscription-required true \ --approval-required true \ --state published # Add the AI Gateway API to the product az apim product api add \ --resource-group my-rg \ --service-name my-apim \ --product-id team-premium \ --api-id ai-gateway # Create a subscription for a specific team az apim subscription create \ --resource-group my-rg \ --service-name my-apim \ --subscription-id ds-team-sub \ --display-name "Data Science Team" \ --product-id "/products/team-premium" \ --state active ``` Apply per-product quota policy on the product scope (not API scope) to enforce monthly limits independently of the per-request rate limit applied at the API level. ## Security properties This pattern satisfies the following controls by construction: | Control | How it is satisfied | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | No direct Azure OpenAI access | Public access disabled on the AOAI resource; only the private endpoint in the AKS subnet can resolve and reach it | | Token validated before AI call | `validate-jwt` runs first in the inbound chain — a malformed or expired token returns HTTP 401 before any backend traffic | | Request quota enforced at the network layer | APIM `rate-limit-by-key` and `quota-by-key` policies; enforced before the request reaches Cognisafe | | Full audit trail | Every LLM exchange (prompt, completion, model, latency, cost, safety score) is written to TimescaleDB by Cognisafe; tamper-evident by row-level append-only policy | | AI-specific threat detection | Cognisafe runs LLM-as-judge scorers asynchronously on every captured exchange — jailbreak, PII, content safety, prompt injection | | Data residency | Azure OpenAI private endpoint, AKS cluster, and Cognisafe all run in the same Azure region; no data transits region boundaries | ### OWASP LLM Top 10 controls addressed | OWASP LLM Risk | Mitigation in this pattern | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | LLM01 — Prompt Injection | Cognisafe's `jailbreak_detection` scorer flags injections in every request; APIM blocks unauthenticated callers entirely | | LLM02 — Insecure Output Handling | Completions logged and scored; alert rules can trigger on detected harmful content | | LLM06 — Sensitive Information Disclosure | `pii_detection` scorer runs on both prompt and completion | | LLM08 — Excessive Agency | Agent tagging + per-agent usage dashboards expose runaway autonomy | | LLM09 — Overreliance | Cost attribution and request-volume alerts surface over-use before it becomes a risk | ## Operational guidance **Log Analytics integration.** Enable APIM diagnostics and stream to a Log Analytics workspace. Correlate APIM request IDs with Cognisafe's `llm_requests.id` for end-to-end tracing: ```bash theme={null} az monitor diagnostic-settings create \ --name apim-to-law \ --resource $(az apim show --name my-apim --resource-group my-rg --query id -o tsv) \ --workspace $(az monitor log-analytics workspace show \ --resource-group my-rg --workspace-name my-law --query id -o tsv) \ --logs '[{"category":"GatewayLogs","enabled":true}]' \ --metrics '[{"category":"AllMetrics","enabled":true}]' ``` **Sentinel alerting.** Forward Cognisafe webhook events (`threat_detected`, `pii_detected`) to a Sentinel custom log table via an Azure Logic App or Event Hub. This puts AI-layer threats into the same SIEM as your infrastructure alerts. **Cost attribution.** Cognisafe attributes token cost by `project_id` and `agent_name`. Use the `/api/v1/costs?group_by=project_id` endpoint to generate monthly chargeback reports per APIM product/subscription. **Scaling.** The Cognisafe proxy is stateless — scale the AKS deployment horizontally. The safety worker is independently scalable via a separate Deployment; each replica pulls from the same Redis queue. APIM scales within its SKU tier. ## Failure modes | Failure | Behaviour | Recovery | | ----------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | Cognisafe proxy pod crash | AKS restarts the pod; APIM receives HTTP 503 and returns it to the client | Set `minReplicas: 2` in HPA; configure APIM retry policy for transient 503s | | Cognisafe API unreachable | Proxy logs the request to a local buffer (configurable); scoring is delayed | Deploy API as a Deployment with 2+ replicas; use PodDisruptionBudget | | Redis unavailable | Scoring jobs are not enqueued; requests are still forwarded and logged to Postgres | Redis Sentinel or Azure Cache for Redis with replication | | Azure OpenAI private endpoint DNS failure | Proxy cannot resolve upstream; returns 502 to APIM | Verify private DNS zone link; deploy at least 2 private endpoint NICs across AZs | | APIM VNet integration failure | Clients receive 503 from APIM | Deploy APIM in zone-redundant configuration (Premium SKU) | APIM's `validate-jwt` policy caches the OpenID Connect metadata document. If you rotate your AAD app registration signing keys, allow up to 5 minutes for APIM to refresh the cache before forcing a pod restart. Do not use the APIM Developer tier for production. It does not support VNet integration or zone redundancy, and has no SLA.