Azure OpenAI + APIM + Cognisafe

This pattern assembles three Azure-native components into a hardened AI gateway. Azure API Management owns identity validation, rate limiting, and request routing. Cognisafe owns AI-specific runtime security — OWASP LLM scoring, jailbreak detection, PII scanning, cost attribution, and the immutable audit trail. Azure OpenAI sits behind a private endpoint and is unreachable from the public internet. No component duplicates another’s responsibility. Use this pattern when your organisation needs to: satisfy Azure Security Benchmark controls on AI workloads, enforce per-team request quotas at the gateway layer, demonstrate an audit trail of every LLM exchange to a compliance team, or integrate AI API access with an existing Azure AD enterprise application model.

Architecture overview

┌─────────────────────────────────────────────────────────────────────┐
│  Client (web app / service / developer)                              │
│  Authorization: Bearer <AAD access_token>                            │
└───────────────────────────┬─────────────────────────────────────────┘
                            │ HTTPS (public)
                            ▼
┌─────────────────────────────────────────────────────────────────────┐
│  Azure API Management (Standard / Premium tier)                      │
│                                                                      │
│  Inbound policy chain:                                               │
│    1. validate-jwt  →  verify AAD token, check audience + issuer     │
│    2. rate-limit-by-key  →  quota per subscription / team            │
│    3. set-header  →  X-Cognisafe-Project-ID from JWT claim           │
│    4. set-header  →  Authorization: Bearer <PROXY_API_KEY>           │
│    5. forward  →  Cognisafe proxy (private ClusterIP)                │
└───────────────────────────┬─────────────────────────────────────────┘
                            │ HTTP (private VNet)
                            ▼
┌─────────────────────────────────────────────────────────────────────┐
│  AKS cluster (private)                                               │
│                                                                      │
│  ┌───────────────────────────────────────┐                           │
│  │  Cognisafe Proxy  (Go, :8080)         │                           │
│  │  - Authenticates PROXY_API_KEY        │                           │
│  │  - Forwards to Azure OpenAI PE        │                           │
│  │  - POSTs /internal/log (async)        │                           │
│  └──────────────┬────────────────────────┘                          │
│                 │                                                    │
│  ┌──────────────▼────────────────────────┐                           │
│  │  Cognisafe API  (FastAPI, :8000)      │                           │
│  │  - Persists request/response          │                           │
│  │  - Pushes scoring job to Redis        │                           │
│  └──────────────┬────────────────────────┘                          │
│                 │                                                    │
│  ┌──────────────▼────────────────────────┐                           │
│  │  Safety Worker  (Python)              │                           │
│  │  - Scores async; writes to Postgres   │                           │
│  └───────────────────────────────────────┘                          │
│                                                                      │
│  PostgreSQL (TimescaleDB)   Redis                                    │
└───────────────────────────┬─────────────────────────────────────────┘
                            │ Private endpoint (RFC 1918)
                            ▼
┌─────────────────────────────────────────────────────────────────────┐
│  Azure OpenAI  (no public access; firewall = deny all public)        │
│  Private endpoint in AKS subnet                                      │
└─────────────────────────────────────────────────────────────────────┘

Response flows back through the same path. APIM passes the LLM response payload through unchanged. Cognisafe scoring runs entirely asynchronously — the client receives the LLM response with no scoring latency.

Component responsibilities

Component	Owns	Does NOT own
Azure AD	Token issuance, user/group identity, MFA	Request content, LLM policy
Azure APIM	JWT validation, rate limiting by subscription, API versioning, developer portal	AI-specific threat detection, cost per-token attribution
Cognisafe Proxy	LLM request forwarding, async logging trigger, PROXY_API_KEY auth	Identity management, coarse rate limiting
Cognisafe API + Worker	Persistence, OWASP LLM scoring, cost attribution, alerting, audit trail	Network-layer access control
Azure OpenAI	LLM inference	Security, logging, rate limiting — all delegated upstream

Implementation

Deploy Azure OpenAI behind a private endpoint

Disable public network access on the Azure OpenAI resource. Create a private endpoint in the same VNet as your AKS cluster.

# Create the Azure OpenAI resource
az cognitiveservices account create \
  --name my-aoai \
  --resource-group my-rg \
  --kind OpenAI \
  --sku S0 \
  --location eastus \
  --custom-domain my-aoai \
  --public-network-access Disabled

# Create a private endpoint in the AKS subnet
az network private-endpoint create \
  --name aoai-pe \
  --resource-group my-rg \
  --vnet-name my-vnet \
  --subnet aks-subnet \
  --private-connection-resource-id $(az cognitiveservices account show \
      --name my-aoai --resource-group my-rg --query id -o tsv) \
  --group-id account \
  --connection-name aoai-pe-conn

# DNS: create private DNS zone and link to VNet
az network private-dns zone create \
  --resource-group my-rg \
  --name privatelink.openai.azure.com

az network private-dns link vnet create \
  --resource-group my-rg \
  --zone-name privatelink.openai.azure.com \
  --name aoai-dns-link \
  --virtual-network my-vnet \
  --registration-enabled false

Verify the private endpoint resolves from inside the AKS cluster:

kubectl run -it --rm dns-test --image=busybox --restart=Never -- \
  nslookup my-aoai.openai.azure.com
# Should resolve to a 10.x.x.x address, not a public IP

Deploy Cognisafe to AKS

Deploy the Cognisafe stack into AKS. The proxy service should be a ClusterIP (not LoadBalancer) — APIM reaches it over the VNet, not the public internet.

# cognisafe-proxy-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: cognisafe-proxy
  namespace: cognisafe
spec:
  type: ClusterIP
  selector:
    app: cognisafe-proxy
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cognisafe-proxy
  namespace: cognisafe
spec:
  replicas: 2
  selector:
    matchLabels:
      app: cognisafe-proxy
  template:
    metadata:
      labels:
        app: cognisafe-proxy
    spec:
      containers:
        - name: proxy
          image: cognisafe/proxy:latest
          env:
            - name: UPSTREAM_URL
              value: "https://my-aoai.openai.azure.com"
            - name: API_BACKEND_URL
              value: "http://cognisafe-api.cognisafe.svc.cluster.local:8000"
            - name: PROXY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: proxy-api-key
          ports:
            - containerPort: 8080

Set UPSTREAM_URL to the Azure OpenAI private endpoint hostname. The proxy resolves this over the private DNS zone configured in Step 1.

Create an APIM instance with VNet integration

APIM must be deployed in the same VNet (or a peered VNet) to reach the Cognisafe ClusterIP. Use Internal mode to keep APIM itself off the public internet, or External mode if you need the developer portal publicly accessible.

az apim create \
  --name my-apim \
  --resource-group my-rg \
  --publisher-email ops@example.com \
  --publisher-name "My Org" \
  --sku-name Standard \
  --virtual-network External \
  --vnet-type External

# Inject APIM into the VNet subnet
az apim update \
  --name my-apim \
  --resource-group my-rg \
  --set virtualNetworkConfiguration.subnetResourceId=$(az network vnet subnet show \
      --resource-group my-rg \
      --vnet-name my-vnet \
      --name apim-subnet \
      --query id -o tsv)

Add the Cognisafe proxy ClusterIP as a named backend. Use the AKS internal load balancer IP or the internal DNS name if APIM is in the same cluster VNet:

az apim backend create \
  --resource-group my-rg \
  --service-name my-apim \
  --backend-id cognisafe-proxy \
  --url "http://<cognisafe-proxy-clusterip>:8080" \
  --protocol http \
  --title "Cognisafe Proxy"

Configure the APIM inbound policy

The inbound policy does five things in sequence: validate the AAD JWT, enforce a quota, extract the project ID from the token claims, inject the Cognisafe API key, and set the backend.Create an API in APIM (e.g. ai-gateway) and apply this policy at the API level:

<policies>
  <inbound>
    <base />

    <!-- 1. Validate the Azure AD JWT -->
    <validate-jwt header-name="Authorization"
                   failed-validation-httpcode="401"
                   failed-validation-error-message="Unauthorized: invalid or missing token"
                   require-expiration-time="true"
                   require-signed-tokens="true">
      <openid-config url="https://login.microsoftonline.com/{tenant-id}/v2.0/.well-known/openid-configuration" />
      <audiences>
        <audience>api://{app-registration-client-id}</audience>
      </audiences>
      <issuers>
        <issuer>https://login.microsoftonline.com/{tenant-id}/v2.0</issuer>
      </issuers>
      <required-claims>
        <claim name="roles" match="any">
          <value>AI.ReadWrite</value>
        </claim>
      </required-claims>
    </validate-jwt>

    <!-- 2. Rate limit: 500 calls / 60 seconds per subscription key -->
    <rate-limit-by-key calls="500"
                       renewal-period="60"
                       counter-key="@(context.Subscription.Id)" />

    <!-- 3. Extract project ID from the 'oid' (object ID) claim and
            map to a Cognisafe project. In practice, maintain a mapping
            in APIM named values or an external lookup. -->
    <set-header name="X-Cognisafe-Project-ID" exists-action="override">
      <value>@(context.Request.Headers.GetValueOrDefault("X-Project-ID",
                context.User?.Id ?? "default"))</value>
    </set-header>

    <!-- 4. Replace Authorization header with the Cognisafe PROXY_API_KEY.
            The original AAD token has served its purpose at this layer. -->
    <set-header name="Authorization" exists-action="override">
      <value>@("Bearer " + context.Variables["cognisafe-proxy-key"])</value>
    </set-header>

    <!-- 5. Route to Cognisafe proxy backend -->
    <set-backend-service backend-id="cognisafe-proxy" />
  </inbound>

  <backend>
    <base />
  </backend>

  <outbound>
    <!-- Strip any internal headers before returning to client -->
    <set-header name="X-Cognisafe-Project-ID" exists-action="delete" />
    <base />
  </outbound>

  <on-error>
    <base />
  </on-error>
</policies>

Store cognisafe-proxy-key as an APIM named value (type: secret). Retrieve it from Azure Key Vault rather than embedding it directly.

Clients authenticate to APIM using an AAD access token scoped to your API registration.

# Create the app registration
az ad app create \
  --display-name "AI Gateway API" \
  --identifier-uris "api://{your-client-id}"

# Expose an API scope
az ad app update \
  --id {your-client-id} \
  --set api.oauth2PermissionScopes='[{
    "id": "'$(uuidgen)'",
    "adminConsentDescription": "Read and write AI API",
    "adminConsentDisplayName": "AI.ReadWrite",
    "isEnabled": true,
    "type": "User",
    "userConsentDescription": "Access AI API",
    "userConsentDisplayName": "AI.ReadWrite",
    "value": "AI.ReadWrite"
  }]'

# Grant admin consent
az ad app permission admin-consent --id {your-client-id}

Client applications acquire a token with:

from azure.identity import ClientSecretCredential

credential = ClientSecretCredential(
    tenant_id="{tenant-id}",
    client_id="{app-client-id}",
    client_secret="{app-client-secret}",
)
token = credential.get_token("api://{your-client-id}/.default")
# Use token.token as the Bearer value in requests to APIM

Configure APIM products and subscriptions for quota tiers

Map your internal team/business unit structure to APIM products. Each product gets its own quota, and each team gets a subscription key tied to that product.

# Create a product for a high-volume team
az apim product create \
  --resource-group my-rg \
  --service-name my-apim \
  --product-id team-premium \
  --product-name "Team Premium" \
  --description "5M tokens/month — data science teams" \
  --subscription-required true \
  --approval-required true \
  --state published

# Add the AI Gateway API to the product
az apim product api add \
  --resource-group my-rg \
  --service-name my-apim \
  --product-id team-premium \
  --api-id ai-gateway

# Create a subscription for a specific team
az apim subscription create \
  --resource-group my-rg \
  --service-name my-apim \
  --subscription-id ds-team-sub \
  --display-name "Data Science Team" \
  --product-id "/products/team-premium" \
  --state active

Apply per-product quota policy on the product scope (not API scope) to enforce monthly limits independently of the per-request rate limit applied at the API level.

Security properties

This pattern satisfies the following controls by construction:

Control	How it is satisfied
No direct Azure OpenAI access	Public access disabled on the AOAI resource; only the private endpoint in the AKS subnet can resolve and reach it
Token validated before AI call	`validate-jwt` runs first in the inbound chain — a malformed or expired token returns HTTP 401 before any backend traffic
Request quota enforced at the network layer	APIM `rate-limit-by-key` and `quota-by-key` policies; enforced before the request reaches Cognisafe
Full audit trail	Every LLM exchange (prompt, completion, model, latency, cost, safety score) is written to TimescaleDB by Cognisafe; tamper-evident by row-level append-only policy
AI-specific threat detection	Cognisafe runs LLM-as-judge scorers asynchronously on every captured exchange — jailbreak, PII, content safety, prompt injection
Data residency	Azure OpenAI private endpoint, AKS cluster, and Cognisafe all run in the same Azure region; no data transits region boundaries

OWASP LLM Top 10 controls addressed

OWASP LLM Risk	Mitigation in this pattern
LLM01 — Prompt Injection	Cognisafe’s `jailbreak_detection` scorer flags injections in every request; APIM blocks unauthenticated callers entirely
LLM02 — Insecure Output Handling	Completions logged and scored; alert rules can trigger on detected harmful content
LLM06 — Sensitive Information Disclosure	`pii_detection` scorer runs on both prompt and completion
LLM08 — Excessive Agency	Agent tagging + per-agent usage dashboards expose runaway autonomy
LLM09 — Overreliance	Cost attribution and request-volume alerts surface over-use before it becomes a risk

Operational guidance

Log Analytics integration. Enable APIM diagnostics and stream to a Log Analytics workspace. Correlate APIM request IDs with Cognisafe’s llm_requests.id for end-to-end tracing:

az monitor diagnostic-settings create \
  --name apim-to-law \
  --resource $(az apim show --name my-apim --resource-group my-rg --query id -o tsv) \
  --workspace $(az monitor log-analytics workspace show \
      --resource-group my-rg --workspace-name my-law --query id -o tsv) \
  --logs '[{"category":"GatewayLogs","enabled":true}]' \
  --metrics '[{"category":"AllMetrics","enabled":true}]'

Sentinel alerting. Forward Cognisafe webhook events (threat_detected, pii_detected) to a Sentinel custom log table via an Azure Logic App or Event Hub. This puts AI-layer threats into the same SIEM as your infrastructure alerts. Cost attribution. Cognisafe attributes token cost by project_id and agent_name. Use the /api/v1/costs?group_by=project_id endpoint to generate monthly chargeback reports per APIM product/subscription. Scaling. The Cognisafe proxy is stateless — scale the AKS deployment horizontally. The safety worker is independently scalable via a separate Deployment; each replica pulls from the same Redis queue. APIM scales within its SKU tier.

Failure modes

Failure	Behaviour	Recovery
Cognisafe proxy pod crash	AKS restarts the pod; APIM receives HTTP 503 and returns it to the client	Set `minReplicas: 2` in HPA; configure APIM retry policy for transient 503s
Cognisafe API unreachable	Proxy logs the request to a local buffer (configurable); scoring is delayed	Deploy API as a Deployment with 2+ replicas; use PodDisruptionBudget
Redis unavailable	Scoring jobs are not enqueued; requests are still forwarded and logged to Postgres	Redis Sentinel or Azure Cache for Redis with replication
Azure OpenAI private endpoint DNS failure	Proxy cannot resolve upstream; returns 502 to APIM	Verify private DNS zone link; deploy at least 2 private endpoint NICs across AZs
APIM VNet integration failure	Clients receive 503 from APIM	Deploy APIM in zone-redundant configuration (Premium SKU)

APIM’s validate-jwt policy caches the OpenID Connect metadata document. If you rotate your AAD app registration signing keys, allow up to 5 minutes for APIM to refresh the cache before forcing a pod restart.

Do not use the APIM Developer tier for production. It does not support VNet integration or zone redundancy, and has no SLA.

Getting Started

SDKs

LLM Providers

Self-hosted Models

Self-hosting

Integrations

Reference Architectures

Deployment Blueprints

Safety & Scoring

Azure OpenAI + APIM + Cognisafe

Architecture overview

Component responsibilities

Implementation

Security properties

OWASP LLM Top 10 controls addressed

Operational guidance

Failure modes

​Architecture overview

​Component responsibilities

​Implementation

​Security properties

​OWASP LLM Top 10 controls addressed

​Operational guidance

​Failure modes

Architecture overview

Component responsibilities

Implementation

Security properties

OWASP LLM Top 10 controls addressed

Operational guidance

Failure modes