> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# AKS Deployment

> Deploy Cognisafe on Azure Kubernetes Service with production-grade configuration

This blueprint walks through a full production deployment of Cognisafe on Azure Kubernetes Service (AKS). It covers namespace layout, Kubernetes manifests for all four components, secret management via Azure Key Vault CSI, TLS termination via cert-manager, autoscaling, and observability. The goal is a cluster you can hand to an on-call engineer and have them understand without tribal knowledge.

## Prerequisites

* AKS cluster running Kubernetes **1.28+** (`az aks get-versions` to verify availability in your region)
* `kubectl` configured against the target cluster (`az aks get-credentials --resource-group <rg> --name <cluster>`)
* Helm 3.12+
* Azure CLI 2.55+ with the `aks-preview` extension
* Azure Container Registry (ACR) attached to the cluster (`az aks update --attach-acr <acr-name>`)
* cert-manager 1.14+ installed in the cluster (see below)
* NGINX Ingress Controller installed (see below)
* Azure Key Vault CSI driver enabled on the cluster

<Note>
  The Key Vault CSI driver and cert-manager are not installed by default on AKS. Install them before applying these manifests. Installation commands are listed in the [Install cluster add-ons](#install-cluster-add-ons) section.
</Note>

## Architecture overview

```
Internet
    │
    ▼
Azure Load Balancer (public IP)
    │
    ▼
NGINX Ingress Controller  (ingress-nginx namespace)
    │  TLS terminated here; cert-manager issues Let's Encrypt certs
    ├──── proxy.cognisafe.yourcompany.com  ──▶  proxy Service (ClusterIP :8080)
    ├──── api.cognisafe.yourcompany.com    ──▶  api Service   (ClusterIP :8000)
    └──── cognisafe.yourcompany.com        ──▶  web Service   (ClusterIP :3000)

Inside cognisafe-system namespace:
  proxy  Deployment (2 replicas)
    └── forwards every LLM call → upstream LLM API (egress)
    └── POSTs /internal/log → api Service (ClusterIP)
  api    Deployment (2 replicas)
    └── reads/writes → PostgreSQL (Azure DB for PostgreSQL Flexible Server, private endpoint)
    └── pushes jobs  → Redis (Azure Cache for Redis, private endpoint)
  web    Deployment (2 replicas)
    └── calls api Service (ClusterIP) server-side
  safety-worker Deployment (3 replicas, HPA: 2–10)
    └── pulls from Redis queue, runs scoring, writes scores to PostgreSQL

Secrets sourced from Azure Key Vault via CSI SecretProviderClass.
ConfigMap holds non-secret env vars shared across deployments.
```

## Install cluster add-ons

<Steps>
  <Step title="Enable Key Vault CSI driver">
    ```bash theme={null}
    az aks enable-addons \
      --resource-group <rg> \
      --name <cluster> \
      --addons azure-keyvault-secrets-provider

    # Verify
    kubectl get pods -n kube-system -l app=secrets-store-csi-driver
    ```
  </Step>

  <Step title="Install cert-manager">
    ```bash theme={null}
    helm repo add jetstack https://charts.jetstack.io
    helm repo update

    helm install cert-manager jetstack/cert-manager \
      --namespace cert-manager \
      --create-namespace \
      --set installCRDs=true \
      --set global.leaderElection.namespace=cert-manager
    ```
  </Step>

  <Step title="Install NGINX Ingress Controller">
    ```bash theme={null}
    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
    helm repo update

    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.replicaCount=2 \
      --set controller.nodeSelector."kubernetes\.io/os"=linux \
      --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
    ```

    Wait for the LoadBalancer IP to be assigned:

    ```bash theme={null}
    kubectl get service ingress-nginx-controller -n ingress-nginx \
      --watch -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
    ```

    Point your DNS A records (`proxy.cognisafe.yourcompany.com`, `api.cognisafe.yourcompany.com`, `cognisafe.yourcompany.com`) at this IP before cert-manager can issue certificates.
  </Step>
</Steps>

## Namespace and ResourceQuota

```yaml theme={null}
# namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cognisafe-system
  labels:
    app.kubernetes.io/managed-by: kubectl
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: cognisafe-quota
  namespace: cognisafe-system
spec:
  hard:
    requests.cpu: "16"
    requests.memory: 32Gi
    limits.cpu: "32"
    limits.memory: 64Gi
    pods: "50"
    services: "20"
    persistentvolumeclaims: "10"
```

```bash theme={null}
kubectl apply -f namespace.yaml
```

## ConfigMap

Non-secret configuration shared across deployments. Update the values for your environment.

```yaml theme={null}
# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cognisafe-config
  namespace: cognisafe-system
data:
  # Proxy
  UPSTREAM_URL: "https://api.openai.com"
  API_BACKEND_URL: "http://api:8000"

  # API
  STRIPE_PRICE_PRO: "price_REPLACE_ME_PRO"
  STRIPE_PRICE_TEAM: "price_REPLACE_ME_TEAM"

  # Safety worker
  SCORER_MODEL: "gpt-4o-mini"

  # Web
  NEXT_PUBLIC_API_URL: "https://api.cognisafe.yourcompany.com"
  API_URL: "http://api:8000"
```

```bash theme={null}
kubectl apply -f configmap.yaml
```

## Secret management via Azure Key Vault CSI

Store all sensitive values in Azure Key Vault, then mount them as Kubernetes Secrets using the Secrets Store CSI driver. This avoids storing secrets in etcd or in manifests.

### Enable workload identity on the cluster

```bash theme={null}
az aks update \
  --resource-group <rg> \
  --name <cluster> \
  --enable-oidc-issuer \
  --enable-workload-identity

OIDC_ISSUER=$(az aks show \
  --resource-group <rg> \
  --name <cluster> \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)
```

### Create a managed identity and Key Vault access policy

```bash theme={null}
az identity create \
  --resource-group <rg> \
  --name cognisafe-kv-identity

IDENTITY_CLIENT_ID=$(az identity show \
  --resource-group <rg> \
  --name cognisafe-kv-identity \
  --query clientId -o tsv)

IDENTITY_OBJECT_ID=$(az identity show \
  --resource-group <rg> \
  --name cognisafe-kv-identity \
  --query principalId -o tsv)

az keyvault set-policy \
  --name <your-keyvault-name> \
  --object-id "$IDENTITY_OBJECT_ID" \
  --secret-permissions get list

# Federated credential so the ServiceAccount can use workload identity
az identity federated-credential create \
  --name cognisafe-federated \
  --identity-name cognisafe-kv-identity \
  --resource-group <rg> \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:cognisafe-system:cognisafe-sa" \
  --audiences api://AzureADTokenExchange
```

### ServiceAccount

```yaml theme={null}
# serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cognisafe-sa
  namespace: cognisafe-system
  annotations:
    azure.workload.identity/client-id: "<IDENTITY_CLIENT_ID>"
  labels:
    azure.workload.identity/use: "true"
```

### SecretProviderClass

```yaml theme={null}
# secretproviderclass.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: cognisafe-kv-secrets
  namespace: cognisafe-system
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "<IDENTITY_CLIENT_ID>"
    keyvaultName: "<your-keyvault-name>"
    tenantId: "<your-tenant-id>"
    objects: |
      array:
        - |
          objectName: cognisafe-postgres-url
          objectType: secret
          objectAlias: POSTGRES_URL
        - |
          objectName: cognisafe-redis-url
          objectType: secret
          objectAlias: REDIS_URL
        - |
          objectName: cognisafe-proxy-api-key
          objectType: secret
          objectAlias: PROXY_API_KEY
        - |
          objectName: cognisafe-stripe-secret-key
          objectType: secret
          objectAlias: STRIPE_SECRET_KEY
        - |
          objectName: cognisafe-stripe-webhook-secret
          objectType: secret
          objectAlias: STRIPE_WEBHOOK_SECRET
        - |
          objectName: cognisafe-internal-api-secret
          objectType: secret
          objectAlias: INTERNAL_API_SECRET
        - |
          objectName: cognisafe-openai-api-key
          objectType: secret
          objectAlias: OPENAI_API_KEY
        - |
          objectName: cognisafe-clerk-secret-key
          objectType: secret
          objectAlias: CLERK_SECRET_KEY
        - |
          objectName: cognisafe-clerk-publishable-key
          objectType: secret
          objectAlias: NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
  secretObjects:
    - secretName: cognisafe-secrets
      type: Opaque
      data:
        - objectName: POSTGRES_URL
          key: POSTGRES_URL
        - objectName: REDIS_URL
          key: REDIS_URL
        - objectName: PROXY_API_KEY
          key: PROXY_API_KEY
        - objectName: STRIPE_SECRET_KEY
          key: STRIPE_SECRET_KEY
        - objectName: STRIPE_WEBHOOK_SECRET
          key: STRIPE_WEBHOOK_SECRET
        - objectName: INTERNAL_API_SECRET
          key: INTERNAL_API_SECRET
        - objectName: OPENAI_API_KEY
          key: OPENAI_API_KEY
        - objectName: CLERK_SECRET_KEY
          key: CLERK_SECRET_KEY
        - objectName: NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
          key: NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
```

<Warning>
  The Kubernetes Secret (`cognisafe-secrets`) is only created after the first pod mounts the CSI volume. You must deploy at least one pod (any of the four components) before the Secret object exists. The manifests below include the CSI volume mount in every Deployment precisely so the Secret is hydrated on first rollout.
</Warning>

## Proxy Deployment

```yaml theme={null}
# deploy-proxy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: proxy
  namespace: cognisafe-system
  labels:
    app: proxy
    app.kubernetes.io/component: proxy
    app.kubernetes.io/part-of: cognisafe
spec:
  replicas: 2
  selector:
    matchLabels:
      app: proxy
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: proxy
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: cognisafe-sa
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: proxy
      containers:
        - name: proxy
          image: <acr-name>.azurecr.io/cognisafe/proxy:latest
          ports:
            - containerPort: 8080
              name: http
          env:
            - name: UPSTREAM_URL
              valueFrom:
                configMapKeyRef:
                  name: cognisafe-config
                  key: UPSTREAM_URL
            - name: API_BACKEND_URL
              valueFrom:
                configMapKeyRef:
                  name: cognisafe-config
                  key: API_BACKEND_URL
            - name: PROXY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: PROXY_API_KEY
          resources:
            requests:
              cpu: 250m
              memory: 128Mi
            limits:
              cpu: "1"
              memory: 256Mi
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health
              port: 8080
            initialDelaySeconds: 3
            periodSeconds: 5
            failureThreshold: 2
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: cognisafe-kv-secrets
---
apiVersion: v1
kind: Service
metadata:
  name: proxy
  namespace: cognisafe-system
spec:
  selector:
    app: proxy
  ports:
    - port: 8080
      targetPort: 8080
      name: http
  type: ClusterIP
```

## API Deployment

```yaml theme={null}
# deploy-api.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: cognisafe-system
  labels:
    app: api
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: cognisafe
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: cognisafe-sa
      initContainers:
        - name: migrate
          image: <acr-name>.azurecr.io/cognisafe/api:latest
          command: ["alembic", "upgrade", "head"]
          env:
            - name: POSTGRES_URL
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: POSTGRES_URL
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/secrets-store
              readOnly: true
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: api
      containers:
        - name: api
          image: <acr-name>.azurecr.io/cognisafe/api:latest
          ports:
            - containerPort: 8000
              name: http
          envFrom:
            - configMapRef:
                name: cognisafe-config
            - secretRef:
                name: cognisafe-secrets
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 1Gi
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 10
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 8
            periodSeconds: 5
            failureThreshold: 2
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: cognisafe-kv-secrets
---
apiVersion: v1
kind: Service
metadata:
  name: api
  namespace: cognisafe-system
spec:
  selector:
    app: api
  ports:
    - port: 8000
      targetPort: 8000
      name: http
  type: ClusterIP
```

<Note>
  The `migrate` init container runs `alembic upgrade head` on every pod start. Because Alembic migrations are idempotent, this is safe with multiple replicas — only the first pod to acquire the advisory lock will run pending migrations; the rest will no-op. This avoids the need for a separate migration Job.
</Note>

## Web Deployment

```yaml theme={null}
# deploy-web.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: cognisafe-system
  labels:
    app: web
    app.kubernetes.io/component: web
    app.kubernetes.io/part-of: cognisafe
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: web
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: cognisafe-sa
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: web
      containers:
        - name: web
          image: <acr-name>.azurecr.io/cognisafe/web:latest
          ports:
            - containerPort: 3000
              name: http
          env:
            - name: NEXT_PUBLIC_API_URL
              valueFrom:
                configMapKeyRef:
                  name: cognisafe-config
                  key: NEXT_PUBLIC_API_URL
            - name: API_URL
              valueFrom:
                configMapKeyRef:
                  name: cognisafe-config
                  key: API_URL
            - name: NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY
            - name: CLERK_SECRET_KEY
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: CLERK_SECRET_KEY
            - name: INTERNAL_API_SECRET
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: INTERNAL_API_SECRET
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: "1"
              memory: 512Mi
          livenessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 15
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /api/health
              port: 3000
            initialDelaySeconds: 10
            periodSeconds: 5
            failureThreshold: 2
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: cognisafe-kv-secrets
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: cognisafe-system
spec:
  selector:
    app: web
  ports:
    - port: 3000
      targetPort: 3000
      name: http
  type: ClusterIP
```

## Safety Worker Deployment

The safety worker has no ingress — it only pulls from Redis and writes to PostgreSQL.

```yaml theme={null}
# deploy-safety-worker.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: safety-worker
  namespace: cognisafe-system
  labels:
    app: safety-worker
    app.kubernetes.io/component: safety-worker
    app.kubernetes.io/part-of: cognisafe
spec:
  replicas: 3
  selector:
    matchLabels:
      app: safety-worker
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: safety-worker
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: cognisafe-sa
      containers:
        - name: safety-worker
          image: <acr-name>.azurecr.io/cognisafe/api:latest
          command: ["python", "workers/safety_scorer.py"]
          env:
            - name: POSTGRES_URL
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: POSTGRES_URL
            - name: REDIS_URL
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: REDIS_URL
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: cognisafe-secrets
                  key: OPENAI_API_KEY
            - name: SCORER_MODEL
              valueFrom:
                configMapKeyRef:
                  name: cognisafe-config
                  key: SCORER_MODEL
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: "2"
              memory: 2Gi
          volumeMounts:
            - name: secrets-store
              mountPath: /mnt/secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: cognisafe-kv-secrets
```

## HorizontalPodAutoscaler for safety-worker

Scale the worker pool based on CPU, since safety scoring is CPU/network-bound during LLM calls.

```yaml theme={null}
# hpa-safety-worker.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: safety-worker-hpa
  namespace: cognisafe-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: safety-worker
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120
```

<Tip>
  If your Redis queue depth is a more meaningful signal than CPU (e.g., burst scoring jobs after a traffic spike), consider adding a custom metric via KEDA (`ScaledObject` targeting the `safety_score_jobs` list length). KEDA's Azure Cache for Redis scaler handles this directly.
</Tip>

## TLS Ingress with cert-manager

### ClusterIssuer (Let's Encrypt production)

```yaml theme={null}
# clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@yourcompany.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx
```

### Ingress

```yaml theme={null}
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cognisafe-ingress
  namespace: cognisafe-system
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    # Force HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    # Rate limiting at ingress layer (supplement APIM if present)
    nginx.ingress.kubernetes.io/limit-rps: "200"
spec:
  tls:
    - hosts:
        - cognisafe.yourcompany.com
        - api.cognisafe.yourcompany.com
        - proxy.cognisafe.yourcompany.com
      secretName: cognisafe-tls
  rules:
    - host: cognisafe.yourcompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 3000
    - host: api.cognisafe.yourcompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8000
    - host: proxy.cognisafe.yourcompany.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: proxy
                port:
                  number: 8080
```

## Apply all manifests

```bash theme={null}
kubectl apply -f namespace.yaml
kubectl apply -f serviceaccount.yaml
kubectl apply -f secretproviderclass.yaml
kubectl apply -f configmap.yaml
kubectl apply -f deploy-proxy.yaml
kubectl apply -f deploy-api.yaml
kubectl apply -f deploy-web.yaml
kubectl apply -f deploy-safety-worker.yaml
kubectl apply -f hpa-safety-worker.yaml
kubectl apply -f clusterissuer.yaml
kubectl apply -f ingress.yaml

# Verify rollout
kubectl rollout status deployment/proxy -n cognisafe-system
kubectl rollout status deployment/api -n cognisafe-system
kubectl rollout status deployment/web -n cognisafe-system
kubectl rollout status deployment/safety-worker -n cognisafe-system
```

## Node pool recommendations

| Pool     | VM SKU                              | Taints/Labels                        | Use                                          |
| -------- | ----------------------------------- | ------------------------------------ | -------------------------------------------- |
| `system` | Standard\_D4s\_v5 (2 nodes, min)    | `CriticalAddonsOnly=true:NoSchedule` | kube-system, ingress-nginx, cert-manager     |
| `app`    | Standard\_D4s\_v5 (2–8 nodes)       | —                                    | proxy, api, web workloads                    |
| `worker` | Standard\_D8s\_v5 spot (0–10 nodes) | `workload=scoring:NoSchedule`        | safety-worker (CPU-intensive, spot-tolerant) |

```bash theme={null}
# Create the spot worker pool
az aks nodepool add \
  --resource-group <rg> \
  --cluster-name <cluster> \
  --name workerpool \
  --node-count 2 \
  --min-count 0 \
  --max-count 10 \
  --enable-cluster-autoscaler \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --node-vm-size Standard_D8s_v5 \
  --node-taints "workload=scoring:NoSchedule" \
  --labels workload=scoring
```

Add the corresponding toleration to the safety-worker Deployment spec:

```yaml theme={null}
tolerations:
  - key: "workload"
    operator: "Equal"
    value: "scoring"
    effect: "NoSchedule"
nodeSelector:
  workload: scoring
```

<Warning>
  Spot nodes can be evicted with 30 seconds notice. The safety worker must handle `SIGTERM` gracefully — finish the current scoring job, then exit. The safety worker loop should catch `KeyboardInterrupt`/`SIGTERM` and drain the in-flight job before stopping. Ensure `terminationGracePeriodSeconds` is set to at least 60 seconds in the pod spec.
</Warning>

## Monitoring: Azure Monitor + Container Insights

```bash theme={null}
az aks enable-addons \
  --resource-group <rg> \
  --name <cluster> \
  --addons monitoring \
  --workspace-resource-id /subscriptions/<sub>/resourcegroups/<rg>/providers/microsoft.operationalinsights/workspaces/<workspace>
```

After enabling Container Insights, navigate to the AKS cluster in the Azure Portal → **Monitoring** → **Insights** to see per-node CPU/memory, per-pod logs, and live event streams.

Key metrics to alert on:

| Metric                                  | Alert threshold | Notes                                       |
| --------------------------------------- | --------------- | ------------------------------------------- |
| `safety-worker` CPU utilization         | > 80% for 5 min | Indicates queue backlog; HPA should kick in |
| `api` p99 latency                       | > 2s            | Check PostgreSQL connection pool            |
| Pod restart count                       | > 3 in 10 min   | OOMKill or crash loop                       |
| Redis queue depth (`safety_score_jobs`) | > 10,000        | Workers not keeping up                      |

```bash theme={null}
# Stream live logs from all proxy pods
kubectl logs -n cognisafe-system -l app=proxy --follow --prefix

# Check HPA status
kubectl get hpa -n cognisafe-system

# Exec into a worker pod to inspect Redis queue depth
kubectl exec -n cognisafe-system \
  $(kubectl get pod -n cognisafe-system -l app=safety-worker -o name | head -1) \
  -- python -c "import redis, os; r=redis.from_url(os.environ['REDIS_URL']); print(r.llen('safety_score_jobs'))"
```