> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# AKS Production Reference Architecture

> Production-grade Cognisafe deployment on Azure Kubernetes Service with HA, auto-scaling, and full observability

This reference architecture describes the canonical production deployment pattern for Cognisafe on Azure Kubernetes Service. It is the pattern a well-run platform team would build: zone-redundant, GitOps-driven, secret-rotation-safe, and observable from day one. Use it as an authoritative baseline, not a starting point you have to work backwards from.

This document is intentionally opinionated. Where there are trade-offs, a specific choice is made and the rationale is given.

## Architecture overview

```text theme={null}
                        Internet
                            │
                            ▼
               Azure Load Balancer (public IP, zone-redundant)
                            │
                            ▼
          NGINX Ingress Controller  (ingress-nginx namespace, 2 replicas)
          TLS terminated here · cert-manager issues Let's Encrypt wildcard
                            │
          ┌─────────────────┼──────────────────────┐
          ▼                 ▼                       ▼
proxy.cognisafe.        api.cognisafe.        app.cognisafe.
yourco.com              yourco.com            yourco.com
          │                 │                       │
          ▼                 ▼                       ▼
    proxy Service     api Service             web Service
    (ClusterIP)       (ClusterIP)             (ClusterIP)
          │                 │                       │
          │           ┌─────┤                       │
          │           ▼     ▼                       │
          │     PostgreSQL  Redis            api Service
          │     Flexible    Azure Cache      (server-side
          │     Server      for Redis        Next.js calls)
          │     (private    (private
          │     endpoint)   endpoint)
          │           ▲
          │           │
    safety-worker (3–20 replicas, HPA)
    pulls from Redis · runs scoring · writes scores to PostgreSQL

Azure Key Vault  ←→  CSI SecretProviderClass  ←→  all Deployments
Azure Container Registry  ←→  AKS kubelet identity (pull)
Azure Monitor + Container Insights  ←→  AKS diagnostics
Prometheus + Grafana (kube-prometheus-stack)  ←→  custom metrics

Node pools:
  system pool  (3× Standard_D4s_v3, across 3 zones)
  user pool    (3–10× Standard_D4s_v3, spot, across 3 zones, HPA-driven)
  [optional]   GPU pool (1–3× Standard_NC4as_T4_v3) for local LLM safety scoring
```

## Why this architecture

### Zone redundancy at every layer

A single AZ failure must not cause an outage. This requires: a zone-redundant load balancer, NGINX Ingress replicas across zones, application pods spread across zones via `topologySpreadConstraints`, a zone-redundant PostgreSQL Flexible Server, and an Azure Cache for Redis Standard C1 (which includes a replica in a separate fault domain). The AKS control plane is zone-redundant by default when you set `--zones 1 2 3` on the node pools.

### Spot instances for the safety worker only

The safety worker is the only component suitable for spot instances. It is stateless, processes jobs from a queue, and handles `SIGTERM` gracefully. The proxy, API, and web services are latency-sensitive and must run on regular-priority nodes to avoid the 30-second eviction notice.

### GitOps, not imperative kubectl

All cluster state is declared in a Helm chart checked into a `deploy/` directory. The CI pipeline runs `helm upgrade --install` on every merge to `main`. There is no "apply some YAML files" step in production — that creates configuration drift. The Helm chart is the single source of truth.

***

## Node pool design

| Pool             | SKU                     | Count         | Priority | Zones   | Taints                               | Workloads                                            |
| ---------------- | ----------------------- | ------------- | -------- | ------- | ------------------------------------ | ---------------------------------------------------- |
| `system`         | Standard\_D4s\_v3       | 3 (fixed)     | Regular  | 1, 2, 3 | `CriticalAddonsOnly=true:NoSchedule` | kube-system, ingress-nginx, cert-manager, monitoring |
| `user`           | Standard\_D4s\_v3       | 3–10 (CA)     | Regular  | 1, 2, 3 | —                                    | proxy, api, web                                      |
| `worker`         | Standard\_D4s\_v3       | 0–10 (CA+HPA) | Spot     | 1, 2, 3 | `workload=scoring:NoSchedule`        | safety-worker                                        |
| `gpu` (optional) | Standard\_NC4as\_T4\_v3 | 1–3           | Regular  | 1       | `hardware=gpu:NoSchedule`            | local LLM safety scoring with GPU acceleration       |

```bash theme={null}
# System pool — created at cluster provisioning time
az aks create \
  --resource-group rg-cognisafe-prod \
  --name aks-cognisafe-prod \
  --location uksouth \
  --kubernetes-version 1.30 \
  --node-count 3 \
  --node-vm-size Standard_D4s_v3 \
  --zones 1 2 3 \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 3 \
  --nodepool-name system \
  --nodepool-taints "CriticalAddonsOnly=true:NoSchedule" \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --attach-acr acr-cognisafe-prod \
  --enable-addons azure-keyvault-secrets-provider,monitoring \
  --workspace-resource-id /subscriptions/<sub>/resourcegroups/rg-cognisafe-prod/providers/microsoft.operationalinsights/workspaces/law-cognisafe-prod \
  --network-plugin azure \
  --network-policy azure \
  --generate-ssh-keys

# User pool (regular priority, zone-spread, autoscaler 3–10)
az aks nodepool add \
  --resource-group rg-cognisafe-prod \
  --cluster-name aks-cognisafe-prod \
  --name userpool \
  --node-count 3 \
  --min-count 3 \
  --max-count 10 \
  --enable-cluster-autoscaler \
  --node-vm-size Standard_D4s_v3 \
  --zones 1 2 3

# Worker pool (spot, autoscaler 0–10)
az aks nodepool add \
  --resource-group rg-cognisafe-prod \
  --cluster-name aks-cognisafe-prod \
  --name workerpool \
  --node-count 0 \
  --min-count 0 \
  --max-count 10 \
  --enable-cluster-autoscaler \
  --priority Spot \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --node-vm-size Standard_D4s_v3 \
  --zones 1 2 3 \
  --node-taints "workload=scoring:NoSchedule" \
  --labels workload=scoring
```

<Warning>
  Spot nodes receive 30 seconds notice before eviction via a scheduled event. The safety worker must catch `SIGTERM` and finish or re-queue its in-flight job before exiting. Set `terminationGracePeriodSeconds: 60` on the safety-worker pod spec. If a worker exits mid-job, the Redis job will remain in the queue and be picked up by another replica.
</Warning>

***

## Namespace design

Three namespaces keep platform, data, and monitoring concerns cleanly separated. RBAC policies are applied per namespace so the monitoring namespace cannot write to platform secrets.

```yaml theme={null}
# namespaces.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: cognisafe-system      # proxy, api, web, safety-worker
  labels:
    app.kubernetes.io/managed-by: helm
---
apiVersion: v1
kind: Namespace
metadata:
  name: cognisafe-data        # only used if running PostgreSQL/Redis in-cluster (not recommended for prod)
  labels:
    app.kubernetes.io/managed-by: helm
---
apiVersion: v1
kind: Namespace
metadata:
  name: cognisafe-monitoring  # kube-prometheus-stack, Grafana, alertmanager
  labels:
    app.kubernetes.io/managed-by: helm
```

<Note>
  In the reference architecture, PostgreSQL and Redis run as fully managed Azure services (PostgreSQL Flexible Server and Azure Cache for Redis), not in-cluster. The `cognisafe-data` namespace is only relevant if you choose to run in-cluster databases for cost or latency reasons — this is not recommended for production.
</Note>

***

## High availability configuration

### Proxy (2+ replicas)

```yaml theme={null}
# Excerpt from Helm chart: values-prod.yaml
proxy:
  replicaCount: 2
  podDisruptionBudget:
    enabled: true
    minAvailable: 1
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: proxy
  resources:
    requests:
      cpu: 250m
      memory: 128Mi
    limits:
      cpu: "1"
      memory: 256Mi
```

### API (2+ replicas)

```yaml theme={null}
api:
  replicaCount: 2
  podDisruptionBudget:
    enabled: true
    minAvailable: 1
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: api
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "2"
      memory: 1Gi
```

### Safety worker (3 replicas minimum, HPA up to 20)

```yaml theme={null}
safetyWorker:
  replicaCount: 3
  tolerations:
    - key: workload
      operator: Equal
      value: scoring
      effect: NoSchedule
  nodeSelector:
    workload: scoring
  terminationGracePeriodSeconds: 60
  resources:
    requests:
      cpu: 500m
      memory: 512Mi
    limits:
      cpu: "2"
      memory: 2Gi
```

```yaml theme={null}
# hpa-safety-worker.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: safety-worker-hpa
  namespace: cognisafe-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: safety-worker
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Pods
          value: 3
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 120
```

<Tip>
  If queue depth is a better signal than CPU for your workload (common during batch red team runs), deploy KEDA with the Azure Cache for Redis scaler targeting the `safety_score_jobs` list length. Set `listLength: 100` as the trigger threshold — this scales one worker replica per 100 queued jobs.
</Tip>

### PostgreSQL Flexible Server (zone-redundant)

```bash theme={null}
az postgres flexible-server create \
  --resource-group rg-cognisafe-prod \
  --name pg-cognisafe-prod \
  --location uksouth \
  --sku-name Standard_D4s_v3 \
  --tier GeneralPurpose \
  --storage-size 128 \
  --version 15 \
  --high-availability ZoneRedundant \
  --zone 1 \
  --standby-zone 2 \
  --active-directory-auth Enabled \
  --password-auth Enabled \
  --backup-retention 30 \
  --geo-redundant-backup Enabled
```

### Azure Cache for Redis (Standard C1 with replica)

```bash theme={null}
az redis create \
  --resource-group rg-cognisafe-prod \
  --name redis-cognisafe-prod \
  --location uksouth \
  --sku Standard \
  --vm-size C1 \
  --enable-non-ssl-port false \
  --minimum-tls-version 1.2
```

***

## Secret management

All secrets live in Azure Key Vault and are injected into pods at runtime via the Secrets Store CSI driver. Secrets are never stored in etcd, never in manifests, and never in environment files committed to source control. Key rotation happens in Key Vault; the CSI driver picks up new values on the next pod restart without a redeployment.

```bash theme={null}
# Provision the Key Vault
az keyvault create \
  --resource-group rg-cognisafe-prod \
  --name kv-cognisafe-prod \
  --location uksouth \
  --sku standard \
  --enable-purge-protection true \
  --retention-days 90

# Store each secret (repeat for all vars)
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-postgres-url   --value "postgresql+asyncpg://..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-redis-url      --value "rediss://..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-proxy-api-key  --value "<random-256-bit>"
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-stripe-secret  --value "sk_live_..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-stripe-webhook --value "whsec_..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-internal-api   --value "<random-256-bit>"
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-openai-key     --value "sk-..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-clerk-secret   --value "sk_live_..."
az keyvault secret set --vault-name kv-cognisafe-prod --name cognisafe-clerk-pub-key  --value "pk_live_..."

# Create workload identity and federated credential
az identity create \
  --resource-group rg-cognisafe-prod \
  --name id-cognisafe-kv

IDENTITY_CLIENT_ID=$(az identity show \
  --resource-group rg-cognisafe-prod \
  --name id-cognisafe-kv \
  --query clientId -o tsv)

IDENTITY_OBJECT_ID=$(az identity show \
  --resource-group rg-cognisafe-prod \
  --name id-cognisafe-kv \
  --query principalId -o tsv)

OIDC_ISSUER=$(az aks show \
  --resource-group rg-cognisafe-prod \
  --name aks-cognisafe-prod \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

az keyvault set-policy \
  --name kv-cognisafe-prod \
  --object-id "$IDENTITY_OBJECT_ID" \
  --secret-permissions get list

az identity federated-credential create \
  --name cognisafe-federated \
  --identity-name id-cognisafe-kv \
  --resource-group rg-cognisafe-prod \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:cognisafe-system:cognisafe-sa" \
  --audiences api://AzureADTokenExchange
```

***

## Certificate management

cert-manager issues and renews a wildcard TLS certificate for `*.cognisafe.yourco.com` using DNS-01 challenge against Azure DNS. This avoids exposing each individual subdomain for HTTP-01 validation.

```yaml theme={null}
# clusterissuer-letsencrypt-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@yourco.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          azureDNS:
            subscriptionID: <subscription-id>
            resourceGroupName: rg-cognisafe-prod
            hostedZoneName: yourco.com
            environment: AzurePublicCloud
            managedIdentity:
              clientID: <identity-client-id>
```

```yaml theme={null}
# certificate-wildcard.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cognisafe-wildcard-tls
  namespace: cognisafe-system
spec:
  secretName: cognisafe-wildcard-tls
  dnsNames:
    - "*.cognisafe.yourco.com"
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
```

***

## Ingress rules

```yaml theme={null}
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: cognisafe-ingress
  namespace: cognisafe-system
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/limit-rps: "500"
    nginx.ingress.kubernetes.io/limit-connections: "100"
spec:
  tls:
    - hosts:
        - proxy.cognisafe.yourco.com
        - api.cognisafe.yourco.com
        - app.cognisafe.yourco.com
      secretName: cognisafe-wildcard-tls
  rules:
    - host: proxy.cognisafe.yourco.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: proxy
                port:
                  number: 8080
    - host: api.cognisafe.yourco.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 8000
    - host: app.cognisafe.yourco.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 3000
```

***

## Deployment pipeline

The deployment pipeline runs on GitHub Actions. The pipeline builds, tests, pushes images to ACR, and performs a Helm upgrade. The Alembic migration runs as a Kubernetes Job before the new API pods are promoted, not as an init container in production (init containers on every replica are acceptable in staging but create race conditions at scale).

```yaml theme={null}
# .github/workflows/deploy.yml
name: Deploy to AKS

on:
  push:
    branches: [main]

env:
  ACR_NAME: acrcognisafeprod
  AKS_CLUSTER: aks-cognisafe-prod
  AKS_RG: rg-cognisafe-prod

jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Build and push images
        run: |
          az acr login --name $ACR_NAME

          # API + worker (same image, different CMD)
          docker build -t $ACR_NAME.azurecr.io/cognisafe/api:$GITHUB_SHA api/
          docker push $ACR_NAME.azurecr.io/cognisafe/api:$GITHUB_SHA

          # Proxy
          docker build -t $ACR_NAME.azurecr.io/cognisafe/proxy:$GITHUB_SHA proxy/
          docker push $ACR_NAME.azurecr.io/cognisafe/proxy:$GITHUB_SHA

          # Web
          docker build -t $ACR_NAME.azurecr.io/cognisafe/web:$GITHUB_SHA web/
          docker push $ACR_NAME.azurecr.io/cognisafe/web:$GITHUB_SHA

  migrate:
    needs: build-and-push
    runs-on: ubuntu-latest
    steps:
      - uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Get AKS credentials
        run: az aks get-credentials --resource-group $AKS_RG --name $AKS_CLUSTER

      - name: Run Alembic migration Job
        run: |
          kubectl create job alembic-migrate-$GITHUB_SHA \
            --from=cronjob/alembic-migrate \
            -n cognisafe-system \
            --image=$ACR_NAME.azurecr.io/cognisafe/api:$GITHUB_SHA

          kubectl wait job/alembic-migrate-$GITHUB_SHA \
            -n cognisafe-system \
            --for=condition=complete \
            --timeout=300s

  deploy:
    needs: migrate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: azure/login@v2
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Get AKS credentials
        run: az aks get-credentials --resource-group $AKS_RG --name $AKS_CLUSTER

      - name: Helm upgrade
        run: |
          helm upgrade --install cognisafe deploy/helm/cognisafe \
            --namespace cognisafe-system \
            --create-namespace \
            --values deploy/helm/cognisafe/values-prod.yaml \
            --set global.image.tag=$GITHUB_SHA \
            --atomic \
            --timeout 10m \
            --wait
```

<Tip>
  `--atomic` on the Helm upgrade automatically rolls back to the previous release if any pod fails its readiness probe within the timeout window. This prevents a bad deploy from leaving the cluster in a partially-upgraded state.
</Tip>

***

## Observability

### Azure Monitor + Container Insights

Container Insights is enabled at cluster creation. It provides per-node and per-pod CPU, memory, network, and disk metrics in Azure Monitor, plus log forwarding to a Log Analytics workspace. Navigate to **AKS → Monitoring → Insights** in the Azure Portal.

### kube-prometheus-stack

```bash theme={null}
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace cognisafe-monitoring \
  --create-namespace \
  --set grafana.adminPassword="<strong-random-password>" \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=managed-premium \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi
```

### Key alerts

| Alert             | Threshold       | Severity | Action                                             |
| ----------------- | --------------- | -------- | -------------------------------------------------- |
| safety-worker CPU | > 80% for 5 min | Warning  | Check Redis queue depth; HPA should have responded |
| safety-worker CPU | > 95% for 2 min | Critical | Manual scale or investigate                        |
| API p99 latency   | > 2 s           | Warning  | Check PostgreSQL connection pool, slow queries     |
| Pod restart count | > 3 in 10 min   | Critical | OOMKill or crash loop; `kubectl describe pod`      |
| Redis queue depth | > 10,000        | Warning  | Workers not keeping up; scale manually             |
| Redis queue depth | > 50,000        | Critical | Workers saturated; investigate backpressure        |
| PVC used          | > 80%           | Warning  | Prometheus storage filling; expand PVC             |

### Custom Cognisafe Prometheus metrics

Expose these metrics from the API service by adding a `/metrics` endpoint (FastAPI + `prometheus-fastapi-instrumentator`):

```python theme={null}
from prometheus_fastapi_instrumentator import Instrumentator

Instrumentator().instrument(app).expose(app, endpoint="/metrics")
```

Then define a `ServiceMonitor` so Prometheus scrapes the API pods:

```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cognisafe-api
  namespace: cognisafe-monitoring
  labels:
    release: kube-prometheus-stack
spec:
  namespaceSelector:
    matchNames:
      - cognisafe-system
  selector:
    matchLabels:
      app: api
  endpoints:
    - port: http
      path: /metrics
      interval: 30s
```

***

## Backup and restore

### Daily pg\_dump to Azure Blob Storage

```bash theme={null}
# Create storage account and container
az storage account create \
  --resource-group rg-cognisafe-prod \
  --name stcognisafebackup \
  --sku Standard_GRS \
  --kind StorageV2

az storage container create \
  --account-name stcognisafebackup \
  --name postgres-backups
```

```yaml theme={null}
# CronJob: runs pg_dump daily at 02:00 UTC
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
  namespace: cognisafe-system
spec:
  schedule: "0 2 * * *"
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cognisafe-sa
          restartPolicy: OnFailure
          containers:
            - name: pg-dump
              image: postgres:15
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: cognisafe-secrets
                      key: POSTGRES_PASSWORD
              command:
                - /bin/bash
                - -c
                - |
                  TIMESTAMP=$(date +%Y%m%d_%H%M%S)
                  pg_dump -h $PGHOST -U $PGUSER -d cognisafe -Fc \
                    | az storage blob upload \
                        --account-name stcognisafebackup \
                        --container postgres-backups \
                        --name "cognisafe_${TIMESTAMP}.dump" \
                        --auth-mode login \
                        --data @-
```

Retention: configure a lifecycle management policy on the storage account to delete blobs older than 30 days.

### Tested restore procedure

<Steps>
  <Step title="List available backups">
    ```bash theme={null}
    az storage blob list \
      --account-name stcognisafebackup \
      --container postgres-backups \
      --output table \
      --auth-mode login
    ```
  </Step>

  <Step title="Download the target dump">
    ```bash theme={null}
    az storage blob download \
      --account-name stcognisafebackup \
      --container postgres-backups \
      --name "cognisafe_20260501_020012.dump" \
      --file /tmp/cognisafe.dump \
      --auth-mode login
    ```
  </Step>

  <Step title="Restore to a recovery PostgreSQL instance">
    ```bash theme={null}
    # Point at the recovery server, NOT production
    pg_restore -h pg-cognisafe-recovery.postgres.database.azure.com \
      -U adminuser \
      -d cognisafe \
      -Fc /tmp/cognisafe.dump \
      --no-owner \
      --no-privileges \
      --verbose
    ```
  </Step>

  <Step title="Verify row counts match expectation">
    ```sql theme={null}
    SELECT schemaname, tablename, n_live_tup
    FROM pg_stat_user_tables
    ORDER BY n_live_tup DESC;
    ```
  </Step>

  <Step title="Promote the recovery instance">
    Update the `POSTGRES_URL` secret in Key Vault to point at the recovery server. Restart all pods to pick up the new connection string.
  </Step>
</Steps>

<Warning>
  Run this procedure quarterly in a dedicated recovery subscription, not against production. An untested backup is not a backup.
</Warning>

***

## Disaster recovery

| Metric                         | Target   | Mechanism                                                                                                                                                           |
| ------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| RTO (Recovery Time Objective)  | 1 hour   | Zone failover is automatic via PostgreSQL zone-redundant HA. Region failover requires manual promotion of geo-replica and AKS redeployment in the secondary region. |
| RPO (Recovery Point Objective) | 24 hours | Daily pg\_dump backup. PostgreSQL Flexible Server also maintains continuous WAL backups (PITR up to 30 days).                                                       |

### Failover procedure (full region loss)

<Steps>
  <Step title="Promote PostgreSQL geo-replica">
    ```bash theme={null}
    az postgres flexible-server replica promote \
      --resource-group rg-cognisafe-dr \
      --name pg-cognisafe-dr
    ```
  </Step>

  <Step title="Deploy AKS cluster in secondary region">
    Run the same Terraform/Bicep that provisions the primary cluster, targeting the secondary region (e.g., `northeurope`).
  </Step>

  <Step title="Update Key Vault secrets">
    Update `cognisafe-postgres-url` and `cognisafe-redis-url` in the secondary region's Key Vault to point at the promoted replicas.
  </Step>

  <Step title="Deploy application via Helm">
    ```bash theme={null}
    helm upgrade --install cognisafe deploy/helm/cognisafe \
      --namespace cognisafe-system \
      --values deploy/helm/cognisafe/values-prod.yaml \
      --set global.image.tag=<last-known-good-tag>
    ```
  </Step>

  <Step title="Reroute DNS">
    Update Azure DNS or Traffic Manager to point `*.cognisafe.yourco.com` at the secondary region's load balancer IP.
  </Step>
</Steps>

***

## Cost estimate

All prices are approximate UK South list pricing as of mid-2026. Actual costs depend on reserved instance discounts, egress, and storage.

| Tier                  | AKS nodes                               | PostgreSQL                             | Redis       | Total estimate |
| --------------------- | --------------------------------------- | -------------------------------------- | ----------- | -------------- |
| Small (\< 1M req/mo)  | 3× D4s\_v3 regular + 2× D4s\_v3 spot    | Standard\_D2s\_v3 (2 vCores)           | Standard C0 | \~£800/mo      |
| Medium (1–10M req/mo) | 3× D4s\_v3 regular + 3–5× D4s\_v3 spot  | Standard\_D4s\_v3 (4 vCores) + standby | Standard C1 | \~£1,800/mo    |
| Large (10M+ req/mo)   | 3× D8s\_v3 regular + 5–10× D4s\_v3 spot | Standard\_D8s\_v3 (8 vCores) + standby | Standard C2 | \~£4,500/mo    |

<Tip>
  Purchase 1-year reserved instances for the system and user node pools. Spot instances on the worker pool already provide 60–80% discount. Reserved instances on the regular pools typically save an additional 30–40% over pay-as-you-go.
</Tip>

***

## Security hardening checklist

* [ ] Network Policy enabled (`--network-policy azure`) — deny all ingress/egress by default, allow only required pod-to-pod paths
* [ ] Private endpoint for PostgreSQL Flexible Server — no public network access
* [ ] Private endpoint for Azure Cache for Redis — TLS 1.2 minimum, no non-SSL port
* [ ] Azure Key Vault firewall — allow only AKS subnet CIDR and deployment pipeline IPs
* [ ] ACR network rules — allow only AKS kubelet subnet
* [ ] Pod Security Standards — set `pod-security.kubernetes.io/enforce: restricted` on `cognisafe-system` namespace
* [ ] Container images built on `distroless` base, non-root UID, read-only root filesystem
* [ ] Dependabot or Renovate enabled on the repository for automated dependency updates
* [ ] Microsoft Defender for Containers enabled on the AKS cluster
* [ ] Audit logging enabled on the AKS control plane, forwarded to Log Analytics
