> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# Grafana & Prometheus Monitoring

> Expose Cognisafe metrics to Prometheus and build Grafana dashboards

## Metrics Endpoint

Cognisafe's FastAPI backend (`api/`) exposes a `/metrics` endpoint in Prometheus text format using [`prometheus-fastapi-instrumentator`](https://github.com/trallnag/prometheus-fastapi-instrumentator).

**Installation** (already included in the official Docker image — this is for self-builders):

```bash theme={null}
pip install prometheus-fastapi-instrumentator==6.1.0
```

**Wiring in `api/main.py`:**

```python theme={null}
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

Instrumentator(
    should_group_status_codes=False,
    should_ignore_untemplated=True,
    should_respect_env_var=False,
    excluded_handlers=["/healthz", "/readyz"],
).instrument(app).expose(app, endpoint="/metrics", include_in_schema=False)
```

The safety worker additionally exports Redis queue depth and processing duration metrics via a lightweight `prometheus_client` exposition loop on port `9091`.

***

## Key Metrics

All Cognisafe metrics use the `cognisafe_` namespace.

### `cognisafe_requests_total`

**Type:** Counter\
**Labels:** `project_id`, `model`, `score_label`

Incremented once per LLM request logged by the proxy. `score_label` is `pass`, `fail`, or `unscored` (the scorer did not run).

```
cognisafe_requests_total{project_id="proj_acme_prod",model="gpt-4o",score_label="fail"} 47
cognisafe_requests_total{project_id="proj_acme_prod",model="gpt-4o",score_label="pass"} 8312
```

### `cognisafe_request_latency_seconds`

**Type:** Histogram\
**Labels:** `project_id`, `model`\
**Buckets:** 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0

End-to-end proxy latency — from first byte received to last byte forwarded to the upstream LLM. Does not include async safety scoring time.

### `cognisafe_safety_score`

**Type:** Gauge\
**Labels:** `project_id`, `scorer`, `severity`

Last-known score value per scorer per project per severity band. Useful for spotting persistent elevated risk.

```
cognisafe_safety_score{project_id="proj_acme_prod",scorer="prompt_injection",severity="critical"} 0.94
```

### `cognisafe_flagged_requests_total`

**Type:** Counter\
**Labels:** `project_id`, `owasp_category`

Incremented each time a scorer produces `score_label=fail`. One request can produce multiple increments if multiple scorers flag it.

```
cognisafe_flagged_requests_total{project_id="proj_acme_prod",owasp_category="LLM01"} 12
cognisafe_flagged_requests_total{project_id="proj_acme_prod",owasp_category="LLM06"} 3
```

### `cognisafe_worker_queue_depth`

**Type:** Gauge\
**Labels:** *(none)*

Current length of the `safety_score_jobs` Redis list. Exported by the safety worker on port `9091`. A sustained value above 100 indicates the worker fleet is under-provisioned.

### `cognisafe_worker_processing_seconds`

**Type:** Histogram\
**Labels:** `scorer`\
**Buckets:** 0.1, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0

Time taken to run a single scorer evaluation, including LLM API call latency inside the scorer.

***

## Prometheus Scrape Configuration

### Standalone Prometheus

```yaml theme={null}
# prometheus.yml
scrape_configs:
  - job_name: cognisafe-api
    static_configs:
      - targets: ["cognisafe-api:8000"]
    metrics_path: /metrics
    scrape_interval: 15s

  - job_name: cognisafe-worker
    static_configs:
      - targets: ["cognisafe-worker:9091"]
    metrics_path: /metrics
    scrape_interval: 15s
```

### Kubernetes ServiceMonitor (kube-prometheus-stack)

Apply after installing the `cognisafe` Helm chart with `serviceMonitor.enabled: true`:

```yaml theme={null}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cognisafe
  namespace: monitoring   # must match Prometheus operator's namespaceSelector
  labels:
    release: kube-prometheus-stack   # must match Prometheus operator's serviceMonitorSelector
spec:
  namespaceSelector:
    matchNames:
      - cognisafe
  selector:
    matchLabels:
      app.kubernetes.io/name: cognisafe
  endpoints:
    - port: http-metrics        # api: 8000
      path: /metrics
      interval: 15s
    - port: worker-metrics      # worker: 9091
      path: /metrics
      interval: 15s
```

<Note>
  The Helm chart creates the `Service` with the `http-metrics` and `worker-metrics` named ports automatically when `serviceMonitor.enabled: true`.
</Note>

***

## Grafana Dashboard

Set up a dashboard with these six panels. Import the JSON at the end of this section, or build manually.

### Panel 1 — Request Rate (req/s, 5 min avg)

**Type:** Stat + Time series combo

```promql theme={null}
# Stat: current rate
sum(rate(cognisafe_requests_total[5m]))

# Time series: per project
sum by (project_id) (rate(cognisafe_requests_total[5m]))
```

### Panel 2 — Flag Rate % (flagged / total)

**Type:** Gauge with colour thresholds

```promql theme={null}
100 * sum(rate(cognisafe_requests_total{score_label="fail"}[5m]))
    / sum(rate(cognisafe_requests_total[5m]))
```

Thresholds: 0–1% green, 1–5% amber, >5% red.

### Panel 3 — OWASP Category Breakdown (last 24h)

**Type:** Bar chart

```promql theme={null}
sort_desc(
  sum by (owasp_category) (
    increase(cognisafe_flagged_requests_total[24h])
  )
)
```

### Panel 4 — Safety Worker Queue Depth

**Type:** Time series with alert annotation

```promql theme={null}
cognisafe_worker_queue_depth
```

Draw a horizontal alert threshold line at 100.

### Panel 5 — Proxy Latency P50 / P95 / P99

**Type:** Time series

```promql theme={null}
# P50
histogram_quantile(0.50, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))

# P95
histogram_quantile(0.95, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))

# P99
histogram_quantile(0.99, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))
```

### Panel 6 — Top 5 Flagged Agents (Table)

**Type:** Table

```promql theme={null}
topk(5,
  sum by (project_id) (
    increase(cognisafe_flagged_requests_total[24h])
  )
)
```

Add a **Last Seen** column by joining against `cognisafe_safety_score` max timestamp. In Grafana, add a second query with `cognisafe_safety_score` and use a **Transform → Merge** to combine.

***

## Dashboard JSON

Abbreviated but fully functional — import via **Grafana → Dashboards → Import → Paste JSON**.

```json theme={null}
{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "type": "datasource",
      "pluginId": "prometheus"
    }
  ],
  "__requires": [
    { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
    { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }
  ],
  "title": "Cognisafe Overview",
  "uid": "cognisafe-overview",
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-24h", "to": "now" },
  "panels": [
    {
      "id": 1,
      "title": "Request Rate (req/s)",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 6 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "sum by (project_id) (rate(cognisafe_requests_total[5m]))",
          "legendFormat": "{{project_id}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "reqps",
          "custom": { "lineWidth": 2 }
        }
      }
    },
    {
      "id": 2,
      "title": "Flag Rate %",
      "type": "gauge",
      "gridPos": { "x": 12, "y": 0, "w": 6, "h": 6 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "100 * sum(rate(cognisafe_requests_total{score_label=\"fail\"}[5m])) / sum(rate(cognisafe_requests_total[5m]))",
          "legendFormat": "Flag rate"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 1 },
              { "color": "red", "value": 5 }
            ]
          }
        }
      }
    },
    {
      "id": 3,
      "title": "OWASP Category Breakdown (24h)",
      "type": "barchart",
      "gridPos": { "x": 0, "y": 6, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "sort_desc(sum by (owasp_category) (increase(cognisafe_flagged_requests_total[24h])))",
          "legendFormat": "{{owasp_category}}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" }
      }
    },
    {
      "id": 4,
      "title": "Worker Queue Depth",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 6, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "cognisafe_worker_queue_depth",
          "legendFormat": "Queue depth"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "custom": {
            "lineWidth": 2,
            "thresholdsStyle": { "mode": "line" }
          },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "red", "value": 100 }
            ]
          }
        }
      }
    },
    {
      "id": 5,
      "title": "Proxy Latency P50 / P95 / P99",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 14, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P50"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P95"
        },
        {
          "expr": "histogram_quantile(0.99, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P99"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    },
    {
      "id": 6,
      "title": "Top 5 Flagged Agents (24h)",
      "type": "table",
      "gridPos": { "x": 12, "y": 14, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "topk(5, sum by (project_id) (increase(cognisafe_flagged_requests_total[24h])))",
          "legendFormat": "{{project_id}}",
          "instant": true
        }
      ],
      "transformations": [
        { "id": "organize", "options": { "renameByName": { "project_id": "Agent / Project", "Value": "Flags (24h)" } } },
        { "id": "sortBy", "options": { "fields": [{ "desc": true, "displayName": "Flags (24h)" }] } }
      ]
    }
  ]
}
```

***

## AlertManager Rules

```yaml theme={null}
# cognisafe-alerts.yaml — load via kube-prometheus-stack PrometheusRule CRD
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cognisafe-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: cognisafe.safety
      interval: 30s
      rules:
        - alert: CognisafeHighFlagRate
          expr: |
            (
              sum(rate(cognisafe_requests_total{score_label="fail"}[5m]))
              /
              sum(rate(cognisafe_requests_total[5m]))
            ) > 0.10
          for: 2m
          labels:
            severity: critical
            team: security
          annotations:
            summary: "Cognisafe flag rate exceeds 10%"
            description: "{{ $value | humanizePercentage }} of requests are being flagged. Check the OWASP breakdown dashboard."
            runbook_url: "https://docs.cognisafe.io/runbooks/high-flag-rate"

        - alert: CognisafeWorkerQueueDepthHigh
          expr: cognisafe_worker_queue_depth > 100
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Safety worker queue depth is {{ $value }}"
            description: "Queue has been above 100 for 5 minutes. Scale the safety_worker deployment or investigate worker errors."
            runbook_url: "https://docs.cognisafe.io/runbooks/worker-queue-depth"

        - alert: CognisafeProxyErrorRateHigh
          expr: |
            sum(rate(http_requests_total{job="cognisafe-api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="cognisafe-api"}[5m]))
            > 0.01
          for: 3m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "Cognisafe API 5xx error rate is {{ $value | humanizePercentage }}"
            description: "More than 1% of API requests are returning 5xx. Check API pod logs and database connectivity."
            runbook_url: "https://docs.cognisafe.io/runbooks/api-errors"
```

***

## Grafana Alerting: PagerDuty + Slack

Configure contact points in **Grafana → Alerting → Contact points**:

```yaml theme={null}
# grafana-provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: cognisafe-pagerduty
    receivers:
      - uid: cognisafe-pd
        type: pagerduty
        settings:
          integrationKey: "${PAGERDUTY_INTEGRATION_KEY}"
          severity: "critical"
          class: "cognisafe-safety"
          component: "safety-scorer"
          group: "cognisafe"
          summary: '{{ template "default.message" . }}'

  - orgId: 1
    name: cognisafe-slack
    receivers:
      - uid: cognisafe-slack
        type: slack
        settings:
          url: "${SLACK_WEBHOOK_URL}"
          channel: "#cognisafe-alerts"
          username: "Cognisafe"
          icon_emoji: ":shield:"
          title: '{{ template "default.title" . }}'
          text: |
            *Alert:* {{ .CommonAnnotations.summary }}
            *Description:* {{ .CommonAnnotations.description }}
            *Runbook:* {{ .CommonAnnotations.runbook_url }}
```

Notification policy:

```yaml theme={null}
# grafana-provisioning/alerting/notification-policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: cognisafe-slack
    group_by: ["alertname", "project_id"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: cognisafe-pagerduty
        matchers:
          - severity = critical
        continue: true   # also send to Slack
      - receiver: cognisafe-slack
        matchers:
          - severity =~ "warning|critical"
```

***

## Grafana Cloud: Remote Write

If you use Grafana Cloud instead of self-hosted Grafana, push metrics via Prometheus `remote_write` — no inbound scrape required:

```yaml theme={null}
# prometheus.yml — append to existing config
remote_write:
  - url: "https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push"
    basic_auth:
      username: "${GRAFANA_CLOUD_METRICS_USER_ID}"
      password: "${GRAFANA_CLOUD_API_KEY}"
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "cognisafe_.*"
        action: keep   # only forward cognisafe metrics to save cost
    queue_config:
      max_samples_per_send: 1000
      max_shards: 4
      capacity: 2500
```

<Tip>
  Set `write_relabel_configs` to `keep` only `cognisafe_.*` metrics to avoid ingesting the full Kubernetes metrics corpus into Grafana Cloud, which reduces your monthly active series bill significantly.
</Tip>