Grafana & Prometheus Monitoring

Metrics Endpoint

Cognisafe’s FastAPI backend (api/) exposes a /metrics endpoint in Prometheus text format using prometheus-fastapi-instrumentator. Installation (already included in the official Docker image — this is for self-builders):

pip install prometheus-fastapi-instrumentator==6.1.0

Wiring in api/main.py:

from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

Instrumentator(
    should_group_status_codes=False,
    should_ignore_untemplated=True,
    should_respect_env_var=False,
    excluded_handlers=["/healthz", "/readyz"],
).instrument(app).expose(app, endpoint="/metrics", include_in_schema=False)

The safety worker additionally exports Redis queue depth and processing duration metrics via a lightweight prometheus_client exposition loop on port 9091.

Key Metrics

All Cognisafe metrics use the cognisafe_ namespace.

`cognisafe_requests_total`

Type: Counter
Labels: project_id, model, score_label Incremented once per LLM request logged by the proxy. score_label is pass, fail, or unscored (the scorer did not run).

cognisafe_requests_total{project_id="proj_acme_prod",model="gpt-4o",score_label="fail"} 47
cognisafe_requests_total{project_id="proj_acme_prod",model="gpt-4o",score_label="pass"} 8312

`cognisafe_request_latency_seconds`

Type: Histogram
Labels: project_id, model
Buckets: 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0 End-to-end proxy latency — from first byte received to last byte forwarded to the upstream LLM. Does not include async safety scoring time.

`cognisafe_safety_score`

Type: Gauge
Labels: project_id, scorer, severity Last-known score value per scorer per project per severity band. Useful for spotting persistent elevated risk.

cognisafe_safety_score{project_id="proj_acme_prod",scorer="prompt_injection",severity="critical"} 0.94

`cognisafe_flagged_requests_total`

Type: Counter
Labels: project_id, owasp_category Incremented each time a scorer produces score_label=fail. One request can produce multiple increments if multiple scorers flag it.

cognisafe_flagged_requests_total{project_id="proj_acme_prod",owasp_category="LLM01"} 12
cognisafe_flagged_requests_total{project_id="proj_acme_prod",owasp_category="LLM06"} 3

`cognisafe_worker_queue_depth`

Type: Gauge
Labels: (none) Current length of the safety_score_jobs Redis list. Exported by the safety worker on port 9091. A sustained value above 100 indicates the worker fleet is under-provisioned.

`cognisafe_worker_processing_seconds`

Type: Histogram
Labels: scorer
Buckets: 0.1, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0 Time taken to run a single scorer evaluation, including LLM API call latency inside the scorer.

Prometheus Scrape Configuration

Standalone Prometheus

# prometheus.yml
scrape_configs:
  - job_name: cognisafe-api
    static_configs:
      - targets: ["cognisafe-api:8000"]
    metrics_path: /metrics
    scrape_interval: 15s

  - job_name: cognisafe-worker
    static_configs:
      - targets: ["cognisafe-worker:9091"]
    metrics_path: /metrics
    scrape_interval: 15s

Kubernetes ServiceMonitor (kube-prometheus-stack)

Apply after installing the cognisafe Helm chart with serviceMonitor.enabled: true:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cognisafe
  namespace: monitoring   # must match Prometheus operator's namespaceSelector
  labels:
    release: kube-prometheus-stack   # must match Prometheus operator's serviceMonitorSelector
spec:
  namespaceSelector:
    matchNames:
      - cognisafe
  selector:
    matchLabels:
      app.kubernetes.io/name: cognisafe
  endpoints:
    - port: http-metrics        # api: 8000
      path: /metrics
      interval: 15s
    - port: worker-metrics      # worker: 9091
      path: /metrics
      interval: 15s

The Helm chart creates the Service with the http-metrics and worker-metrics named ports automatically when serviceMonitor.enabled: true.

Grafana Dashboard

Set up a dashboard with these six panels. Import the JSON at the end of this section, or build manually.

Panel 1 — Request Rate (req/s, 5 min avg)

Type: Stat + Time series combo

# Stat: current rate
sum(rate(cognisafe_requests_total[5m]))

# Time series: per project
sum by (project_id) (rate(cognisafe_requests_total[5m]))

Panel 2 — Flag Rate % (flagged / total)

Type: Gauge with colour thresholds

100 * sum(rate(cognisafe_requests_total{score_label="fail"}[5m]))
    / sum(rate(cognisafe_requests_total[5m]))

Thresholds: 0–1% green, 1–5% amber, >5% red.

Panel 3 — OWASP Category Breakdown (last 24h)

Type: Bar chart

sort_desc(
  sum by (owasp_category) (
    increase(cognisafe_flagged_requests_total[24h])
  )
)

Panel 4 — Safety Worker Queue Depth

Type: Time series with alert annotation

cognisafe_worker_queue_depth

Draw a horizontal alert threshold line at 100.

Panel 5 — Proxy Latency P50 / P95 / P99

Type: Time series

# P50
histogram_quantile(0.50, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))

# P95
histogram_quantile(0.95, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))

# P99
histogram_quantile(0.99, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))

Panel 6 — Top 5 Flagged Agents (Table)

Type: Table

topk(5,
  sum by (project_id) (
    increase(cognisafe_flagged_requests_total[24h])
  )
)

Add a Last Seen column by joining against cognisafe_safety_score max timestamp. In Grafana, add a second query with cognisafe_safety_score and use a Transform → Merge to combine.

Dashboard JSON

Abbreviated but fully functional — import via Grafana → Dashboards → Import → Paste JSON.

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "type": "datasource",
      "pluginId": "prometheus"
    }
  ],
  "__requires": [
    { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "10.0.0" },
    { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }
  ],
  "title": "Cognisafe Overview",
  "uid": "cognisafe-overview",
  "version": 1,
  "refresh": "30s",
  "time": { "from": "now-24h", "to": "now" },
  "panels": [
    {
      "id": 1,
      "title": "Request Rate (req/s)",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 0, "w": 12, "h": 6 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "sum by (project_id) (rate(cognisafe_requests_total[5m]))",
          "legendFormat": "{{project_id}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "reqps",
          "custom": { "lineWidth": 2 }
        }
      }
    },
    {
      "id": 2,
      "title": "Flag Rate %",
      "type": "gauge",
      "gridPos": { "x": 12, "y": 0, "w": 6, "h": 6 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "100 * sum(rate(cognisafe_requests_total{score_label=\"fail\"}[5m])) / sum(rate(cognisafe_requests_total[5m]))",
          "legendFormat": "Flag rate"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percent",
          "min": 0,
          "max": 100,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "yellow", "value": 1 },
              { "color": "red", "value": 5 }
            ]
          }
        }
      }
    },
    {
      "id": 3,
      "title": "OWASP Category Breakdown (24h)",
      "type": "barchart",
      "gridPos": { "x": 0, "y": 6, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "sort_desc(sum by (owasp_category) (increase(cognisafe_flagged_requests_total[24h])))",
          "legendFormat": "{{owasp_category}}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "short" }
      }
    },
    {
      "id": 4,
      "title": "Worker Queue Depth",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 6, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "cognisafe_worker_queue_depth",
          "legendFormat": "Queue depth"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "short",
          "custom": {
            "lineWidth": 2,
            "thresholdsStyle": { "mode": "line" }
          },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "green", "value": null },
              { "color": "red", "value": 100 }
            ]
          }
        }
      }
    },
    {
      "id": 5,
      "title": "Proxy Latency P50 / P95 / P99",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 14, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "histogram_quantile(0.50, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P50"
        },
        {
          "expr": "histogram_quantile(0.95, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P95"
        },
        {
          "expr": "histogram_quantile(0.99, sum by (le) (rate(cognisafe_request_latency_seconds_bucket[5m])))",
          "legendFormat": "P99"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    },
    {
      "id": 6,
      "title": "Top 5 Flagged Agents (24h)",
      "type": "table",
      "gridPos": { "x": 12, "y": 14, "w": 12, "h": 8 },
      "datasource": "${DS_PROMETHEUS}",
      "targets": [
        {
          "expr": "topk(5, sum by (project_id) (increase(cognisafe_flagged_requests_total[24h])))",
          "legendFormat": "{{project_id}}",
          "instant": true
        }
      ],
      "transformations": [
        { "id": "organize", "options": { "renameByName": { "project_id": "Agent / Project", "Value": "Flags (24h)" } } },
        { "id": "sortBy", "options": { "fields": [{ "desc": true, "displayName": "Flags (24h)" }] } }
      ]
    }
  ]
}

AlertManager Rules

# cognisafe-alerts.yaml — load via kube-prometheus-stack PrometheusRule CRD
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cognisafe-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: cognisafe.safety
      interval: 30s
      rules:
        - alert: CognisafeHighFlagRate
          expr: |
            (
              sum(rate(cognisafe_requests_total{score_label="fail"}[5m]))
              /
              sum(rate(cognisafe_requests_total[5m]))
            ) > 0.10
          for: 2m
          labels:
            severity: critical
            team: security
          annotations:
            summary: "Cognisafe flag rate exceeds 10%"
            description: "{{ $value | humanizePercentage }} of requests are being flagged. Check the OWASP breakdown dashboard."
            runbook_url: "https://docs.cognisafe.io/runbooks/high-flag-rate"

        - alert: CognisafeWorkerQueueDepthHigh
          expr: cognisafe_worker_queue_depth > 100
          for: 5m
          labels:
            severity: warning
            team: platform
          annotations:
            summary: "Safety worker queue depth is {{ $value }}"
            description: "Queue has been above 100 for 5 minutes. Scale the safety_worker deployment or investigate worker errors."
            runbook_url: "https://docs.cognisafe.io/runbooks/worker-queue-depth"

        - alert: CognisafeProxyErrorRateHigh
          expr: |
            sum(rate(http_requests_total{job="cognisafe-api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="cognisafe-api"}[5m]))
            > 0.01
          for: 3m
          labels:
            severity: critical
            team: platform
          annotations:
            summary: "Cognisafe API 5xx error rate is {{ $value | humanizePercentage }}"
            description: "More than 1% of API requests are returning 5xx. Check API pod logs and database connectivity."
            runbook_url: "https://docs.cognisafe.io/runbooks/api-errors"

Grafana Alerting: PagerDuty + Slack

Configure contact points in Grafana → Alerting → Contact points:

# grafana-provisioning/alerting/contact-points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: cognisafe-pagerduty
    receivers:
      - uid: cognisafe-pd
        type: pagerduty
        settings:
          integrationKey: "${PAGERDUTY_INTEGRATION_KEY}"
          severity: "critical"
          class: "cognisafe-safety"
          component: "safety-scorer"
          group: "cognisafe"
          summary: '{{ template "default.message" . }}'

  - orgId: 1
    name: cognisafe-slack
    receivers:
      - uid: cognisafe-slack
        type: slack
        settings:
          url: "${SLACK_WEBHOOK_URL}"
          channel: "#cognisafe-alerts"
          username: "Cognisafe"
          icon_emoji: ":shield:"
          title: '{{ template "default.title" . }}'
          text: |
            *Alert:* {{ .CommonAnnotations.summary }}
            *Description:* {{ .CommonAnnotations.description }}
            *Runbook:* {{ .CommonAnnotations.runbook_url }}

Notification policy:

# grafana-provisioning/alerting/notification-policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: cognisafe-slack
    group_by: ["alertname", "project_id"]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: cognisafe-pagerduty
        matchers:
          - severity = critical
        continue: true   # also send to Slack
      - receiver: cognisafe-slack
        matchers:
          - severity =~ "warning|critical"

Grafana Cloud: Remote Write

If you use Grafana Cloud instead of self-hosted Grafana, push metrics via Prometheus remote_write — no inbound scrape required:

# prometheus.yml — append to existing config
remote_write:
  - url: "https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push"
    basic_auth:
      username: "${GRAFANA_CLOUD_METRICS_USER_ID}"
      password: "${GRAFANA_CLOUD_API_KEY}"
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "cognisafe_.*"
        action: keep   # only forward cognisafe metrics to save cost
    queue_config:
      max_samples_per_send: 1000
      max_shards: 4
      capacity: 2500

Set write_relabel_configs to keep only cognisafe_.* metrics to avoid ingesting the full Kubernetes metrics corpus into Grafana Cloud, which reduces your monthly active series bill significantly.

Getting Started

SDKs

LLM Providers

Self-hosted Models

Self-hosting

Integrations

Reference Architectures

Deployment Blueprints

Safety & Scoring

Grafana & Prometheus Monitoring

Metrics Endpoint

Key Metrics

`cognisafe_requests_total`

`cognisafe_request_latency_seconds`

`cognisafe_safety_score`

`cognisafe_flagged_requests_total`

`cognisafe_worker_queue_depth`

`cognisafe_worker_processing_seconds`

Prometheus Scrape Configuration

Standalone Prometheus

Kubernetes ServiceMonitor (kube-prometheus-stack)

Grafana Dashboard

Panel 1 — Request Rate (req/s, 5 min avg)

Panel 2 — Flag Rate % (flagged / total)

Panel 3 — OWASP Category Breakdown (last 24h)

Panel 4 — Safety Worker Queue Depth

Panel 5 — Proxy Latency P50 / P95 / P99

Panel 6 — Top 5 Flagged Agents (Table)

Dashboard JSON

AlertManager Rules

Grafana Alerting: PagerDuty + Slack

Grafana Cloud: Remote Write

​Metrics Endpoint

​Key Metrics

​cognisafe_requests_total

​cognisafe_request_latency_seconds

​cognisafe_safety_score

​cognisafe_flagged_requests_total

​cognisafe_worker_queue_depth

​cognisafe_worker_processing_seconds

​Prometheus Scrape Configuration

​Standalone Prometheus

​Kubernetes ServiceMonitor (kube-prometheus-stack)

​Grafana Dashboard

​Panel 1 — Request Rate (req/s, 5 min avg)

​Panel 2 — Flag Rate % (flagged / total)

​Panel 3 — OWASP Category Breakdown (last 24h)

​Panel 4 — Safety Worker Queue Depth

​Panel 5 — Proxy Latency P50 / P95 / P99

​Panel 6 — Top 5 Flagged Agents (Table)

​Dashboard JSON

​AlertManager Rules

​Grafana Alerting: PagerDuty + Slack

​Grafana Cloud: Remote Write

Metrics Endpoint

Key Metrics

`cognisafe_requests_total`

`cognisafe_request_latency_seconds`

`cognisafe_safety_score`

`cognisafe_flagged_requests_total`

`cognisafe_worker_queue_depth`

`cognisafe_worker_processing_seconds`

Prometheus Scrape Configuration

Standalone Prometheus

Kubernetes ServiceMonitor (kube-prometheus-stack)

Grafana Dashboard

Panel 1 — Request Rate (req/s, 5 min avg)

Panel 2 — Flag Rate % (flagged / total)

Panel 3 — OWASP Category Breakdown (last 24h)

Panel 4 — Safety Worker Queue Depth

Panel 5 — Proxy Latency P50 / P95 / P99

Panel 6 — Top 5 Flagged Agents (Table)

Dashboard JSON

AlertManager Rules

Grafana Alerting: PagerDuty + Slack

Grafana Cloud: Remote Write