> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# vLLM

> Route vLLM inference server traffic through Cognisafe for runtime security and observability.

## Overview

vLLM is a high-throughput inference engine for LLMs with PagedAttention. It exposes an OpenAI-compatible API and is one of the most widely deployed self-hosted inference backends.

Cognisafe intercepts all vLLM traffic transparently — no changes to your vLLM deployment required.

## Quick start

### Run vLLM

```bash theme={null}
pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --host 0.0.0.0
```

vLLM's OpenAI-compatible server listens on `http://localhost:8000/v1`.

### Proxy configuration

```bash theme={null}
# Cognisafe proxy env vars
UPSTREAM_URL=http://vllm-host.internal:8000
```

### SDK setup

```python theme={null}
import cognisafe
from openai import OpenAI

cognisafe.configure(
    api_key="csk_your_key_here",
    project_id="my-app",
    proxy_url="http://cognisafe-proxy:8080",
)
cognisafe.patch_openai()

client = OpenAI(
    api_key="not-required",  # vLLM doesn't validate API keys by default
    base_url="http://cognisafe-proxy:8080/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",  # exact model name vLLM was started with
    messages=[{"role": "user", "content": "Hello"}],
)
```

<Note>
  vLLM uses the full model name as the `model` field (e.g. `meta-llama/Llama-3.1-8B-Instruct`). Make sure your application code uses the same name that vLLM was started with.
</Note>

## Multi-model deployment

vLLM supports serving multiple models via the `--served-model-name` flag or by running multiple instances. A common pattern is to deploy one vLLM instance per model and route via Cognisafe:

```bash theme={null}
# Instance 1 — Llama on :8000
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b \
  --port 8000

# Instance 2 — Mistral on :8001
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --served-model-name mistral-7b \
  --port 8001
```

Run a Cognisafe proxy per upstream, or use a load balancer in front of a single proxy.

## Kubernetes deployment

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: cognisafe-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - --model=meta-llama/Llama-3.1-8B-Instruct
            - --port=8000
            - --host=0.0.0.0
            - --max-model-len=8192
          ports:
            - containerPort: 8000
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "20Gi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 15
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: cognisafe-system
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000
```

## API key enforcement

By default, vLLM accepts any API key. To enforce a specific key:

```bash theme={null}
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --api-key your-internal-vllm-key
```

Set this key as `VLLM_API_KEY` in your environment and configure the Cognisafe proxy to forward it:

```bash theme={null}
# Cognisafe proxy env vars
UPSTREAM_API_KEY=your-internal-vllm-key
```

<Tip>
  Restrict network access to the vLLM service using a Kubernetes NetworkPolicy — only allow ingress from the Cognisafe proxy pod. This ensures all traffic goes through Cognisafe and cannot bypass it.
</Tip>

```yaml theme={null}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-ingress
  namespace: cognisafe-system
spec:
  podSelector:
    matchLabels:
      app: vllm
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: cognisafe-proxy
      ports:
        - port: 8000
```

## Air-gapped safety scoring

vLLM works well as a local scoring backend. Run a second vLLM instance with a safety-focused model:

```bash theme={null}
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-Guard-3-8B \
  --served-model-name llama-guard \
  --port 8001
```

```bash theme={null}
# safety_worker env vars
OPENAI_API_KEY=dummy
SCORER_MODEL=llama-guard
OPENAI_BASE_URL=http://vllm-scoring.cognisafe-system.svc.cluster.local:8001/v1
```

Llama Guard 3 is the recommended open-source scoring model — it is specifically fine-tuned for safety classification across the MLCommons hazard taxonomy, which maps closely to OWASP LLM categories.