vLLM

Overview

vLLM is a high-throughput inference engine for LLMs with PagedAttention. It exposes an OpenAI-compatible API and is one of the most widely deployed self-hosted inference backends. Cognisafe intercepts all vLLM traffic transparently — no changes to your vLLM deployment required.

Quick start

Run vLLM

pip install vllm

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --host 0.0.0.0

vLLM’s OpenAI-compatible server listens on http://localhost:8000/v1.

Proxy configuration

# Cognisafe proxy env vars
UPSTREAM_URL=http://vllm-host.internal:8000

SDK setup

import cognisafe
from openai import OpenAI

cognisafe.configure(
    api_key="csk_your_key_here",
    project_id="my-app",
    proxy_url="http://cognisafe-proxy:8080",
)
cognisafe.patch_openai()

client = OpenAI(
    api_key="not-required",  # vLLM doesn't validate API keys by default
    base_url="http://cognisafe-proxy:8080/v1",
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",  # exact model name vLLM was started with
    messages=[{"role": "user", "content": "Hello"}],
)

vLLM uses the full model name as the model field (e.g. meta-llama/Llama-3.1-8B-Instruct). Make sure your application code uses the same name that vLLM was started with.

Multi-model deployment

vLLM supports serving multiple models via the --served-model-name flag or by running multiple instances. A common pattern is to deploy one vLLM instance per model and route via Cognisafe:

# Instance 1 — Llama on :8000
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --served-model-name llama-3.1-8b \
  --port 8000

# Instance 2 — Mistral on :8001
python -m vllm.entrypoints.openai.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --served-model-name mistral-7b \
  --port 8001

Run a Cognisafe proxy per upstream, or use a load balancer in front of a single proxy.

Kubernetes deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
  namespace: cognisafe-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args:
            - --model=meta-llama/Llama-3.1-8B-Instruct
            - --port=8000
            - --host=0.0.0.0
            - --max-model-len=8192
          ports:
            - containerPort: 8000
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "20Gi"
          readinessProbe:
            httpGet:
              path: /health
              port: 8000
            initialDelaySeconds: 90
            periodSeconds: 15
          volumeMounts:
            - name: model-cache
              mountPath: /root/.cache/huggingface
      volumes:
        - name: model-cache
          persistentVolumeClaim:
            claimName: model-cache-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: cognisafe-system
spec:
  selector:
    app: vllm
  ports:
    - port: 8000
      targetPort: 8000

API key enforcement

By default, vLLM accepts any API key. To enforce a specific key:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --api-key your-internal-vllm-key

Set this key as VLLM_API_KEY in your environment and configure the Cognisafe proxy to forward it:

# Cognisafe proxy env vars
UPSTREAM_API_KEY=your-internal-vllm-key

Restrict network access to the vLLM service using a Kubernetes NetworkPolicy — only allow ingress from the Cognisafe proxy pod. This ensures all traffic goes through Cognisafe and cannot bypass it.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vllm-ingress
  namespace: cognisafe-system
spec:
  podSelector:
    matchLabels:
      app: vllm
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: cognisafe-proxy
      ports:
        - port: 8000

Air-gapped safety scoring

vLLM works well as a local scoring backend. Run a second vLLM instance with a safety-focused model:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-Guard-3-8B \
  --served-model-name llama-guard \
  --port 8001

# safety_worker env vars
OPENAI_API_KEY=dummy
SCORER_MODEL=llama-guard
OPENAI_BASE_URL=http://vllm-scoring.cognisafe-system.svc.cluster.local:8001/v1

Llama Guard 3 is the recommended open-source scoring model — it is specifically fine-tuned for safety classification across the MLCommons hazard taxonomy, which maps closely to OWASP LLM categories.

Getting Started

SDKs

LLM Providers

Self-hosted Models

Self-hosting

Integrations

Reference Architectures

Deployment Blueprints

Safety & Scoring

Overview

Quick start

Run vLLM

Proxy configuration

SDK setup

Multi-model deployment

Kubernetes deployment

API key enforcement

Air-gapped safety scoring

​Overview

​Quick start

​Run vLLM

​Proxy configuration

​SDK setup

​Multi-model deployment

​Kubernetes deployment

​API key enforcement

​Air-gapped safety scoring

Overview

Quick start

Run vLLM

Proxy configuration

SDK setup

Multi-model deployment

Kubernetes deployment

API key enforcement

Air-gapped safety scoring