> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# How it works

> Architecture overview: proxy, scoring worker, and dashboard.

## Two patching modes

Cognisafe uses different integration strategies depending on whether the LLM provider's API is OpenAI-compatible.

### Proxy mode

For OpenAI-compatible providers (OpenAI, Mistral, Ollama), the SDK rewrites the provider client's `base_url` to point to the Cognisafe proxy running on port 8080. The Go proxy:

1. Receives the request from your app
2. Forwards it to the upstream LLM provider (unchanged)
3. Streams the response back to your app
4. In parallel, POSTs the full request and response payload to `POST /internal/log` on the FastAPI backend — non-blocking, so your app gets the response first

This means **zero added latency** on the round trip to the LLM.

### Direct mode

For providers whose API isn't OpenAI-compatible (Anthropic, Cohere), the SDK wraps the provider client's `create` / `messages.create` method. After the LLM responds:

1. Your app receives the response immediately
2. The SDK fires a background task that ships the payload to `POST /internal/log`

A small amount of work happens on the SDK side, but it does not block the return of the response to your code.

| Provider  | Mode   | How                                             |
| --------- | ------ | ----------------------------------------------- |
| OpenAI    | Proxy  | `base_url` rewritten to `http://localhost:8080` |
| Mistral   | Proxy  | `base_url` rewritten to `http://localhost:8080` |
| Ollama    | Proxy  | `base_url` rewritten to `http://localhost:8080` |
| Anthropic | Direct | `messages.create` wrapped                       |
| Cohere    | Direct | `chat` method wrapped                           |

## Async scoring pipeline

Safety scoring happens entirely off the hot path:

1. `POST /internal/log` receives the payload and writes it to the `llm_requests` table in PostgreSQL (TimescaleDB).
2. It immediately pushes a job to the `safety_score_jobs` Redis queue.
3. One or more `safety_worker` processes read from the queue and run PyRIT scorers (currently `content_safety`, `pii_detection`, `jailbreak_detection`, `data_poisoning`, `vector_weakness`, `unbounded_consumption`).
4. Scorer results are written to the `safety_scores` table — one row per scorer per request.
5. The dashboard polls for new scores and displays them.

The safety workers are horizontally scalable: run as many as you need with `docker compose up --scale safety_worker=N`.

## Data model

### `llm_requests`

Every LLM API call captured by the proxy or SDK. A TimescaleDB hypertable partitioned on `created_at` for efficient time-range queries.

Key columns: `id`, `project_id`, `model`, `provider`, `prompt_tokens`, `completion_tokens`, `cost_usd`, `latency_ms`, `request_body` (JSONB), `response_body` (JSONB), `created_at`.

### `safety_scores`

One row per scorer per request. Foreign key to `llm_requests`.

Key columns: `id`, `request_id`, `scorer_name`, `score_value` (1–5 Likert), `score_label` (`safe` / `unsafe` / `unscored`), `rationale`, `created_at`.

### `subscriptions`

Per-project billing state managed by Stripe webhooks.

Key columns: `project_id`, `tier` (`free` / `pro` / `team` / `enterprise`), `stripe_customer_id`, `stripe_subscription_id`, `requests_this_period`, `period_reset_at`.

Rate limiting is enforced in `POST /internal/log` — the API returns HTTP 429 when `requests_this_period` exceeds the tier limit.

## The `@cognisafe.trace` decorator

For non-standard LLM calls — custom HTTP clients, internal models, fine-tuned endpoints — use the generic decorator:

```python theme={null}
import cognisafe

@cognisafe.trace(model="gpt-4o-fine-tuned")
def call_my_model(prompt: str) -> str:
    # any code that calls an LLM
    response = requests.post("https://my-internal-model/v1/chat", json={"prompt": prompt})
    return response.json()["output"]
```

The decorator captures the function's input and output and ships them to `/internal/log` in the background.

## Request flow diagram

```
Your app
   │
   │  (OpenAI call with patched base_url)
   ▼
proxy/ (Go, :8080)
   │  ◄── forwards response to your app immediately
   │
   │  POST /internal/log  (non-blocking)
   ▼
api/ (FastAPI, :8000)
   │  ── writes to llm_requests (PostgreSQL)
   │  ── pushes job to Redis queue
   ▼
safety_worker/ (Python)
   │  ── pulls from safety_score_jobs queue
   │  ── runs PyRIT scorers
   │  ── writes to safety_scores (PostgreSQL)
   ▼
web/ (Next.js, :3000)
   └── reads from PostgreSQL, renders dashboard
```