Skip to main content

​
Two patching modes

Cognisafe uses different integration strategies depending on whether the LLM provider’s API is OpenAI-compatible.

​
Proxy mode

For OpenAI-compatible providers (OpenAI, Mistral, Ollama), the SDK rewrites the provider client’s base_url to point to the Cognisafe proxy running on port 8080. The Go proxy:
  1. Receives the request from your app
  2. Forwards it to the upstream LLM provider (unchanged)
  3. Streams the response back to your app
  4. In parallel, POSTs the full request and response payload to POST /internal/log on the FastAPI backend — non-blocking, so your app gets the response first
This means zero added latency on the round trip to the LLM.

​
Direct mode

For providers whose API isn’t OpenAI-compatible (Anthropic, Cohere), the SDK wraps the provider client’s create / messages.create method. After the LLM responds:
  1. Your app receives the response immediately
  2. The SDK fires a background task that ships the payload to POST /internal/log
A small amount of work happens on the SDK side, but it does not block the return of the response to your code.

​
Async scoring pipeline

Safety scoring happens entirely off the hot path:
  1. POST /internal/log receives the payload and writes it to the llm_requests table in PostgreSQL (TimescaleDB).
  2. It immediately pushes a job to the safety_score_jobs Redis queue.
  3. One or more safety_worker processes read from the queue and run LLM-as-judge scorers. Which scorers run depends on the account tier, up to the full OWASP LLM Top 10 set (content_safety, pii_detection, jailbreak_detection, system_prompt_leakage, supply_chain, excessive_agency, hallucination, data_poisoning, vector_weakness, unbounded_consumption).
  4. Scorer results are written to the safety_scores table — one row per scorer per request.
  5. The dashboard polls for new scores and displays them.
The worker scores from the request text carried on the Redis job, not from the stored row — so accounts with zero content retention still get full scoring even though no prompt or response body is persisted. The safety workers are horizontally scalable: run as many as you need with docker compose up --scale safety_worker=N.

​
Data model

​
llm_requests

Every LLM API call captured by the proxy or SDK. A TimescaleDB hypertable partitioned on created_at for efficient time-range queries. Key columns: id, project_id, model, provider, prompt_tokens, completion_tokens, cost_usd, latency_ms, request_body, response_body, created_at. On managed cloud the body columns are AES-256-GCM encrypted at rest and erased after 90 days (or never written, with zero content retention).

​
safety_scores

One row per scorer per request. Foreign key to llm_requests. Key columns: id, request_id, scorer_name, score_value (1–5 Likert), score_label (safe / unsafe / unscored), rationale, created_at.

​
subscriptions

Per-project billing state managed by Stripe webhooks. Key columns: project_id, tier (free / pro / team / enterprise), stripe_customer_id, stripe_subscription_id, requests_this_period, period_reset_at. Rate limiting is enforced in POST /internal/log — the API returns HTTP 429 when requests_this_period exceeds the tier limit.

​
The @cognisafe.trace decorator

For non-standard LLM calls — custom HTTP clients, internal models, fine-tuned endpoints — use the generic decorator:
The decorator captures the function’s input and output and ships them to /internal/log in the background.

​
Request flow diagram