> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# Hugging Face (TGI & Inference Endpoints)

> Route Hugging Face Text Generation Inference and Inference Endpoints traffic through Cognisafe.

## Overview

Hugging Face exposes two deployment targets that Cognisafe can intercept:

* **Text Generation Inference (TGI)** — self-hosted inference server, OpenAI-compatible API on `/v1/`
* **Inference Endpoints** — managed hosting on HF infrastructure, also OpenAI-compatible

Both speak the same protocol as the Cognisafe proxy, so the setup is identical — point `UPSTREAM_URL` at your TGI instance or Inference Endpoint URL.

## Text Generation Inference (self-hosted)

TGI exposes an OpenAI-compatible endpoint at `http://<host>:8080/v1` by default.

### Docker deployment

```bash theme={null}
docker run --gpus all -p 8080:80 \
  -v $HOME/.cache/huggingface:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct
```

### Proxy configuration

```bash theme={null}
# Cognisafe proxy env vars
UPSTREAM_URL=http://tgi-host.internal:8080
```

### SDK setup

```python theme={null}
import cognisafe
from openai import OpenAI

cognisafe.configure(
    api_key="csk_your_key_here",
    project_id="my-app",
    proxy_url="http://localhost:8080",  # Cognisafe proxy
)
cognisafe.patch_openai()

client = OpenAI(api_key="hf-token-not-needed-for-local-tgi")

response = client.chat.completions.create(
    model="tgi",  # TGI ignores the model name — it serves whatever was loaded
    messages=[{"role": "user", "content": "Summarise this document..."}],
)
```

<Note>
  TGI serves one model at a time. The `model` field in the request is ignored — TGI always responds with the model it was started with.
</Note>

## Hugging Face Inference Endpoints (managed)

Inference Endpoints expose a URL like `https://xyz.us-east-1.aws.endpoints.huggingface.cloud`.

### Proxy configuration

```bash theme={null}
UPSTREAM_URL=https://xyz.us-east-1.aws.endpoints.huggingface.cloud
```

### SDK setup

```python theme={null}
import cognisafe
from openai import OpenAI

cognisafe.configure(
    api_key="csk_your_key_here",
    project_id="prod",
    proxy_url="http://your-cognisafe-proxy:8080",
)
cognisafe.patch_openai()

client = OpenAI(api_key="hf_your_token_here")  # HF token forwarded by proxy

response = client.chat.completions.create(
    model="tgi",
    messages=[{"role": "user", "content": "Hello"}],
)
```

The Cognisafe proxy forwards the `Authorization: Bearer hf_...` header to the Inference Endpoint unchanged.

## Air-gapped scoring

In air-gapped environments, point the safety worker's scoring model at a local TGI instance:

```bash theme={null}
# safety_worker env vars
OPENAI_API_KEY=dummy          # any non-empty value
SCORER_MODEL=tgi              # model name TGI ignores anyway
OPENAI_BASE_URL=http://tgi-scoring-host.internal:8080/v1
```

<Tip>
  For scoring, run a separate TGI instance with a dedicated safety-focused model (e.g. `meta-llama/Llama-Guard-3-8B`) rather than your production model. Llama Guard is specifically trained for safety classification and is significantly more accurate than general-purpose models for OWASP LLM scoring.
</Tip>

## Kubernetes deployment

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tgi
spec:
  replicas: 1
  selector:
    matchLabels:
      app: tgi
  template:
    spec:
      containers:
        - name: tgi
          image: ghcr.io/huggingface/text-generation-inference:latest
          args:
            - --model-id=meta-llama/Llama-3.1-8B-Instruct
            - --port=8080
          ports:
            - containerPort: 8080
          resources:
            limits:
              nvidia.com/gpu: "1"
          env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
---
apiVersion: v1
kind: Service
metadata:
  name: tgi
spec:
  selector:
    app: tgi
  ports:
    - port: 8080
      targetPort: 8080
```

Set `UPSTREAM_URL=http://tgi.default.svc.cluster.local:8080` on the Cognisafe proxy deployment.