> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognisafe.uk/llms.txt
> Use this file to discover all available pages before exploring further.

# NVIDIA NIM

> Secure NVIDIA NIM inference microservices with Cognisafe runtime monitoring.

## Overview

NVIDIA NIM (NVIDIA Inference Microservices) packages optimised LLMs as containers exposing an OpenAI-compatible API. NIMs run on-premises on NVIDIA GPU infrastructure or on NGC-hosted endpoints.

Because NIM speaks the OpenAI API protocol, Cognisafe intercepts all traffic transparently — no NIM configuration changes required.

## Self-hosted NIM

### Prerequisites

* NVIDIA GPU with CUDA 12.x
* Docker with NVIDIA Container Toolkit
* NGC API key from [ngc.nvidia.com](https://ngc.nvidia.com)

### Run a NIM container

```bash theme={null}
export NGC_API_KEY=nvapi_your_key_here

docker run --gpus all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
```

NIM exposes the OpenAI-compatible API at `http://localhost:8000/v1`.

### Proxy configuration

```bash theme={null}
# Cognisafe proxy env vars
UPSTREAM_URL=http://nim-host.internal:8000
```

### SDK setup

```python theme={null}
import cognisafe
from openai import OpenAI

cognisafe.configure(
    api_key="csk_your_key_here",
    project_id="my-app",
    proxy_url="http://cognisafe-proxy:8080",
)
cognisafe.patch_openai()

client = OpenAI(
    api_key="nvapi_your_key_here",  # forwarded to NIM by the proxy
    base_url="http://cognisafe-proxy:8080/v1",
)

response = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[{"role": "user", "content": "Explain quantum entanglement."}],
)
```

## NGC-hosted NIM endpoints

NVIDIA hosts NIM endpoints at `https://integrate.api.nvidia.com/v1`. These are useful for testing before deploying your own GPU infrastructure.

```bash theme={null}
UPSTREAM_URL=https://integrate.api.nvidia.com
```

```python theme={null}
client = OpenAI(
    api_key="nvapi_your_key_here",
    base_url="http://cognisafe-proxy:8080/v1",
)

response = client.chat.completions.create(
    model="nvidia/llama-3.1-nemotron-70b-instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
```

<Note>
  The Cognisafe proxy strips and rewrites the `base_url` — your client always points at the proxy, and the proxy forwards to `UPSTREAM_URL`. This means your application code never contains the NIM endpoint URL directly.
</Note>

## Kubernetes with NVIDIA GPU Operator

```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nim-llama
  namespace: cognisafe-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nim-llama
  template:
    metadata:
      labels:
        app: nim-llama
    spec:
      runtimeClassName: nvidia
      containers:
        - name: nim
          image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
          ports:
            - containerPort: 8000
          env:
            - name: NGC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: ngc-secret
                  key: api-key
          resources:
            limits:
              nvidia.com/gpu: "1"
              memory: "24Gi"
            requests:
              nvidia.com/gpu: "1"
          readinessProbe:
            httpGet:
              path: /v1/health/ready
              port: 8000
            initialDelaySeconds: 60
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: nim-llama
  namespace: cognisafe-system
spec:
  selector:
    app: nim-llama
  ports:
    - port: 8000
      targetPort: 8000
```

Set `UPSTREAM_URL=http://nim-llama.cognisafe-system.svc.cluster.local:8000` on the Cognisafe proxy.

<Warning>
  NIM containers are large (10–30 GB). Use a dedicated node pool with local NVMe storage and pre-pull the image via a DaemonSet to avoid cold-start delays in production.
</Warning>

## Air-gapped safety scoring with NIM

NVIDIA publishes a NIM for content safety:

```bash theme={null}
docker run --gpus all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8001:8000 \
  nvcr.io/nim/nvidia/llama-3.1-nemo-guardrails:latest
```

Configure the safety worker to use it:

```bash theme={null}
OPENAI_API_KEY=nvapi_your_key_here
SCORER_MODEL=nvidia/llama-3.1-nemo-guardrails
OPENAI_BASE_URL=http://nim-guardrails-host.internal:8001/v1
```

This keeps all scoring on-premises — no OpenAI dependency.