Skip to main content
To test a model, Alethia needs to know how to talk to it. That’s what an LLM Configuration stores: the provider, the model name, and your API key. LLM Configurations live at the team level, so every project in the team can use them.

​
Supported providers

Out of the box, Alethia supports:
Need a provider not on the list? Use the Custom option. You provide the endpoint URL, headers, and the JSON paths to read/write — Alethia handles the rest.

​
Adding a configuration

1

Go to Organization → LLM Configurations

You’ll see all the configurations your team has set up.
2

Click 'Add Configuration'

Choose the provider from the dropdown.
3

Fill in the details

  • Name — A friendly label (e.g., “GPT-4o Production”)
  • Model name — The exact model identifier from the provider
  • API Key — Pasted from your provider’s dashboard
  • Mark as test model — ✅ if this is a model you want to test, ❌ if it’s a judge
  • Generator model (optional) — ✅ if this model should power AI prompt generation
  • Judge prompt (optional) — Custom instructions for judging behavior. Leave blank for the default.
4

Test the connection

Click Test Connection. Alethia sends a tiny ping to the provider with your key.
  • ✅ Green — your model is reachable.
  • ❌ Red — most often a wrong key or wrong model name. Double-check both.
5

Save

Done. Your model is now available to any project in the team.

​
Model roles

Alethia uses LLMs in three roles:

Test model

The model whose safety you’re evaluating. Each project has exactly one. Marked Is test model: ✅.

Judge model

The model that grades responses from the test model. Each project uses up to three. Marked Is test model: ❌.

Generator model

The model that drafts adversarial prompts for the library. Flag it Generator model: ✅. See AI prompt generation.
A single LLM Configuration can serve any of these. For example, you might add gpt-4o once and use it as a test model in Project A and as a judge in Project B.
Don’t use the same model as both test and judge in the same project. It’s biased — the model will tend to grade itself favorably. Pick judges from different model families.

​
Custom providers

If your organization runs a self-hosted model or uses a provider Alethia doesn’t natively support, choose Custom. You’ll need: Use Test Connection to verify before saving.

​
Where do my API keys live?

API keys are encrypted with AES-256-GCM before being written to the database. They’re never returned in any API response or page — anywhere you’d see a key, it shows as sk-...**** instead. The only place a key is decrypted is in memory, momentarily, when Alethia is about to call the provider. It’s never written to logs or shown in the UI.
Best practice: create a dedicated API key just for Alethia at your provider, separate from your production key. That way you can rotate it without affecting other systems, and you can see Alethia’s usage cleanly in your provider’s dashboard.

​
Editing or rotating a key

1
Open the configuration from Organization → LLM Configurations.
2
Click Edit.
3
Paste the new key. The old encrypted value is replaced.
4
Click Test Connection to verify the new key works.
You don’t need to update any project — they reference the configuration by name, not by key.