Skip to main content
This guide walks you through your very first safety test, end to end. By the time you finish, you’ll have:
  • A team set up
  • An LLM configured (the model you want to test)
  • A judge model added
  • Your first safety test result on screen
You’ll need at least one API key for the LLM you want to test. Most providers (OpenAI, Anthropic, Google, etc.) give you a key in your account settings.

​
Step 1 — Create your account

1

Sign up

Go to your Alethia login page and click Register. Use your work email.
2

Verify your email

Click the confirmation link in your inbox.
3

Create your organization

Give your organization a name (e.g., “Acme Corp”). This is the top-level container for everything you do in Alethia.
Your organization is automatically given its first team. You can rename it or create more teams later under Organization → Teams.

​
Step 2 — Add an LLM you want to test

This is the model whose safety you’re going to evaluate.
1

Open LLM Configurations

From the sidebar, go to Organization → LLM Configurations.
2

Click 'Add Configuration'

Pick the provider (OpenAI, Anthropic, Google, Mistral, DeepSeek, or Custom).
3

Fill in the details

  • Name: Something memorable, like “GPT-4o (production)”
  • Model: The exact model identifier from your provider (e.g., gpt-4o, claude-sonnet-4)
  • API Key: Paste your API key (it gets encrypted before being saved)
  • Mark as test model: ✅ Yes
4

Test the connection

Click Test Connection. A green check means Alethia can reach your model.

​
Step 3 — Add at least one judge

Judges are the AI models that will grade whether the test response is safe or unsafe. You need at least three judges to use the default consensus method, but you can start with one to try things out. Repeat Step 2, but this time:
  • Mark as test model: ❌ No
  • Judge prompt: Leave blank to use Alethia’s default judge prompt (recommended)
Use a different model family for your judges than your test model. If you’re testing GPT-4o, use Claude or Gemini as judges. This avoids “self-grading” bias.

​
Step 4 — Create your first project

A project is a focused testing workspace tied to one specific model.
1

Click 'New Project'

From the sidebar, go to Projects → New Project.
2

Name your project

Example: “GPT-4o safety check — Q2 2026”
3

Pick the target model

Select the LLM you added in Step 2.
4

Pick your judges

Select up to three judge models from Step 3.
5

Choose a voting method

Leave it as Majority for now — the safest default.
6

Create the project

You’ll land on the project overview.

​
Step 5 — Run your first test

Your team comes pre-loaded with a default prompt library of 18 categories and 137 subcategories. You don’t need to write any prompts to start.
1

Open Run Tests

Inside your project, click the Run tab in the sidebar.
2

Pick a single prompt

Browse the categories tree. Pick something simple to start, like a prompt under harmful_content → hate_speech.
3

Click 'Run Test'

Alethia will:
  1. Send the prompt to your test model
  2. Capture the response
  3. Send the response to your judges
  4. Calculate a final verdict
You’ll see this happen live, usually in under 30 seconds.

​
Step 6 — Read the result

You’ll see something like: If you disagree with the verdict, you can override it as a human. Learn more in Human Overrides.

​
What’s next?

Run a full category

Test dozens of prompts at once.

Compare models

Test two models against the same prompts.

Schedule tests

Run safety checks every week automatically.

EU AI Act

Get fully compliant with the new regulation.