Skip to main content
Once your project has a target model, judges, and prompts, you’re ready to test. Alethia supports two ways to run:

Single Test

Pick one prompt and run it. Result returns in seconds. Perfect for quick checks.

Category Test

Pick an entire category. Alethia runs every prompt in it, streaming results live.

​
Running a single test

1

Open the Run tab

Inside your project, click Run.
2

Browse the prompt tree

Drill down: category → subcategory → individual prompt.
3

Click 'Run Test'

The test starts. You’ll see a live progress bar:
  1. Sending prompt to your test model
  2. Capturing the response
  3. Sending the response to each judge (in parallel)
  4. Calculating the consensus verdict
4

Read the result

The full result appears on the same screen — verdict, confidence, judge breakdown, and reasoning.
A single test usually completes in 10–40 seconds, depending on the providers’ latency.

​
Running a category test

Want to test a model against an entire category — say, all 9 bias_and_fairness prompts at once? Use a category test.
1

Open the Run tab

Same as a single test.
2

Click on a category (not a single prompt)

Then click Run Category.
3

Confirm the run

Alethia shows you exactly how many prompts will run and (if you’re using token-billed judges) the estimated cost.
4

Watch results stream in live

Tests run sequentially to respect provider rate limits. The progress bar updates in real time, and each completed test appears in the results table immediately.
You can leave the page during a category test. The job runs in the background. When you come back to the Results tab, you’ll see everything that completed.

​
How the live updates work

Alethia streams test progress to your browser using Server-Sent Events. You’ll see events like:
  • 🚀 Test started — “Sent prompt to GPT-4o”
  • ⚖️ Judge complete — “Claude judged: safe (92%)”
  • ✅ Test complete — “Final verdict: safe (88%)”
  • 📊 Category progress — “15 of 100 prompts done”
This means no refreshing, no polling — you see what’s happening as it happens.

​
What happens behind the scenes

For each prompt, Alethia does this:
Read more in Multi-Judge Consensus.

​
Stopping a category test

If you need to stop a long-running category test:
1
Go to the Results tab.
2
Find the running job at the top of the list.
3
Click Cancel.
Tests already completed remain saved. Tests not yet started are skipped.

​
Tips for productive testing

Start with one category. Don’t run all 137 default prompts on day one — pick a category that matters most for your use case (e.g., professional_ethics for a healthcare chatbot) and start there.
Add severity tags to prompts. When you upload custom prompts, marking them as low / medium / high / critical makes filtering and risk scoring much more useful.
Don’t over-test. Each test costs API tokens at your provider. Use scheduled tests to run a regular, sampled subset weekly instead of the full library every day.

​
Common errors