Skip to main content
A single AI deciding whether another AI is safe or unsafe is risky — judges have biases, blind spots, and bad days. Alethia uses a panel of three judges instead, then combines their votes into a single verdict. This page explains how the system works, the three voting methods, and how to choose between them.

​
The three judges

Each project picks up to three judge models during configuration. Best practice:
  • Use different model families (e.g., Claude + Gemini + DeepSeek), not three from the same vendor
  • Don’t reuse the test model as a judge — that’s self-grading
  • Keep the default judge prompt unless you have a specific reason to customize it
Each judge independently returns:

​
The three voting methods

When you configure your project, you pick one of these:
Rule: 2 of 3 judges must agree on the verdict. The verdict they agree on becomes the final verdict.Confidence: The average of the agreeing judges’ confidence scores.Use when: You want a balanced, robust default that handles a single judge being wrong.→ Final: Safe (87.5%) — Claude and Gemini agree.

​
Which method should I use?

Just starting out?

Use Majority. It’s the most balanced default and works well in 90% of cases.

High-stakes decisions?

Use Unanimous. Anything inconclusive lands on a human reviewer for a final call.

Lots of low-confidence noise?

Use Weighted. Soft votes don’t drown out a strong, confident vote.
You can change the voting method on a project at any time from Project → Configure.

​
Minimum confidence threshold

Each project also has a minimum confidence threshold (default: 70%). When a verdict’s confidence falls below this number, Alethia flags it in the UI as a low-confidence verdict — a hint that a human should take a look.

​
What if a judge fails?

Sometimes a judge call fails (network error, malformed response, rate limit). Alethia handles this gracefully:
  • If at least 2 of 3 judges return successfully, the verdict is calculated normally with the available judges.
  • If fewer than 2 judges return, the test run is marked failed and you can re-run it.
  • You’re never charged for failed judge evaluations on team-provided models — only successful ones count.

​
Calibration with rubrics

Behind the scenes, every judge call includes a judge rubric — a structured set of criteria that helps the judge decide what counts as safe vs unsafe for that specific category. Rubrics are seeded by Alethia and aligned with EU AI Act safety domains. They’re what keep judges from making up their own definitions of “safe.” You don’t see rubrics in the UI directly — they’re applied automatically based on the prompt’s category. But it’s part of why the judges’ verdicts tend to converge: they’re all working from the same definition.

​
A note on bias

Even with three judges, AI judging has limits. The platform is designed to make this transparent:
  • Every judge’s individual verdict and reasoning is shown — you can spot when one model has a systematic bias
  • Humans can override anything — and every override is recorded with the user’s name and reasoning (learn more)
  • You can change judges over time and re-run tests to see if results shift
Trust, but verify. Treat the consensus verdict as a strong signal — not a final authority.