> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alethiaai.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Judge Consensus

> How Alethia combines three independent AI judges into one trustworthy verdict.

A single AI deciding whether another AI is safe or unsafe is risky — judges have biases, blind spots, and bad days. Alethia uses **a panel of three judges** instead, then combines their votes into a single verdict.

This page explains how the system works, the three voting methods, and how to choose between them.

## The three judges

Each project picks up to **three judge models** during configuration. Best practice:

* Use **different model families** (e.g., Claude + Gemini + DeepSeek), not three from the same vendor
* **Don't reuse the test model as a judge** — that's self-grading
* Keep the **default judge prompt** unless you have a specific reason to customize it

Each judge independently returns:

| Field          | What it means                                  |
| -------------- | ---------------------------------------------- |
| **Verdict**    | `safe`, `unsafe`, or `partial`                 |
| **Confidence** | A number from 0 to 100 — how sure the judge is |
| **Reasoning**  | A short text explanation of why                |

## The three voting methods

When you configure your project, you pick one of these:

<Tabs>
  <Tab title="Majority (default)">
    **Rule:** 2 of 3 judges must agree on the verdict. The verdict they agree on becomes the final verdict.

    **Confidence:** The average of the agreeing judges' confidence scores.

    **Use when:** You want a balanced, robust default that handles a single judge being wrong.

    | Judge    | Verdict | Confidence |
    | -------- | ------- | ---------- |
    | Claude   | Safe    | 90         |
    | Gemini   | Safe    | 85         |
    | DeepSeek | Unsafe  | 80         |

    → **Final: Safe (87.5%)** — Claude and Gemini agree.
  </Tab>

  <Tab title="Unanimous">
    **Rule:** All 3 judges must agree. Any disagreement → verdict is `inconclusive`.

    **Confidence:** Average of all three judges (when they agree).

    **Use when:** You need maximum certainty — e.g., for a high-risk medical or legal use case where any disagreement deserves human review.

    | Judge    | Verdict | Confidence |
    | -------- | ------- | ---------- |
    | Claude   | Safe    | 90         |
    | Gemini   | Safe    | 85         |
    | DeepSeek | Unsafe  | 80         |

    → **Final: Inconclusive** — judges disagree.
  </Tab>

  <Tab title="Weighted by Confidence">
    **Rule:** Each judge votes, but the size of their vote scales with their confidence. Safe = +1 × confidence. Unsafe = −1 × confidence. Sum the scores; positive → safe, negative → unsafe, zero → inconclusive.

    **Use when:** You want a low-confidence vote to count less than a high-confidence one.

    | Judge    | Verdict | Confidence | Weighted score |
    | -------- | ------- | ---------- | -------------- |
    | Claude   | Safe    | 90         | +0.90          |
    | Gemini   | Safe    | 85         | +0.85          |
    | DeepSeek | Unsafe  | 80         | −0.80          |

    Sum = +0.95 → **Final: Safe**
  </Tab>
</Tabs>

## Which method should I use?

<CardGroup cols={3}>
  <Card title="Just starting out?" icon="circle-play">
    Use **Majority**. It's the most balanced default and works well in 90% of cases.
  </Card>

  <Card title="High-stakes decisions?" icon="triangle-exclamation">
    Use **Unanimous**. Anything inconclusive lands on a human reviewer for a final call.
  </Card>

  <Card title="Lots of low-confidence noise?" icon="chart-mixed">
    Use **Weighted**. Soft votes don't drown out a strong, confident vote.
  </Card>
</CardGroup>

You can change the voting method on a project at any time from **Project → Configure**.

## Minimum confidence threshold

Each project also has a **minimum confidence threshold** (default: 70%). When a verdict's confidence falls below this number, Alethia flags it in the UI as a low-confidence verdict — a hint that a human should take a look.

## What if a judge fails?

Sometimes a judge call fails (network error, malformed response, rate limit). Alethia handles this gracefully:

* If **at least 2 of 3** judges return successfully, the verdict is calculated normally with the available judges.
* If **fewer than 2** judges return, the test run is marked `failed` and you can re-run it.
* **You're never charged** for failed judge evaluations on team-provided models — only successful ones count.

## Calibration with rubrics

Behind the scenes, every judge call includes a **judge rubric** — a structured set of criteria that helps the judge decide what counts as safe vs unsafe for that specific category. Rubrics are seeded by Alethia and aligned with EU AI Act safety domains. They're what keep judges from making up their own definitions of "safe."

You don't see rubrics in the UI directly — they're applied automatically based on the prompt's category. But it's part of why the judges' verdicts tend to converge: they're all working from the same definition.

## A note on bias

Even with three judges, AI judging has limits. The platform is designed to make this transparent:

* **Every judge's individual verdict and reasoning is shown** — you can spot when one model has a systematic bias
* **Humans can override anything** — and every override is recorded with the user's name and reasoning ([learn more](/testing/human-overrides))
* **You can change judges over time** and re-run tests to see if results shift

Trust, but verify. Treat the consensus verdict as a strong signal — not a final authority.