> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alethiaai.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Comparing Models Side by Side

> Pick two or more projects and see exactly how their safety profiles differ.

The Comparison Engine lets you put two or more projects next to each other and see exactly how the models compare on safety. This is the primary way to answer questions like:

* "Is GPT-4o safer than Claude for our use case?"
* "Did the new fine-tune actually improve over the old one?"
* "Has Gemini's safety drifted between January and April?"

## Why projects matter for comparison

Comparison only works because of how Alethia is structured:

* All projects in a team share the **same Prompt Library**.
* Each project targets a **single, specific model**.

So if Project A targets GPT-4o and Project B targets Claude, both projects have run the same prompts. That's a fair fight — and the comparison results are meaningful.

<Warning>
  **You can't compare projects from different teams.** Different teams have different prompt libraries, so the comparison wouldn't be apples-to-apples.
</Warning>

## Running a comparison

<Steps>
  <Step title="Open the Compare tab">
    From the left sidebar.
  </Step>

  <Step title="Select 2 or more projects">
    They must be in the same team. You can compare up to 5 projects at once.
  </Step>

  <Step title="Apply optional filters">
    Narrow the comparison to specific:

    * **Categories** — only compare on `bias_and_fairness`, for example
    * **Severity** — only `high` and `critical` prompts
    * **Date range** — only test runs from a given window
  </Step>

  <Step title="Click 'Compare'">
    The comparison loads in a few seconds.
  </Step>
</Steps>

## Reading the comparison

You'll see three main sections:

### 1. Summary

A side-by-side card for each project showing:

| Metric                           | What it means                                     |
| -------------------------------- | ------------------------------------------------- |
| **Total tests**                  | How many test runs are included in the comparison |
| **Safe / Unsafe / Inconclusive** | Counts                                            |
| **Safe rate**                    | Percentage of tests with `safe` effective verdict |
| **Human overrides**              | How many tests have been overridden by a human    |

<Tip>
  Always look at **Safe rate**, not just absolute counts. If Project A ran 1000 tests and Project B ran 100, the absolute "unsafe" count isn't comparable — but the rate is.
</Tip>

### 2. Category breakdown

A table showing the safe rate per category for each project. This is where you'll find the most interesting differences:

```
                 GPT-4o    Claude    Δ
─────────────────────────────────────
harmful_content   95%      98%      +3
bias_and_fairness 78%      92%     +14   ← Claude wins big here
child_safety      99%      99%       0
adversarial_test  82%      71%     -11   ← GPT-4o wins here
```

### 3. Visual charts

* **Safe rate by category** — grouped bar chart
* **Verdict distribution** — stacked bar chart per project
* **Confidence histograms** — see how confident judges were across each project

You can hover any chart segment to see exact values.

## Important: effective verdicts

Comparison numbers always use the **effective verdict** — the human override if there is one, otherwise the AI consensus verdict. This means human review work is reflected in your comparisons.

If you want to see the difference, the test detail panel shows both verdicts side by side.

## Useful comparison patterns

<CardGroup cols={2}>
  <Card title="A vs B" icon="left-right">
    Compare two competitor models on the same prompts. Best for vendor selection.
  </Card>

  <Card title="Old vs New" icon="clock-rotate-left">
    Compare a project from before a model update against one from after. Catches regressions.
  </Card>

  <Card title="Baseline vs Mitigation" icon="shield-check">
    Compare a project before and after applying a safety mitigation. Quantifies the improvement.
  </Card>

  <Card title="Production vs Fine-tune" icon="microchip-ai">
    Compare your hosted model against a custom fine-tune to validate the fine-tune doesn't make things worse.
  </Card>
</CardGroup>

## Exporting comparison results

Click **Export** to download:

* **CSV** — flat data, one row per category × project
* **JSON** — full structured data
* **PDF** — a formatted comparison report (good for sharing with stakeholders)

## Comparing across long time spans

If you've been running scheduled tests for several months and want to compare *the same project against itself over time*, use the **date range filter** instead of comparing two projects:

* Set the filter to "Jan 1 – Mar 31" → see Q1 results
* Set the filter to "Apr 1 – Jun 30" → see Q2 results
* Bookmark each filtered view to share or revisit

This is the cleanest way to track a single model's safety trend over time.