Skip to main content
The Comparison Engine lets you put two or more projects next to each other and see exactly how the models compare on safety. This is the primary way to answer questions like:
  • “Is GPT-4o safer than Claude for our use case?”
  • “Did the new fine-tune actually improve over the old one?”
  • “Has Gemini’s safety drifted between January and April?”

​
Why projects matter for comparison

Comparison only works because of how Alethia is structured:
  • All projects in a team share the same Prompt Library.
  • Each project targets a single, specific model.
So if Project A targets GPT-4o and Project B targets Claude, both projects have run the same prompts. That’s a fair fight — and the comparison results are meaningful.
You can’t compare projects from different teams. Different teams have different prompt libraries, so the comparison wouldn’t be apples-to-apples.

​
Running a comparison

1

Open the Compare tab

From the left sidebar.
2

Select 2 or more projects

They must be in the same team. You can compare up to 5 projects at once.
3

Apply optional filters

Narrow the comparison to specific:
  • Categories — only compare on bias_and_fairness, for example
  • Severity — only high and critical prompts
  • Date range — only test runs from a given window
4

Click 'Compare'

The comparison loads in a few seconds.

​
Reading the comparison

You’ll see three main sections:

​
1. Summary

A side-by-side card for each project showing:
Always look at Safe rate, not just absolute counts. If Project A ran 1000 tests and Project B ran 100, the absolute “unsafe” count isn’t comparable — but the rate is.

​
2. Category breakdown

A table showing the safe rate per category for each project. This is where you’ll find the most interesting differences:

​
3. Visual charts

  • Safe rate by category — grouped bar chart
  • Verdict distribution — stacked bar chart per project
  • Confidence histograms — see how confident judges were across each project
You can hover any chart segment to see exact values.

​
Important: effective verdicts

Comparison numbers always use the effective verdict — the human override if there is one, otherwise the AI consensus verdict. This means human review work is reflected in your comparisons. If you want to see the difference, the test detail panel shows both verdicts side by side.

​
Useful comparison patterns

A vs B

Compare two competitor models on the same prompts. Best for vendor selection.

Old vs New

Compare a project from before a model update against one from after. Catches regressions.

Baseline vs Mitigation

Compare a project before and after applying a safety mitigation. Quantifies the improvement.

Production vs Fine-tune

Compare your hosted model against a custom fine-tune to validate the fine-tune doesn’t make things worse.

​
Exporting comparison results

Click Export to download:
  • CSV — flat data, one row per category × project
  • JSON — full structured data
  • PDF — a formatted comparison report (good for sharing with stakeholders)

​
Comparing across long time spans

If you’ve been running scheduled tests for several months and want to compare the same project against itself over time, use the date range filter instead of comparing two projects:
  • Set the filter to “Jan 1 – Mar 31” → see Q1 results
  • Set the filter to “Apr 1 – Jun 30” → see Q2 results
  • Bookmark each filtered view to share or revisit
This is the cleanest way to track a single model’s safety trend over time.