The Alethia loop
1
Add a model
You connect any LLM you want to test — OpenAI, Anthropic, Google, Mistral, DeepSeek, or a custom endpoint. API keys are encrypted before they ever touch the database.
2
Create a project
A project is a focused workspace for testing one specific model (e.g., “GPT-4o, Q2 2026”). You also pick the judges that will grade responses.
3
Pick prompts
Use the default Prompt Library (137 prompts across 18 safety categories) or upload your own. You can run a single prompt or an entire category at once.
4
Run tests
Alethia sends each prompt to your model, captures the response, and forwards it to three independent AI judges. Each judge grades the response as safe, unsafe, or partial.
5
Get a verdict
A consensus algorithm combines the three judges’ opinions into a final verdict and a confidence score.
6
Review and override
A human reviewer can override any verdict. Every override is permanently logged for auditability.
7
Document and report
For high-risk AI systems, the Risk Management module turns test results into a structured EU AI Act compliance report.
The big picture
Why three judges?
A single AI judge can be biased, miss nuance, or simply have a bad day. Alethia uses three independent judges from different model families by default. This gives you:- Robustness — one judge being wrong doesn’t change the verdict
- Transparency — you see how each judge reasoned, side by side
- Defensibility — when an auditor asks “how did you decide this was safe?”, you have three professional opinions instead of one
Where does EU AI Act compliance fit in?
If you’re testing a high-risk AI system under the EU AI Act, the platform layers a full risk management workflow on top of testing:1
Classify the system
Decide which EU AI Act tier the system falls under (Prohibited, High-Risk, Limited-Risk, or Minimal-Risk).
2
Maintain a risk register
List every identified risk. The system can suggest risks based on your test results.
3
Run risk assessments
Link test runs to specific risks, calculate failure rates, and document your findings.
4
Track mitigations
Record what you’ve done to reduce each risk, then re-test to verify it worked.
5
Generate a compliance report
Export a structured report aligned with EU AI Act Articles 9, 11, and 14.
Where does my data live?
- API keys — encrypted with AES-256 before being stored. Never returned in API responses (you’ll always see them masked, like
sk-...****). - Test results — scoped to the project they were run in. Other teams in your organization can’t see your project unless they’re members of your team.
- Audit logs — append-only. Nothing in the audit log can be edited or deleted, only added to.
Ready to try it?
Quickstart
Run your first test in under 10 minutes.
Core Concepts
Understand organizations, teams, and projects.