Skip to main content

​
Getting started

No. Alethia is a web application — everything is done through the UI. The only “technical” thing you need to do is paste an API key from the LLM provider you want to test (OpenAI, Anthropic, etc.).
Two costs to think about:
  1. The API cost at your provider — every test sends one prompt to your test model and three prompts to your judges. The cost is whatever those four calls cost at your provider’s pricing. Usually a few cents per test.
  2. Alethia’s plan cost — depends on your plan. The free tier is generous for getting started.
For team-provided judge models (where Alethia handles the API costs), you pay in tokens — see Token Usage & Billing for details.
Out of the box: OpenAI, Anthropic, Google, Mistral, DeepSeek. Anything else (custom hosting, your own fine-tunes, less common providers) works through the Custom provider — you give Alethia the endpoint URL, headers, and JSON paths to use.
No. If your AI use case is low-risk (a spam filter, a video game NPC), you can skip the risk module entirely. It’s there for the people who need it.

​
Testing

A single judge can be wrong, biased, or have a bad day. Three judges give you a robust verdict that doesn’t depend on any one model. See Multi-Judge Consensus.
Technically yes, but it’s a bad idea. Models tend to grade themselves favorably. Always use judges from different model families than your test model.
  • Unsafe — the consensus says the model’s response was unsafe.
  • Inconclusive — the judges couldn’t agree (under the Unanimous voting method) or the weighted votes canceled out (under Weighted).
Both deserve a human eye, but inconclusive results in particular often need an override one way or the other.
You can’t edit the AI verdict — that’s locked. But you can override it as a human, with a written reason. The override becomes the effective verdict from then on. The original AI verdict is still visible. See Human Overrides.
Deletion is restricted by design — it would break the audit trail. You can archive an entire project to remove it from active dashboards while keeping the data.

​
Prompt library

The categories ship empty by design — different organizations have very different testing needs (a healthcare AI tester needs different prompts than a finance AI tester). You add prompts via single entry or bulk JSON upload.Alethia provides starter prompt packs you can import — ask support if you’d like one for your domain.
Because it’s what makes model comparisons meaningful — same prompts, different models, fair comparison. If you really need a different prompt set, create a new team.
Not directly through the UI. But you can export prompts from one team as JSON and bulk-upload them to another team.

​
Risk management

  1. Classify the project (Risk Classification)
  2. Get the classification approved by a Team Admin
  3. Run a baseline category test on the most relevant prompts
  4. Use Suggest Risks to get a starter risk register
  5. Assess each risk with the test data
  6. Plan mitigations for elevated risks
  7. Schedule recurring tests to keep evidence flowing
The platform looks at:
  • The failure rate in the relevant category (more failures → higher likelihood)
  • The severity of the prompts that failed (high-severity prompts → higher impact)
  • The EU AI Act risk domain mapped to that category
All suggestions are pre-filled but fully editable — you have the final say.
No, and no platform can. Compliance is a determination made by your own internal compliance team and (where required) a notified body. Alethia gives you the tooling, evidence, and documentation you need to support that determination.

​
Account & teams

Yes. One Alethia account can be a member of multiple organizations and multiple teams within each. Use the team switcher at the top of the sidebar to move between them.
Your profile is removed from all teams. Anything you created (test runs, risk entries, overrides) stays — it’s reattributed to “Deleted User” so the audit trail isn’t broken.
  • Org Admin manages the whole organization — creating teams, inviting members across teams, but not billing.
  • Team Admin manages one specific team — its projects, members, API keys, prompts, and risk approvals.
A person can be both at once.

​
Security

Encrypted at rest with AES-256-GCM. Keys are never shown again in the UI — they always appear masked (e.g., sk-...****). The only place they’re decrypted is briefly in memory when Alethia is calling the provider.
No. Projects are team-scoped, and team members can’t see other teams’ projects unless they’ve explicitly been added as a member.
30 days, with sliding expiration (each request resets the clock). You can revoke any active session at any time from your Account Settings → Security.

​
Billing

Org Owners and Admins can see usage at Organization → Billing. Token transactions for team-provided judge models are itemized.
Tokens are reserved before each test starts. If your balance is too low, the test is blocked before it begins — no surprise overdrafts. You’ll see a clear error message and can top up.

​
Still stuck?

Glossary

Look up unfamiliar terms.

Troubleshooting

Specific error messages and fixes.
If you can’t find your answer, email support@alethia.ai with as much detail as you can — error messages, what you were trying to do, and a screenshot if relevant.