> ## Documentation Index
> Fetch the complete documentation index at: https://docs.alethiaai.app/llms.txt
> Use this file to discover all available pages before exploring further.

# Running Tests

> Run a single test or an entire category and watch results stream in live.

Once your project has a target model, judges, and prompts, you're ready to test. Alethia supports two ways to run:

<CardGroup cols={2}>
  <Card title="Single Test" icon="play">
    Pick one prompt and run it. Result returns in seconds. Perfect for quick checks.
  </Card>

  <Card title="Category Test" icon="layer-group">
    Pick an entire category. Alethia runs every prompt in it, streaming results live.
  </Card>
</CardGroup>

## Running a single test

<Steps>
  <Step title="Open the Run tab">
    Inside your project, click **Run**.
  </Step>

  <Step title="Browse the prompt tree">
    Drill down: category → subcategory → individual prompt.
  </Step>

  <Step title="Click 'Run Test'">
    The test starts. You'll see a live progress bar:

    1. Sending prompt to your test model
    2. Capturing the response
    3. Sending the response to each judge (in parallel)
    4. Calculating the consensus verdict
  </Step>

  <Step title="Read the result">
    The full result appears on the same screen — verdict, confidence, judge breakdown, and reasoning.
  </Step>
</Steps>

A single test usually completes in **10–40 seconds**, depending on the providers' latency.

## Running a category test

Want to test a model against an entire category — say, all 9 `bias_and_fairness` prompts at once? Use a category test.

<Steps>
  <Step title="Open the Run tab">
    Same as a single test.
  </Step>

  <Step title="Click on a category (not a single prompt)">
    Then click **Run Category**.
  </Step>

  <Step title="Confirm the run">
    Alethia shows you exactly how many prompts will run and (if you're using token-billed judges) the estimated cost.
  </Step>

  <Step title="Watch results stream in live">
    Tests run **sequentially** to respect provider rate limits. The progress bar updates in real time, and each completed test appears in the results table immediately.
  </Step>
</Steps>

<Tip>
  You can leave the page during a category test. The job runs in the background. When you come back to the **Results** tab, you'll see everything that completed.
</Tip>

## How the live updates work

Alethia streams test progress to your browser using **Server-Sent Events**. You'll see events like:

* 🚀 **Test started** — "Sent prompt to GPT-4o"
* ⚖️ **Judge complete** — "Claude judged: safe (92%)"
* ✅ **Test complete** — "Final verdict: safe (88%)"
* 📊 **Category progress** — "15 of 100 prompts done"

This means no refreshing, no polling — you see what's happening as it happens.

## What happens behind the scenes

For each prompt, Alethia does this:

```
   Step 1   ─►  Send prompt to your TEST MODEL
                Capture the response

   Step 2   ─►  Send the [prompt + response] to JUDGE 1, JUDGE 2, JUDGE 3
                (all three in parallel)

   Step 3   ─►  Each judge returns: verdict + confidence + reasoning

   Step 4   ─►  Apply your project's voting method
                Produce a single FINAL VERDICT and CONFIDENCE

   Step 5   ─►  Save the test run + each judge evaluation
                (immutable record — kept for audit)

   Step 6   ─►  (optional) Generate a RISK SIGNAL if the verdict is unsafe
                — flags the relevant EU AI Act risk domain for review
```

Read more in [Multi-Judge Consensus](/testing/consensus-and-judges).

## Stopping a category test

If you need to stop a long-running category test:

<Steps>
  <Step>Go to the **Results** tab.</Step>
  <Step>Find the running job at the top of the list.</Step>
  <Step>Click **Cancel**.</Step>
</Steps>

Tests already completed remain saved. Tests not yet started are skipped.

## Tips for productive testing

<Tip>
  **Start with one category.** Don't run all 137 default prompts on day one — pick a category that matters most for your use case (e.g., `professional_ethics` for a healthcare chatbot) and start there.
</Tip>

<Tip>
  **Add severity tags to prompts.** When you upload custom prompts, marking them as `low / medium / high / critical` makes filtering and risk scoring much more useful.
</Tip>

<Tip>
  **Don't over-test.** Each test costs API tokens at your provider. Use [scheduled tests](/testing/schedules) to run a regular, sampled subset weekly instead of the full library every day.
</Tip>

## Common errors

| Error                | What it means                                     | Fix                                                                                |
| -------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `INVALID_API_KEY`    | The key for that LLM no longer works              | Edit the LLM Configuration and paste a fresh key                                   |
| `RATE_LIMITED`       | Provider is throttling                            | Alethia auto-retries with backoff. If it persists, slow down by spacing out tests. |
| `MODEL_UNAVAILABLE`  | Provider is having an outage                      | Check the provider's status page                                                   |
| `MALFORMED_RESPONSE` | A judge returned something that wasn't valid JSON | Usually self-resolving. If it keeps happening, change the judge model.             |
| `TIMEOUT`            | The provider took longer than 60 seconds          | Try again. If consistent, the model is overloaded.                                 |