One-time scheduled test
Pick a single date and time in the future. Alethia runs the test then.
Recurring schedule
Pick a cadence (daily, weekly, monthly). Alethia keeps running the same test on that schedule until you stop it.
Why schedule tests?
- Catch regressions automatically. Providers update models without notice. A weekly run on the same prompts catches behavior changes the moment they happen.
- Build a compliance trail. A monthly recurring run on your high-risk prompts is exactly the kind of “continuous risk monitoring” the EU AI Act expects.
- Save engineering time. Set it once, forget it, review the results when you need them.
Creating a one-time scheduled test
1
Go to the Schedules tab
Inside your project.
2
Click 'Schedule a Test'
Choose One-time.
3
Pick what to run
A single prompt or an entire category.
4
Pick when
Date, time, and timezone.
5
Save
The schedule appears in the Pending list. You can edit or cancel it any time before it runs.
Creating a recurring schedule
1
Click 'Schedule a Test'
Choose Recurring.
2
Pick what to run
A single prompt or an entire category. Recurring schedules are great for category tests — they give you week-over-week coverage data.
3
Pick the cadence
- Daily — every N days at the time you choose
- Weekly — pick the days of the week
- Monthly — pick a day of the month (1–31; the schedule auto-skips months that don’t have that day)
4
Pick the time of day
24-hour format. Pick a timezone (defaults to UTC).
5
Set start and (optional) end
- Starts at — when the first run happens (defaults to now)
- Ends at — when to stop, or a maximum number of occurrences
6
Save
The schedule shows up in the Active list with the next run time.
Managing schedules
In the Schedules tab you can:- Pause / resume a recurring schedule (toggle the switch)
- Edit the cadence, time, or end conditions
- Cancel a one-time schedule before it runs
- View history — every run a recurring schedule has produced, linking back to the test results
What if I edit the prompt while a recurring schedule runs?
Each individual scheduled run uses the prompt as it exists at the moment of execution, not as it was when the schedule was created. So if you update a prompt’s content tomorrow, future scheduled runs will use the new content. Past test results are unaffected — they keep referencing the prompt as it was when they ran.What if a scheduled run fails?
Failed runs appear in the Results tab with a red status. The recurring schedule keeps going — a single failure doesn’t pause the schedule. Common failure reasons:- The provider’s API was down at the moment of execution
- The API key has been rotated and isn’t updated in the LLM Configuration
- A judge model has been deprecated
How does this fit with EU AI Act compliance?
Article 9 of the EU AI Act requires a “continuous, iterative risk management process”. Recurring scheduled tests are how you do this in practice:- A weekly run on safety-critical categories produces ongoing evidence of monitoring
- The test history (visible in Schedules → History) is the evidence trail
- Each run can feed into a periodic risk assessment automatically
- Daily smoke test on a small set of critical prompts
- Weekly full run of all categories tied to the system’s risk register
- Monthly post-assessment to summarize trends and update the risk register
Limits
If you hit a limit, contact support or upgrade your plan from Organization → Billing.