Scopehaven
Request a first test
RELEASE CHECKLIST
AI agent release checklist: what to retest when the model, prompt or tools change
An agent can change behavior without a line of your application code changing: a new model version, an edited system prompt, a new tool or a refreshed knowledge base. Treat each one as a release. Rerun the permission checks you already have, compare the result with the last accepted run, and block on regressions, including a fix that blocks normal work.
Short answer: before each release, list what changed, rerun the same forbidden checks and allowed-use controls on the release candidate with the same identities and data, and compare against the last accepted run. Block on any new failure, any failed allowed-use control, and any inconclusive result on a boundary you have promised a customer.
This checklist assumes you already have permission checks from a first test or an earlier release. If you don't, start with how to test an AI agent before release.
01 / CHANGES
Changes that call for a rerun.
- Model. A new model, version or provider, or changed settings such as temperature or reasoning effort.
- Instructions. The system prompt, tool descriptions or routing rules.
- Tools. A tool added or removed, or given new parameters, scopes or credentials.
- Permissions. New roles, plans, approval limits or tenant rules.
- Data. A new retrieval source, a rebuilt index or new document types.
- Framework. An agent framework or SDK upgrade, including changes to retries, memory or tool calling.
If you cannot say which of these changed, assume all of them did.
02 / PREPARE
Before the run.
- Record the release candidate. Build, model identifier, prompt version, tool list and configuration.
- Reuse the baseline's setup. The same test identities, customer accounts and synthetic data as the last accepted run.
- Add checks for anything new. Each new tool or permission gets at least one forbidden check and one allowed-use control.
- Keep the old checks. Removing a check is a coverage change. Record it rather than letting it disappear.
03 / COMPARE
Rerun the same checks and compare like with like.
Run the full set on the release candidate, then compare it with the last accepted run. A comparison means something only when the checks, rules, identities and evaluation are the same. If any of them changed, record it and treat the run as a new baseline rather than a pass.
| Result | What it means | Release decision |
|---|---|---|
| A forbidden check newly fails | A boundary that held before does not hold now. | Block. |
| An allowed-use control fails | Normal work broke, often because a change blocks too much. | Block. |
| A check is inconclusive | The evidence the check needs was not available. | Block if the boundary is one you have promised; otherwise record it. |
| A failing check now passes | A possible repair. | Count it only if the allowed-use controls still pass. |
| No change | Every check has the same result as the last accepted run. | Proceed, with the coverage limits recorded. |
Model output varies from run to run. On the paths that matter most, run each check more than once: a single pass on a path that can vary is weaker evidence than several.
04 / GATE
Write the release gate before the results arrive.
Agree the blocking rules in advance, and apply them automatically where you can, for example as a required check in your deployment pipeline. If a deadline can override the gate without anyone recording the decision, it is only a recommendation.
Keep each release's run, comparison and decision together, so a customer's security reviewer can see what was tested on the version they use.
05 / AFTER
After release.
- Keep the accepted run. It is the baseline for the next change.
- Record what was not covered. For example, tools whose effects you could not observe, or paths you could not test.
- Turn reports into checks. When someone reports a permission problem, write a check for it before the fix ships, then keep that check in the set.
Where Scopehaven fits
Scopehaven is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Scopehaven runs allowed-use controls alongside forbidden checks, so a fix that over-blocks a covered workflow fails instead of passing. A run is never labelled fixed when the run, suite, policy or evaluator identities are incompatible. Saved runs can be exported as JSON, HTML or SARIF; formats organize the evidence but do not expand what was tested.
On your own app, what a run can establish depends on the connection and the evidence available. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions
Do I need to rerun everything for a small prompt change?
Rerun at least the checks for data boundaries and for tools with real effects. A small prompt change can still change which tools the model calls, and with which arguments.
What if the provider updates the model behind the same name?
Record whatever model identifier the provider returns, and rerun on a schedule as well as on your own releases. If you cannot pin the version, note that the result applies to the model that served the run.
How do I keep this from slowing every release?
Keep a fast core set for every change and run the full set for model, tool and permission changes. Automate the comparison, so people review only failures, failed controls and inconclusive results.
Does a clean comparison mean the release is safe?
No. It means the boundaries you tested held as before, under the conditions you tested. Untested tools, actions and conditions remain open questions.