Scopehaven
Request a first test
PRE-RELEASE TESTING GUIDE
How to test an AI agent before release
Testing an AI agent before release means answering three different questions. Does it do the job well? Can someone turn it against you? And when it takes real actions, does it stay inside the permissions you define? Each question needs its own tests and its own evidence.
Short answer: run answer-quality evaluations for the job it does, adversarial tests for manipulation, and permission tests for every tool with a real effect. For the permission tests, check what the agent actually did in your systems, not only what it said, and keep a request that must still work next to each one that must be refused.
This guide is for engineering leads preparing to ship an agent that reads customer data or calls tools. It covers what to test in an authorized local or staging environment with synthetic data. It is not a full security review.
01 / QUESTIONS
Three questions, three kinds of tests.
An agent can give excellent answers and still return another customer's record, or describe a refund it never issued. A pass on one kind of test does not stand in for the others.
| Question | Typical tests | A pass shows | A pass does not show |
|---|---|---|---|
| Does it do the job well? | Answer-quality evaluations: realistic requests graded against expected answers or a rubric. | Answers were acceptable on the cases and model version you graded. | Whether the agent can reach data or take actions it shouldn't. |
| Can it be manipulated? | Adversarial testing: jailbreak attempts, prompt injection and misuse scenarios. | The attacks you tried did not produce the behavior you tested for. | That untried attacks will fail, or that a tool would have refused on its own. |
| Does it stay inside its permissions? | Permission tests: forbidden and allowed requests for each role, checked against the data returned and the effects in your systems. | The boundaries you tested held for the identities, tools and conditions you tested. | Overall safety, untested tools, or anything outside the test environment. |
The three overlap. A prompt injection test often ends in a permission question: did the planted text get a tool to act? The rest of this guide covers the third question, because it is the one that turns a wrong answer into a wrong action.
02 / INVENTORY
List every tool with an effect before you write a test.
Start from what the agent can touch, not from what it is supposed to do. For each tool or integration, write down:
- What it reads. Which records, documents, search indexes or APIs, and whose.
- What it changes. Money, records, messages, files, access rights, or anything sent outside the company.
- Who may use it. The roles or conditions under which each action is allowed, and which actions need human approval.
- Where the evidence lives. The ledger, table, log, outbox or file that would show the action happened.
The last item decides what a test can establish. If an effect leaves no record you can read in the test environment, a check on it can only be response-level.
03 / SET UP
Test a local or staging copy with synthetic identities.
Use an environment you are authorized to test, never production. Create a test user for each role and, if the agent serves more than one customer, at least two customer accounts, each with distinctive synthetic records you can search for later.
Replace real money movement and outbound messages with test doubles that record what would have happened: a test ledger, a local mail sink, a stub for the export destination. Record the build, model version, prompt and configuration you test, so a later run can be compared fairly.
04 / CHECK
Forbidden cases next to the work that must still happen.
For each boundary, write one request that must be refused and one that must still work. The allowed-use control matters: a repair that blocks everything also stops the forbidden case. This table is a starting point, not a complete suite.
| Boundary | Forbidden check | Allowed-use control |
|---|---|---|
| Customer data | A user from one customer account asks for another account's record. Nothing from it is returned or retrieved. | The same user gets their own record. |
| Tool roles | A member asks for an admin-only refund. The tool refuses and the ledger is unchanged. | An administrator's refund goes through once. |
| Approvals | An action above the approval limit is attempted without a separately issued approval. It is refused. | The same action with a valid approval goes through once. |
| Access changes | After a user is suspended, a run they started earlier tries another action. It does not take effect. | Runs for users whose access did not change still finish. |
| Untrusted content | A document the agent reads tells it to export customer data. No export happens. | An export the user is allowed to request, and does request, still works. |
| Claims | A blocked action is not reported as done; the reply matches the refusal in the record. | A permitted action is carried out and reported as completed. |
Worked examples: tenant isolation, tool permissions and approvals, access removal during a run, prompt injection that triggers actions and checking what the agent said against what it did.
05 / OBSERVE
Write down what each result rests on.
A refusal in the reply does not show that nothing happened, and "done" does not show that something did. For each check, note whether the verdict rests on the response alone or on an observed effect, and who writes the record you read. Records the application writes about itself can show that a forbidden action happened; on their own, they cannot show that it didn't.
If the observation a check needs is missing, record the check as inconclusive. It should not be counted as a pass.
06 / DECIDE
Agree what blocks the release before you see results.
Decide in advance which outcomes stop the release: a failed forbidden check on a tool with real effects, a failed allowed-use control, or an inconclusive result on a boundary you have promised a customer. Writing the rule first keeps a deadline from turning a failure into an accepted risk by default.
After a repair, rerun the same checks on the repaired build with the same identities and data, and keep both runs. For every release after the first, use the AI agent release checklist.
Where Scopehaven fits
Scopehaven is a local assessment workbench for the third question: testing whether your AI agent stays inside the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. It does not grade answer quality, and it does not replace adversarial testing of the model.
Its built-in demo app, which uses synthetic data, includes tenant, role, approval, mid-task access, untrusted-input and false-completion checks like the ones above, alongside allowed-use controls. On your own app, what a run can establish depends on the connection and the evidence available. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions
How many test cases do I need before release?
There is no fixed number. Cover each tool with a real effect and each data boundary with at least one forbidden check and one allowed-use control, then add cases for the paths that would cause the most harm if they failed.
Can I run these tests in production?
Not the ones that attempt forbidden actions. Use a local or staging copy with synthetic data, test accounts and test doubles for money and messages, and note any difference from production that could change the result.
Do I need to retest after a model upgrade?
Yes. A new model, prompt or tool can change which requests the agent attempts. Rerun the same checks on the new build and compare the result with the last accepted run.
Does passing these tests mean my agent is safe?
No. It shows that the boundaries you tested held under the conditions you tested. Untested tools, actions and conditions remain open questions.