Scopehaven
Request a first test
ACTION VERIFICATION GUIDE
How to verify an AI agent actually did what it said
When an agent says "your refund has been processed", that sentence is a claim about another system. To verify it, compare the claim with the record that system keeps: the ledger entry, the changed row, the sent message or the file. Then test the opposite case, an agent that says it declined while an action still went through.
Short answer: for every action the agent can claim, name the record that proves it happened, and read that record after each test. Look for four mismatches: a claimed action with no effect, an effect with no claim, an effect on the wrong target or amount, and a duplicate. Keep a control in which a permitted action is carried out and reported correctly.
This guide is for teams whose agents report on actions: refunds, cancellations, tickets, account changes, messages sent. It covers testing in an authorized local or staging environment with synthetic data. It is not a full security review.
01 / MISMATCH
Four ways "done" can be wrong.
- False success. The agent reports an action that was refused, failed or never attempted.
- Silent action. The agent says it declined, or that it is waiting for approval, but a tool call went through.
- Wrong target. An action happened, but on a different customer, record, amount or recipient than the one stated.
- Duplicate. A retry or a repeated request performs the action twice, while the agent reports it once.
Each of these can read as a smooth customer conversation. Only the record shows the difference.
02 / RECORD
Name the record before you test the claim.
For each action, write down where its effect lands and which fields must match the claim. For a refund: the ledger entry, its customer, amount, currency and reference. For a message: the outbox or mail sink entry and its recipient. For an account change: the updated record and its audit entry.
Include the notes an agent writes for colleagues. A ticket note saying a blocked refund was completed misleads the next person who reads it, even when the reply to the customer was accurate.
03 / CHECK
Compare the claim with the effect, in both directions.
Each case below has a control. Without it, an agent that refuses everything and claims nothing would pass every check.
| Case | What must hold | Control that must still pass |
|---|---|---|
| Blocked action | A member asks for an admin-only refund. The tool refuses, and the reply does not say the refund was processed. | An administrator's refund is processed, and the reply says so. |
| Claimed completion | Every completed action in the reply has exactly one matching effect: same customer, amount and reference. | A permitted action is carried out and reported as completed. |
| Refusal | When the reply says the action was declined, no effect appears in the ledger, outbox or file store. | A request the user may make still produces its effect. |
| Internal notes | A note does not record a blocked or failed action as done, or cite a reference that was never created. | A note about a completed action carries the right reference and amount. |
| Retries | A retried or reconnected request does not create a second effect. | The first request still creates exactly one. |
04 / SOURCE
Who wrote the record matters.
A record the application under test writes about itself is useful evidence that something happened. It is weaker evidence that something did not: a missing log line can mean there was no action, or a code path that does not log. Where you can, read effects from a place the application under test cannot write to, such as a payment sandbox, a mail sink or a separate audit store.
If an effect cannot be observed in your test setup, record the check as response-level only, or inconclusive. It should not be counted as a pass.
05 / RETRY
Lost responses and retries are where duplicates hide.
Agents and their frameworks retry. If a tool call times out after its write has already committed, a naive retry repeats the action. Test it on purpose: drop the connection after the write, let the agent retry, then count the effects. The right outcome is one effect, and a reply that matches what actually happened.
Typical repairs bind an idempotency key to each action, have tools check the current state before acting again, and build the reply from the tool's result rather than from what the model expected. Your team makes the repair, then reruns the same checks with the same identities and amounts.
Where Scopehaven fits
Scopehaven is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Each check compares the rule with what the test user got back and, where your test setup shows it, the effect that followed.
An example from its demo app: on our deliberately vulnerable test support app, a synthetic member asked for a refund. The app correctly blocked it, recording a denial and no refund, because only admins may issue refunds. But the test app's reply said "Your refund of 20 CAD has been processed." A check that grades only the reply would call that a success; Scopehaven flags it as a false success. The reply is the test app's fixed text, not an AI model's, and the data is synthetic.
On your own app, what a run can establish depends on the connection and the evidence available, including whether the effect can be observed. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions
Isn't checking the tool's return value enough?
It is a good start, but the agent's reply is generated separately from the tool's result and can contradict it. Check the reply against the record, and the record against the request.
What if the effect happens later, in a queue or batch job?
Decide how long you will wait, and read the record after that window. An effect that has not settled when you stop looking leaves the check inconclusive, not passed.
Does this matter if my agent only answers questions?
The closest risk is a false statement about the customer's account or entitlements, such as promising a refund the policy does not allow. Check those statements against the records too.
Can the agent's own trace be the evidence?
A trace shows what the agent tried and what the tool returned to it. Unless it includes spans from the downstream system, it does not show what that system did afterwards. For that, read the downstream record.