Scopehaven
Request a first test
ACCESS REMOVAL GUIDE
How to test what an AI agent does after you remove a user's access
Removing access happens at one moment, but an agent's work takes time. A run that started a minute before a user was suspended may still be planning, waiting for approval or about to call a tool. Test that the change reaches all of that work, not only the user's next request.
Short answer: start an agent run as a test user, remove their access partway through, and check whether anything the run does afterwards takes effect: tool calls, writes, queued jobs, cached reads. Then confirm that a run for a user whose access did not change still completes.
This guide is for teams whose agents run multi-step tasks, background jobs or approval flows on behalf of signed-in users. It covers testing in an authorized local or staging environment with synthetic data. It is not a full security review.
01 / WHERE
Five places old access can survive.
- In-flight runs. An agent loop that checked permissions once, at the start, keeps acting on that decision.
- Paused and queued actions. An action waiting for approval, a retry or a scheduled job commits after the change.
- Caches. A cached record or search result is served after the user lost the right to read it.
- Sessions and tokens. A session, delegated token or service credential the agent holds for the user stays valid.
- Earlier approvals. An approval issued before the change still authorizes the action after it.
02 / DEFINE
Decide when removal must take effect.
Write the rule down before testing. Does removal stop the next tool call, the next step, or only new requests? Must a paused action fail, or wait for a fresh approval? Must an export already in progress stop? More than one answer can be reasonable; a rule nobody wrote down cannot be tested.
Then choose the access changes to test: a suspended account, a removed role, a user moved to another team or customer account, a disconnected integration. Use the same path production uses to make the change, such as the admin API or the identity provider's test tenant.
03 / CHECK
Check old work, not only new requests.
Pair each forbidden check with an allowed-use control, so a repair that stops everyone's work does not look like a pass. This table is a starting point, not a complete suite.
| Situation | Forbidden check | Allowed-use control |
|---|---|---|
| New request | After suspension, the user's next request is refused. | Users who were not suspended still get answers. |
| In-flight run | A run started before the suspension tries to write afterwards. The write does not happen. | A run whose user keeps access completes and writes once. |
| Paused action | An action paused for approval before the user's role was removed does not commit once it is approved. | A paused action whose user's access is unchanged still commits once approved. |
| Cached read | A record the user read before losing access is not served again from a cache. | Users whose access did not change still get cached results. |
| Delegated tokens | A token issued to the agent on that user's behalf stops working, or expires within the window your rule allows. | Tokens held for other users keep working. |
| Account move | After moving to another customer account, the user cannot reach the old account's records. | They can read records in the new account. |
04 / TIME
Hit the gap between deciding and acting.
A test that removes access and then starts a new run covers the easy case. The harder case is a run that has already decided to act: it has read a record and not yet written, or its action is waiting for approval.
Hold the run at that point with a test hook, a slowed tool or a manual approval step. Change the access, release the run, then read the effect stores, and wait long enough for queued work to settle before you decide. Record when the access changed and when each effect landed, so the result can be reviewed later.
05 / RETEST
Check access at each action, then rerun.
Typical repairs check the user's current access inside the tool immediately before each effect commits, ideally in the same transaction, rather than once per run. They also invalidate caches and tokens when access changes, and re-check a paused action's access and approval when it commits. Your team makes the repair.
Rerun the same timed checks with the same identities. Removal should now stop the pending actions, and the allowed-use controls should still complete.
06 / PILOT
What an internal pilot found.
In an internal pilot, we used Scopehaven's approach on a pinned version of an open-source AI workspace on our own machine, with its real server, a local model and synthetic data. We suspended a test user. New requests from that user were correctly refused, but in 2 of 14 test cases an agent run started before the suspension still wrote a file afterward. We made a partial, local repair, and the same 14 checks then passed, including the ones confirming that normal use still works. We verified the repair only on the file-write path we tested; other tools and full session termination were not tested.
We're not naming the project. This is one finding in one pinned version, reproduced and partly repaired on our machine with synthetic data. It is not a claim about the project's current release. It does show why the in-flight case needs its own check: the new-request check alone would have passed.
Where Scopehaven fits
Scopehaven is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Its built-in demo app, which uses synthetic data, includes a check that revocation takes effect before cached access, and a mid-task check in which a paused refund must not commit after the user's access is removed, next to a control in which the user's access is unchanged and the paused refund still commits.
On your own app, what a run can establish depends on the connection and the evidence available. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions
Isn't ending the user's session enough?
It stops new requests on that session. Work already running, queued jobs, cached data and tokens issued to the agent can each outlive it, so test them separately.
How quickly should removal take effect?
That is a policy choice. Possible rules include stopping the next tool call, or letting a read finish while refusing any new write. Write the rule down, then test against it.
Do I need a real identity provider to test this?
Not necessarily. A local copy with test accounts works if access is removed through the same path production uses. If production removes access through an identity provider, use its test tenant.
Does a passing test mean revocation works everywhere?
No. It shows that the paths and timings you tested held. Other tools, background jobs and session types remain open questions until they are tested.