Scopehaven
Request a first test
LOCAL TESTING GUIDE
How to test an AI agent's permissions locally, without uploading your data
You can test whether an AI agent stays inside its permissions without sending your code, customer data or results to a testing service. Run a copy of the app on your own machine, give it synthetic data, catch everything it would send out, and read the results from its own records.
Short answer: run the app and its database on your machine or in a private staging environment, seed synthetic users and records, point email, webhooks and payment calls at local stand-ins, and read each result from the app's own records. If the app calls a hosted model, prompts still go to that provider, so use synthetic data, or a local model, when that matters.
This guide is for teams that cannot, or would rather not, share their app or data with an outside testing service. It covers an authorized local or staging copy, never production. It is not a full security review.
01 / INVENTORY
Know what could leave your machine.
Before the first test, list every way data can leave during a run:
- The model call. If the app uses a hosted model, each prompt, including any records retrieved into it, goes to that provider under its terms.
- Outbound effects. Email, webhooks, payment calls and other integrations the agent can trigger.
- Telemetry. Error reporting, analytics and tracing services the app already sends data to.
- The testing tool itself. Some testing services need a hosted endpoint, an account, or a copy of your traces.
For each one, decide whether to keep it, replace it with a local stand-in, or switch it off for the test, and write the decision down.
02 / SET UP
Run a local copy with synthetic data.
- Run the app on your machine. Use your normal build, from source or a container, with its own database. Never point a test at production or at a copy of production data.
- Seed synthetic identities. One test user per role and, if you serve more than one customer, at least two customer accounts, each with records that contain distinctive values you can search for later.
- Sign in the real way. Test users should authenticate the same way real users do, so the checks exercise your actual authorization code rather than a shortcut.
- Record the setup. Build, configuration, model, prompt and tool versions, so a later rerun can be compared with this one.
03 / STAND-INS
Catch everything the agent would send.
Local stand-ins keep effects on your machine and give you a record to read.
| Effect | Local stand-in | What it lets you check |
|---|---|---|
| A local mail catcher that stores messages instead of delivering them. | Whether a message was sent, to whom and with what content. | |
| Webhooks and outbound HTTP | A local receiver that logs each request. | Exports and forwards that would have left. |
| Payments and refunds | The payment provider's test mode, or a test ledger in your database. | Whether money would have moved, how much and how many times. |
| Files | A dedicated test folder. | Files written, changed or deleted. |
| Third-party APIs | The provider's sandbox, or a stub that records requests. | Which calls the agent made and with what arguments. |
If an effect cannot be replaced, block it during the test and record it as not observed, so a check that depends on it reads as inconclusive rather than as a pass.
04 / MODEL
Decide how to handle the model call.
- A local model keeps prompts on your machine. Checks on boundaries enforced in your code, such as tenant filters and role checks inside tools, still exercise that code. Model behavior differs from the one you ship.
- A hosted model with synthetic data uses the same model you ship, and nothing real is in the prompts, but the prompts do go to the provider.
Boundaries that live in your code should hold whichever model drives the agent. If a check passes only with one model, find out what the code is relying on. For behavior that depends on the model itself, such as how often planted instructions work, also test the model you ship.
05 / EVIDENCE
Read results from the app's own records.
For each check, keep the reply together with the records that settle it: database rows, the audit log, the test ledger, the mail catcher and the webhook log. With synthetic data, these records can be kept and shared inside your team without exposing customers.
Store runs where your other engineering records live, and keep the before and after runs of a fix side by side. If a check needs an effect you could not observe locally, record it as inconclusive.
Where Scopehaven fits
Scopehaven is a local assessment workbench for testing the permissions you define, on the actions you check, and rerunning the same checks to show whether a fix held, including the actions that should still work. Runs and evidence stay on your machine; the workbench does not upload them to Scopehaven.
On your own app, what a run can establish depends on the connection and the evidence available. Scopehaven does not certify your application or guarantee it is free from vulnerabilities.
Common questions
Will local tests miss production problems?
Some. Configuration, identity providers, data volume and network rules can differ. Local tests show whether the boundaries in your code hold for the checks you ran. Keep a list of what differs and test those parts in staging.
Can I use a copy of production data?
Avoid it. Synthetic records with distinctive values make leaks easy to spot and keep real customer data out of test logs and model prompts.
What if the agent calls services I cannot run locally?
Use the provider's test mode or a stub that records the request. If neither exists, block the call during the test and treat that effect as not observed.
Is a local model good enough for permission tests?
For boundaries enforced in your code, usually. For results that depend on the model, test the model you ship as well.