Scopehaven
Request a first test
READING THE EVIDENCE
What should an AI agent security test report show?
A useful permission-testing report connects a specific rule to an observed result, identifies the evidence behind it, and makes the limits clear. A passing label without that context is hard to assess.
About this walkthrough: the screenshot below comes from Scopehaven's built-in demo app with a deliberate permission bug and synthetic data. It is an existing saved-run illustration, not a customer assessment, a new test run, or a downloadable assessment report.
Start with the rule, then inspect the record.
In this synthetic demo, the rule is that one tenant's user must not receive another tenant's record. The displayed run shows a failed tenant check and the record returned to the test user. The purpose of the screenshot is to show the relationship between the rule, observation, and verdict.
The fields below are a review checklist. They do not assert that every connector or export can supply every kind of evidence.
- Scope and identity
- The application and build, configuration, test identity, data, and relevant suite or policy version.
- Permission rule
- Who may perform the action, on which resource, and under what conditions.
- Expected and observed outcome
- What the check required, what the available evidence showed, and whether the conclusion is pass, fail, or inconclusive.
- Evidence source
- The response or effect observed, who controls that source, and how the observation is linked to this action.
- Observation limits
- The time window, coverage gaps, unsupported conditions, and effects that may still be pending.
- Allowed-use controls
- Evidence that the covered actions users are permitted to take still work.
“Backend observed” describes coverage.
It does not, by itself, mean a permission boundary passed. Inspect whether the observation source can support the particular conclusion. A record controlled by the application under test may be useful context without being independent proof that the boundary held.
A response can establish which data was returned. A statement about a committed file or saved record needs an appropriate observation of that effect. A statement that no effect occurred also needs sufficient coverage and a defined observation window.
Ask what is missing. An inconclusive check, unsupported observation, or untested action should stay visible. None should silently become a passing result.
A repair comparison needs compatible evidence.
Identify the original and repaired application builds and confirm that the policies, checks, identities, and evaluation requirements are comparable. Review the original failure, the retest evidence, and the controls for permitted work together.
- The failed boundary must satisfy the check's evidence requirements on the repaired build.
- The covered allowed-use controls must still pass.
- Changed assumptions, unsupported checks, and observation gaps must remain explicit.
Scopehaven supports comparing compatible runs. Your team makes the repair and reviews what the retest establishes. See how it works.
Give the reviewer a scoped answer.
For a customer security questionnaire, connect each claim to the particular check and evidence that supports it. State which application build was assessed and which behavior remains outside the assessment. A report is evidence for a review; it is not a compliance sign-off.
Keep sensitive records, credentials, and personal data out of public examples. The screenshot on this page uses synthetic data; a real engagement needs its own agreed handling and access rules.
Scopehaven's workbench saves runs and evidence locally. Saved runs can be exported as JSON, HTML, or SARIF; formats organize the evidence but do not expand what was tested.