Scopehaven
Book 20 minutes
Failure library
Sixteen kinds of agent failure from public reports, each written as a check and shown to fail on a deliberately broken build of our own test app. With outside measurements of the problem.
Read the failure libraryRESEARCH AND EVIDENCE
Research, evidence and the rules we publish by.
This page lists what Scopehaven publishes about its own testing: what a result means, how to check one, how we report problems in other people's software, and what we keep private and why.
THE PAGES
Where to start.
Evaluation protocol v0
What PASS, FAIL and INCONCLUSIVE mean, the four evidence rules, the report fields, expiry per build, the re-checker and a redacted sample bundle.
Read the protocolSample report
A report from our synthetic test app before and after a fix, with signed evidence and the assessor-pack version beside it. Labelled synthetic.
Read the sample reportDisclosure policy
How we report problems we find in open-source software: private notice, 90 days plus 30, extensions on request, credit, no scoreboard.
Read the disclosure policyWorks with, and when you don't need us
What other evaluation tools document, with dated links, where Scopehaven fits beside them, and when to skip it.
Read the comparisonPublic cases
Six public cases where an AI product or agent exposed data or took an action it shouldn't have, and the control each one illustrates. None involved Scopehaven.
Read the six casesRESEARCH NOTES
Notes published so far.
- An internal pilot on one open-source AI workspace, written before the disclosure policy; it names no project.
Notes on software we did not write follow the disclosure policy: no project is named, and no count of affected projects is given, before its deadline.
WHAT WE PUBLISH
Results, the rules behind them, and how to check them.
- Results, each with its sample size, pinned versions, dates, conditions and nulls.
- The evaluation protocol: what each verdict means and the evidence rules at rule level.
- The re-checker, inside each evidence pack, and one redacted sample bundle.
- The failure library and its ledger, updated as public reports come in.
- Maintainers' advisories that credit our reports, once they publish them.
WHAT WE DO NOT PUBLISH, AND WHY
The how stays private. So do other people's details.
| What | Why |
|---|---|
| How our observers read an app's records | It is how the product works. |
| Our check packs and templates | They are the product. |
| Real evidence bundles | A real bundle shows which tables and records were read on someone's app. |
| Our test sets of bypass attempts and held-out cases | Publishing them would teach evasion and let a tool be tuned to the test. |
| Results on someone else's software before its deadline | The maintainers get time to fix it before we publish, under the disclosure policy. |
| Customers' and prospects' names, and their results | They are not ours to share. |
The full method is available to assessors and in diligence, under NDA.
FUNDING AND EDITORIAL STATEMENT
Who writes these pages, and who pays for them.
These pages are written by Scopehaven, which is built by Yusef Syed in Toronto and is not yet incorporated. No vendor, model provider, maintainer or customer paid for, reviewed or edited them, and none can change a computed result.
[YUSEF: one sentence on funding, for example "Scopehaven has taken no outside funding."]
Each page shows the date it was last updated, and a correction is dated on the page it corrects.