Evaluation protocol v0: what a Scopehaven verdict means.

1 / VERDICTS

Three verdicts, and one rule about them.

Two other statuses can appear: error, when the harness itself failed, and not applicable, when a check does not apply to the app. Neither counts as passing.

2 / EVIDENCE RULES

Four rules every verdict follows.

1. The check must have fired. The step that exposes the failure ran and the deciding record was read. A check that never ran reads "not exercised", never pass.
2. An allowed-use twin must pass beside it. Each forbidden check is paired with the same kind of action by someone entitled to it, which must still succeed, so a build that refuses everything cannot pass.
3. Custody decides what a record can show. Each record is labelled by who could have written it: the app under test, an observer deriving it from records the app cannot author, or a source outside the app such as the request path or a payment provider. A boundary pass that rests only on records the app could write reads inconclusive.
4. Unproven means inconclusive. When the records cannot decide, the verdict is inconclusive and the gap is named.

Human notes are stored separately and cannot change a computed verdict.

Each test user's login is proven before and after its checks.

3 / SCOPE AND LIMITS

What a verdict covers, and what it does not.

• Scopehaven is a private alpha, tested on our own apps and open-source templates with synthetic data, not yet on a customer's app.
• A verdict covers the named check, on the named build, with the named test users and data, for the records the report says were read. It says nothing about checks that were not run or records nobody observed.
• It runs before a release, on a local copy of the app. It is not production monitoring, not a pen test and not a certification.
• Scripted test replies are used where a report says so, and then a verdict says nothing about how a particular AI model would behave.
• We have measured how often our checks pass when they should fail on our own adversarial test app. That measurement is still a draft, so it is shared in diligence under NDA, not here.

4 / THE REPORT

What a run export contains.

5 / EXPIRY

A verdict holds for one build.

A verdict applies to the build, suite, policy and test data its run names. When any of them changes, the verdict no longer applies and the checks are run again.

Saved pilot records are re-verified each time they are shown; if the app source or the evaluator changed since a record was saved, it is marked stale and reports no outcome instead of a pass.

6 / CHECK IT YOURSELF

Signed evidence and the re-checker.

A Scopehaven report can ship with a signed evidence bundle. A small Python script, which uses the standard library and can be read in full, re-checks that the verdicts follow from that bundle and that the bundle is unchanged since we signed it. It does not show how the evidence was gathered; that stays with us.

The script ships inside each evidence pack with the exact commands to run. Two commands do the work:

• The signature step shows that the holder of the signing key produced these exact files. Confirm the key id with us over a separate channel; there is no key rotation or transparency log yet.
• The re-derivation step recomputes each verdict it can from the kept observations and refuses the bundle if a stored verdict disagrees. Verdicts it cannot recompute are listed by name with the reason, never passed silently.
• Before trusting its own answer, the script runs tampered copies of the bundle (a flipped verdict, a resealed forged verdict, a dropped control, an added row) through the same code and must reject them.
• What it cannot show: that the observations are true, that the right records were read, or that the app is safe. A key holder who consistently fabricated the observations would get verdicts that follow from them.

On the synthetic sample behind our sample report, re-run on October 29, 2026, the script re-derived all 88 verdicts across both builds and rejected 8 of 8 tampered copies.

7 / A REDACTED SAMPLE BUNDLE

One bundle you can open, from our synthetic test app.

The redacted sample bundle (JSON) comes from the same signed sample: our owned support-agent test app, suite 2.3.0, on a deliberately vulnerable build and its repaired build. Each evidence row is reduced to its record class, time, hash and custody level. Check ids, observer identifiers, query text and record contents are removed.

Redaction changes the bytes, so this copy cannot be re-checked. It names the SHA-256 of each signed original and of the signature envelope, and the public key, so you can see which signed files it came from.

8 / DRILL PROTOCOLS

Written and sealed before the run.

When we test a kind of failure on open-source apps we did not write, the protocol is written down, its digest recorded and timestamped before any counted run, and a later change is a new version. Results name the version they ran under, report nulls and documented behaviour with the same weight as findings, and follow our disclosure policy.