Proof & evals
Scored runs, newest first.
/ Historytracked
| When | Data | Run | Score | Cost |
|---|---|---|---|---|
| Oct 24, 2026latest | 20 outside PDFs (world/pack) | Sonnet 5.5 end to end, 15 simulated days | 12 / 15 checks | $1.22 |
| Oct 24, 2026 · 05:15 | Demo world · 132 documents | One strong model, every document (baseline) | 8 / 9 | $0.31 |
| Oct 24, 2026 · 04:42 | Demo world on Postgres | Live Sonnet agents + Haiku verifier | 8 / 9 | $2.41 |
| Oct 24, 2026 · 01:25 | 8 planted bad alerts | Verifier: none / code rules / live | 8 / 8 stopped | $0.015 |
| Oct 24, 2026 · 00:56 | Demo world · 9 planted scenarios | Rules workflow + Haiku verifier | 9 / 9 | $0.09 |
Earlier run: the generated demo worldNine planted scenarios, the baselines and the verifier study (Oct 24, 2026)
/ Workflow vs. “just ask a model”same answer key
| System | Scenarios | Cause accuracy | Moment recheck | Cost | Run |
|---|---|---|---|---|---|
| AccountableAI workflow | 9/9 | 100% | yes | $0.00 | Rules-mode harness · current local run |
| Live verifier | 9/9 | 100% | yes | $0.0883 | Haiku verifier run · 59 calls |
| One strong model, all docs | 8/9 | 83% | no | $0.3075 | Opus baseline · 132 documents, 2 calls |
The baseline saw all the same documents. It missed an on-time vendor-side case and rechecked more than one alert after the changed fact.
/ The verifier earns its placeplanted bad drafts
Approved every planted bad draft
Caught wording and citation failures
Blocked or refuted every planted bad draft
Bad drafts include accusations, invented quotes, fake sources, unsupported citations, claims contradicted by bank/vendor proof, and attempts to blame Keyway Management for authorized or owner-side delays.
/ Safety boundary
Inspect the guardrails ↗Zero is the only acceptable number for prohibited actions.
The policy gate refuses money movement, outbound reminders outside approved templates, self-approval, and unverified bank-detail changes—before an agent’s text can become an action.