Both evaluations were rerun on 2026-08-17 against the current build. Engine source digest f71643897ca3…. The two results must always be cited together; see the V3 contract for what the scores do and do not mean.
Frozen 40-conversation holdout; threshold selected on a disjoint 40-conversation development split (80 developer-authored conversations total)
AUROC
0.995
95% CI 0.978–1.000
Sensitivity
0.900
95% CI 0.750–1.000
Specificity
1.000
95% CI 1.000–1.000
Developer-authored synthetic fixtures written with knowledge of the detector taxonomy. Useful for reproducibility and regression detection, not independent evidence of safety.
pnpm validate:v3
2,000 independently authored conversations (HH-RLHF red-team attempts)
AUROC
0.465
95% CI 0.441–0.489
Sensitivity
0.244
95% CI 0.217–0.271
Specificity
0.723
95% CI 0.696–0.752
The proxy uses human attack-success ratings, not V3 category labels, so it is an imperfect proxy. The failure still shows Q is not currently a general harm score and does not replace ROC/AUROC analysis or deployment-specific validation.
node packages/npm/validation/v3/run-anthropic-proxy.mjs <red_team_attempts.jsonl.gz>
An earlier hand-labeled fixture corpus was withdrawn from the working tree and now exists only in repository history (commit c976e98). The counts below describe that archived snapshot as of 2026-05; nothing at the current HEAD substantiates them independently, which is why the validation runs above are the evidence of record. Browse the archived snapshot.
Fixture files
Canonical JSONL files in the archived preliminary corpus.
Labeled turns
Each turn includes input, output, labels, and review notes.
Controls
Expected-safe or bounded examples for false-positive review.
Positive turns
Examples marked with one or more behavioral risk labels.
Observed labels
Distinct risk categories represented in the archived corpus.
Control rate
Share reserved for expected-safe or bounded examples.
Planned work before stronger evaluation claims.
Compare detector output against independently labeled fixtures.
Publish precision, recall, and F1 by category.
Add independent second-rater review on a representative subset.
Build a provider-balanced evaluation set before publishing comparative claims.
Two validation runs — cited together, pass and fail — plus an archived preliminary corpus and the milestones that remain before stronger claims. GitHub is the source of record.
