What changes when the LLM judge and the System One model are switched on.
Before any judged number is believable the harness has to reproduce the published one. This is that check, run on the locked source-disjoint test partition after materializing the corpus from its pinned revision and verifying every manifest digest against the lock.
| Metric | Published | Reproduced | Match |
|---|---|---|---|
| F1 | 13.74% | 13.74% | matches |
| Precision | 60.75% | 60.75% | matches |
| Recall | 7.75% | 7.75% | matches |
| False-positive rate | 7.71% | 7.71% | matches |
| Samples | 1,384 | 1,384 | matches |
| Malicious | 839 | 839 | matches |
| Benign | 545 | 545 | matches |
| True positive | False positive | False negative | True negative | Scan errors |
|---|---|---|---|---|
| 65 | 42 | 774 | 503 | 0 |
The population digest is pinned in the dataset lock, so a shrunken or substituted population cannot pass unnoticed.
population_sha256 e77564f010fe55ee368af237fabc89b94ca308bd0b3a8c49f052716609f89399
Scanner version n/a, CEL mode off.
Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.
| Corpus | Population | Recall | Precision | F1 | FPR | Errors |
|---|---|---|---|---|---|---|
| HarmfulSkillBench | 200 bad, 0 harmless | 4.00% | no harmless class | no harmless class | no harmless class | 0 |
| MaliciousSkillBench balanced subset | 400 bad, 400 harmless | 33.00% | 94.29% | 48.89% | 2.00% | 0 |
| MaliciousSkillBench source-disjoint (the published test split) | 839 bad, 545 harmless | 7.99% | 59.29% | 14.08% | 8.44% | 0 |
| OpenSkillRisk: contextually risky | 214 bad, 0 harmless | 32.71% | no harmless class | no harmless class | no harmless class | 0 |
| OpenSkillRisk: obviously malicious | 160 bad, 0 harmless | 30.00% | no harmless class | no harmless class | no harmless class | 0 |
Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.
Each row is a published deterministic figure re-measured from the pinned corpus revision. Reproducing these was the precondition for believing any judged number, because a harness that cannot reproduce the baseline cannot be trusted with a new arm.
| Corpus | Population | Previously published | Re-measured | Agrees |
|---|---|---|---|---|
| MaliciousSkillBench source-disjoint | 1,384 packages | F1 13.74% | F1 13.742% | yes |
| MaliciousSkillBench source-disjoint | 1,384 packages | precision 60.75% | 60.748% | yes |
| MaliciousSkillBench source-disjoint | 1,384 packages | recall 7.75% | 7.747% | yes |
| MaliciousSkillBench source-disjoint | 1,384 packages | FPR 7.71% | 7.706% | yes |
| NotInject | 339 benign text cases | 0.00% flag rate at MEDIUM+ | 0.00% | yes |
| InjecAgent | 1,054 canonical signals | 100.00% signal recall | 100.00% | yes |
| In-Page Prompt Injection | 1,101 canonical groups | 99.36% signal recall | 99.36% | yes |
| HarmfulSkillBench | 200 positive-risk skills | 3.50% MEDIUM+ | 3.5% block rate | yes |
| OpenSkillRisk | 374 positive-risk skills | 28.90% HIGH+, 35.74% MEDIUM+ | 30.0% and 32.7% detection | yes |
What is good: All nine figures reproduce. That is the only reason the judged numbers elsewhere on this site are worth reading: the same code path produced both.
| Corpus | Population | Reason |
|---|---|---|
| Bundled skills snapshot | 111 installed skills | the lock is stale against the currently installed applications, and refreshing it requires human review; forcing a refresh would change the corpus identity and break comparability |
| DataDog malicious packages | 5 selected positives | quarantine-only acquisition policy, and too small to move any figure |
| MaliciousAgentSkillsBench | 98,380 rows | metadata only, with no skill content to scan |