What changes when the LLM judge and the System One model are switched on.
Both models were served locally with vLLM on four NVIDIA H200 GPUs (150 GB each), 96 CPU cores: OpenJev, the System One weights, on two cards, and Gemma 4 26B-A4B as the LLM judge on the other two, each as independent replicas because either model fits on one card. The deterministic scanner ran on the host CPUs. Nothing left the cluster.
| Judge | F1 | Precision | Recall | FPR |
|---|---|---|---|---|
| Gemma 4 26B, local vLLM | 64.19% | 81.63% | 52.88% | 18.20% |
| Gemma 4 26B, Bedrock mantle (earlier) | 61.20% | 81.50% | 49.00% | 17.60% |
MaliciousSkillBench source-disjoint split, single pass, core rules, detection at MEDIUM or above. The two routes agree, which is what licenses using the local run for everything below.
The first local run lost 46.5% of records (644 of 1,384) to finish_reason=length. Under grammar-constrained JSON decoding the model emitted a complete object and then padded it with whitespace until it reached the 8,192-token limit. Serving with disable_any_whitespace took that to 0.6% (8 records). On eight records that had all failed, the patched server answered 8 of 8 and an unpatched control 4 of 8.
OpenJev was asked eight boolean threat questions per skill, with the probability of true read from its next-token distribution. Its prompt_injection answer screens every record, and the judge runs only on those above a threshold. The threshold was chosen on one hash-selected half of the records (655) to keep recall within two points of the judge alone while skipping the most benign records, and every figure below is from the other half (721).
| Held-out half | F1 | Precision | Recall | FPR | Judge calls |
|---|---|---|---|---|---|
| Judge alone | 61.15% | 76.22% | 51.05% | 23.13% | 100% |
| Jev screen, then judge | 62.59% | 86.07% | 49.18% | 11.56% | 60.7% |
The screen skips 70.7% of benign records. It is not only a cost saving: many records the judge flags wrongly are ones OpenJev scores as harmless, so the cascade vetoes judge false positives and precision rises while recall moves little. This is one corpus split in half; the threshold has not been measured on the other labelled corpora and should not be assumed to transfer.
What is not: this threshold was chosen on half of MaliciousSkillBench's frozen test split, and the benchmark's terms forbid selecting thresholds on test members. The reported half was unseen, but the half used for selection was test data. The screen is re-selected on train/validation in the next section, and those figures supersede these.
A record is flagged only when the screen passes it and the judge flags it. Two screens: OpenJev's prompt_injection probability alone, and a logistic regression over all eight probe logits fitted on train/validation. For each judge prompt the threshold keeps at least 97% of the judge's own train/validation recall at the lowest train/validation false-positive rate; the frozen test split was then scored once, and nothing was chosen on it.
| Held-out test, MEDIUM+ | Recall | FPR | Precision | F1 | Judge calls | Real-skill flags | Real-skill judge calls |
|---|---|---|---|---|---|---|---|
| Judge alone | 65.7% | 12.5% | 89.0% | 75.6% | 100% | 10.45% | 100.0% |
| prompt_injection screen (p ≥ 0.04835) | 59.5% | 3.7% | 96.1% | 73.5% | 60% | 3.15% | 7.9% |
| Logistic, all eight probes (score ≥ -0.4536) | 61.3% | 2.4% | 97.5% | 75.3% | 58% | 1.75% | 3.1% |
| Held-out test, MEDIUM+ | Recall | FPR | Precision | F1 | Judge calls | Real-skill flags | Real-skill judge calls |
|---|---|---|---|---|---|---|---|
| Judge alone | 51.3% | 16.1% | 82.9% | 63.4% | 100% | 12.80% | 100.0% |
| prompt_injection screen (p ≥ 0.06092) | 48.0% | 3.3% | 95.7% | 63.9% | 58% | 2.60% | 7.0% |
| Logistic, all eight probes (score ≥ -0.0264) | 48.8% | 2.0% | 97.4% | 65.0% | 56% | 1.30% | 2.5% |
| Held-out test, MEDIUM+ | Recall | FPR | Real-skill flags | Real-skill judge calls |
|---|---|---|---|---|
| Judge alone | 62.1% | 10.5% | 7.60% | 100.0% |
| prompt_injection screen | 54.2% | 2.9% | 2.35% | 6.0% |
| Logistic, all eight probes | 56.8% | 1.7% | 1.35% | 2.5% |
What is good: Selected on train/validation, a screen cuts the judge's test false-positive rate by 71% to 88% for 2.5 to 6.2 points of recall, and on real skills it sends only 2.5% to 7.9% of records to the judge at all. The eight-probe logistic screen is the stronger of the two on every population.
What is not: The logistic screen needs all eight probes per skill, which were scored on the labelled splits and the 2,000-skill sample; the full-corpus run scored prompt_injection only, so the full-corpus cascade uses that screen. Real-skill rates are flag rates, not false-positive rates.
| Screen | Train/val AUC | Test AUC |
|---|---|---|
logistic | 0.973 | 0.915 |
obfuscation | 0.967 | 0.908 |
prompt_injection | 0.959 | 0.881 |
security_control_change | 0.935 | 0.859 |
supply_chain | 0.897 | 0.793 |
destructive_command | 0.830 | 0.791 |
remote_hidden_execution | 0.877 | 0.726 |
data_exfiltration | 0.792 | 0.648 |
sensitive_data_access | 0.734 | 0.588 |
Every finding was recorded with the analyzer that produced it and stored as Parquet, queried with DuckDB. The record-level rates reproduce the published deterministic baseline (7.71% false-positive rate at HIGH or above), which checks that the new pipeline measures the same thing as the old one.
| Analyzer | MEDIUM+ on harmless | MEDIUM+ on malicious | Harmless per malicious |
|---|---|---|---|
| correlation | 35 | 29 | 1.21 |
| static | 10 | 30 | 0.33 |
| pipeline | 10 | 33 | 0.30 |
| Analyzer | Rule | Harmless | Malicious | Precision |
|---|---|---|---|---|
| correlation | CORRELATED_NETWORK_EXECUTION_FLOW | 31 | 7 | 18.4% |
| pipeline | PIPELINE_TAINT_FLOW | 10 | 9 | 47.4% |
| correlation | CORRELATED_SENSITIVE_NETWORK_FLOW | 4 | 22 | 84.6% |
| static | ACTIVE_DYNAMIC_EXECUTION | 4 | 18 | 81.8% |
| static | ACTIVE_REMOTE_ACQUIRE_EXECUTE | 1 | 1 | 50.0% |
| static | FILE_MAGIC_MISMATCH | 1 | 0 | 0.0% |
| Configuration | F1 | Precision | Recall | FPR |
|---|---|---|---|---|
| Shipped, all rules | 13.49% | 58.18% | 7.63% | 8.44% |
| Demote CORRELATED_NETWORK_EXECUTION_FLOW | 13.29% | 77.22% | 7.27% | 3.30% |
| Also demote PIPELINE_TAINT_FLOW | 11.50% | 80.00% | 6.20% | 2.39% |
correlation is the only analyzer that fires more often on harmless records than malicious ones, and one rule accounts for almost all of it. The earlier conclusion that rule-level suppression could buy at most 0.2 F1 points came from data that recorded only a record's set of rule ids, with no analyzer and no per-finding severity; it was a limit of the data, not of the rules. F1 barely moves because deterministic recall is low either way, so precision and false-positive rate are the metrics to read.
1,876,662 real published skills, the whole usable gitskills corpus, each scanned in full by every deterministic analyzer with the shipped scanner, again after the first tuning pass, and a third time with the final tree. Between the shipped and the final scanner 38,275 records left MEDIUM or above and 7 entered it. These are unlabelled, so a flag rate bounds the false-positive rate from above rather than measuring it.
| Threshold | Shipped (95% interval) | First pass | Final (95% interval) | Relative change |
|---|---|---|---|---|
| CRITICAL | 0.629% 0.617 to 0.640 | 0.598% | 0.425% 0.416 to 0.434 | -32.4% |
| HIGH+ | 2.469% 2.447 to 2.491 | 2.312% | 1.987% 1.967 to 2.007 | -19.5% |
| MEDIUM+ | 4.184% 4.155 to 4.212 | 2.466% | 2.144% 2.124 to 2.165 | -48.7% |
| LOW+ | 5.213% 5.181 to 5.245 | 3.770% | 3.451% 3.425 to 3.477 | -33.8% |
| INFO+ | 90.365% 90.323 to 90.407 | 89.639% | 89.616% 89.572 to 89.659 | -0.8% |
The later passes were first measured by rescanning only the records each change could touch and overlaying them on the first-pass scan. That estimate was 2.153%; the full rescan gives 2.144%. Every record the full scan flags was in the estimate, which over-counted 168 records (130 by YARA_jailbreak_generic, 38 by FIND_EXEC_PATTERN) whose changes landed after, or outside, the rescanned sets.
Shipped scanner against the final scanner, static arm, MEDIUM or above. Every corpus the harness holds is shown, including those where the tuning costs recall. The rules were designed on the train/validation split and on real skills; the test split was not used to design them.
| Corpus | Records | Recall | FPR | Unlabelled flag rate |
|---|---|---|---|---|
| msb-trainval development split (rules were designed on it) | 6,594 | 33.14% → 31.45% | 2.17% → 1.05% | — → — |
| msb-source-disjoint frozen test split (held out) | 1,384 | 7.99% → 7.99% | 8.44% → 4.22% | — → — |
| msb-balanced-800 overlaps the test split (137 records) | 800 | 33.00% → 31.00% | 2.00% → 1.25% | — → — |
| openskillrisk-corpus | 374 | 31.55% → 29.95% | — → — | — → — |
| mcp-tool-poisoning | 50 | 40.00% → 40.00% | 0.00% → 0.00% | — → — |
| harmfulskillbench-corpus unlabelled for this purpose | 200 | — → — | — → — | 4.00% → 3.50% |
| real-world-skills | 12,498 | — → — | — → — | 3.76% → 2.49% |
| gitskills-sample | 199,998 | — → — | — → — | 2.23% → 0.52% |
What is not: the recall cost on train/validation is 89 malicious packages that no longer reach MEDIUM. The findings that carried them: 44 by SUPPLY_CHAIN_UNPINNED_DEPENDENCY, 19 by SOCIAL_ENG_ANTHROPIC_IMPERSONATION, 12 by PIPELINE_TAINT_FLOW, 6 by FILE_MAGIC_MISMATCH, 4 by CORRELATED_NETWORK_EXECUTION_FLOW + PIPELINE_TAINT_FLOW, 3 by CORRELATED_SENSITIVE_NETWORK_FLOW, 1 by PROMPT_INJECTION_IGNORE_INSTRUCTIONS. Most are hygiene or mention findings rather than the malicious behaviour — an unpinned dependency, a mention of Anthropic, a text label in Markdown — and the judge is the layer meant to catch these skills. Keeping a rule firing on the wrong evidence for its incidental recall would keep the false positives that come with it.
The LLM judge, with the shipped prompt, read 1,876,206 of them (563 could not be analysed and are excluded, not counted as clean; 96,519 were read with partial coverage, the prompt budget leaving a file out) and flagged 12.70% 12.65 to 12.75 at MEDIUM or above.
Its package verdicts: SAFE 80.7%, SUSPICIOUS 17.3%, MALICIOUS 2.0%. Almost all of its false positives read as findings about what a skill is for — a browser-automation skill can run JavaScript, a payments skill can move money — which is what the new prompt below addresses.
The final deterministic scanner and the judge agree on 16,321 flagged records; 23,829 are flagged only by the rules and 221,924 only by the judge (Cohen's κ 0.084). The two layers look at different things, which is why neither is ground truth for the other.
Deterministic flags alone: 2.14%. Deterministic flags plus the judge behind the OpenJev screen: 5.19%.
For the rules that flag the most real skills at MEDIUM or above: how many records each rule alone drives to that tier (removing the rule would clear them), and the share the judge clears. The judge is not ground truth — on the labelled split its own false-positive rate is about 18% and its recall 53% — so this ranks rules for review rather than measuring their error.
| Analyzer | Rule | Records | Sole driver | Judged | Judge clears |
|---|---|---|---|---|---|
| correlation | CORRELATED_NETWORK_EXECUTION_FLOW | 7,556 | 4,380 | 7,549 | 43.6% 42.5 to 44.7 |
| static | ACTIVE_DYNAMIC_EXECUTION | 6,968 | 5,734 | 6,937 | 63.8% 62.7 to 64.9 |
| pipeline | PIPELINE_TAINT_FLOW | 5,438 | 2,184 | 5,415 | 54.0% 52.6 to 55.3 |
| correlation | CORRELATED_SENSITIVE_NETWORK_FLOW | 4,237 | 3,590 | 4,226 | 59.7% 58.3 to 61.2 |
| pipeline | COMPOUND_FIND_EXEC | 2,327 | 1,401 | 2,326 | 70.2% 68.3 to 72.0 |
| static | ACTIVE_REMOTE_ACQUIRE_EXECUTE | 2,236 | 1,715 | 2,225 | 68.4% 66.4 to 70.3 |
| static | PROMPT_INJECTION_IGNORE_INSTRUCTIONS | 2,045 | 1,319 | 2,038 | 39.7% 37.6 to 41.9 |
| static | SECRET_CONNECTION_STRING | 1,495 | 1,312 | 1,493 | 71.0% 68.6 to 73.2 |
| static | FIND_EXEC_PATTERN | 1,395 | 474 | 1,393 | 68.8% 66.3 to 71.2 |
| static | YARA_jailbreak_generic | 1,365 | 1,047 | 1,356 | 70.4% 67.9 to 72.8 |
| static | YARA_sql_injection_generic | 1,185 | 1,049 | 1,178 | 59.7% 56.8 to 62.4 |
| pipeline | COMPOUND_FETCH_EXECUTE | 1,030 | 391 | 1,028 | 53.4% 50.3 to 56.4 |
| static | ACTIVE_OS_PERSISTENCE_DIRECTIVE | 1,020 | 812 | 1,017 | 64.0% 61.0 to 66.9 |
| static | YARA_tool_chaining_abuse_generic | 949 | 516 | 946 | 42.0% 38.9 to 45.1 |
| static | YARA_autonomy_abuse_generic | 932 | 808 | 931 | 88.2% 86.0 to 90.1 |
| static | YARA_prompt_injection_generic | 872 | 379 | 872 | 60.6% 57.3 to 63.7 |
| static | PROMPT_INJECTION_CONCEALMENT | 859 | 406 | 859 | 73.2% 70.2 to 76.1 |
| static | YARA_command_injection_generic | 751 | 470 | 744 | 46.2% 42.7 to 49.8 |
| static | YARA_system_manipulation_generic | 684 | 500 | 682 | 55.3% 51.5 to 59.0 |
| static | YARA_credential_harvesting_generic | 670 | 520 | 665 | 58.5% 54.7 to 62.2 |
OpenJev screens each skill and the judge runs only above a threshold on its prompt_injection probability. The threshold, 0.0609, was selected on MaliciousSkillBench train/validation for the prompt this corpus was judged with: keep at least 97% of the judge's recall at the lowest false-positive rate. On real skills it sends far fewer records to the judge than it did on the labelled split, so the whole trade-off is shown.
| Screen threshold | Judge calls | Judge flags kept | MEDIUM+ flag rate |
|---|---|---|---|
| 0.005 | 34.6% | 70.9% | 9.00% |
| 0.01 | 23.2% | 56.7% | 7.20% |
| 0.02 | 15.4% | 43.3% | 5.50% |
| 0.035 | 11.1% | 34.4% | 4.36% |
| 0.0484 | 9.1% | 29.9% | 3.80% |
| 0.0609 | 7.8% | 26.8% | 3.40% |
| 0.1 | 5.5% | 20.6% | 2.62% |
| 0.2 | 3.2% | 13.3% | 1.69% |
| 0.35 | 1.7% | 8.3% | 1.05% |
| 0.5 | 1.0% | 5.8% | 0.73% |
OpenJev's score ranks the judge's flags with AUC 0.778 over 1,876,206 skills both models read. Judge alone: 12.70%.
The scan policy has hundreds of settings; most users need one decision, how much review capacity they have. Two presets sit beside the default. Every rule was scored by how many real skills it alone drives to MEDIUM or above that the judge also cleared, against how many malicious train/validation packages it alone detects; rules were demoted greedily in that order and the path was cut at two points. A demoted rule is reported at LOW, still visible but not gating. Real-skill rates below are exact on the final full scan, because a pack only lowers the severity of listed rules.
| Preset | Rules reported at LOW | Real-skill MEDIUM+ (rules) | Train/val recall | Train/val FPR |
|---|---|---|---|---|
balanced | 0 | 2.144% 2.124 to 2.165 | 31.45% | 1.05% |
low-noise | 11 | 1.930% 1.911 to 1.950 | 31.35% | 1.05% |
quiet | 19 | 1.331% 1.314 to 1.347 | 29.78% | 0.15% |
Use them with --policy low-noise or --policy quiet. low-noise also caps judge findings the model itself rates low-confidence; quiet additionally caps findings it labels contextual risk.
The threat-analysis prompt now lists what is not a finding on its own — the capability the skill exists to provide, the breadth of its declared permissions, installing dependencies from a registry, requiring API keys, hypothetical injection, quality issues — and, just as explicitly, the misuse to report even when it is phrased as routine: a bundled or downloaded script the agent must run automatically, instructions to hide actions or override other instructions, and data sent where the stated purpose does not need it. It was written from real skills and MaliciousSkillBench train/validation, then scored once on the frozen test split.
| Judge, MEDIUM+ | Train/val recall | Train/val FPR | Test recall | Test FPR | Test precision | Test F1 | Real-skill flags |
|---|---|---|---|---|---|---|---|
| Shipped prompt | 86.2% | 28.2% | 51.6% | 16.1% | 83.1% | 63.7% | 12.80% |
| New prompt | 88.8% | 24.0% | 65.9% | 12.5% | 89.0% | 75.8% | 10.45% |
Each finding carries the model's own verdict (TRUE_POSITIVE or CONTEXTUAL_RISK) and confidence. Capping either below MEDIUM is an operating point, not a free win: it cuts false positives on every population and costs recall, more on test than on train/validation.
| New prompt, MEDIUM+ | Train/val recall | Train/val FPR | Test recall | Test FPR | Real-skill flags |
|---|---|---|---|---|---|
| No cap | 88.8% | 24.0% | 65.9% | 12.5% | 10.45% |
| Low-confidence findings at LOW | 86.3% | 16.6% | 62.3% | 10.5% | 7.60% |
| Contextual-risk findings at LOW | 83.8% | 6.9% | 49.6% | 4.2% | 3.00% |
Gemma 4 26B-A4B via Bedrock mantle, verdict repair on (the default). Train/validation: 600 malicious and 1,338 benign MaliciousSkillBench packages. Test: the 1,384-package source-disjoint split. Real skills: 2,000 sampled uniformly from gitskills, a flag rate that bounds the false-positive rate from above.
Every figure on this page, and the per-rule tables behind it, is in results/large-scale.json: counts, rates, thresholds and rule identifiers only. No skill content, record identifier, finding text, prompt or model rationale is published, and the file is refused at build time if one appears.
results/SHA256SUMS holds its SHA-256, so a copy can be checked against the file this page was rendered from.
Scanner trees: shipped 5b696a1, first tuning pass f3a42f3, final 9c08673; report built at 5807c36de84b.
| Step | Script |
|---|---|
| Score OpenJev's eight probes | evals/experiments/c3_openjev_local.py |
| Run the judge alone over a corpus | evals/runners/judge_only.py |
| Scan a corpus with one tree | evals/runners/cross_tool_benchmark.py |
| Build the store | evals/experiments/f1_full_corpus_store.py |
| Analyse every real skill | evals/experiments/f2_full_corpus_analysis.py |
| Check the overlay against the full scan | evals/experiments/f3_overlay_check.py |
| Shipped against final on labelled corpora | evals/experiments/f4_labelled_ab.py |
| The preset demotion path | evals/experiments/f5_policy_pack_path.py |
| Judge prompt and caps | evals/experiments/a2_judge_prompt_caps.py |
| Select the OpenJev screen | evals/experiments/c4_openjev_screen.py |
| Assemble and check this report | evals/publish/large_scale_report.py |