What changes when the LLM judge and the System One model are switched on.
Each entry is a change measured against the shipped configuration on the same records, with whatever control is needed to say what caused the change rather than only that something did.
Every figure on this page is from one corpus: the MaliciousSkillBench source-disjoint split. Each table states its population, and the numbers are specific to it. Detection is counted at MEDIUM or above throughout. Results on the other corpora are on the head-to-head page, and nothing here should be read as holding on them until it is measured there.
The full split, so this is directly comparable to the head-to-head page, where the shipped judge scores 61.2% F1 against SkillSpector's 72.1% on the same records.
Corpus: msb-source-disjoint · 545 benign, 839 malicious · judge model bedrock-mantle/google.gemma-4-26b-a4b
Hypothesis. Does the gain transfer to Gemma 4 26B, and do two further focuses drawn from the records SkillSpector catches and we miss convert them?
| Arm | Passes | F1 | Precision | Recall | FPR | Errors |
|---|---|---|---|---|---|---|
| Three passes, specialized prompts | 3 | 68.29% | 85.48% | 56.85% | 14.86% | 0 |
| Five passes, specialized prompts | 5 | 71.02% | 85.17% | 60.91% | 16.33% | 0 |
Five passes, specialized prompts: 71.0% F1 at 85.2% precision and 60.9% recall, with a 16.3% false-positive rate.
Corpus: msb-source-disjoint · 545 benign, 839 malicious · judge model bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
Hypothesis. Does the arm comparison measured on the 400-record subsample hold on the full 1,384-record split with the same model.
| Arm | Passes | F1 | Precision | Recall | FPR | Errors |
|---|---|---|---|---|---|---|
| Three passes, specialized prompts | 3 | 81.44% | 86.89% | 76.64% | 17.80% | 0 |
Three passes, specialized prompts: 81.4% F1 at 86.9% precision and 76.6% recall, with a 17.8% false-positive rate.
A 400-record subsample, run first because it takes a fraction of the calls. Absolute values differ from the topline because the population differs; only the differences between arms are meaningful here. The sampling control lives in this section rather than the topline ones, so the split between sampling and specialization is read from this table.
Corpus: msb-source-disjoint · 150 benign, 250 malicious · judge model bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
Hypothesis. The recall gap comes from running one generic judge where SkillSpector runs four specialized semantic analyzers, and a same-prompt control separates that from simple sampling variance.
| Arm | Passes | F1 | F1 change | Precision | Recall | FPR | Errors |
|---|---|---|---|---|---|---|---|
| One pass, shipped prompt | 1 | 72.22% | — | 85.71% | 62.40% | 17.33% | 0 |
| Three passes, same prompt (control) | 3 | 79.23% | +7.0 | 85.25% | 74.00% | 21.33% | 0 |
| Three passes, specialized prompts | 3 | 83.06% | +10.8 | 85.90% | 80.40% | 22.00% | 0 |
The total gain is +10.8 F1 points. The control arm runs the shipped prompt three times and unions the findings, which accounts for +7.0 of it: one pass simply misses findings a second pass returns, so part of the gain is sampling rather than prompt design. Specialization adds the remaining +3.8 points at equal call count. Recall moves from 62.4% to 80.4% while precision holds at 85.7% against 85.9%; the false-positive rate rises from 17.3% to 22.0%, so it is not free. Without the control this would have been reported as a prompt-design result, and most of it is not.
Records the judge could not read, across the 1,384-record source-disjoint split. The failure is emitted as an INFO finding, and INFO does not gate.
| Arm | Unread | Share | Malicious | Harmless | Passed the gate unread |
|---|---|---|---|---|---|
| Three specialized passes, Haiku 4.5 | 92 | 6.60% | 76 | 16 | 51 |
| Five specialized passes, Gemma 4 26B | 51 | 3.70% | 32 | 19 | 47 |
When a skill exceeds the context budget, or the request fails, the scanner emits that as an INFO finding. INFO does not gate, so the skill passes. Most of the affected records here are malicious, which makes this a fail-open rather than a reporting nicety: a skill can evade the semantic stage by being large enough not to fit.
The fix is not a severity change, because raising it would put ordinary large skills through the same gate. Un-analysed content needs to be reported as a coverage gap the caller can choose to gate on. SkillSpector does exactly that through its analysis_completeness record and --fail-on-incomplete flag; this scanner has the same information and currently reports it in a form that passes.
Strided one-in-five sample of msb-source-disjoint, Gemma 4 26B, detection at MEDIUM or above. The meta-analyzer can only demote. Reaching this measurement required fixing a wiring bug that had made it a silent no-op on this route.
| Arm | F1 | Precision | Recall | FPR |
|---|---|---|---|---|
| Judge only | 61.90% | 83.90% | 49.10% | 15.00% |
| Judge plus meta-analyzer | 46.00% | 77.10% | 32.70% | 14.70% |
The stage applied on 219 of 274 records and altered findings on 64, trading 16.4 points of recall for 0.3 points of false-positive rate. It removes far more real detections than false ones, so off remains the right default.
Getting to that number required fixing a bug first. On this route the meta path built its request handler without supplying a schema, so the handler loaded its default: the analyzer's schema. Meta requests were constrained to the wrong shape, the model returned findings and a verdict instead of the meta delta, every batch failed contract validation, and each was silently retained unchanged. Meta reported that it ran while changing nothing, which reads exactly like meta agreeing with every finding. Any earlier result that enabled meta on this route measured a no-op.
120 source-disjoint records, Gemma 4 26B. Five passes cost 5.1x the input tokens of one, which is the expected ratio.
| Configuration | Input tokens per skill | Output tokens per skill |
|---|---|---|
| Single judge pass | 4,838 | 300 |
| Five specialized passes | 24,703 | 1,548 |
The same change — five specialized judge passes against one — measured per corpus on Gemma 4 26B, detection at MEDIUM or above. Populations differ, so read each row against its own baseline rather than across rows.
| Corpus (population) | F1 before | F1 after | Recall before | Recall after | FPR before | FPR after |
|---|---|---|---|---|---|---|
| msb-source-disjoint (839 bad / 545 harmless) | 61.20% | 71.02% | 49.00% | 60.91% | 17.60% | 16.33% |
| msb-balanced-800 (400 bad / 400 harmless) | 77.76% | 79.53% | 80.00% | 84.00% | 25.75% | 27.25% |
| openskillrisk-corpus (374 bad / 0 harmless) | 92.84% | 93.30% | 86.63% | 87.43% | 0.00% | 0.00% |
| harmfulskillbench-corpus (200 bad / 0 harmless) | not permitted | not permitted | 60.00% | 57.00% | no harmless class | no harmless class |
The change helps on every corpus measured and never costs precision, but the size of the gain tracks how much headroom the single pass left. On source-disjoint, where one pass reached only 49.0% recall, decomposition adds 9.8 F1 points. On the balanced subset, where one pass already reached 80.0%, it adds 1.7. On OpenSkillRisk, at 86.6%, it adds 0.5.
So the honest reading is narrower than the headline: decomposition recovers recall the single pass was missing rather than raising a ceiling. Quoting the source-disjoint figure as the improvement would overstate what it does on corpora the scanner already handles well.
HarmfulSkillBench is the exception that matters: there recall falls, 60.0% to 57.0%. The change is not uniformly beneficial, and that corpus is the one where the risk is harmful content rather than a technical capability, which is the kind of judgement the added focuses were not written for. Its dataset terms permit only recall, so no F1 or false-positive rate is quoted for it.
12,500 published skills sampled from abersbail/ai-skill-md-dataset-500 (MIT) and FayeZC/SkillMD-138K (CC-BY-4.0). Unlabelled, so these are flag rates and an upper bound on the false-positive rate. Static analysis only. This scanner is measured on all 12,498 usable records; the SkillSpector row is from the 4,379 it was run on.
| Configuration | CRITICAL | HIGH+ | MEDIUM+ | INFO+ |
|---|---|---|---|---|
| skill-scanner, shipped static rules (n=12,498) | 0.42% | 2.46% | 3.76% | 88.21% |
| SkillSpector, all static analyzers (n=4,379) | 0.11% | 12.90% | 26.42% | 26.76% |
On the population users actually scan, the shipped rules are roughly seven times quieter at MEDIUM or above: 3.76% against 26.42%. More than a quarter of real published skills reach MEDIUM or above under SkillSpector's static analyzers. That is the practical counterpart to the precision lead visible on the labelled corpora.
A larger sample of the same two sources tightens that interval; it does not test it. A third population does. On 200,000 skills from mvaccargiu/gitskills, which indexes skills found in public git repositories, the MEDIUM+ rate is 2.228% with a 95% interval of 2.164% to 2.293%, and HIGH+ is 0.522% against 2.46%. The MEDIUM+ intervals do not overlap, so the two populations genuinely differ and 3.76% is not a general real-world false-positive rate. The direction is favourable, but the figure has to be quoted with its population. Two caveats cut against treating the lower number as the better one: SKILL_LOAD_FALLBACK_USED fires on 16.89% of those records, so the dataset indexes candidate Markdown files rather than only well-formed skills and differs in kind as well as origin, and MANIFEST_MISSING_LICENSE fires on 79.77%, which is what puts its INFO+ rate at 97%. That sample also has a populated LOW tier, 1.66%, where these corpora have none.
The estimate needed the larger sample to settle. At 1,100 records the MEDIUM+ rate read 2.00%, at 4,500 it read 3.40%, and at 12,498 it is 3.76% with a 95% interval of 3.44% to 4.11%. The first sample understated it by nearly half, which is why the interval is quoted rather than the point estimate alone.
The INFO row changed a decision. Nearly every real skill receives an INFO finding, and nothing lands at LOW on these corpora, so severity is effectively bimodal on them — though the gitskills sample below shows that is corpus-specific rather than structural. On MaliciousSkillBench, moving the gate down to INFO looks attractive and raises F1 from 81.4% to 83.2%. On real skills the same change would flag 88.2% of everything scanned, so it was rejected. A corpus of labelled contrast pairs could not have shown that.
The same three-pass change measured on both models, same corpus and population.
| Judge model | F1 before | F1 after | Recall before | Recall after | FPR before | FPR after |
|---|---|---|---|---|---|---|
| Claude Haiku 4.5 | 58.70% | 81.40% | 45.80% | 76.60% | 15.60% | 17.80% |
| Gemma 4 26B | 61.20% | 68.30% | 49.00% | 56.90% | 17.60% | 14.90% |
Decomposition helps on both models and costs precision on neither, but the size of the gain is not portable: Haiku 4.5 gains 22.7 F1 points and Gemma 4 26B gains 7.1 on the same corpus, population and arms. On Gemma 4 the false-positive rate actually falls, 17.6% to 14.9%, and precision rises from 81.5% to 85.5%.
The consequence for reading this page is that the headline figure belongs to a model. Our Haiku arm at 81.4% sits above SkillSpector's 72.9% on Haiku; our Gemma 4 arm at 68.3% sits below its 72.1% on Gemma 4. Quoting one without the model would be misleading.
Corpus: msb-source-disjoint · Same corpus and threshold as the head-to-head page, so these are comparable.
| Configuration | F1 | Precision | Recall | FPR |
|---|---|---|---|---|
| skill-scanner, three specialized judge passes | 81.44% | 86.89% | 76.64% | 17.80% |
| skill-scanner, shipped single-pass judge (Gemma 4 26B) | 61.20% | 81.50% | 49.00% | 17.60% |
| skill-scanner, five specialized passes (Gemma 4 26B) | 71.00% | 85.20% | 60.90% | 16.30% |
| SkillSpector, all analyzers (Gemma 4 26B) | 72.10% | 65.70% | 80.00% | 66.50% |
Decomposing the judge moves F1 from 58.7% to 81.4% on this corpus, and recall from 45.8% to 76.6%, while the false-positive rate stays close to where it was: 15.6% before, 17.8% after. That is the change that matters, because the recall deficit was the whole of the gap to SkillSpector and the precision lead was the thing not to spend closing it.
Read the comparison carefully. SkillSpector reaches 84.6% recall here against our 76.6%, so it still finds more. It does so at a 73.6% false-positive rate against our 17.8%, which is why the F1 ordering reverses. Neither number alone describes the tools.
Three passes cost three times the model calls. No cost figure is quoted because the analyzer's usage counter was found to report the most recent call rather than the sum, which makes the recorded token totals unreliable; that needs fixing before cost can be compared.
These are judge-side changes measured with the shipped core rule pack. They do not address the false-positive rate, which rule-level suppression was separately found unable to move: benign records that flag fire four rules on average and never a single rule alone, so their rule profile is not separable from that of true positives.