Skill scanner evaluation results

Summary

The Cisco AI Defense skill scanner has three stages that can look at a skill: a set of deterministic rules, a System One model, and a full LLM judge. Only the rules had been measured. This is what happens when the other two are switched on, measured on the same corpora and with the same code that produced the published rule-only numbers.

Tuned against every real skill

Every usable skill in the gitskills corpus, 1,876,662 of them, was scanned by every deterministic analyzer and read by the LLM judge and the System One model served on local GPUs. The deterministic rules were then tuned from those results, checked on every labelled corpus, and the whole corpus rescanned with the final tree. The large-scale page has the detail.

4.18% → 2.14%

Real skills the rules flag at MEDIUM+, shipped to final

8.44% → 4.22%

Rules false-positive rate on the held-out test split

63.7% → 75.8%

Judge F1 on the held-out test split, new prompt

1.75%

Of 2,000 sampled real skills, flagged by the judge behind the OpenJev screen; 3.1% reach the judge

The first evaluation, on the locked test corpus

13.74%

F1 with rules alone

37.68%

F1 with the judge added

78.49%

Precision with the judge added

14.00%

How much a judged run moves against itself

2.6x

More quality per dollar from the System One model

The short version

Switching on the judge nearly tripled F1 on the locked test corpus, from 13.74% to 37.68%, and precision went up rather than down.
The System One model is the better deal for anything automated. It roughly doubled F1 over the rules, produced no false positives at all in our runs, and processed the whole 1,384-package test corpus in 42 seconds for 18 cents. Later measurements qualify this: asked a single question over raw skill source, the hosted model did not separate malicious from benign skills (AUC 0.25 to 0.31), so the shipped --system-one-endpoint tier is advisory only; asked eight choice questions, OpenJev screens well in front of the judge, as the large-scale page shows.
The meta-judge did nothing. It ran on 78% of packages and produced no suppressions and no additions. Later traced to a schema-wiring defect that made it a no-op; once fixed it cost 16.4 points of recall, so it stays off.
Nothing here is stable below 14%. Repeating one judged configuration over the same packages changed the verdict on 14% of them, so smaller differences are noise.

Why this was worth measuring

A skill is an unusually open format. It can contain any code, in any language, for any legitimate reason, so most of what a rule matches on is ordinary software. That makes precision hard to hold and makes it tempting to assume an LLM reviewing the output would mostly add noise. On the development split, where rule precision was already 99%, that assumption looked safe. It turned out to be wrong on the harder test split, where the rules were missing far more than they were over-flagging.

Every stage, on every corpus we could label

Detection is the share of skills in that group that got flagged at all. Hover any column name for its definition.

Corpus	Population	Rules	System One model	Judge
HarmfulSkillBench	200 skills	4.00%	26.13%	56.00%
MaliciousSkillBench balanced subset	400 malicious, 400 benign	33.00%	54.31%	81.75%
OpenSkillRisk: contextually risky	214 skills	32.71%	11.00%	77.10%
OpenSkillRisk: obviously malicious	160 skills	30.00%	95.77%	100.00%

What is good: Both models beat the rules everywhere except one place, and the judge beats everything everywhere. On the obviously-malicious half of OpenSkillRisk the judge caught all of it.

What is not: The System One model is much worse than the rules on contextually risky skills, catching 11% against the rules' 33%. It is a confident filter for clear cases, not a substitute for judgement on ambiguous ones.

Where to look next

How to configure it is the practical answer: which tiers to turn on for which job.
Deterministic baseline shows the previously published figures re-measured, which is what makes the rest credible.
Judge and meta-judge and System One model give the per-corpus detail.
Experiments covers the fifteen side questions, including the four that came out negative.
Reproduce it has the actual commands, in order, and the four mistakes that cost us a run each.

Source figures for the rule-only baseline: https://huggingface.co/blog/Vineethsain/tuning-cisco-skill-scanner. Earlier work on System One models: https://huggingface.co/spaces/Vineethsain/defenseclaw-system-one.