What this is
What do the two skill scanners do, and what does this page measure?
AI coding agents call tools. They run shell commands, edit files and call APIs. Some calls are destructive, or come from a compromised trajectory. A guard looks at each call before it runs and decides allow, confirm (ask a human) or block.
An agent skill is a packaged set of instructions and code that an agent can load. It is usually a SKILL.md file plus scripts. A skill can hold any code for a legitimate reason, so a malicious one is hard to spot. Once loaded, it can make the agent run a bundled script, hide its actions or send data away. A skill scanner checks a skill before it is installed.
Cisco AI Defense Skill Scanner is open source (github.com/cisco-ai-defense/skill-scanner). It has three stages: deterministic rules, a Decision Model (OpenJev) and an LLM judge. NVIDIA SkillSpector v2.11.2 runs 27 analyzers. It reports findings as LOW, MEDIUM, HIGH or CRITICAL, the same scale Cisco Skill Scanner uses.
The two tools, stage by stage
- Rulesmatch known patterns in milliseconds
- Decision Model (OpenJev)answers fixed threat questions with probabilities
- LLM judgereads the skill and writes findings
- 27 analyzersstatic checks, with or without an LLM
- FindingsLOW to CRITICAL, plus a 0 to 100 risk score
This page measures each Cisco Skill Scanner stage alone and chained. It then runs both tools head-to-head on the same judge models. The labelled test set is the held-out MaliciousSkillBench split: 839 malicious and 545 benign skills. Flag rates come from 1,876,662 unlabelled real skills.
Which scanner setup catches the most malicious skills without burying reviewers in false alarms?
It is written for engineers choosing what to deploy. Sibling Spaces: System One benchmarks Decision Models for the tool-call guard. Small models for tool calls asks if a small local model can guard alone.
The answer
Which setup catches the most malicious skills without burying reviewers?
NVIDIA SkillSpector finds more malicious agent skills than Cisco Skill Scanner. It also flags two of every three benign ones. Cisco Skill Scanner's rules alone catch 8% of held-out malicious skills. The Decision Model (OpenJev) can act as an OpenJev screen in front of the LLM judge. That closes most of the gap at a fraction of the noise.
Short answer
Keep the tuned rules on everywhere. Put the OpenJev screen in front of the judge. Use NVIDIA SkillSpector only where a long review queue is acceptable.
- Rules alone 65 of 839 held-out malicious skills caught, recall 7.75%, at a false-positive rate of 7.71%.
- Screen plus judge 61.3% recall at 2.4% false positives, F1 75.3%, with 3.1% of real skills reaching the judge.
- NVIDIA SkillSpector 80.0% recall on the same Gemma 4 judge, at a 66.5% false-positive rate.
- Real skills the rules flag at MEDIUM+, of 1,876,662 4.18%to2.14% 78,513 to 40,245 skills, shipped to tuned rules
- F1 on the held-out test split 13.7%to75.3% Rules alone to screen plus judge
- Real skills that reach the judge behind the screen 3.1% 63 of 2,000 sampled real skills
NVIDIA SkillSpector buys its recall with false positives; the screened judge sits far to the left
Recall against false-positive rate, MaliciousSkillBench source-disjoint test split (839 malicious, 545 benign)
How to read it. Each point is one configuration. Higher means more malicious skills caught; further left means fewer benign skills flagged. Circles are Cisco Skill Scanner setups, coloured by stage; diamonds are NVIDIA SkillSpector. Hover or tab to a point for precision, F1 and cost. Switch the axis to Log to spread the points crowded near zero.
Hover or tab to a point for precision, F1 and cost.
What it shows. The screened judge catches 61.3% at 2.4% false positives; NVIDIA SkillSpector catches 80.0% at 66.5%.
Why it matters. The horizontal axis is your review queue. Each benign skill flagged is a person reading a safe skill. NVIDIA SkillSpector's extra 19 points of recall cost about 64 points of false positives. If you can read every flag, it misses less. If you cannot, the screened judge is the better trade. The chapters below show where each number comes from.
Three stages, one scanner
What does each stage of Cisco Skill Scanner catch, and what does it miss?
The rules are fast and precise but blind to intent; the judge sees intent and pays in false positives.
The stages run in order. The rules match known patterns. The Decision Model (OpenJev) answers fixed threat questions with probabilities. The judge reads the whole skill. A later stage can raise a decision but never lower it.
A quarter of skills are settled before the judge is asked
Share of skills reaching each stage of the rules, Decision Model (OpenJev) and judge chain, MaliciousSkillBench (MSB) balanced subset (800)
How to read it. Each bar is one stage, in the order they run. Its length is the share of the 800 skills that reach it. The label gives the share passed on by the stage before.
What it shows. 83% of skills reach the Decision Model (OpenJev) and 75% reach the judge.
The judge flags the most on every corpus; the Decision Model (OpenJev) trails the rules on contextual risk
Detection: share of skills flagged at all, per corpus and stage, each stage run alone
How to read it. Each row is a corpus, and only its malicious skills count. The three dots show the share each stage flags when run alone. Further right is better. Hover a row for its population.
What it shows. On contextually risky skills the Decision Model (OpenJev) catches 11.0% against the rules' 32.7%.
Rules alone miss 774 of 839 held-out malicious skills
Shipped rules, block at HIGH+, source-disjoint test split (1,384 skills, no scan errors)
How to read it. Rows are the true label; columns are the rules' verdict. Each row is shaded by its own share, so the miss rate on malicious skills reads directly.
What it shows. The rules block 65 of 839 malicious skills and wrongly block 42 of 545 benign ones.
Adding the judge to the shipped rules lifts F1 from 13.7% to 37.7% at the block gate. Precision rises too, from 60.7% to 78.5%. Counted at MEDIUM+, the judge alone reaches 49.0% recall at 17.8% false positives.
Why it matters. The rules are a floor, not a filter. They are free and reproducible, but they miss more than nine in ten held-out malicious skills. The judge is what finds intent. The Decision Model (OpenJev) is sure about obvious attacks and weak on contextual ones. That suits a screen in front of the judge, not a final verdict.
Earlier meta-judge statements Superseded
How this was measured
Detection per corpus comes from each stage run alone over the full corpus. OpenSkillRisk and HarmfulSkillBench have no benign class, so only detection is reported for them. The chain funnel uses the 800-skill balanced subset under the any-intervention lens. The full three-stage chain scored F1 81.5% there at 22.3% false positives; stopping after the Decision Model (OpenJev) kept F1 77.1% at 2.0%.
Head-to-head with NVIDIA SkillSpector
On the same skills and the same judge model, which tool catches more, and at what cost?
On the same judge model, NVIDIA SkillSpector catches 31 points more malicious skills and flags 49 points more benign ones.
Both tools ran at shipped defaults with every analyzer on, neither tuned on these labels. Only records both tools scored are counted. Both report LOW, MEDIUM, HIGH or CRITICAL, so a severity threshold means the same on both sides. MSB is MaliciousSkillBench. Each point below is one severity lens.
In most panels NVIDIA SkillSpector sits higher and further right: more recall, far more false positives
ROC space per arm and corpus: recall against false-positive rate, four severity lenses per tool
How to read it. Each panel is one arm and corpus. Circles are Cisco Skill Scanner, diamonds NVIDIA SkillSpector; lines join a tool's lenses from strictest to loosest. Up is more recall, right is more false positives. OpenSkillRisk and HarmfulSkillBench have no benign class, so they appear only in the next chart.
What it shows. On the Gemma test split at MEDIUM+, Cisco Skill Scanner catches 49.0% at 17.6% false positives. NVIDIA SkillSpector catches 80.0% at 66.5%.
Cisco Skill Scanner trails on recall at MEDIUM+ and HIGH+ in every arm; looser gates are mixed
Paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, with 95% bootstrap intervals
How to read it. Each row is one arm and corpus. The dot is the recall difference on the same records; the line is its 95% interval. Left of zero means NVIDIA SkillSpector caught more. Switch the lens to Any finding: the Gemma and static rows cross to the right.
What it shows. At MEDIUM+ on the Gemma test split, Cisco Skill Scanner trails by 31.0 points (95% interval 27.5 to 34.2).
When the tools disagree, it is almost always NVIDIA SkillSpector flagging alone
Records by which tool flags them at MEDIUM+, per arm and corpus
How to read it. Each bar is one arm and corpus, scaled to 100%. Its segments are records flagged only by Cisco Skill Scanner, by both, only by NVIDIA SkillSpector, or by neither. Hover a segment for the count.
What it shows. On the Gemma test split, 542 records are flagged only by NVIDIA SkillSpector and 24 only by Cisco Skill Scanner.
Cisco Skill Scanner's rules take 15 ms a skill; NVIDIA SkillSpector's static pass takes 2.5 s
Mean wall-clock seconds per skill, log scale, source-disjoint test split
How to read it. Bars show mean seconds per skill on a log scale, so each gridline is ten times the last. Each pair compares the two tools within one arm. Cisco Skill Scanner's in-process Gemma arm was not instrumented, so it has no bar.
What it shows. On Claude Haiku 4.5, Cisco Skill Scanner takes 13.8 s a skill and NVIDIA SkillSpector 39.6 s.
The lens decides the winner. Counting any finding at all, Cisco Skill Scanner's Gemma arm catches 11.7 points more, at 90.7% false positives. On 4,379 real skills, NVIDIA SkillSpector's static analyzers flag 26.4% at MEDIUM+. Cisco Skill Scanner's shipped rules flag 3.76% of 12,498.
Why it matters. NVIDIA SkillSpector suits a team that reads every flag. Its extra recall comes with about two in three benign skills flagged. Cisco Skill Scanner at MEDIUM+ misses more but flags far less, and runs faster per skill. Neither tool is a superset of the other, so each flags skills the other misses.
Integrity: rows, usable results and errors per tool
How this was measured
Tuning the rules on 1.88M real skills
Can the rules flag fewer real skills without missing more malicious ones?
Tuning halved how often the rules flag real skills, and the held-out split lost no detection.
Every usable skill in gitskills, 1,876,662 of them, was scanned by every rule and read by the judge. Rules the judge kept clearing were demoted, then the corpus was rescanned. Two passes took the tree from 5b696a1 through f3a42f3 to 9c08673.
MEDIUM+ flags on real skills fell 49%, most of it in the first pass
Flag rate by severity tier over 1,876,662 real skills, with Wilson 95% intervals
How to read it. Each line is a severity tier, from the shipped rules through two tuning passes. Height is the share of real skills flagged at that tier or above; whiskers are 95% intervals. INFO+ stays near 90% throughout and is in the data table only.
What it shows. MEDIUM+ flags fell from 4.18% to 2.14% of real skills.
On labelled corpora, false positives fell by up to half while recall barely moved
Shipped (5b696a1) and final (9c08673) rules per corpus; MSB is MaliciousSkillBench
How to read it. Each row is a corpus. One dot is the shipped rules and the other the final rules, joined by a line. Switch the measure to Recall to check that the catches held, and the gate to HIGH+ for the block view.
What it shows. On the held-out split, benign skills flagged fell from 46 to 23 of 545 while 67 of 839 stayed caught.
Even after tuning, the judge clears over half of what most busy rules flag
Share of a rule's MEDIUM+ real skills the judge rates clean (judge clears), final tree, rules ordered by skills flagged
How to read it. Each row is a rule in the final tree, busiest first. The dot is the share of its flagged real skills the judge rates clean; the line is a 95% interval. Choose All 57 to see the quieter rules.
What it shows. For 16 of the 20 busiest rules, the judge rates over half of the flagged real skills clean.
Rules and judge rarely flag the same skill: kappa 0.084 after tuning
Real skills flagged at MEDIUM+ by either, split by who flags them (Cohen's kappa 0.072 shipped, 0.084 final)
How to read it. Each bar covers the real skills flagged by the rules, the judge or both, scaled to 100%. The first bar uses the shipped rules and the second the final rules.
What it shows. The judge flags 221,924 skills the final rules leave alone; the rules flag 23,829 it does not.
The judge calls 80.7% of real skills safe and 2.0% malicious
Judge package verdicts over 1,876,206 real skills, shipped prompt
How to read it. One bar, split by the judge's verdict on each real skill: safe, suspicious or malicious. Hover a segment for the count.
What it shows. 17.3% are suspicious, and reading every real skill once took 7.09 billion input tokens.
Why it matters. Run the tuned rules. They halve the flags on real skills and lose nothing on the held-out split. The judge is not ground truth. A high clear rate marks a rule to review, not a proven false positive. The low agreement means the rules and the judge find different skills. Run both if you can afford the judge.
How this was measured
gitskills is unlabelled, so a flag rate there bounds the false-positive rate from above. The judge read every skill once with the shipped prompt on four H200 GPUs. 563 rows failed and are excluded. Demotions were chosen on the train/validation split and real skills; the held-out split was scored once.
The Decision Model (OpenJev) as a screen for the judge
Can the Decision Model (OpenJev) decide which skills the judge needs to read?
Screening with the Decision Model (OpenJev) cuts the judge's false positives from 12.5% to 2.4% for 4.4 points of recall.
OpenJev answers eight threat questions per skill (the eight-probe format). A logistic regression over those answers decides which skills the judge reads. Thresholds were chosen on train/validation to keep 97% of the judge's recall.
Every screen moves the judge left; the logistic screen moves it furthest
Judge alone and judge behind each screen, test split (1,379 records)
How to read it. Each arrow runs from the judge alone to the same judge behind a screen, with the same severity cap. Up is more recall, left is fewer false positives. Switch the axis to Real skills sent to judge to see the work each screen saves.
What it shows. With the new prompt, the judge alone catches 65.7% at 12.5% false positives; behind the logistic screen, 61.3% at 2.4%.
At the chosen 0.0609 threshold, 7.8% of real skills reach the judge
Cascade over 1,876,206 real skills by cascade threshold, prompt-injection probe, shipped prompt
How to read it. The horizontal axis is the screen threshold, on a log scale. The lines show real skills sent to the judge, the judge's flags kept, and real skills the cascade flags. The vertical mark is the chosen threshold.
What it shows. Raising the threshold sends fewer skills to the judge and keeps fewer of its flags. At 0.0609 the cascade flags 3.4% of real skills.
The logistic combination ranks best, and every probe loses ground on test
AUC of each OpenJev probe, train/validation against the held-out test split
How to read it. Each row is one OpenJev probe, or the logistic combination of all eight. The two dots are its AUC on train/validation and on test; 0.5 is chance and 1 is perfect.
What it shows. Sensitive data access drops from 0.734 to 0.588, close to chance.
The screen is measured, not shipped. The shipped Decision Model tier (the --system-one-endpoint option) records a probability and cannot change a finding.
Why it matters. The screen is the cheapest way to shrink the judge's queue. It sends a small share of real skills to the judge and keeps most of the recall. Probes lose ground on new data, so use the logistic combination, not one probe. Until the screen ships, expect the judge's own false-positive rate in production.
The first screen, picked on test data Superseded
Llama Prompt Guard 2 as a pre-filter Rejected
Closing the gap with a decomposed judge
Does splitting the judge into specialised passes close the gap to NVIDIA SkillSpector?
Three specialised judge passes on Claude Haiku 4.5 reach F1 81.4%, above NVIDIA SkillSpector's 72.9% on the same model.
A decomposed judge runs one pass per focus and unions the findings. A sampling control separates what specialisation adds from simply asking three times.
Two thirds of the gain comes from asking more than once
F1 on a 400-record subsample (250 malicious, 150 benign), Claude Haiku 4.5
How to read it. The first bar is F1 for one judge pass. Each floating step adds the gain from one change; the last bar is the total.
What it shows. Sampling adds 7.0 points and specialisation 3.8, for 10.8 in total.
Decomposition helps both models, and Haiku far more than Gemma
Recall (%): one judge pass against the decomposed judge, source-disjoint test split
How to read it. Each line joins one judge pass, on the left, to the decomposed judge, on the right. Switch Compare to see each corpus, and Measure to see F1 or false positives.
What it shows. Haiku's recall rises from 45.8% to 76.6%; HarmfulSkillBench permits recall only.
It is not free. Five specialised passes used 24,703 input and 1,548 output tokens per skill, against 4,838 and 300 for one pass. A usage-counter bug means no cost is quoted for three passes.
Why it matters. Use the decomposed judge for high-stakes reviews only. It costs three to five times the tokens of one pass. Two thirds of its gain comes from simply asking three times. Its false-positive rate on Haiku, 17.8%, stays far below NVIDIA SkillSpector's 73.6% on the same model.
What to deploy
Which stages and which preset should you turn on for your job?
Review at MEDIUM+ and block at HIGH+; pick the preset by how many reviews you can afford.
Each preset demotes more rules to LOW. Balanced demotes none, low-noise 11, quiet 19 plus two judge caps. The judge lifts every preset's recall several-fold.
Under the balanced preset, the judge lifts held-out review recall from 8% to 67%
Recall against false-positive rate per preset, rules only and with the judge, test split (1,384)
How to read it. Each arrow runs from a preset with rules only to the same preset with the judge on. Up is more recall, left is fewer false positives. Switch the action to Block at HIGH+; the bar chart below follows.
What it shows. With the judge, balanced reaches 66.7% review recall at 15.4% false positives; quiet reaches 50.3% at 7.2%.
With the judge, quiet sends 3.9% of real skills to review; balanced sends 11.7%
Real-skill flag rate per preset, 2,000 sampled real skills, follows the action toggle above
How to read it. Each pair of bars is one preset: rules only, then rules plus judge. Length is the share of sampled real skills flagged under the action chosen above.
What it shows. Without the judge, every preset flags 2.0% of real skills or fewer at MEDIUM+.
Why it matters. Start from your review capacity. The flag rate on real skills sizes your queue, so scan a sample of your own skills to confirm it. If every flag gets read, use balanced with the judge. If reviewers are scarce, use quiet with the judge, never quiet alone. The tables below give the setting and command for each case.
Settings per stage
Setups by use case
Noise floor and caveats
How large must a difference be before you trust it?
A judged run disagrees with itself on 14% of skills, so smaller differences between judged arms are noise.
- Judged verdicts that flip across 5 repeats 14% 28 of 200 skills, mostly allow to confirm
- Rule verdicts that flip across repeats 0% 0 of 200 skills
This is the noise floor: one judged configuration, the same 200 skills, temperature zero.
Why it matters. Treat any judged difference under 14% of skills as a tie. Choose between such options on cost or queue size instead. The rules are deterministic, so a change in their numbers is real.
Ways to misread these results
- Do not compare two judged runs on a difference smaller than 14% of skills.
- Do not quote an F1 without its lens. Whether a review flag counts as a catch can invert the ranking.
- Do not treat an unanalysed skill as clean. A skill the judge could not read passes the gate as an INFO finding; route those to review.
- Do not carry a flag rate between populations. The final rules flag 2.14% of all gitskills but 0.52% of the 200,000-skill sample.
- Do not use the quiet preset without the judge. Its extra demotions halve what the rules alone catch on the held-out split.
- Do not read the balanced subset as a fair fight. The rules were tuned on MaliciousSkillBench; NVIDIA SkillSpector never saw it.
- Do not take NVIDIA SkillSpector's supply-chain recall as final. Its transitive mode was off under the corpus safety terms.
- Do not run the judge without verdict repair. Without it, analyses that contradict themselves are discarded, which flatters precision.