Cisco AI Defense skill scanner: evaluation results

Does the System One model earn its place between the rules and the judge?

Scored on a label-balanced subset of 400 malicious and 400 benign packages, so both error directions are visible at once. Every chain is reported under both lenses, because they reorder the ranking: block-only counts just a hard block as a catch, while any-intervention also counts a confirm, which is the right question when a confirm routes to review.

The cascade short-circuits only on a block. A deterministic confirm escalates to the next tier and the tiers are merged so a later one can raise a decision but never lower it.

Block-only lens

Any-intervention lens

The last column reads in stage order. Where it says "rules 100%; System One 91%", the rules examined every skill and 9% were already settled by a clear block, so the System One model was only asked about the remaining 91%. That is the saving from putting cheap stages first.

What the two lenses show

System One model: every metric, corpus by corpus

Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.

Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.

The System One model against the rules, detection only

What is good: It nearly matches the judge on clear malice at a fraction of the cost, reaching 95.8% on the obviously-malicious half of OpenSkillRisk, and in these runs it never flagged a harmless skill.

What is not: On contextually risky skills it detected 11% where the rules detected 33%. Whatever it is good at, it is not resolving ambiguity, and a deployment that relies on it alone will miss exactly the cases that need a judgement call.

What context is worth

Each recipe adds one layer and nothing else, and an unknown field is rejected rather than ignored, so a recipe cannot silently fall back to the full context and report a difference that was never sent.

Context is worth a few points of recall for roughly a quarter more input tokens, and the gain is monotonic across the recipes. On an unlabeled real-world corpus the same sweep showed nothing at all, because the model allowed every package at every recipe; an ablation cannot discriminate on a population where every answer is the same.

Llama Prompt Guard 2 as a pre-filter

meta-llama/Llama-Prompt-Guard-2-22M is a 22M-parameter DeBERTa-v2 classifier, 283 MB, with a 512-token context, so it needs no GPU: this run is CPU-only and took 45 minutes for 12,500 skills on 16 cores. Each skill is split into 512-token windows overlapping by 128 and scored by its maximum window probability, which favours detection, so a low recall cannot be blamed on the chunking.

Separation on 839 malicious and 545 benign records

It is effectively silent. AUC is 0.611, so there is a faint signal and it points the right way — unlike the System One result above, which is inverted — but the probabilities never reach a threshold anything could act on. Two of 1,384 records fire at 0.5 and both are benign.

One number here is misleading, so it is stated rather than quoted

Sweeping every threshold, the best achievable F1 is 77.4%, which is higher than the shipped single judge's 61.2% on the same records. That is class balance, not detection: it occurs at a threshold of 0.0015 with an 89.0% false-positive rate, which is very nearly “flag everything”, and the corpus is 60.6% malicious. The trivial always-flag classifier scores 75.5% F1 here, so the model beats it by 1.9 points. At the judge's own operating point of 17.6% false-positive rate its recall is 22.3% against the judge's 49.0%, and at a 1% false-positive rate its recall is 0.1%.

On real published skills it is not merely inert

The three highest-scoring records — 0.998, 0.995 and 0.993 — are a Chinese-language novel-writing checklist, a conversation-memory skill, and a prompt optimizer. All three are benign, and two causes account for the pattern. Skills containing more than 2% CJK characters (976 of 12,500) are flagged at 1.64% against 0.10% for the rest, a sixteen-fold difference, and sixteen of the 27 flagged records are CJK-heavy though such skills are 7.8% of the corpus. Separately, a prompt-engineering skill reads like an instruction-override attempt because directing a model is the product. Mean scores are nearly identical across the two groups, so the effect sits entirely in the high-confidence tail.

The same measurement, applied to this scanner

The deterministic rules flag CJK-heavy skills less often, not more: 0.58x at MEDIUM or above and 0.32x at HIGH or above, against Prompt Guard's sixteen-fold in the other direction. That is the expected shape for rules keyed on code constructs — curl | bash, base64 decoding, credential paths — which do not care what language the prose around them is in.

It does not follow that the scanner is unbiased here, and this corpus cannot settle it. These records are unlabelled, so a lower flag rate is consistent with two different explanations: those skills genuinely do less risky work, or the rules under-detect when the surrounding text is non-English. Separating them needs labelled non-English skills, which no corpus screened so far provides. The defensible claim is the narrow one — there is no evidence of over-flagging non-English skills, and the direction of any error is towards silence rather than noise. The INFO row moves the other way, which is consistent with the one hygiene rule that dominates that tier.

So enabling it as a screen would penalise non-English and prompt-engineering skills while detecting essentially none of the labelled malicious ones. This is a statement about fit, not about the model. Prompt Guard detects prompt-injection and jailbreak attempts in text; whether a skill is safe to install is a different question, and a skill can be malicious through a download-and-execute chain or a credential sink without containing anything an injection classifier would recognise. It is not integrated.