Cisco AI Defense skill scanner: evaluation results

What fraction of deterministic findings are actually real?

Two adjudicators from different vendors differ by roughly fivefold with non-overlapping intervals, and chance-corrected agreement is near zero, so no single model can serve as the arbiter.

The corpus is unlabeled real-world content, so this estimates a rate rather than establishing truth.

Does one batched request per skill match one request per finding?

The batched form reproduces the aggregate rate at a third of the requests, but agrees with the per-finding arm no better than chance on which findings are real. Sound for an estimate, unsound for a per-finding suppression decision.

Which links in the cascade earn their cost?

The full chain of rules, System One model and judge scores highest on both lenses, but the judge buys its recall with an order-of-magnitude rise in false positives. Stopping after the System One model retains almost all of the quality at a fraction of the false-positive rate, which is the better trade when most content is legitimate. The System One model also absorbs some of the judge's calls.

Scored on 400 malicious and 400 benign packages drawn from the whole snapshot, so these are not directly comparable to the published test-partition figures.

Can false positives be removed by gating on the model's own confidence?

No. The disposition format already produces no false positives, so there is nothing to gate, and on the eight-probe format no confidence floor removes a false positive without also losing a true detection. The model's confidence does not separate its own mistakes.

Does telling the model to judge by detected content type beat a generic prompt?

Yes. Naming the instruction surface explicitly and directing the model to judge each file by its detected content type rather than its name gave the best any-intervention F1 and the highest recall, at no cost to precision.

The generic variant still wins on the block-only lens, because it blocks where the others confirm.

Does the self-hosted System One model match the hosted one?

Not answered, and the reason is infrastructure rather than the model. The self-hosted model does speak the same protocol, and on a clear case it agreed with the hosted one at lower confidence. But a full sweep returned usable answers for only a sixth of requests, and a single real request later timed out entirely at two minutes. The serving process was also replaced partway through by a different one on the same ports. Every usable answer was 'allow', which does not match the hosted model's behaviour on the same packages and is a further sign the deployment rather than the model was being measured. Publishing any figure from that would describe the machine, so none is published.

Needs an uncontended machine, or coordination with whoever owns the work on that box. Nothing there was stopped or reconfigured to make room.

Does remembering a decision save work on a re-scan?

excluding the line number reuses 6.8 times more decisions than the existing identity, but still loses a substantial minority of decisions, so memory reduces judge calls without eliminating re-adjudication

a five-line header inserted in every text file plus re-indentation of every line, which is more aggressive than typical churn

Can a dismissal be generalised into a safe suppression rule?

Rarely, at this granularity. Most candidates would have silenced real detections: replaying them over a labelled population rejected the large majority for suppressing findings on malicious packages at HIGH or CRITICAL severity. Only a small fraction of benign findings can be suppressed safely by rule and file role, so useful generalisation needs finer typed facts than that.

Shadow-mode proposals only. Nothing is applied; every entry goes to human review.

Can an organisation express a policy the generic rules cannot?

Yes. Every other domain control is an allowlist that demotes; a denied-destination section was added that promotes instead, matching on the registrable domain at label boundaries so a lookalike domain is not flagged. Asked to generalise one observed host, the model widened it to the wildcard form, named the owning entity, and identified a sibling domain the same entity owns that a naive registrable-domain extraction would have missed.

Inert unless an organisation configures it, so the default scan is unchanged. Proposals go to review, never applied silently.

Which model decisions deserve to become deterministic rules?

A large share of what the model confidently blocks has no matching rule, so the same judgement is re-derived on every call. On the disposition format every single such case is genuinely malicious, which makes it a clean rule-authoring queue.

Candidates and evidence only. Rules stay human-authored.

Can the policy be fitted automatically without losing a detection?

Barely. Only one single-rule change both improved F1 and left every package caught on a HIGH or CRITICAL malicious finding still caught, and the gain was within rounding. Meanwhile a large share of the tempting changes would have dropped a real detection, which is exactly the trade a tuner optimising F1 alone would have made.

Proposed diff only. Nothing is applied, and every line carries its measured effect.

Does a stateful scanner work, and is it safe?

It works, and the first version was unsafe. A drift-tolerant fingerprint means one dismissal generalises across the corpus, which is the point of it: a bounded set of model dismissals hid most findings on a re-scan. But unrestricted, that generalisation also silenced dozens of HIGH and CRITICAL detections on malicious packages. Adding a ceiling, so a model dismissal can never suppress a high-severity finding while a human review still can, keeps almost all of the saving and loses none of those detections.

Adjudications are model-made and labelled as such; none is ground truth.

How far must two judged arms differ before the difference is real?

Further than most of the differences worth arguing about. Repeating one configuration over the same packages changed the package verdict on a substantial minority of them, almost always between allowing and asking for review. The deterministic core did not move at all, so the variation is the model rather than the harness. Any gap between two judged arms smaller than this cannot be distinguished from run-to-run noise, which is what makes the meta-judge's apparent effect indistinguishable from nothing.

Temperature zero is not determinism. A difference between two judged arms smaller than the flip rate here cannot be distinguished from run-to-run variation.

What does the judge do on a corpus built to separate clear risk from contextual risk?

It closes almost the whole gap on the clear cases and behaves as it should on the ambiguous ones. On skills the corpus authors labelled obviously malicious, the rules caught under a third and the judge caught every one. On skills labelled contextually risky, where the right answer genuinely depends on context, it caught a large majority but not all, which is the discrimination you would want rather than a flat improvement.

Both groups are positive-risk, so this corpus supports a detection rate and not a false-positive rate. Its licence permits exactly that use.

What does the judge do on a corpus the rules almost entirely miss?

It finds most of what the rules cannot see. The deterministic core flagged a few percent of these skills, matching the figure published for it, and adding the judge raised detection more than tenfold. This corpus is the clearest case for the judge earning its cost, because the rules have almost no purchase on it at all.

Its licence permits a positive-risk detection diagnostic only; no F1 or false-positive claim.