Cisco AI Defense skill scanner: evaluation results

Corpus shape decides how a skill can be packed

The benchmark corpora store skills as one or two text fields, not as directories, so they almost always fit a System One model's context whole. Real-world skills do not. Any packing or cost conclusion drawn from the small golden fixtures would be wrong about real input.

Population	Median files	Median scanner-relevant bytes	Fits a 32k-token window whole
MaliciousSkillBench rows	1 (single text field)	2,936	essentially all
ClawHub rows	2 (skill_md + bundle)	24,173	most
Golden fixtures	3	1,797	100%
Real-world skills	80	576,745	16%

The packing ladder

Three rungs, tried in order. A sends the whole record when it fits, which covers essentially every benchmark row. B fills the budget by priority when it does not: SKILL.md first because it fits 96% of the time and is the instruction surface where prompt injection lives, then scripts it references, then remaining code ordered by detected content type rather than file extension, then documentation truncated hardest. C falls back to one request per candidate finding for the long tail.

Measured, the ladder puts every benchmark row on rung A and 84% of real-world skills on rung B.

Rules that hold on every rung

• The matched evidence is never truncated; only the window around it is, and truncation keeps windows containing security-relevant terms so an obfuscated payload's middle survives.
• An oversize request is never sent. A provider that truncates and still answers answers confidently about content it never saw, so the oversize path returns not-applicable and falls back to the deterministic decision.
• Untrusted sample text is neutralized so a malicious manifest cannot close the harness's own delimiters and have its instructions read as policy.
• No benchmark label, and no other tier's verdict, ever enters the context. Including a verdict would destroy arm independence and make cascade scoring circular.

Judge model, verified rather than assumed

Property	What was measured
One model, one route	google.gemma-4-26b-a4b is reachable only on the Bedrock mantle OpenAI-compatible route. It is absent from list-foundation-models, there is no model-listing endpoint, and no other Gemma 4 size exists, so the model id is a pinned constant.
Strict schema works, after one removal	Schema-constrained JSON succeeds once uniqueItems is stripped; the mantle validator rejects it outright and the request would otherwise degrade silently to loose JSON mode.
Context is ample	A 124,989-byte request was accepted at 40,855 prompt tokens, so the scanner's 100,000-character budget fits with headroom.
Prompt caching does not engage	cached_tokens and cache_write_tokens stayed at zero across two byte-identical 4,859-token requests, so the largest assumed cost lever is unavailable on this route.
Code tokenizes worse than prose	Measured at 3.06 bytes per token on repetitive Python, not the 4.0 a naive estimator assumes, which is why budgets are enforced in bytes at a pessimistic 2 bytes per token.

Reproducibility discipline

• Every analysis is gated on a completion attestation and on the on-disk digest matching the manifest, so a partial run cannot be published as a finished one.
• Results are written to new paths; no scorecard or prediction file is modified in place.
• Integrity counters travel with every result: row counts, error rows, unparsable rows and the size of any unjudgeable subset.
• Per-corpus licence restrictions are honoured. Several forbid exactly the metric one would reach for first, so the permitted metrics are recorded per corpus rather than left to discipline.
• Samples are never executed, and no sample content is republished.