Appendix: corpora, method, experiments, cost, reproduce, glossary

Corpora

Each corpus is reported only with the figure its licence permits. The source-disjoint MaliciousSkillBench split is the held-out test set: it was scored once and used to design nothing. Unlabelled real skills give flag rates, which bound the false-positive rate from above.

How to read the first table: one row per corpus, with its size, its labels and what its licence allows. A corpus with no benign class can report recall or detection, never precision or false positives. HarmfulSkillBench allows recall only.

Records both tools scored, per cross-tool arm

The head-to-head counts a record only when both tools returned a usable result. A crashed scan looks like zero findings, so counting it as clean would credit a tool for a failure.

Previously published baselines, re-measured

The rule-only figures were re-measured from pinned corpus revisions before anything new was trusted. Three listed corpora were not re-measured, for the reasons given.

Method

Three stages of Cisco Skill Scanner can inspect a skill: deterministic rules, the Decision Model (OpenJev, jev-1.13.0) and an LLM judge. Older pages and the results dataset call the Decision Model "System One model" or "jev"; the tables here use the new name. The judge is Gemma 4 26B-A4B on the Bedrock mantle route unless Claude Haiku 4.5 is named. A chain lets a later stage raise a decision but never lower it.

NVIDIA SkillSpector v2.11.2 ran with all 27 analyzers on. It emits findings plus a 0 to 100 risk score, and gates an install above 50. Both tools grade findings LOW, MEDIUM, HIGH or CRITICAL. So detection at a fixed severity is the main lens, and each tool's own gate is reported beside it.

Two scoring lenses recur. Block-only counts a catch only when the skill is blocked. Any-intervention also counts a review flag as a catch. Severity gates are MEDIUM or above (review) and HIGH or above (the shipped block gate). The ranking of configurations can invert between lenses, so every figure names its lens.

The judge model, verified rather than assumed

How skills are packed into a request

Discipline

Measurement notes

What a skill looks like

Experiments E1 to E16

Sixteen side questions, one row each. Statuses are as measured; E3's detail report and E1's adjudicator detail block were absent or empty in the old build and are not repeated. E5 below is the context ablation for the Decision Model (OpenJev).

How to read the ledger: each row asks one question and gives the measured answer. "Not answered" means the run could not complete, not that the answer was negative. Negative answers are kept, because they also tell you what not to deploy.

E5: what context is worth to the Decision Model (OpenJev)

Each recipe adds one kind of context to what the Decision Model (OpenJev) reads. Compare F1 against input tokens: more context costs tokens on every skill.

Cost and tokens, reconciled

One judge pass costs 4,838 input and 300 output tokens per skill, measured on 120 source-disjoint records. That is the current figure. Totals divided by records give the other per-skill rows. No cost is quoted for three decomposed passes because the analyzer's usage counter reported the last call rather than the sum.

What each stage costs on the 1,384-skill test split

Wall clock and hardware

Reproduce

The commands below rebuild every figure from the corpora. Samples are never executed and no sample content is published. Run them in order; each step writes new files and never edits an earlier result.

Requirements

Commands, in order

Scripts for the large-scale run

Guarantees

Mistakes that each cost a run

Glossary

Every column definition that was a hover tooltip on the old pages, and every term the method defines.