# NVIDIA SkillSpector vs Cisco Skill Scanner

Cisco AI Defense Skill Scanner (Cisco Skill Scanner below) and NVIDIA SkillSpector both scan agent skills: packaged instructions and code an AI agent can load. Both grade findings LOW, MEDIUM, HIGH or CRITICAL, so a severity threshold means the same thing on both sides.

Regenerated from `data/skillspector.json` (2026-10-16). Cells a run did not measure say so ("not instrumented", "not permitted", "no harmless class") instead of printing zero.

Both scanners run over one population of skill directories, byte-identical on each side, with every analyzer each tool ships enabled and neither tuned on these labels. Three arms: static analysis only, then both tools on Gemma 4 26B, then both on Claude Haiku 4.5.

Only records where both tools returned a usable result are scored. A crashed scan, an unparsable payload and a run with analyzers switched off all present as zero findings, so scoring them as clean would credit specificity that was never earned.

## What was compared

|  | Version | Configuration |
|---|---|---|
| NVIDIA SkillSpector | v2.11.2 (94f5cc80679c45a1fa4774e2d37ad663ca0be337) | All 27 analyzers enabled; built-in YARA; live OSV; baselines off; resource ceilings raised; --transitive off (corpus safety terms) |
| Cisco Skill Scanner | 4f8bec3 | Shipped core rule pack; judge + meta-judge; escalate-only verdict repair; policy suppressions empty; VirusTotal and AI Defense off (no credential) |

## Four decision lenses

Cisco Skill Scanner emits findings plus a package verdict. NVIDIA SkillSpector emits findings plus a 0-100 risk score and gates an install above 50; its score accumulates with diminishing returns per rule, so one severe finding need not gate.

A gate-to-gate comparison therefore measures two thresholds rather than two engines. Detection at a fixed severity is the primary lens; each tool's own gate is reported alongside it.

## Same model: Gemma 4 26B

Both tools on bedrock-mantle/google.gemma-4-26b-a4b.

### MaliciousSkillBench, source-disjoint split

1367 records scored on both tools: 528 benign, 839 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Gemma 4 26B) | 49.0% | 81.5% | 61.2% | 17.6% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector (Gemma 4 26B) | 80.0% | 65.7% | 72.1% | 66.5% |
| Any finding at HIGH or above | Cisco Skill Scanner (Gemma 4 26B) | 25.3% | 80.0% | 38.4% | 10.0% |
| Any finding at HIGH or above | NVIDIA SkillSpector (Gemma 4 26B) | 75.4% | 69.4% | 72.3% | 52.8% |
| Each tool's own shipped install gate | Cisco Skill Scanner (Gemma 4 26B) | 25.3% | 80.0% | 38.4% | 10.0% |
| Each tool's own shipped install gate | NVIDIA SkillSpector (Gemma 4 26B) | 38.9% | 74.4% | 51.1% | 21.2% |
| Block on any finding at all | Cisco Skill Scanner (Gemma 4 26B) | 92.6% | 61.9% | 74.2% | 90.7% |
| Block on any finding at all | NVIDIA SkillSpector (Gemma 4 26B) | 80.9% | 64.4% | 71.7% | 71.0% |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −31.0 points, 95% CI [−34.2, −27.5]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −50.2 points, 95% CI [−53.9, −46.5]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −13.6 points, 95% CI [−16.3, −10.7]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +11.7 points, 95% CI [+8.7, +14.9]: separable.

Agreement at MEDIUM: 58.6%; complementarity 41.4% (both 480, neither 321, only Cisco Skill Scanner 24, only NVIDIA SkillSpector 542). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | not instrumented | not instrumented | not instrumented |
| NVIDIA SkillSpector (Gemma 4 26B) | 26.018 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | 1384 | 1384 | 0 | 0 | 0 |
| NVIDIA SkillSpector (Gemma 4 26B) | 1384 | 1367 | 0 | 17 | 1043 |

### MaliciousSkillBench, balanced subset

793 records scored on both tools: 397 benign, 396 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Gemma 4 26B) | 82.1% | 78.5% | 80.2% | 22.4% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector (Gemma 4 26B) | 94.4% | 57.5% | 71.5% | 69.5% |
| Any finding at HIGH or above | Cisco Skill Scanner (Gemma 4 26B) | 65.9% | 93.2% | 77.2% | 4.8% |
| Any finding at HIGH or above | NVIDIA SkillSpector (Gemma 4 26B) | 86.4% | 63.2% | 73.0% | 50.1% |
| Each tool's own shipped install gate | Cisco Skill Scanner (Gemma 4 26B) | 65.9% | 93.2% | 77.2% | 4.8% |
| Each tool's own shipped install gate | NVIDIA SkillSpector (Gemma 4 26B) | 71.0% | 80.5% | 75.4% | 17.1% |
| Block on any finding at all | Cisco Skill Scanner (Gemma 4 26B) | 97.0% | 50.7% | 66.6% | 94.0% |
| Block on any finding at all | NVIDIA SkillSpector (Gemma 4 26B) | 95.2% | 54.6% | 69.4% | 79.1% |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −12.4 points, 95% CI [−16.2, −8.7]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −20.5 points, 95% CI [−25.0, −15.8]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −5.1 points, 95% CI [−9.1, −1.0]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +1.8 points, 95% CI [−0.3, +4.3]: not separable.

Agreement at MEDIUM: 66.0%; complementarity 34.1% (both 397, neither 126, only Cisco Skill Scanner 17, only NVIDIA SkillSpector 253). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | not instrumented | not instrumented | not instrumented |
| NVIDIA SkillSpector (Gemma 4 26B) | 23.174 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | 800 | 800 | 0 | 0 | 0 |
| NVIDIA SkillSpector (Gemma 4 26B) | 800 | 793 | 0 | 7 | 674 |

### OpenSkillRisk

343 records scored on both tools: 188 contextually risky, 155 obviously malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Gemma 4 26B) | 85.7% | no harmless class | no harmless class | no harmless class |
| Any finding at MEDIUM or above | NVIDIA SkillSpector (Gemma 4 26B) | 95.9% | no harmless class | no harmless class | no harmless class |
| Any finding at HIGH or above | Cisco Skill Scanner (Gemma 4 26B) | 71.4% | no harmless class | no harmless class | no harmless class |
| Any finding at HIGH or above | NVIDIA SkillSpector (Gemma 4 26B) | 86.3% | no harmless class | no harmless class | no harmless class |
| Each tool's own shipped install gate | Cisco Skill Scanner (Gemma 4 26B) | 71.4% | no harmless class | no harmless class | no harmless class |
| Each tool's own shipped install gate | NVIDIA SkillSpector (Gemma 4 26B) | 77.8% | no harmless class | no harmless class | no harmless class |
| Block on any finding at all | Cisco Skill Scanner (Gemma 4 26B) | 99.4% | no harmless class | no harmless class | no harmless class |
| Block on any finding at all | NVIDIA SkillSpector (Gemma 4 26B) | 98.2% | no harmless class | no harmless class | no harmless class |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −10.2 points, 95% CI [−14.0, −6.4]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −14.9 points, 95% CI [−19.0, −10.8]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −6.4 points, 95% CI [−10.5, −2.3]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +1.2 points, 95% CI [−0.3, +2.9]: not separable.

Agreement at MEDIUM: 86.9%; complementarity 13.1% (both 289, neither 9, only Cisco Skill Scanner 5, only NVIDIA SkillSpector 40). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | not instrumented | not instrumented | not instrumented |
| NVIDIA SkillSpector (Gemma 4 26B) | 23.519 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | 374 | 374 | 0 | 0 | 0 |
| NVIDIA SkillSpector (Gemma 4 26B) | 374 | 343 | 0 | 31 | 302 |

### HarmfulSkillBench

196 records scored on both tools: 196 unlabelled.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Gemma 4 26B) | 56.1% | not permitted | not permitted | not permitted |
| Any finding at MEDIUM or above | NVIDIA SkillSpector (Gemma 4 26B) | 84.2% | not permitted | not permitted | not permitted |
| Any finding at HIGH or above | Cisco Skill Scanner (Gemma 4 26B) | 38.8% | not permitted | not permitted | not permitted |
| Any finding at HIGH or above | NVIDIA SkillSpector (Gemma 4 26B) | 57.6% | not permitted | not permitted | not permitted |
| Each tool's own shipped install gate | Cisco Skill Scanner (Gemma 4 26B) | 38.8% | not permitted | not permitted | not permitted |
| Each tool's own shipped install gate | NVIDIA SkillSpector (Gemma 4 26B) | 29.1% | not permitted | not permitted | not permitted |
| Block on any finding at all | Cisco Skill Scanner (Gemma 4 26B) | 98.5% | not permitted | not permitted | not permitted |
| Block on any finding at all | NVIDIA SkillSpector (Gemma 4 26B) | 87.8% | not permitted | not permitted | not permitted |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −28.1 points, 95% CI [−35.7, −20.7]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −18.9 points, 95% CI [−26.5, −11.2]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +9.7 points, 95% CI [+2.0, +17.3]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +10.7 points, 95% CI [+6.6, +15.3]: separable.

Agreement at MEDIUM: 62.8%; complementarity 37.2% (both 101, neither 22, only Cisco Skill Scanner 9, only NVIDIA SkillSpector 64). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | not instrumented | not instrumented | not instrumented |
| NVIDIA SkillSpector (Gemma 4 26B) | 17.079 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Gemma 4 26B) | 200 | 200 | 0 | 0 | 0 |
| NVIDIA SkillSpector (Gemma 4 26B) | 200 | 196 | 0 | 4 | 161 |

## Same model: Claude Haiku 4.5

Both tools on us.anthropic.claude-haiku-4-5 via Bedrock. The Cisco Skill Scanner rows are the Haiku judge arm (the old page labelled them "core rules").

### MaliciousSkillBench, source-disjoint split

1322 records scored on both tools: 518 benign, 804 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Claude Haiku 4.5) | 45.8% | 82.0% | 58.7% | 15.6% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 84.6% | 64.1% | 72.9% | 73.6% |
| Any finding at HIGH or above | Cisco Skill Scanner (Claude Haiku 4.5) | 26.9% | 86.1% | 40.9% | 6.8% |
| Any finding at HIGH or above | NVIDIA SkillSpector | 74.5% | 67.5% | 70.8% | 55.8% |
| Each tool's own shipped install gate | Cisco Skill Scanner (Claude Haiku 4.5) | 26.9% | 86.1% | 40.9% | 6.8% |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 54.9% | 74.6% | 63.2% | 29.0% |
| Block on any finding at all | Cisco Skill Scanner (Claude Haiku 4.5) | 74.6% | 57.4% | 64.9% | 85.9% |
| Block on any finding at all | NVIDIA SkillSpector | 84.6% | 64.1% | 72.9% | 73.6% |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −38.8 points, 95% CI [−42.7, −35.2]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −47.6 points, 95% CI [−51.4, −43.9]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −28.0 points, 95% CI [−31.7, −24.4]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −10.0 points, 95% CI [−13.8, −6.2]: separable.

Agreement at MEDIUM: 49.3%; complementarity 50.7% (both 420, neither 232, only Cisco Skill Scanner 29, only NVIDIA SkillSpector 641). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Claude Haiku 4.5) | 13.763 | 13,367,982 | 2,107,191 |
| NVIDIA SkillSpector | 39.585 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Claude Haiku 4.5) | 1384 | 1351 | 0 | 33 | 44 |
| NVIDIA SkillSpector | 1384 | 1355 | 0 | 29 | 1045 |

### MaliciousSkillBench, balanced subset

766 records scored on both tools: 388 benign, 378 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (Claude Haiku 4.5) | 80.7% | 73.9% | 77.1% | 27.8% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 97.9% | 53.9% | 69.5% | 81.4% |
| Any finding at HIGH or above | Cisco Skill Scanner (Claude Haiku 4.5) | 63.2% | 89.2% | 74.0% | 7.5% |
| Any finding at HIGH or above | NVIDIA SkillSpector | 89.4% | 61.3% | 72.8% | 54.9% |
| Each tool's own shipped install gate | Cisco Skill Scanner (Claude Haiku 4.5) | 63.2% | 89.2% | 74.0% | 7.5% |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 81.5% | 73.7% | 77.4% | 28.3% |
| Block on any finding at all | Cisco Skill Scanner (Claude Haiku 4.5) | 92.9% | 50.4% | 65.3% | 89.2% |
| Block on any finding at all | NVIDIA SkillSpector | 97.9% | 53.9% | 69.5% | 81.7% |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −17.2 points, 95% CI [−21.2, −13.4]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −26.2 points, 95% CI [−31.0, −21.3]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −18.3 points, 95% CI [−22.8, −13.5]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −5.0 points, 95% CI [−7.4, −2.4]: separable.

Agreement at MEDIUM: 62.8%; complementarity 37.2% (both 407, neither 74, only Cisco Skill Scanner 6, only NVIDIA SkillSpector 279). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (Claude Haiku 4.5) | 19.865 | 8,650,465 | 1,814,253 |
| NVIDIA SkillSpector | 30.982 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (Claude Haiku 4.5) | 800 | 783 | 0 | 17 | 34 |
| NVIDIA SkillSpector | 800 | 783 | 0 | 17 | 674 |

## Static only, no model

Deterministic analyzers on both sides. No LLM.

### MaliciousSkillBench, source-disjoint split

1368 records scored on both tools: 529 benign, 839 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (core rules) | 8.0% | 60.4% | 14.1% | 8.3% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 34.3% | 56.1% | 42.6% | 42.5% |
| Any finding at HIGH or above | Cisco Skill Scanner (core rules) | 7.8% | 61.9% | 13.8% | 7.6% |
| Any finding at HIGH or above | NVIDIA SkillSpector | 14.5% | 60.1% | 23.4% | 15.3% |
| Each tool's own shipped install gate | Cisco Skill Scanner (core rules) | 7.8% | 61.9% | 13.8% | 7.6% |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 1.3% | 57.9% | 2.6% | 1.5% |
| Block on any finding at all | Cisco Skill Scanner (core rules) | 60.6% | 52.5% | 56.2% | 87.0% |
| Block on any finding at all | NVIDIA SkillSpector | 34.3% | 55.4% | 42.4% | 43.9% |

- **Any finding at MEDIUM or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −26.3 points, 95% CI [−29.8, −23.2]: separable.
- **Any finding at HIGH or above**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, −6.8 points, 95% CI [−9.3, −4.1]: separable.
- **Each tool's own shipped install gate**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +6.4 points, 95% CI [+4.8, +8.3]: separable.
- **Block on any finding at all**: paired recall difference, Cisco Skill Scanner minus NVIDIA SkillSpector, +26.2 points, 95% CI [+21.6, +31.0]: separable.

Agreement at MEDIUM: 66.2%; complementarity 33.8% (both 81, neither 825, only Cisco Skill Scanner 30, only NVIDIA SkillSpector 432). Complementarity is the share of records exactly one tool flags.

| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (core rules) | 0.015 | not instrumented | not instrumented |
| NVIDIA SkillSpector | 2.453 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (core rules) | 1384 | 1384 | 0 | 0 | 11 |
| NVIDIA SkillSpector | 1384 | 1368 | 0 | 16 | 1043 |

### MaliciousSkillBench, balanced subset

794 records scored on both tools: 397 benign, 397 malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (core rules) | 33.0% | 94.2% | 48.9% | 2.0% |
| Any finding at MEDIUM or above | Cisco Skill Scanner (all rule packs) | 94.5% | 53.0% | 67.9% | 83.9% |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 48.1% | 62.6% | 54.4% | 28.7% |
| Any finding at HIGH or above | Cisco Skill Scanner (core rules) | 31.2% | 95.4% | 47.1% | 1.5% |
| Any finding at HIGH or above | Cisco Skill Scanner (all rule packs) | 91.4% | 56.5% | 69.8% | 70.5% |
| Any finding at HIGH or above | NVIDIA SkillSpector | 30.5% | 75.6% | 43.5% | 9.8% |
| Each tool's own shipped install gate | Cisco Skill Scanner (core rules) | 31.2% | 95.4% | 47.1% | 1.5% |
| Each tool's own shipped install gate | Cisco Skill Scanner (all rule packs) | 91.4% | 56.5% | 69.8% | 70.5% |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 8.6% | 91.9% | 15.7% | 0.8% |
| Block on any finding at all | Cisco Skill Scanner (core rules) | 81.9% | 47.9% | 60.4% | 89.2% |
| Block on any finding at all | Cisco Skill Scanner (all rule packs) | 99.8% | 50.1% | 66.7% | 99.5% |
| Block on any finding at all | NVIDIA SkillSpector | 50.1% | 63.2% | 55.9% | 29.2% |


| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (core rules) | 0.017 | not instrumented | not instrumented |
| Cisco Skill Scanner (all rule packs) | 0.280 | not instrumented | not instrumented |
| NVIDIA SkillSpector | 2.399 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (core rules) | 800 | 800 | 0 | 0 | 18 |
| Cisco Skill Scanner (all rule packs) | 800 | 800 | 0 | 0 | 18 |
| NVIDIA SkillSpector | 800 | 794 | 0 | 6 | 674 |

### OpenSkillRisk

335 records scored on both tools: 182 contextually risky, 153 obviously malicious.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (core rules) | 25.1% | no harmless class | no harmless class | no harmless class |
| Any finding at MEDIUM or above | Cisco Skill Scanner (all rule packs) | 96.4% | no harmless class | no harmless class | no harmless class |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 77.9% | no harmless class | no harmless class | no harmless class |
| Any finding at HIGH or above | Cisco Skill Scanner (core rules) | 21.5% | no harmless class | no harmless class | no harmless class |
| Any finding at HIGH or above | Cisco Skill Scanner (all rule packs) | 91.6% | no harmless class | no harmless class | no harmless class |
| Any finding at HIGH or above | NVIDIA SkillSpector | 62.7% | no harmless class | no harmless class | no harmless class |
| Each tool's own shipped install gate | Cisco Skill Scanner (core rules) | 21.5% | no harmless class | no harmless class | no harmless class |
| Each tool's own shipped install gate | Cisco Skill Scanner (all rule packs) | 91.6% | no harmless class | no harmless class | no harmless class |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 33.4% | no harmless class | no harmless class | no harmless class |
| Block on any finding at all | Cisco Skill Scanner (core rules) | 97.6% | no harmless class | no harmless class | no harmless class |
| Block on any finding at all | Cisco Skill Scanner (all rule packs) | 100.0% | no harmless class | no harmless class | no harmless class |
| Block on any finding at all | NVIDIA SkillSpector | 78.2% | no harmless class | no harmless class | no harmless class |


| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (core rules) | 0.018 | not instrumented | not instrumented |
| Cisco Skill Scanner (all rule packs) | 0.266 | not instrumented | not instrumented |
| NVIDIA SkillSpector | 2.457 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (core rules) | 374 | 374 | 0 | 0 | 5 |
| Cisco Skill Scanner (all rule packs) | 374 | 365 | 0 | 9 | 14 |
| NVIDIA SkillSpector | 374 | 343 | 0 | 31 | 302 |

### HarmfulSkillBench

197 records scored on both tools: 197 unlabelled.

| Lens | Tool | Recall | Precision | F1 | False-positive rate |
|---|---|---|---|---|---|
| Any finding at MEDIUM or above | Cisco Skill Scanner (core rules) | 4.1% | not permitted | not permitted | not permitted |
| Any finding at MEDIUM or above | Cisco Skill Scanner (all rule packs) | 90.9% | not permitted | not permitted | not permitted |
| Any finding at MEDIUM or above | NVIDIA SkillSpector | 56.9% | not permitted | not permitted | not permitted |
| Any finding at HIGH or above | Cisco Skill Scanner (core rules) | 3.5% | not permitted | not permitted | not permitted |
| Any finding at HIGH or above | Cisco Skill Scanner (all rule packs) | 83.8% | not permitted | not permitted | not permitted |
| Any finding at HIGH or above | NVIDIA SkillSpector | 17.3% | not permitted | not permitted | not permitted |
| Each tool's own shipped install gate | Cisco Skill Scanner (core rules) | 3.5% | not permitted | not permitted | not permitted |
| Each tool's own shipped install gate | Cisco Skill Scanner (all rule packs) | 83.8% | not permitted | not permitted | not permitted |
| Each tool's own shipped install gate | NVIDIA SkillSpector | 2.5% | not permitted | not permitted | not permitted |
| Block on any finding at all | Cisco Skill Scanner (core rules) | 96.5% | not permitted | not permitted | not permitted |
| Block on any finding at all | Cisco Skill Scanner (all rule packs) | 100.0% | not permitted | not permitted | not permitted |
| Block on any finding at all | NVIDIA SkillSpector | 56.9% | not permitted | not permitted | not permitted |


| Tool | Mean seconds per skill | Input tokens | Output tokens |
|---|---|---|---|
| Cisco Skill Scanner (core rules) | 0.017 | not instrumented | not instrumented |
| Cisco Skill Scanner (all rule packs) | 0.325 | not instrumented | not instrumented |
| NVIDIA SkillSpector | 2.400 | not instrumented | not instrumented |

| Tool | Rows | Usable | Errors | Capability degraded | Reported partial |
|---|---|---|---|---|---|
| Cisco Skill Scanner (core rules) | 200 | 200 | 0 | 0 | 19 |
| Cisco Skill Scanner (all rule packs) | 200 | 200 | 0 | 0 | 19 |
| NVIDIA SkillSpector | 200 | 197 | 0 | 3 | 160 |

## Limits on what these numbers support

Our rule packs were tuned against MaliciousSkillBench during development. NVIDIA SkillSpector has not seen it. The source-disjoint split and the corpora outside MSB are therefore the ones that carry weight, and figures on the balanced subset should be read as home ground. NVIDIA SkillSpector was tuned on a private 31,000-skill set that is not published, so contamination can be bounded in one direction only and is disclosed rather than measured in the other.

Enabling every rule pack Cisco Skill Scanner ships is not its strongest configuration: it is a triage setting rather than a gating one, so the shipped core profile is what appears above; the ATR pack is also excluded from source-disjoint claims by the corpus terms.

One NVIDIA SkillSpector capability is absent from these runs. Its --transitive mode follows references out of a skill, which on this corpus would mean fetching attacker-controlled URLs, and the corpus safety defaults forbid it. Its supply-chain recall would likely be higher with that enabled, so the figures here understate it on that axis.

MaliciousSkillBench stores each record as one SKILL.md. Analyzers that operate on bundled scripts or MCP manifests have nothing to read, on both sides, and report not_applicable rather than failing. Three of NVIDIA SkillSpector's 27 analyzers are inert on that corpus for this reason. OpenSkillRisk and HarmfulSkillBench carry the multi-file signal.

NVIDIA SkillSpector reports analysis as partial on most MSB records. Coverage is 100%, nothing is left uninspected, and the ledger exceptions are non-fatal: each SKILL.md references files the corpus does not ship, and the tool says so. That is a reporting capability Cisco Skill Scanner does not have, and the partial rate is recorded in the integrity table rather than treated as a defect or used to exclude rows.

Finding counts are not compared directly. NVIDIA SkillSpector applies diminishing returns per rule and caps at three occurrences, so its issue count and Cisco Skill Scanner's finding count measure different things. Distinct rules fired and records flagged are used instead.

## Run notes

- Both tools ran at shipped defaults with every capability enabled, and neither was tuned on these labels. The configuration is recorded in each run manifest so it can be checked rather than trusted.
- Only records where both tools returned a usable result are scored. A crashed scan, an unparsable payload, or a run with analyzers switched off all present as “no findings”, and scoring those as clean would award free specificity.
- Our Gemma 4 column reuses the per-record rows from the previously published judge run rather than re-scanning. The corpora are byte-identical, verified by digest over 400 records; only directory names changed and label sidecars moved out of scan scope.
- Gemma 4 is reachable only through the Bedrock mantle route, which authenticates with SigV4. NVIDIA SkillSpector speaks OpenAI-compatible with a static bearer token, so a loopback signing proxy was used for that arm alone. Its Haiku arm used its native Bedrock provider.
- HarmfulSkillBench figures on our side predate a fix that moved a label-bearing _meta.json out of the scanned directory, so that corpus is marked label-contaminated for us and should be read with that in mind.

Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content. Evaluation only: nothing here is approved for training, synthetic generation, teacher context, distillation or redistribution.