What changes when the LLM judge and the System One model are switched on.
Run the deterministic rules on every skill, and add the LLM judge wherever a person reads the output. Block at HIGH and send MEDIUM to review: with the judge on, a MEDIUM threshold catches twice the malicious skills that HIGH does, and also flags more than twice the harmless ones. Pick the preset by review capacity: low-noise for your own skills, balanced for vetting third-party skills, and quiet only with the judge, when the review queue is the binding constraint.
| Stage | Recommended setting | Why | Cost per skill | What to watch |
|---|---|---|---|---|
| Deterministic rules | on by default; --policy low-noise for your own skills | Free, reproducible and milliseconds per skill. Tuned against every real skill: 4.18% to 2.14% of real skills flagged at MEDIUM+, and the held-out false-positive rate 8.44% to 4.22% with recall unchanged. | milliseconds per skill | Alone, it catches 8% of held-out malicious skills: a floor for unambiguous patterns, not a judge of intent. |
| LLM judge | SKILL_SCANNER_LLM_MODEL=... skill-scanner scan DIR --use-llm --fail-on-severity high | The new prompt separates a skill's purpose from misuse: held-out recall 65.9% at 12.5% FPR (F1 75.8%, from 63.7% with the shipped prompt). | about 3,800 input and 90 output tokens per skill | Flags about 10% of real skills at MEDIUM+: review those, block only at HIGH. |
| LLM caps | llm_analysis.low_confidence_max_severity / contextual_risk_max_severity: LOW (on in low-noise / quiet) | Report the judge's low-confidence findings at LOW (real-skill flags 10.5% to 7.6%), or its contextual-risk findings (to 3.0%); quiet applies both. | none | The contextual cap costs 16 points of held-out recall; use it when the queue, not the attacker, is the constraint. |
| Verdict repair | on by default; leave it on | A SAFE verdict with findings contradicts itself, and the strict path discarded the analysis, almost always on benign skills. Repair escalates to SUSPICIOUS and keeps the findings. | none | Escalate-only by design; SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=0 restores the strict path. |
| Meta-analyzer | leave --enable-meta off | Once a schema-wiring defect that made it a no-op was fixed, it cost 16.4 points of recall for 0.3 points of false-positive rate. | a second model call per skill | Any earlier result with meta on over the mantle route measured a no-op. |
| Decomposed judge | --llm-decompose, for high-stakes reviews only | One pass per focus, findings unioned: held-out recall 49.0% to 60.9% with the earlier prompt. | three to five times the tokens | A control arm attributed about two thirds of the gain to sampling variance; not re-measured with the new prompt. |
| OpenJev screen | measured, not shipped; --system-one-endpoint is advisory only | Selected on train/validation, an eight-probe screen in front of the judge gave 61.3% held-out recall at 2.4% FPR, with 3.1% of real skills reaching the judge. | a small local model per skill | Not a product configuration yet: the shipped tier records a probability and cannot change a finding. |
skill-scanner scan ./my-skill --policy low-noise (pre-commit: "policy": "low-noise", "severity_threshold": "high")
No keys, nothing leaves the machine, milliseconds per skill. low-noise reports at LOW the 11 rules that most often flag a real skill the judge considers harmless, and on the held-out split gives up one detection of 839.
Expect: 1.85% of real skills blocked at HIGH, 1.93% flagged at MEDIUM+.
reusable workflow with policy: low-noise, fail_on_severity: high (SARIF annotations show MEDIUM)
Your own skills are trusted for intent; the gate is for mistakes and copied-in patterns, which the rules catch deterministically, so a failing build is reproducible.
Expect: About 1.9 failing skills in a hundred on the real-world population measured here; more if your skills install software or handle credentials.
skill-scanner scan DIR --use-llm --fail-on-severity high --format json (review everything at MEDIUM+)
Most malicious skills contain nothing a pattern can prove -- an instruction to run a bundled script on a trigger, or to send data where the purpose does not need it -- which is what the judge reads for.
Expect: Held out: 66.7% of malicious skills reach review at 15.4% FPR, 33.7% are blocked at 6.4%. Real skills: 11.7% reviewed, 3.8% blocked.
skill-scanner scan DIR --use-llm --policy quiet --fail-on-severity high
quiet demotes 19 rules and caps the judge's low-confidence and contextual findings at LOW, so they are still reported but do not queue. Never use it without the judge: alone, its extra demotions halve what the rules catch on the held-out split.
Expect: Held out: 50.3% reach review at 7.2% FPR, 33.1% blocked. Real skills: 3.9% reviewed, 2.9% blocked.
skill-scanner scan-all DIR --recursive --policy low-noise, plus the judge where the budget allows
The judge is the cost at this scale: 7.1 billion input tokens for 1.88 million skills. The rules took about ninety minutes on one 16-core host.
Expect: 1.93% of real skills from the rules at MEDIUM+; the judge agrees with 47% of them where it read the whole skill.
SKILL_SCANNER_LLM_PROVIDER=openai SKILL_SCANNER_LLM_BASE_URL=http://127.0.0.1:8000/v1 skill-scanner scan DIR --use-llm
Gemma 4 26B served locally with vLLM reproduces its hosted results. Serve it with xgrammar and disable_any_whitespace, or it pads its JSON to the token limit and loses 46.5% of analyses.
Expect: The rates above, with nothing sent off the host.
skill-scanner scan DIR --policy strict --rule-packs atr promptguard --use-llm --llm-decompose --format json --detailed
Every community pack raised recall to 73.8% on a sample of the test split and the benign flag rate to 92.5%: useful for finding everything a skill might do, useless as a gate.
Expect: Most skills flag; read the output, never block on it.
A scan with the judge on flags a skill when either layer reports MEDIUM or above, so each preset is measured as that union, with its rule demotions and LLM caps applied. Held out: MaliciousSkillBench's frozen test split, used to design nothing. Real skills: 2,000 sampled uniformly, a flag rate that bounds the false-positive rate from above.
| Configuration | Review at MEDIUM+: recall | FPR | Real skills | Block at HIGH+: recall | FPR | Real skills |
|---|---|---|---|---|---|---|
balanced, rules only | 8.0% | 4.2% | 2.00% | 7.7% | 4.0% | 1.85% |
balanced + judge | 66.7% | 15.4% | 11.65% | 33.7% | 6.4% | 3.80% |
low-noise, rules only | 7.9% | 4.2% | 1.65% | 7.7% | 4.0% | 1.65% |
low-noise + judge | 63.2% | 13.4% | 8.65% | 33.7% | 6.4% | 3.60% |
quiet, rules only | 4.2% | 3.7% | 1.10% | 3.7% | 3.3% | 1.00% |
quiet + judge | 50.3% | 7.2% | 3.85% | 33.1% | 5.5% | 2.90% |
That first pitfall is measured, not hypothetical: repeating one judged configuration 5 times over the same packages changed the verdict on 14.00% of them, while the deterministic rules did not move once.
The full guide, with the commands for each setup, is Recommended settings in the repository.