Reproduce it

Reproducing these results

Every figure on this site comes from code in the scanner repository, run against corpora pinned by revision and verified by content digest. Nothing is hand-entered. These are the actual commands, in the order they need to run.

What you need first

Requirement	Detail
The scanner, installed from source	git clone the repository, create a virtual environment, then `pip install -e .` plus the pinned `huggingface-hub` and `pyarrow`. Building the CEL helper needs Go 1.27.1 or newer.
AWS credentials that can reach Bedrock	The judge model is served only on the Bedrock mantle OpenAI-compatible route and authenticates with SigV4, so the normal credential chain is enough. No API key is involved.
A hosted System One credential, for that arm only	One environment variable, read from wherever you keep secrets. The deterministic and judge results need nothing beyond AWS.

The commands, in order

1. Acquire the corpus

python evals/datasets/materialize_malicious_skill_bench.py --source-dir SRC --output-dir OUT

Downloads only the ten files named in the committed inspection profile, checks each byte hash against the dataset lock, and writes one inert SKILL.md per row. It refuses to proceed on a hash mismatch. Expect 9,740 declared artifacts, 9,737 usable, 3 quarantined.

2. Reproduce the published baseline

python evals/runners/public_dataset_benchmark.py --snapshot-dir OUT --dataset-id ProtectSkills/MaliciousSkillBench --profile release --cel-mode off --output baseline.json

This must produce F1 13.742%, precision 60.748%, recall 7.747% on the source-disjoint track, and a population digest matching the lock. If it does not, stop: nothing downstream is trustworthy.

3. Run the judged arms

SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=1 python evals/runners/judged_dataset_benchmark.py --snapshot-dir OUT --dataset-id ProtectSkills/MaliciousSkillBench --cel-mode off --arm core_judge --arm core_judge_meta --arm core_meta --judge-model bedrock-mantle/google.gemma-4-26b-a4b --output judged.json

About an hour per arm on 1,384 packages. Each arm is written as it finishes, so a later failure does not discard earlier work. The report is stamped non-blocking and can never be mistaken for release evidence.

4. Run the System One arm

python evals/system_one/runner.py --corpus CORPUS --context K7 --instruction I4 --question SQ2 --model jev-1.13.0 --model-revision jev-1.13.0 --concurrency 8 --max-usd 5 --output jev.jsonl

42 seconds for the whole 1,384-package track. The runner writes a completion attestation beside the predictions, and the scorer refuses to read a run that did not finish.

5. Emit the rule and judge stages in the same shape

python evals/system_one/deterministic.py --corpus CORPUS --tier deterministic --output rules.jsonl

Add --judge-model to produce the judge stage instead. This exists so all three stages share case identifiers and can be composed.

6. Score the chains

python evals/system_one/scorer.py --labels LABELS.json --chain "rules_then_jev=rules.jsonl,jev.jsonl" --output cascade.json

Repeat --chain for each combination. Every chain is reported under both lenses. The composition short-circuits only on a hard block and never lets a later stage lower an earlier decision.

7. Establish the noise floor

python evals/experiments/e16_stability.py --corpus CORPUS --max-skills 200 --repeats 5 --output stability.json

Run this before comparing any two judged configurations. It also re-runs the deterministic stage as a control, which must not move at all.

8. Rebuild this site

python evals/publish/build_space.py --output SITE --baseline baseline.json --judged-arm ... --cost cost-comparison.json

Renders from the result files only. A section whose file is missing, or whose completion attestation is false, prints as missing rather than as zeros.

What the harness guarantees

Samples are never executed. The corpus lock sets that as a default and the materializer writes text only, at mode 0600.
Corpora are pinned by revision and verified by content digest, so a shrunken or substituted population cannot pass unnoticed.
Results are written to new paths. No scorecard or prediction file is ever modified in place.
Every prediction row is value-free: identifiers, actions and probabilities, never sample text.
Per-corpus licence restrictions are honoured. Several forbid exactly the metric one would reach for first, so each corpus is reported only with the figure its licence permits.

Four things that will waste your afternoon

Each of these cost us a run before we found it.

Set SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=1 for any judged run. Without it the judge contradicts itself on roughly a fifth of harmless skills and its whole analysis is discarded for those, which silently flatters its measured precision.
Do not run scripts from a shared temporary directory. A stray module there can shadow a real library on the import path; one did, and it made the judge fail silently with zero recorded tokens.
The judge's token usage must be read from the LLM analyzer, not the meta analyzer. Reading only the latter reports zero tokens for the judge arm and makes its cost unmeasurable.
Check the meta-judge's routing decision, not its routing reason text. Keying on the reason reported that meta had never run when it had run on most packages.