{
 "meta": {
  "id": "reproduce",
  "title": "Reproducing these results",
  "sources": [
   "reproduce.html#1",
   "reproduce.html (prose)",
   "large-scale.html#17",
   "methodology.html#2",
   "methodology.html (prose)",
   {
    "repo": "Vineethsain/skill-scanner-vs-skillspector",
    "path": "results/large-scale.json",
    "sha256": "e48f9cb342c05d2d13286c6b91c165411994d6febd71e14d1fe54c6f339d32b5"
   }
  ],
  "extracted_at": "2026-10-16T14:34:41+00:00",
  "notes": [
   "commands: the eight steps in order, verbatim; requirements: reproduce.html#1; scripts: large-scale.html#17.",
   "judge_model_properties: methodology.html#2; packing_rules and discipline: methodology prose.",
   "pitfalls: the four from reproduce.html; measurement_notes: large-scale.json notes (AppleDouble sidecars, endpoint quarantine, etc.)."
  ],
  "page_sha256": {
   "large-scale.html": "2101600e4bc7833d4222bc8f066a548661443464a1ee4981cac5d07372d25a62",
   "methodology.html": "a7a87af52e9ed8b91f308bf430c8247a235812bb6673c06ac189580cca8d2d3e",
   "reproduce.html": "95e1be0a878f1f106476790d86c38e29a1e0ec5a836c0f8434f9fb6ff5c81c99"
  }
 },
 "data": {
  "requirements": [
   {
    "requirement": "The scanner, installed from source",
    "detail": "git clone the repository, create a virtual environment, then `pip install -e .` plus the pinned `huggingface-hub` and `pyarrow`. Building the CEL helper needs Go 1.27.1 or newer."
   },
   {
    "requirement": "AWS credentials that can reach Bedrock",
    "detail": "The judge model is served only on the Bedrock mantle OpenAI-compatible route and authenticates with SigV4, so the normal credential chain is enough. No API key is involved."
   },
   {
    "requirement": "A hosted System One credential, for that arm only",
    "detail": "One environment variable, read from wherever you keep secrets. The deterministic and judge results need nothing beyond AWS."
   }
  ],
  "commands": [
   {
    "step": 1,
    "title": "Acquire the corpus",
    "command": "python evals/datasets/materialize_malicious_skill_bench.py --source-dir SRC --output-dir OUT",
    "note": "Downloads only the files named in the committed inspection profile, checks each byte hash against the dataset lock, writes one inert SKILL.md per row (expected artifact counts: see README gaps)."
   },
   {
    "step": 2,
    "title": "Reproduce the published baseline",
    "command": "python evals/runners/public_dataset_benchmark.py --snapshot-dir OUT --dataset-id ProtectSkills/MaliciousSkillBench --profile release --cel-mode off --output baseline.json",
    "note": "Must produce F1 13.742%, precision 60.748%, recall 7.747% on the source-disjoint track and a population digest matching the lock."
   },
   {
    "step": 3,
    "title": "Run the judged arms",
    "command": "SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=1 python evals/runners/judged_dataset_benchmark.py --snapshot-dir OUT --dataset-id ProtectSkills/MaliciousSkillBench --cel-mode off --arm core_judge --arm core_judge_meta --arm core_meta --judge-model bedrock-mantle/google.gemma-4-26b-a4b --output judged.json",
    "note": "Each arm is written as it finishes; the report is stamped non-blocking (wall clock per arm: see README gaps)."
   },
   {
    "step": 4,
    "title": "Run the System One arm",
    "command": "python evals/system_one/runner.py --corpus CORPUS --context K7 --instruction I4 --question SQ2 --model jev-1.13.0 --model-revision jev-1.13.0 --concurrency 8 --max-usd 5 --output jev.jsonl",
    "note": "42 seconds for the whole 1,384-package track (RESULTS.md stage table); writes a completion attestation."
   },
   {
    "step": 5,
    "title": "Emit the rule and judge stages in the same shape",
    "command": "python evals/system_one/deterministic.py --corpus CORPUS --tier deterministic --output rules.jsonl",
    "note": "Add --judge-model to produce the judge stage instead."
   },
   {
    "step": 6,
    "title": "Score the chains",
    "command": "python evals/system_one/scorer.py --labels LABELS.json --chain \"rules_then_jev=rules.jsonl,jev.jsonl\" --output cascade.json",
    "note": "Repeat --chain per combination; every chain is reported under both lenses."
   },
   {
    "step": 7,
    "title": "Establish the noise floor",
    "command": "python evals/experiments/e16_stability.py --corpus CORPUS --max-skills 200 --repeats 5 --output stability.json",
    "note": "Run before comparing any two judged configurations; re-runs the deterministic stage as a control."
   },
   {
    "step": 8,
    "title": "Rebuild this site",
    "command": "python evals/publish/build_space.py --output SITE --baseline baseline.json --judged-arm ... --cost cost-comparison.json",
    "note": "Renders from the result files only; a missing section prints as missing rather than zeros."
   }
  ],
  "guarantees": [
   "Samples are never executed. The corpus lock sets that as a default and the materializer writes text only, at mode 0600.",
   "Corpora are pinned by revision and verified by content digest, so a shrunken or substituted population cannot pass unnoticed.",
   "Results are written to new paths. No scorecard or prediction file is ever modified in place.",
   "Every prediction row is value-free: identifiers, actions and probabilities, never sample text.",
   "Per-corpus licence restrictions are honoured. Several forbid exactly the metric one would reach for first, so each corpus is reported only with the figure its licence permits.Four things that will waste your afternoon"
  ],
  "pitfalls": [
   "Set SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=1 for any judged run. Without it the judge contradicts itself on roughly a fifth of harmless skills and its whole analysis is discarded for those, which silently flatters its measured precision.",
   "Do not run scripts from a shared temporary directory. A stray module there can shadow a real library on the import path; one did, and it made the judge fail silently with zero recorded tokens.",
   "The judge's token usage must be read from the LLM analyzer, not the meta analyzer. Reading only the latter reports zero tokens for the judge arm and makes its cost unmeasurable.",
   "Check the meta-judge's routing decision, not its routing reason text. Keying on the reason reported that meta had never run when it had run on most packages. Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content."
  ],
  "scripts": [
   {
    "step": "Score OpenJev's eight probes",
    "script": "evals/experiments/c3_openjev_local.py"
   },
   {
    "step": "Run the judge alone over a corpus",
    "script": "evals/runners/judge_only.py"
   },
   {
    "step": "Scan a corpus with one tree",
    "script": "evals/runners/cross_tool_benchmark.py"
   },
   {
    "step": "Build the store",
    "script": "evals/experiments/f1_full_corpus_store.py"
   },
   {
    "step": "Analyse every real skill",
    "script": "evals/experiments/f2_full_corpus_analysis.py"
   },
   {
    "step": "Check the overlay against the full scan",
    "script": "evals/experiments/f3_overlay_check.py"
   },
   {
    "step": "Shipped against final on labelled corpora",
    "script": "evals/experiments/f4_labelled_ab.py"
   },
   {
    "step": "The preset demotion path",
    "script": "evals/experiments/f5_policy_pack_path.py"
   },
   {
    "step": "Judge prompt and caps",
    "script": "evals/experiments/a2_judge_prompt_caps.py"
   },
   {
    "step": "Select the OpenJev screen",
    "script": "evals/experiments/c4_openjev_screen.py"
   },
   {
    "step": "Assemble and check this report",
    "script": "evals/publish/large_scale_report.py"
   }
  ],
  "judge_model_properties": [
   {
    "property": "One model, one route",
    "measured": "google.gemma-4-26b-a4b is reachable only on the Bedrock mantle OpenAI-compatible route. It is absent from list-foundation-models, there is no model-listing endpoint, and no other Gemma 4 size exists, so the model id is a pinned constant."
   },
   {
    "property": "Strict schema works, after one removal",
    "measured": "Schema-constrained JSON succeeds once uniqueItems is stripped; the mantle validator rejects it outright and the request would otherwise degrade silently to loose JSON mode."
   },
   {
    "property": "Context is ample",
    "measured": "A 124,989-byte request was accepted at 40,855 prompt tokens, so the scanner's 100,000-character budget fits with headroom."
   },
   {
    "property": "Prompt caching does not engage",
    "measured": "cached_tokens and cache_write_tokens stayed at zero across two byte-identical 4,859-token requests, so the largest assumed cost lever is unavailable on this route."
   },
   {
    "property": "Code tokenizes worse than prose",
    "measured": "Measured at 3.06 bytes per token on repetitive Python, not the 4.0 a naive estimator assumes, which is why budgets are enforced in bytes at a pessimistic 2 bytes per token."
   }
  ],
  "packing_rules": [
   "The matched evidence is never truncated; only the window around it is, and truncation keeps windows containing security-relevant terms so an obfuscated payload's middle survives.",
   "An oversize request is never sent. A provider that truncates and still answers answers confidently about content it never saw, so the oversize path returns not-applicable and falls back to the deterministic decision.",
   "Untrusted sample text is neutralized so a malicious manifest cannot close the harness's own delimiters and have its instructions read as policy.",
   "No benchmark label, and no other tier's verdict, ever enters the context. Including a verdict would destroy arm independence and make cascade scoring circular. Judge model, verified rather than assumed"
  ],
  "discipline": [
   "Every analysis is gated on a completion attestation and on the on-disk digest matching the manifest, so a partial run cannot be published as a finished one.",
   "Results are written to new paths; no scorecard or prediction file is modified in place.",
   "Integrity counters travel with every result: row counts, error rows, unparsable rows and the size of any unjudgeable subset.",
   "Per-corpus licence restrictions are honoured. Several forbid exactly the metric one would reach for first, so the permitted metrics are recorded per corpus rather than left to discipline.",
   "Samples are never executed, and no sample content is republished. Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content."
  ],
  "measurement_notes": [
   "Judge figures on the labelled splits are single pass, core rules, meta-analyzer off, detection at MEDIUM or above.",
   "The first cascade threshold was selected on half of the frozen test split; it is re-selected on train/validation, and those figures supersede it.",
   "The deterministic rules and the judge prompt were designed on MaliciousSkillBench train/validation and real skills; the frozen test split was scored once and used to design nothing in the later passes.",
   "Unlabelled populations give flag rates, which bound the false-positive rate from above.",
   "An early deterministic run read 100% false-positive rate because copying the corpus from macOS added an AppleDouble '._SKILL.md' to every record, which the scanner correctly flagged; the sidecars were removed.",
   "Endpoint protection quarantined malicious SKILL.md files from local copies of the benchmark, silently turning malicious records into empty ones; every figure here was produced on a Linux analysis host."
  ],
  "provenance": {
   "scanner_commits": {
    "shipped": "5b696a1",
    "first_pass": "f3a42f3",
    "final": "9c08673"
   },
   "report_builder_commit": "5807c36de84b028a912624ed46fab0d0c67d49d0",
   "published_as": "results/large-scale.json"
  },
  "scanner_version_note": "Scanner version n/a, CEL mode off (deterministic.html).",
  "go_version": "1.27.1",
  "repository": "https://github.com/cisco-ai-defense/skill-scanner",
  "recommended_settings_doc": "https://github.com/cisco-ai-defense/skill-scanner/blob/main/docs/user-guide/recommended-settings.md"
 }
}