{ "meta": { "id": "tables_raw", "title": "Every HTML table on the old Space, typed", "sources": [ "index.html", "deterministic.html", "judged.html", "cascade.html", "large-scale.html", "cross-tool.html", "improvements.html", "experiments.html", "recommendations.html", "methodology.html", "reproduce.html" ], "extracted_at": "2026-10-16T14:34:41+00:00", "notes": [ "source is '#' (1-based, document order).", "Percentages are fractions; counts are ints; 'n/a', 'not instrumented', 'not permitted', 'no harmless class' and dashes are null with the literal in `raw`.", "pair = 'before → after'; percent_ci = value with 95% interval; population = corpus size cell; labelled = several labelled percentages in one cell; stages = 'rules 100%; judge 84%'.", "before_text/after_text are the prose blocks adjacent to the table (paired-difference lines live in after_text).", "svgs lists the 5 inline bar charts; each duplicates the table named in duplicates_table." ], "page_sha256": { "index.html": "ea2c1d462effa0d10c18016d0285ae5002a82e61b9d12aa0aa095343cc82cede", "deterministic.html": "752550a10cd6b5f5a0c465f1de4da1168521cefc519bedba7cd5c006f67e4cc4", "judged.html": "c8a6f6c0912a7791db4b80f3195dd316bc5603fbae3ecc0ad9297232724f0ebd", "cascade.html": "c1a4d8157eba904912e05da37e55dd3c9a4abc313064bd6dc0180ecce526bdc1", "large-scale.html": "2101600e4bc7833d4222bc8f066a548661443464a1ee4981cac5d07372d25a62", "cross-tool.html": "e7f8433087a55e382dc51895d350412a45ce19124d12beda376a47a1d55509aa", "improvements.html": "dbb68812c9eca9d368773fb520761af6c33836e65c5b7bcc35d84a0e642fcfb6", "experiments.html": "6b3dcd0ea2e5ff7b991df0410c5c929e998bee5fa4fc2865fcfa6ff3e486d42d", "recommendations.html": "cbb0c3dbc9b80000a67a18f13ae85a94a88827748e65758fc04f17fb9a8ce2da", "methodology.html": "a7a87af52e9ed8b91f308bf430c8247a235812bb6673c06ac189580cca8d2d3e", "reproduce.html": "95e1be0a878f1f106476790d86c38e29a1e0ec5a836c0f8434f9fb6ff5c81c99" } }, "data": { "tables": [ { "source": "index.html#1", "page": "index.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Every stage, on every corpus we could label" }, "before_text": "Detection is the share of skills in that group that got flagged at all. Hover any column name for its definition.", "after_text": "What is good: Both models beat the rules everywhere except one place, and the judge beats everything everywhere. On the obviously-malicious half of OpenSkillRisk the judge caught all of it. What is not: The System One model is much worse than the rules on contextually risky skills, catching 11% against the rules' 33%. It is a confident filter for clear cases, not a substitute for judgement on ambiguous ones.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Rules", "tooltip": null }, { "name": "System One model", "tooltip": null }, { "name": "Judge", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "200 skills", "type": "population", "value": { "n": 200, "unit": "skills" }, "col": "Population" }, { "raw": "4.00%", "type": "percent", "value": 0.04, "col": "Rules" }, { "raw": "26.13%", "type": "percent", "value": 0.2613, "col": "System One model" }, { "raw": "56.00%", "type": "percent", "value": 0.56, "col": "Judge" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "400 malicious, 400 benign", "type": "population", "value": { "positives": 400, "negatives": 400 }, "col": "Population" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Rules" }, { "raw": "54.31%", "type": "percent", "value": 0.5431, "col": "System One model" }, { "raw": "81.75%", "type": "percent", "value": 0.8175, "col": "Judge" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "214 skills", "type": "population", "value": { "n": 214, "unit": "skills" }, "col": "Population" }, { "raw": "32.71%", "type": "percent", "value": 0.3271, "col": "Rules" }, { "raw": "11.00%", "type": "percent", "value": 0.11, "col": "System One model" }, { "raw": "77.10%", "type": "percent", "value": 0.771, "col": "Judge" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "160 skills", "type": "population", "value": { "n": 160, "unit": "skills" }, "col": "Population" }, { "raw": "30.00%", "type": "percent", "value": 0.3, "col": "Rules" }, { "raw": "95.77%", "type": "percent", "value": 0.9577, "col": "System One model" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Judge" } ] ], "kind": "grid" }, { "source": "deterministic.html#1", "page": "deterministic.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Reproducing the published baseline" }, "before_text": "Before any judged number is believable the harness has to reproduce the published one. This is that check, run on the locked source-disjoint test partition after materializing the corpus from its pinned revision and verifying every manifest digest against the lock.", "after_text": "", "columns": [ { "name": "Metric", "tooltip": null }, { "name": "Published", "tooltip": null }, { "name": "Reproduced", "tooltip": null }, { "name": "Match", "tooltip": null } ], "rows": [ [ { "raw": "F1", "type": "text", "value": "F1", "col": "Metric" }, { "raw": "13.74%", "type": "percent", "value": 0.1374, "col": "Published" }, { "raw": "13.74%", "type": "percent", "value": 0.1374, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "Precision", "type": "text", "value": "Precision", "col": "Metric" }, { "raw": "60.75%", "type": "percent", "value": 0.6075, "col": "Published" }, { "raw": "60.75%", "type": "percent", "value": 0.6075, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "Recall", "type": "text", "value": "Recall", "col": "Metric" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Published" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "False-positive rate", "type": "text", "value": "False-positive rate", "col": "Metric" }, { "raw": "7.71%", "type": "percent", "value": 0.0771, "col": "Published" }, { "raw": "7.71%", "type": "percent", "value": 0.0771, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "Samples", "type": "text", "value": "Samples", "col": "Metric" }, { "raw": "1,384", "type": "number", "value": 1384, "col": "Published" }, { "raw": "1,384", "type": "number", "value": 1384, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "Malicious", "type": "text", "value": "Malicious", "col": "Metric" }, { "raw": "839", "type": "number", "value": 839, "col": "Published" }, { "raw": "839", "type": "number", "value": 839, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ], [ { "raw": "Benign", "type": "text", "value": "Benign", "col": "Metric" }, { "raw": "545", "type": "number", "value": 545, "col": "Published" }, { "raw": "545", "type": "number", "value": 545, "col": "Reproduced" }, { "raw": "matches", "type": "text", "value": "matches", "col": "Match" } ] ], "kind": "grid" }, { "source": "deterministic.html#2", "page": "deterministic.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Reproducing the published baseline", "h3": "Confusion counts" }, "before_text": "", "after_text": "", "columns": [ { "name": "True positive", "tooltip": null }, { "name": "False positive", "tooltip": null }, { "name": "False negative", "tooltip": null }, { "name": "True negative", "tooltip": null }, { "name": "Scan errors", "tooltip": null } ], "rows": [ [ { "raw": "65", "type": "number", "value": 65, "col": "True positive" }, { "raw": "42", "type": "number", "value": 42, "col": "False positive" }, { "raw": "774", "type": "number", "value": 774, "col": "False negative" }, { "raw": "503", "type": "number", "value": 503, "col": "True negative" }, { "raw": "0", "type": "number", "value": 0, "col": "Scan errors" } ] ], "kind": "grid" }, { "source": "deterministic.html#3", "page": "deterministic.html", "index": 3, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Reproducing the published baseline", "h3": "Deterministic rules: every metric, corpus by corpus" }, "before_text": "Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.", "after_text": "Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "200 bad, 0 harmless", "type": "population", "value": { "positives": 200, "negatives": 0 }, "col": "Population" }, { "raw": "4.00%", "type": "percent", "value": 0.04, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "400 bad, 400 harmless", "type": "population", "value": { "positives": 400, "negatives": 400 }, "col": "Population" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Recall" }, { "raw": "94.29%", "type": "percent", "value": 0.9429, "col": "Precision" }, { "raw": "48.89%", "type": "percent", "value": 0.4889, "col": "F1" }, { "raw": "2.00%", "type": "percent", "value": 0.02, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench source-disjoint (the published test split)", "type": "text", "value": "MaliciousSkillBench source-disjoint (the published test split)", "col": "Corpus" }, { "raw": "839 bad, 545 harmless", "type": "population", "value": { "positives": 839, "negatives": 545 }, "col": "Population" }, { "raw": "7.99%", "type": "percent", "value": 0.0799, "col": "Recall" }, { "raw": "59.29%", "type": "percent", "value": 0.5929, "col": "Precision" }, { "raw": "14.08%", "type": "percent", "value": 0.1408, "col": "F1" }, { "raw": "8.44%", "type": "percent", "value": 0.0844, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "214 bad, 0 harmless", "type": "population", "value": { "positives": 214, "negatives": 0 }, "col": "Population" }, { "raw": "32.71%", "type": "percent", "value": 0.3271, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "160 bad, 0 harmless", "type": "population", "value": { "positives": 160, "negatives": 0 }, "col": "Population" }, { "raw": "30.00%", "type": "percent", "value": 0.3, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ] ], "kind": "grid" }, { "source": "deterministic.html#4", "page": "deterministic.html", "index": 4, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Re-measuring every previously published figure" }, "before_text": "Each row is a published deterministic figure re-measured from the pinned corpus revision. Reproducing these was the precondition for believing any judged number, because a harness that cannot reproduce the baseline cannot be trusted with a new arm.", "after_text": "What is good: All nine figures reproduce. That is the only reason the judged numbers elsewhere on this site are worth reading: the same code path produced both.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Previously published", "tooltip": null }, { "name": "Re-measured", "tooltip": null }, { "name": "Agrees", "tooltip": null } ], "rows": [ [ { "raw": "MaliciousSkillBench source-disjoint", "type": "text", "value": "MaliciousSkillBench source-disjoint", "col": "Corpus" }, { "raw": "1,384 packages", "type": "population", "value": { "n": 1384, "unit": "packages" }, "col": "Population" }, { "raw": "F1 13.74%", "type": "labelled", "value": [ 0.1374 ], "col": "Previously published" }, { "raw": "F1 13.742%", "type": "labelled", "value": [ 0.13742 ], "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "MaliciousSkillBench source-disjoint", "type": "text", "value": "MaliciousSkillBench source-disjoint", "col": "Corpus" }, { "raw": "1,384 packages", "type": "population", "value": { "n": 1384, "unit": "packages" }, "col": "Population" }, { "raw": "precision 60.75%", "type": "labelled", "value": [ 0.6075 ], "col": "Previously published" }, { "raw": "60.748%", "type": "percent", "value": 0.60748, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "MaliciousSkillBench source-disjoint", "type": "text", "value": "MaliciousSkillBench source-disjoint", "col": "Corpus" }, { "raw": "1,384 packages", "type": "population", "value": { "n": 1384, "unit": "packages" }, "col": "Population" }, { "raw": "recall 7.75%", "type": "labelled", "value": [ 0.0775 ], "col": "Previously published" }, { "raw": "7.747%", "type": "percent", "value": 0.07747, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "MaliciousSkillBench source-disjoint", "type": "text", "value": "MaliciousSkillBench source-disjoint", "col": "Corpus" }, { "raw": "1,384 packages", "type": "population", "value": { "n": 1384, "unit": "packages" }, "col": "Population" }, { "raw": "FPR 7.71%", "type": "labelled", "value": [ 0.0771 ], "col": "Previously published" }, { "raw": "7.706%", "type": "percent", "value": 0.07706, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "NotInject", "type": "text", "value": "NotInject", "col": "Corpus" }, { "raw": "339 benign text cases", "type": "population", "value": { "n": 339, "unit": "benign text cases" }, "col": "Population" }, { "raw": "0.00% flag rate at MEDIUM+", "type": "labelled", "value": [ 0.0 ], "col": "Previously published" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "InjecAgent", "type": "text", "value": "InjecAgent", "col": "Corpus" }, { "raw": "1,054 canonical signals", "type": "population", "value": { "n": 1054, "unit": "canonical signals" }, "col": "Population" }, { "raw": "100.00% signal recall", "type": "labelled", "value": [ 1.0 ], "col": "Previously published" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "In-Page Prompt Injection", "type": "text", "value": "In-Page Prompt Injection", "col": "Corpus" }, { "raw": "1,101 canonical groups", "type": "population", "value": { "n": 1101, "unit": "canonical groups" }, "col": "Population" }, { "raw": "99.36% signal recall", "type": "labelled", "value": [ 0.9936 ], "col": "Previously published" }, { "raw": "99.36%", "type": "percent", "value": 0.9936, "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "200 positive-risk skills", "type": "population", "value": { "n": 200, "unit": "positive-risk skills" }, "col": "Population" }, { "raw": "3.50% MEDIUM+", "type": "labelled", "value": [ 0.035 ], "col": "Previously published" }, { "raw": "3.5% block rate", "type": "labelled", "value": [ 0.035 ], "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ], [ { "raw": "OpenSkillRisk", "type": "text", "value": "OpenSkillRisk", "col": "Corpus" }, { "raw": "374 positive-risk skills", "type": "population", "value": { "n": 374, "unit": "positive-risk skills" }, "col": "Population" }, { "raw": "28.90% HIGH+, 35.74% MEDIUM+", "type": "labelled", "value": [ 0.289, 0.3574 ], "col": "Previously published" }, { "raw": "30.0% and 32.7% detection", "type": "labelled", "value": [ 0.3, 0.327 ], "col": "Re-measured" }, { "raw": "yes", "type": "text", "value": "yes", "col": "Agrees" } ] ], "kind": "grid" }, { "source": "deterministic.html#5", "page": "deterministic.html", "index": 5, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Re-measuring every previously published figure", "h3": "Three we could not re-measure, and why" }, "before_text": "", "after_text": "Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Reason", "tooltip": null } ], "rows": [ [ { "raw": "Bundled skills snapshot", "type": "text", "value": "Bundled skills snapshot", "col": "Corpus" }, { "raw": "111 installed skills", "type": "population", "value": { "n": 111, "unit": "installed skills" }, "col": "Population" }, { "raw": "the lock is stale against the currently installed applications, and refreshing it requires human review; forcing a refresh would change the corpus identity and break comparability", "type": "text", "value": "the lock is stale against the currently installed applications, and refreshing it requires human review; forcing a refresh would change the corpus identity and break comparability", "col": "Reason" } ], [ { "raw": "DataDog malicious packages", "type": "text", "value": "DataDog malicious packages", "col": "Corpus" }, { "raw": "5 selected positives", "type": "population", "value": { "n": 5, "unit": "selected positives" }, "col": "Population" }, { "raw": "quarantine-only acquisition policy, and too small to move any figure", "type": "text", "value": "quarantine-only acquisition policy, and too small to move any figure", "col": "Reason" } ], [ { "raw": "MaliciousAgentSkillsBench", "type": "text", "value": "MaliciousAgentSkillsBench", "col": "Corpus" }, { "raw": "98,380 rows", "type": "population", "value": { "n": 98380, "unit": "rows" }, "col": "Population" }, { "raw": "metadata only, with no skill content to scan", "type": "text", "value": "metadata only, with no skill content to scan", "col": "Reason" } ] ], "kind": "grid" }, { "source": "judged.html#1", "page": "judged.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Enabling the judge and the meta-judge" }, "before_text": "Four arms over one population: the deterministic core alone, the core plus the LLM analyzer, the core plus both the analyzer and the meta-judge, and the core plus the meta-judge with the analyzer left off. The core-only arm exists to prove the harness before any judged number is read. This track is explicitly non-blocking. The release path asserts that the analyzer factory keeps the judge off by default, and that assertion still holds; the judge is enabled per-arm by the evaluation runner alone.", "after_text": "", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Meta invoked", "tooltip": null }, { "name": "Suppressed", "tooltip": null }, { "name": "Added", "tooltip": null } ], "rows": [ [ { "raw": "core_only (baseline)", "type": "text", "value": "core_only (baseline)", "col": "Arm" }, { "raw": "13.74%", "type": "percent", "value": 0.1374, "col": "F1" }, { "raw": "60.75%", "type": "percent", "value": 0.6075, "col": "Precision" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Recall" }, { "raw": "7.71%", "type": "percent", "value": 0.0771, "col": "FPR" }, { "raw": "n/a", "type": "null", "value": null, "col": "Meta invoked" }, { "raw": "0", "type": "number", "value": 0, "col": "Suppressed" }, { "raw": "0", "type": "number", "value": 0, "col": "Added" } ], [ { "raw": "core_judge", "type": "text", "value": "core_judge", "col": "Arm" }, { "raw": "37.68%", "type": "percent", "value": 0.3768, "col": "F1" }, { "raw": "78.49%", "type": "percent", "value": 0.7849, "col": "Precision" }, { "raw": "24.79%", "type": "percent", "value": 0.2479, "col": "Recall" }, { "raw": "10.46%", "type": "percent", "value": 0.1046, "col": "FPR" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Meta invoked" }, { "raw": "0", "type": "number", "value": 0, "col": "Suppressed" }, { "raw": "0", "type": "number", "value": 0, "col": "Added" } ], [ { "raw": "core_judge_meta", "type": "text", "value": "core_judge_meta", "col": "Arm" }, { "raw": "37.24%", "type": "percent", "value": 0.3724, "col": "F1" }, { "raw": "78.24%", "type": "percent", "value": 0.7824, "col": "Precision" }, { "raw": "24.43%", "type": "percent", "value": 0.2443, "col": "Recall" }, { "raw": "10.46%", "type": "percent", "value": 0.1046, "col": "FPR" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Meta invoked" }, { "raw": "0", "type": "number", "value": 0, "col": "Suppressed" }, { "raw": "0", "type": "number", "value": 0, "col": "Added" } ], [ { "raw": "core_meta", "type": "text", "value": "core_meta", "col": "Arm" }, { "raw": "13.74%", "type": "percent", "value": 0.1374, "col": "F1" }, { "raw": "60.75%", "type": "percent", "value": 0.6075, "col": "Precision" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Recall" }, { "raw": "7.71%", "type": "percent", "value": 0.0771, "col": "FPR" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Meta invoked" }, { "raw": "0", "type": "number", "value": 0, "col": "Suppressed" }, { "raw": "0", "type": "number", "value": 0, "col": "Added" } ] ], "kind": "grid" }, { "source": "judged.html#2", "page": "judged.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Enabling the judge and the meta-judge", "h3": "LLM judge: every metric, corpus by corpus" }, "before_text": "Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.", "after_text": "Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "200 bad, 0 harmless", "type": "population", "value": { "positives": 200, "negatives": 0 }, "col": "Population" }, { "raw": "56.00%", "type": "percent", "value": 0.56, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "400 bad, 400 harmless", "type": "population", "value": { "positives": 400, "negatives": 400 }, "col": "Population" }, { "raw": "81.75%", "type": "percent", "value": 0.8175, "col": "Recall" }, { "raw": "78.61%", "type": "percent", "value": 0.7861, "col": "Precision" }, { "raw": "80.15%", "type": "percent", "value": 0.8015, "col": "F1" }, { "raw": "22.25%", "type": "percent", "value": 0.2225, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench source-disjoint (the published test split)", "type": "text", "value": "MaliciousSkillBench source-disjoint (the published test split)", "col": "Corpus" }, { "raw": "839 bad, 545 harmless", "type": "population", "value": { "positives": 839, "negatives": 545 }, "col": "Population" }, { "raw": "48.99%", "type": "percent", "value": 0.4899, "col": "Recall" }, { "raw": "80.91%", "type": "percent", "value": 0.8091, "col": "Precision" }, { "raw": "61.02%", "type": "percent", "value": 0.6102, "col": "F1" }, { "raw": "17.80%", "type": "percent", "value": 0.178, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "214 bad, 0 harmless", "type": "population", "value": { "positives": 214, "negatives": 0 }, "col": "Population" }, { "raw": "77.10%", "type": "percent", "value": 0.771, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "160 bad, 0 harmless", "type": "population", "value": { "positives": 160, "negatives": 0 }, "col": "Population" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ] ], "kind": "grid" }, { "source": "judged.html#3", "page": "judged.html", "index": 3, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Enabling the judge and the meta-judge", "h3": "The judge against the rules, detection only" }, "before_text": "The locked corpus above is one population. These are the others, each with the figure its licence permits. Detection is the share of skills flagged at all.", "after_text": "What is good: The judge improves detection on every corpus, and the gain is largest exactly where the rules are weakest. On HarmfulSkillBench the rules found 4% and the judge found 56%. What is not: It is not free. On the locked corpus the share of harmless skills carrying an actionable finding roughly doubled, so this belongs behind a review queue rather than an automatic block.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Rules", "tooltip": null }, { "name": "Judge", "tooltip": null }, { "name": "Change", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "200 skills", "type": "population", "value": { "n": 200, "unit": "skills" }, "col": "Population" }, { "raw": "4.00%", "type": "percent", "value": 0.04, "col": "Rules" }, { "raw": "56.00%", "type": "percent", "value": 0.56, "col": "Judge" }, { "raw": "+52.0 points", "type": "points", "value": 52.0, "col": "Change" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "400 malicious, 400 benign", "type": "population", "value": { "positives": 400, "negatives": 400 }, "col": "Population" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Rules" }, { "raw": "81.75%", "type": "percent", "value": 0.8175, "col": "Judge" }, { "raw": "+48.8 points", "type": "points", "value": 48.8, "col": "Change" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "214 skills", "type": "population", "value": { "n": 214, "unit": "skills" }, "col": "Population" }, { "raw": "32.71%", "type": "percent", "value": 0.3271, "col": "Rules" }, { "raw": "77.10%", "type": "percent", "value": 0.771, "col": "Judge" }, { "raw": "+44.4 points", "type": "points", "value": 44.4, "col": "Change" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "160 skills", "type": "population", "value": { "n": 160, "unit": "skills" }, "col": "Population" }, { "raw": "30.00%", "type": "percent", "value": 0.3, "col": "Rules" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Judge" }, { "raw": "+70.0 points", "type": "points", "value": 70.0, "col": "Change" } ] ], "kind": "grid" }, { "source": "cascade.html#1", "page": "cascade.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Does the System One model earn its place between the rules and the judge?", "h3": "Block-only lens" }, "before_text": "", "after_text": "", "columns": [ { "name": "Chain", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null }, { "name": "Skills reaching each stage", "tooltip": null } ], "rows": [ [ { "raw": "rules_only", "type": "text", "value": "rules_only", "col": "Chain" }, { "raw": "95.42%", "type": "percent", "value": 0.9542, "col": "Precision" }, { "raw": "31.25%", "type": "percent", "value": 0.3125, "col": "Recall" }, { "raw": "47.08%", "type": "percent", "value": 0.4708, "col": "F1" }, { "raw": "1.50%", "type": "percent", "value": 0.015, "col": "False-positive rate" }, { "raw": "rules 100%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 } ], "col": "Skills reaching each stage" } ], [ { "raw": "judge_only", "type": "text", "value": "judge_only", "col": "Chain" }, { "raw": "93.24%", "type": "percent", "value": 0.9324, "col": "Precision" }, { "raw": "65.50%", "type": "percent", "value": 0.655, "col": "Recall" }, { "raw": "76.95%", "type": "percent", "value": 0.7695, "col": "F1" }, { "raw": "4.75%", "type": "percent", "value": 0.0475, "col": "False-positive rate" }, { "raw": "judge 100%", "type": "stages", "value": [ { "stage": "judge", "share": 1.0 } ], "col": "Skills reaching each stage" } ], [ { "raw": "jev_only", "type": "text", "value": "jev_only", "col": "Chain" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "34.00%", "type": "percent", "value": 0.34, "col": "Recall" }, { "raw": "50.75%", "type": "percent", "value": 0.5075, "col": "F1" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "False-positive rate" }, { "raw": "System One 99%", "type": "stages", "value": [ { "stage": "System One", "share": 0.99 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_judge", "type": "text", "value": "rules_then_judge", "col": "Chain" }, { "raw": "93.24%", "type": "percent", "value": 0.9324, "col": "Precision" }, { "raw": "65.50%", "type": "percent", "value": 0.655, "col": "Recall" }, { "raw": "76.95%", "type": "percent", "value": 0.7695, "col": "F1" }, { "raw": "4.75%", "type": "percent", "value": 0.0475, "col": "False-positive rate" }, { "raw": "rules 100%; judge 84%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "judge", "share": 0.84 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_jev", "type": "text", "value": "rules_then_jev", "col": "Chain" }, { "raw": "96.97%", "type": "percent", "value": 0.9697, "col": "Precision" }, { "raw": "48.00%", "type": "percent", "value": 0.48, "col": "Recall" }, { "raw": "64.21%", "type": "percent", "value": 0.6421, "col": "F1" }, { "raw": "1.50%", "type": "percent", "value": 0.015, "col": "False-positive rate" }, { "raw": "rules 100%; System One 83%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "System One", "share": 0.83 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_jev_then_judge", "type": "text", "value": "rules_then_jev_then_judge", "col": "Chain" }, { "raw": "93.40%", "type": "percent", "value": 0.934, "col": "Precision" }, { "raw": "67.25%", "type": "percent", "value": 0.6725, "col": "Recall" }, { "raw": "78.20%", "type": "percent", "value": 0.782, "col": "F1" }, { "raw": "4.75%", "type": "percent", "value": 0.0475, "col": "False-positive rate" }, { "raw": "rules 100%; System One 83%; judge 75%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "System One", "share": 0.83 }, { "stage": "judge", "share": 0.75 } ], "col": "Skills reaching each stage" } ] ], "kind": "grid" }, { "source": "cascade.html#2", "page": "cascade.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Does the System One model earn its place between the rules and the judge?", "h3": "Any-intervention lens" }, "before_text": "", "after_text": "The last column reads in stage order. Where it says \"rules 100%; System One 91%\", the rules examined every skill and 9% were already settled by a clear block, so the System One model was only asked about the remaining 91%. That is the saving from putting cheap stages first.", "columns": [ { "name": "Chain", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null }, { "name": "Skills reaching each stage", "tooltip": null } ], "rows": [ [ { "raw": "rules_only", "type": "text", "value": "rules_only", "col": "Chain" }, { "raw": "94.29%", "type": "percent", "value": 0.9429, "col": "Precision" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Recall" }, { "raw": "48.89%", "type": "percent", "value": 0.4889, "col": "F1" }, { "raw": "2.00%", "type": "percent", "value": 0.02, "col": "False-positive rate" }, { "raw": "rules 100%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 } ], "col": "Skills reaching each stage" } ], [ { "raw": "judge_only", "type": "text", "value": "judge_only", "col": "Chain" }, { "raw": "78.61%", "type": "percent", "value": 0.7861, "col": "Precision" }, { "raw": "81.75%", "type": "percent", "value": 0.8175, "col": "Recall" }, { "raw": "80.15%", "type": "percent", "value": 0.8015, "col": "F1" }, { "raw": "22.25%", "type": "percent", "value": 0.2225, "col": "False-positive rate" }, { "raw": "judge 100%", "type": "stages", "value": [ { "stage": "judge", "share": 1.0 } ], "col": "Skills reaching each stage" } ], [ { "raw": "jev_only", "type": "text", "value": "jev_only", "col": "Chain" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "53.50%", "type": "percent", "value": 0.535, "col": "Recall" }, { "raw": "69.71%", "type": "percent", "value": 0.6971, "col": "F1" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "False-positive rate" }, { "raw": "System One 99%", "type": "stages", "value": [ { "stage": "System One", "share": 0.99 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_judge", "type": "text", "value": "rules_then_judge", "col": "Chain" }, { "raw": "78.61%", "type": "percent", "value": 0.7861, "col": "Precision" }, { "raw": "81.75%", "type": "percent", "value": 0.8175, "col": "Recall" }, { "raw": "80.15%", "type": "percent", "value": 0.8015, "col": "F1" }, { "raw": "22.25%", "type": "percent", "value": 0.2225, "col": "False-positive rate" }, { "raw": "rules 100%; judge 84%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "judge", "share": 0.84 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_jev", "type": "text", "value": "rules_then_jev", "col": "Chain" }, { "raw": "96.97%", "type": "percent", "value": 0.9697, "col": "Precision" }, { "raw": "64.00%", "type": "percent", "value": 0.64, "col": "Recall" }, { "raw": "77.11%", "type": "percent", "value": 0.7711, "col": "F1" }, { "raw": "2.00%", "type": "percent", "value": 0.02, "col": "False-positive rate" }, { "raw": "rules 100%; System One 83%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "System One", "share": 0.83 } ], "col": "Skills reaching each stage" } ], [ { "raw": "rules_then_jev_then_judge", "type": "text", "value": "rules_then_jev_then_judge", "col": "Chain" }, { "raw": "79.06%", "type": "percent", "value": 0.7906, "col": "Precision" }, { "raw": "84.00%", "type": "percent", "value": 0.84, "col": "Recall" }, { "raw": "81.45%", "type": "percent", "value": 0.8145, "col": "F1" }, { "raw": "22.25%", "type": "percent", "value": 0.2225, "col": "False-positive rate" }, { "raw": "rules 100%; System One 83%; judge 75%", "type": "stages", "value": [ { "stage": "rules", "share": 1.0 }, { "stage": "System One", "share": 0.83 }, { "stage": "judge", "share": 0.75 } ], "col": "Skills reaching each stage" } ] ], "kind": "grid" }, { "source": "cascade.html#3", "page": "cascade.html", "index": 3, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Does the System One model earn its place between the rules and the judge?", "h3": "System One model: every metric, corpus by corpus" }, "before_text": "Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.", "after_text": "Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "199 bad, 0 harmless", "type": "population", "value": { "positives": 199, "negatives": 0 }, "col": "Population" }, { "raw": "26.13%", "type": "percent", "value": 0.2613, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "1", "type": "number", "value": 1, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "394 bad, 397 harmless", "type": "population", "value": { "positives": 394, "negatives": 397 }, "col": "Population" }, { "raw": "54.31%", "type": "percent", "value": 0.5431, "col": "Recall" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "70.39%", "type": "percent", "value": 0.7039, "col": "F1" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "FPR" }, { "raw": "9", "type": "number", "value": 9, "col": "Errors" } ], [ { "raw": "MaliciousSkillBench source-disjoint (the published test split)", "type": "text", "value": "MaliciousSkillBench source-disjoint (the published test split)", "col": "Corpus" }, { "raw": "819 bad, 517 harmless", "type": "population", "value": { "positives": 819, "negatives": 517 }, "col": "Population" }, { "raw": "16.73%", "type": "percent", "value": 0.1673, "col": "Recall" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "28.66%", "type": "percent", "value": 0.2866, "col": "F1" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "FPR" }, { "raw": "48", "type": "number", "value": 48, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "209 bad, 0 harmless", "type": "population", "value": { "positives": 209, "negatives": 0 }, "col": "Population" }, { "raw": "11.00%", "type": "percent", "value": 0.11, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "5", "type": "number", "value": 5, "col": "Errors" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "142 bad, 0 harmless", "type": "population", "value": { "positives": 142, "negatives": 0 }, "col": "Population" }, { "raw": "95.77%", "type": "percent", "value": 0.9577, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR" }, { "raw": "18", "type": "number", "value": 18, "col": "Errors" } ] ], "kind": "grid" }, { "source": "cascade.html#4", "page": "cascade.html", "index": 4, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Does the System One model earn its place between the rules and the judge?", "h3": "The System One model against the rules, detection only" }, "before_text": "", "after_text": "What is good: It nearly matches the judge on clear malice at a fraction of the cost, reaching 95.8% on the obviously-malicious half of OpenSkillRisk, and in these runs it never flagged a harmless skill. What is not: On contextually risky skills it detected 11% where the rules detected 33%. Whatever it is good at, it is not resolving ambiguity, and a deployment that relies on it alone will miss exactly the cases that need a judgement call.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Population", "tooltip": null }, { "name": "Rules", "tooltip": null }, { "name": "System One model", "tooltip": null }, { "name": "Change", "tooltip": null } ], "rows": [ [ { "raw": "HarmfulSkillBench", "type": "text", "value": "HarmfulSkillBench", "col": "Corpus" }, { "raw": "199 skills", "type": "population", "value": { "n": 199, "unit": "skills" }, "col": "Population" }, { "raw": "4.00%", "type": "percent", "value": 0.04, "col": "Rules" }, { "raw": "26.13%", "type": "percent", "value": 0.2613, "col": "System One model" }, { "raw": "+22.1 points", "type": "points", "value": 22.1, "col": "Change" } ], [ { "raw": "MaliciousSkillBench balanced subset", "type": "text", "value": "MaliciousSkillBench balanced subset", "col": "Corpus" }, { "raw": "400 malicious, 400 benign", "type": "population", "value": { "positives": 400, "negatives": 400 }, "col": "Population" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Rules" }, { "raw": "54.31%", "type": "percent", "value": 0.5431, "col": "System One model" }, { "raw": "+21.3 points", "type": "points", "value": 21.3, "col": "Change" } ], [ { "raw": "OpenSkillRisk: contextually risky", "type": "text", "value": "OpenSkillRisk: contextually risky", "col": "Corpus" }, { "raw": "209 skills", "type": "population", "value": { "n": 209, "unit": "skills" }, "col": "Population" }, { "raw": "32.71%", "type": "percent", "value": 0.3271, "col": "Rules" }, { "raw": "11.00%", "type": "percent", "value": 0.11, "col": "System One model" }, { "raw": "-21.7 points", "type": "points", "value": -21.7, "col": "Change" } ], [ { "raw": "OpenSkillRisk: obviously malicious", "type": "text", "value": "OpenSkillRisk: obviously malicious", "col": "Corpus" }, { "raw": "142 skills", "type": "population", "value": { "n": 142, "unit": "skills" }, "col": "Population" }, { "raw": "30.00%", "type": "percent", "value": 0.3, "col": "Rules" }, { "raw": "95.77%", "type": "percent", "value": 0.9577, "col": "System One model" }, { "raw": "+65.8 points", "type": "points", "value": 65.8, "col": "Change" } ] ], "kind": "grid" }, { "source": "cascade.html#5", "page": "cascade.html", "index": 5, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Does the System One model earn its place between the rules and the judge?", "h3": "What context is worth" }, "before_text": "Each recipe adds one layer and nothing else, and an unknown field is rejected rather than ignored, so a recipe cannot silently fall back to the full context and report a difference that was never sent.", "after_text": "Context is worth a few points of recall for roughly a quarter more input tokens, and the gain is monotonic across the recipes. On an unlabeled real-world corpus the same sweep showed nothing at all, because the model allowed every package at every recipe; an ablation cannot discriminate on a population where every answer is the same.", "columns": [ { "name": "Recipe", "tooltip": null }, { "name": "Adds", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Input tokens", "tooltip": null } ], "rows": [ [ { "raw": "K0", "type": "text", "value": "K0", "col": "Recipe" }, { "raw": "control: packed file content only", "type": "text", "value": "control: packed file content only", "col": "Adds" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "45.92%", "type": "percent", "value": 0.4592, "col": "Recall" }, { "raw": "62.94%", "type": "percent", "value": 0.6294, "col": "F1" }, { "raw": "2,062,360", "type": "number", "value": 2062360, "col": "Input tokens" } ], [ { "raw": "K1", "type": "text", "value": "K1", "col": "Recipe" }, { "raw": "declared purpose", "type": "text", "value": "declared purpose", "col": "Adds" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "48.09%", "type": "percent", "value": 0.4809, "col": "Recall" }, { "raw": "64.95%", "type": "percent", "value": 0.6495, "col": "F1" }, { "raw": "2,495,808", "type": "number", "value": 2495808, "col": "Input tokens" } ], [ { "raw": "K3", "type": "text", "value": "K3", "col": "Recipe" }, { "raw": "derived package shape", "type": "text", "value": "derived package shape", "col": "Adds" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "48.35%", "type": "percent", "value": 0.4835, "col": "Recall" }, { "raw": "65.18%", "type": "percent", "value": 0.6518, "col": "F1" }, { "raw": "2,544,772", "type": "number", "value": 2544772, "col": "Input tokens" } ], [ { "raw": "K7", "type": "text", "value": "K7", "col": "Recipe" }, { "raw": "packing tier, the full recipe", "type": "text", "value": "packing tier, the full recipe", "col": "Adds" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Precision" }, { "raw": "49.36%", "type": "percent", "value": 0.4936, "col": "Recall" }, { "raw": "66.10%", "type": "percent", "value": 0.661, "col": "F1" }, { "raw": "2,552,672", "type": "number", "value": 2552672, "col": "Input tokens" } ] ], "kind": "grid" }, { "source": "cascade.html#6", "page": "cascade.html", "index": 6, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Llama Prompt Guard 2 as a pre-filter", "h3": "Separation on 839 malicious and 545 benign records" }, "before_text": "", "after_text": "It is effectively silent. AUC is 0.611, so there is a faint signal and it points the right way — unlike the System One result above, which is inverted — but the probabilities never reach a threshold anything could act on. Two of 1,384 records fire at 0.5 and both are benign .", "columns": [ { "name": "Threshold", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "0.5", "type": "number", "value": 0.5, "col": "Threshold" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Recall" }, { "raw": "—", "type": "null", "value": null, "col": "Precision" }, { "raw": "0.37%", "type": "percent", "value": 0.0037, "col": "False-positive rate" } ], [ { "raw": "0.8", "type": "number", "value": 0.8, "col": "Threshold" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Recall" }, { "raw": "—", "type": "null", "value": null, "col": "Precision" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "False-positive rate" } ], [ { "raw": "0.9", "type": "number", "value": 0.9, "col": "Threshold" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Recall" }, { "raw": "—", "type": "null", "value": null, "col": "Precision" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "False-positive rate" } ], [ { "raw": "0.99", "type": "number", "value": 0.99, "col": "Threshold" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "Recall" }, { "raw": "—", "type": "null", "value": null, "col": "Precision" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cascade.html#7", "page": "cascade.html", "index": 7, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Llama Prompt Guard 2 as a pre-filter", "h3": "On real published skills it is not merely inert" }, "before_text": "", "after_text": "The three highest-scoring records — 0.998, 0.995 and 0.993 — are a Chinese-language novel-writing checklist, a conversation-memory skill, and a prompt optimizer. All three are benign, and two causes account for the pattern. Skills containing more than 2% CJK characters (976 of 12,500) are flagged at 1.64% against 0.10% for the rest, a sixteen-fold difference, and sixteen of the 27 flagged records are CJK-heavy though such skills are 7.8% of the corpus. Separately, a prompt-engineering skill reads like an instruction-override attempt because directing a model is the product. Mean scores are nearly identical across the two groups, so the effect sits entirely in the high-confidence tail.", "columns": [ { "name": "Threshold", "tooltip": null }, { "name": "Records flagged", "tooltip": null }, { "name": "Rate", "tooltip": null } ], "rows": [ [ { "raw": "0.5", "type": "number", "value": 0.5, "col": "Threshold" }, { "raw": "27", "type": "number", "value": 27, "col": "Records flagged" }, { "raw": "0.22% [0.148, 0.314]", "type": "percent_ci", "value": 0.0022, "ci95": [ 0.00148, 0.00314 ], "col": "Rate" } ], [ { "raw": "0.8", "type": "number", "value": 0.8, "col": "Threshold" }, { "raw": "14", "type": "number", "value": 14, "col": "Records flagged" }, { "raw": "0.11% [0.067, 0.188]", "type": "percent_ci", "value": 0.0011, "ci95": [ 0.00067, 0.00188 ], "col": "Rate" } ], [ { "raw": "0.9", "type": "number", "value": 0.9, "col": "Threshold" }, { "raw": "11", "type": "number", "value": 11, "col": "Records flagged" }, { "raw": "0.09% [0.049, 0.158]", "type": "percent_ci", "value": 0.0009, "ci95": [ 0.00049, 0.00158 ], "col": "Rate" } ], [ { "raw": "0.99", "type": "number", "value": 0.99, "col": "Threshold" }, { "raw": "3", "type": "number", "value": 3, "col": "Records flagged" }, { "raw": "0.02% [0.008, 0.071]", "type": "percent_ci", "value": 0.0002, "ci95": [ 8e-05, 0.00071 ], "col": "Rate" } ] ], "kind": "grid" }, { "source": "cascade.html#8", "page": "cascade.html", "index": 8, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Llama Prompt Guard 2 as a pre-filter", "h3": "The same measurement, applied to this scanner" }, "before_text": "", "after_text": "The deterministic rules flag CJK-heavy skills less often, not more: 0.58x at MEDIUM or above and 0.32x at HIGH or above, against Prompt Guard's sixteen-fold in the other direction. That is the expected shape for rules keyed on code constructs — curl | bash , base64 decoding, credential paths — which do not care what language the prose around them is in. It does not follow that the scanner is unbiased here, and this corpus cannot settle it. These records are unlabelled, so a lower flag rate is consistent with two different explanations: those skills genuinely do less risky work, or the rules under-detect when the surrounding text is non-English. Separating them needs labelled non-English skills, which no corpus screened so far provides. The defensible claim is the narrow one — there is no evidence of over -flagging non-English skills, and the direction of any error is towards silence rather than noise. The INFO row moves the other way, which is consistent with the one hygiene rule that dominates that tier. So enabling it as a screen would penalise non-English and prompt-engineering skills while detecting essentially none of the labelled malicious ones. This is a statement about fit, not about the model. Prompt Guard detects prompt-injection and jailbreak attempts in text; whether a skill is safe to install is a different question, and a skill can be malicious through a download-and-execute chain or a credential sink without containing anything an injection classifier would recognise. It is not integrated. Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content.", "columns": [ { "name": "Group", "tooltip": null }, { "name": "Records", "tooltip": null }, { "name": "MEDIUM+", "tooltip": null }, { "name": "HIGH+", "tooltip": null }, { "name": "INFO+", "tooltip": null } ], "rows": [ [ { "raw": "More than 2% CJK", "type": "labelled", "value": [ 0.02 ], "col": "Group" }, { "raw": "976", "type": "number", "value": 976, "col": "Records" }, { "raw": "2.25%", "type": "percent", "value": 0.0225, "col": "MEDIUM+" }, { "raw": "0.82%", "type": "percent", "value": 0.0082, "col": "HIGH+" }, { "raw": "94.57%", "type": "percent", "value": 0.9457, "col": "INFO+" } ], [ { "raw": "Rest", "type": "text", "value": "Rest", "col": "Group" }, { "raw": "11,522", "type": "number", "value": 11522, "col": "Records" }, { "raw": "3.89%", "type": "percent", "value": 0.0389, "col": "MEDIUM+" }, { "raw": "2.60%", "type": "percent", "value": 0.026, "col": "HIGH+" }, { "raw": "87.68%", "type": "percent", "value": 0.8768, "col": "INFO+" } ] ], "kind": "grid" }, { "source": "large-scale.html#1", "page": "large-scale.html", "index": 1, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The judge reproduces locally, once a decoding fault is fixed" }, "before_text": "", "after_text": "MaliciousSkillBench source-disjoint split, single pass, core rules, detection at MEDIUM or above. The two routes agree, which is what licenses using the local run for everything below. The first local run lost 46.5% of records (644 of 1,384) to finish_reason=length . Under grammar-constrained JSON decoding the model emitted a complete object and then padded it with whitespace until it reached the 8,192-token limit. Serving with disable_any_whitespace took that to 0.6% (8 records). On eight records that had all failed, the patched server answered 8 of 8 and an unpatched control 4 of 8.", "columns": [ { "name": "Judge", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null } ], "rows": [ [ { "raw": "Gemma 4 26B, local vLLM", "type": "text", "value": "Gemma 4 26B, local vLLM", "col": "Judge" }, { "raw": "64.19%", "type": "percent", "value": 0.6419, "col": "F1" }, { "raw": "81.63%", "type": "percent", "value": 0.8163, "col": "Precision" }, { "raw": "52.88%", "type": "percent", "value": 0.5288, "col": "Recall" }, { "raw": "18.20%", "type": "percent", "value": 0.182, "col": "FPR" } ], [ { "raw": "Gemma 4 26B, Bedrock mantle (earlier)", "type": "text", "value": "Gemma 4 26B, Bedrock mantle (earlier)", "col": "Judge" }, { "raw": "61.20%", "type": "percent", "value": 0.612, "col": "F1" }, { "raw": "81.50%", "type": "percent", "value": 0.815, "col": "Precision" }, { "raw": "49.00%", "type": "percent", "value": 0.49, "col": "Recall" }, { "raw": "17.60%", "type": "percent", "value": 0.176, "col": "FPR" } ] ], "kind": "grid" }, { "source": "large-scale.html#2", "page": "large-scale.html", "index": 2, "headings": { "h1": "Large-scale run on local GPUs", "h2": "A System One screen in front of the judge halves its false-positive rate" }, "before_text": "OpenJev was asked eight boolean threat questions per skill, with the probability of true read from its next-token distribution. Its prompt_injection answer screens every record, and the judge runs only on those above a threshold. The threshold was chosen on one hash-selected half of the records (655) to keep recall within two points of the judge alone while skipping the most benign records, and every figure below is from the other half (721).", "after_text": "Judge alone Jev screen, then judge The screen skips 70.7% of benign records. It is not only a cost saving: many records the judge flags wrongly are ones OpenJev scores as harmless, so the cascade vetoes judge false positives and precision rises while recall moves little. This is one corpus split in half; the threshold has not been measured on the other labelled corpora and should not be assumed to transfer. What is not: this threshold was chosen on half of MaliciousSkillBench's frozen test split, and the benchmark's terms forbid selecting thresholds on test members. The reported half was unseen, but the half used for selection was test data. The screen is re-selected on train/validation in the next section, and those figures supersede these.", "columns": [ { "name": "Held-out half", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Judge calls", "tooltip": null } ], "rows": [ [ { "raw": "Judge alone", "type": "text", "value": "Judge alone", "col": "Held-out half" }, { "raw": "61.15%", "type": "percent", "value": 0.6115, "col": "F1" }, { "raw": "76.22%", "type": "percent", "value": 0.7622, "col": "Precision" }, { "raw": "51.05%", "type": "percent", "value": 0.5105, "col": "Recall" }, { "raw": "23.13%", "type": "percent", "value": 0.2313, "col": "FPR" }, { "raw": "100%", "type": "percent", "value": 1.0, "col": "Judge calls" } ], [ { "raw": "Jev screen, then judge", "type": "text", "value": "Jev screen, then judge", "col": "Held-out half" }, { "raw": "62.59%", "type": "percent", "value": 0.6259, "col": "F1" }, { "raw": "86.07%", "type": "percent", "value": 0.8607, "col": "Precision" }, { "raw": "49.18%", "type": "percent", "value": 0.4918, "col": "Recall" }, { "raw": "11.56%", "type": "percent", "value": 0.1156, "col": "FPR" }, { "raw": "60.7%", "type": "percent", "value": 0.607, "col": "Judge calls" } ] ], "kind": "grid" }, { "source": "large-scale.html#3", "page": "large-scale.html", "index": 3, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The OpenJev screen, re-selected on train/validation", "h3": "With the new judge prompt" }, "before_text": "", "after_text": "", "columns": [ { "name": "Held-out test, MEDIUM+", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Judge calls", "tooltip": null }, { "name": "Real-skill flags", "tooltip": null }, { "name": "Real-skill judge calls", "tooltip": null } ], "rows": [ [ { "raw": "Judge alone", "type": "text", "value": "Judge alone", "col": "Held-out test, MEDIUM+" }, { "raw": "65.7%", "type": "percent", "value": 0.657, "col": "Recall" }, { "raw": "12.5%", "type": "percent", "value": 0.125, "col": "FPR" }, { "raw": "89.0%", "type": "percent", "value": 0.89, "col": "Precision" }, { "raw": "75.6%", "type": "percent", "value": 0.756, "col": "F1" }, { "raw": "100%", "type": "percent", "value": 1.0, "col": "Judge calls" }, { "raw": "10.45%", "type": "percent", "value": 0.1045, "col": "Real-skill flags" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "Real-skill judge calls" } ], [ { "raw": "prompt_injection screen (p ≥ 0.04835)", "type": "text", "value": "prompt_injection screen (p ≥ 0.04835)", "col": "Held-out test, MEDIUM+" }, { "raw": "59.5%", "type": "percent", "value": 0.595, "col": "Recall" }, { "raw": "3.7%", "type": "percent", "value": 0.037, "col": "FPR" }, { "raw": "96.1%", "type": "percent", "value": 0.961, "col": "Precision" }, { "raw": "73.5%", "type": "percent", "value": 0.735, "col": "F1" }, { "raw": "60%", "type": "percent", "value": 0.6, "col": "Judge calls" }, { "raw": "3.15%", "type": "percent", "value": 0.0315, "col": "Real-skill flags" }, { "raw": "7.9%", "type": "percent", "value": 0.079, "col": "Real-skill judge calls" } ], [ { "raw": "Logistic, all eight probes (score ≥ -0.4536)", "type": "text", "value": "Logistic, all eight probes (score ≥ -0.4536)", "col": "Held-out test, MEDIUM+" }, { "raw": "61.3%", "type": "percent", "value": 0.613, "col": "Recall" }, { "raw": "2.4%", "type": "percent", "value": 0.024, "col": "FPR" }, { "raw": "97.5%", "type": "percent", "value": 0.975, "col": "Precision" }, { "raw": "75.3%", "type": "percent", "value": 0.753, "col": "F1" }, { "raw": "58%", "type": "percent", "value": 0.58, "col": "Judge calls" }, { "raw": "1.75%", "type": "percent", "value": 0.0175, "col": "Real-skill flags" }, { "raw": "3.1%", "type": "percent", "value": 0.031, "col": "Real-skill judge calls" } ] ], "kind": "grid" }, { "source": "large-scale.html#4", "page": "large-scale.html", "index": 4, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The OpenJev screen, re-selected on train/validation", "h3": "With the shipped judge prompt" }, "before_text": "", "after_text": "", "columns": [ { "name": "Held-out test, MEDIUM+", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Judge calls", "tooltip": null }, { "name": "Real-skill flags", "tooltip": null }, { "name": "Real-skill judge calls", "tooltip": null } ], "rows": [ [ { "raw": "Judge alone", "type": "text", "value": "Judge alone", "col": "Held-out test, MEDIUM+" }, { "raw": "51.3%", "type": "percent", "value": 0.513, "col": "Recall" }, { "raw": "16.1%", "type": "percent", "value": 0.161, "col": "FPR" }, { "raw": "82.9%", "type": "percent", "value": 0.829, "col": "Precision" }, { "raw": "63.4%", "type": "percent", "value": 0.634, "col": "F1" }, { "raw": "100%", "type": "percent", "value": 1.0, "col": "Judge calls" }, { "raw": "12.80%", "type": "percent", "value": 0.128, "col": "Real-skill flags" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "Real-skill judge calls" } ], [ { "raw": "prompt_injection screen (p ≥ 0.06092)", "type": "text", "value": "prompt_injection screen (p ≥ 0.06092)", "col": "Held-out test, MEDIUM+" }, { "raw": "48.0%", "type": "percent", "value": 0.48, "col": "Recall" }, { "raw": "3.3%", "type": "percent", "value": 0.033, "col": "FPR" }, { "raw": "95.7%", "type": "percent", "value": 0.957, "col": "Precision" }, { "raw": "63.9%", "type": "percent", "value": 0.639, "col": "F1" }, { "raw": "58%", "type": "percent", "value": 0.58, "col": "Judge calls" }, { "raw": "2.60%", "type": "percent", "value": 0.026, "col": "Real-skill flags" }, { "raw": "7.0%", "type": "percent", "value": 0.07, "col": "Real-skill judge calls" } ], [ { "raw": "Logistic, all eight probes (score ≥ -0.0264)", "type": "text", "value": "Logistic, all eight probes (score ≥ -0.0264)", "col": "Held-out test, MEDIUM+" }, { "raw": "48.8%", "type": "percent", "value": 0.488, "col": "Recall" }, { "raw": "2.0%", "type": "percent", "value": 0.02, "col": "FPR" }, { "raw": "97.4%", "type": "percent", "value": 0.974, "col": "Precision" }, { "raw": "65.0%", "type": "percent", "value": 0.65, "col": "F1" }, { "raw": "56%", "type": "percent", "value": 0.56, "col": "Judge calls" }, { "raw": "1.30%", "type": "percent", "value": 0.013, "col": "Real-skill flags" }, { "raw": "2.5%", "type": "percent", "value": 0.025, "col": "Real-skill judge calls" } ] ], "kind": "grid" }, { "source": "large-scale.html#5", "page": "large-scale.html", "index": 5, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The OpenJev screen, re-selected on train/validation", "h3": "New prompt, low-confidence findings capped at LOW" }, "before_text": "", "after_text": "What is good: Selected on train/validation, a screen cuts the judge's test false-positive rate by 71% to 88% for 2.5 to 6.2 points of recall, and on real skills it sends only 2.5% to 7.9% of records to the judge at all. The eight-probe logistic screen is the stronger of the two on every population. What is not: The logistic screen needs all eight probes per skill, which were scored on the labelled splits and the 2,000-skill sample; the full-corpus run scored prompt_injection only, so the full-corpus cascade uses that screen. Real-skill rates are flag rates, not false-positive rates.", "columns": [ { "name": "Held-out test, MEDIUM+", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Real-skill flags", "tooltip": null }, { "name": "Real-skill judge calls", "tooltip": null } ], "rows": [ [ { "raw": "Judge alone", "type": "text", "value": "Judge alone", "col": "Held-out test, MEDIUM+" }, { "raw": "62.1%", "type": "percent", "value": 0.621, "col": "Recall" }, { "raw": "10.5%", "type": "percent", "value": 0.105, "col": "FPR" }, { "raw": "7.60%", "type": "percent", "value": 0.076, "col": "Real-skill flags" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "Real-skill judge calls" } ], [ { "raw": "prompt_injection screen", "type": "text", "value": "prompt_injection screen", "col": "Held-out test, MEDIUM+" }, { "raw": "54.2%", "type": "percent", "value": 0.542, "col": "Recall" }, { "raw": "2.9%", "type": "percent", "value": 0.029, "col": "FPR" }, { "raw": "2.35%", "type": "percent", "value": 0.0235, "col": "Real-skill flags" }, { "raw": "6.0%", "type": "percent", "value": 0.06, "col": "Real-skill judge calls" } ], [ { "raw": "Logistic, all eight probes", "type": "text", "value": "Logistic, all eight probes", "col": "Held-out test, MEDIUM+" }, { "raw": "56.8%", "type": "percent", "value": 0.568, "col": "Recall" }, { "raw": "1.7%", "type": "percent", "value": 0.017, "col": "FPR" }, { "raw": "1.35%", "type": "percent", "value": 0.0135, "col": "Real-skill flags" }, { "raw": "2.5%", "type": "percent", "value": 0.025, "col": "Real-skill judge calls" } ] ], "kind": "grid" }, { "source": "large-scale.html#6", "page": "large-scale.html", "index": 6, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The OpenJev screen, re-selected on train/validation", "h3": "Each probe on its own" }, "before_text": "", "after_text": "", "columns": [ { "name": "Screen", "tooltip": null }, { "name": "Train/val AUC", "tooltip": null }, { "name": "Test AUC", "tooltip": null } ], "rows": [ [ { "raw": "logistic", "type": "text", "value": "logistic", "col": "Screen" }, { "raw": "0.973", "type": "number", "value": 0.973, "col": "Train/val AUC" }, { "raw": "0.915", "type": "number", "value": 0.915, "col": "Test AUC" } ], [ { "raw": "obfuscation", "type": "text", "value": "obfuscation", "col": "Screen" }, { "raw": "0.967", "type": "number", "value": 0.967, "col": "Train/val AUC" }, { "raw": "0.908", "type": "number", "value": 0.908, "col": "Test AUC" } ], [ { "raw": "prompt_injection", "type": "text", "value": "prompt_injection", "col": "Screen" }, { "raw": "0.959", "type": "number", "value": 0.959, "col": "Train/val AUC" }, { "raw": "0.881", "type": "number", "value": 0.881, "col": "Test AUC" } ], [ { "raw": "security_control_change", "type": "text", "value": "security_control_change", "col": "Screen" }, { "raw": "0.935", "type": "number", "value": 0.935, "col": "Train/val AUC" }, { "raw": "0.859", "type": "number", "value": 0.859, "col": "Test AUC" } ], [ { "raw": "supply_chain", "type": "text", "value": "supply_chain", "col": "Screen" }, { "raw": "0.897", "type": "number", "value": 0.897, "col": "Train/val AUC" }, { "raw": "0.793", "type": "number", "value": 0.793, "col": "Test AUC" } ], [ { "raw": "destructive_command", "type": "text", "value": "destructive_command", "col": "Screen" }, { "raw": "0.830", "type": "number", "value": 0.83, "col": "Train/val AUC" }, { "raw": "0.791", "type": "number", "value": 0.791, "col": "Test AUC" } ], [ { "raw": "remote_hidden_execution", "type": "text", "value": "remote_hidden_execution", "col": "Screen" }, { "raw": "0.877", "type": "number", "value": 0.877, "col": "Train/val AUC" }, { "raw": "0.726", "type": "number", "value": 0.726, "col": "Test AUC" } ], [ { "raw": "data_exfiltration", "type": "text", "value": "data_exfiltration", "col": "Screen" }, { "raw": "0.792", "type": "number", "value": 0.792, "col": "Train/val AUC" }, { "raw": "0.648", "type": "number", "value": 0.648, "col": "Test AUC" } ], [ { "raw": "sensitive_data_access", "type": "text", "value": "sensitive_data_access", "col": "Screen" }, { "raw": "0.734", "type": "number", "value": 0.734, "col": "Train/val AUC" }, { "raw": "0.588", "type": "number", "value": 0.588, "col": "Test AUC" } ] ], "kind": "grid" }, { "source": "large-scale.html#7", "page": "large-scale.html", "index": 7, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Where the deterministic false positives come from" }, "before_text": "Every finding was recorded with the analyzer that produced it and stored as Parquet, queried with DuckDB. The record-level rates reproduce the published deterministic baseline (7.71% false-positive rate at HIGH or above), which checks that the new pipeline measures the same thing as the old one.", "after_text": "", "columns": [ { "name": "Analyzer", "tooltip": null }, { "name": "MEDIUM+ on harmless", "tooltip": null }, { "name": "MEDIUM+ on malicious", "tooltip": null }, { "name": "Harmless per malicious", "tooltip": null } ], "rows": [ [ { "raw": "correlation", "type": "text", "value": "correlation", "col": "Analyzer" }, { "raw": "35", "type": "number", "value": 35, "col": "MEDIUM+ on harmless" }, { "raw": "29", "type": "number", "value": 29, "col": "MEDIUM+ on malicious" }, { "raw": "1.21", "type": "number", "value": 1.21, "col": "Harmless per malicious" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "10", "type": "number", "value": 10, "col": "MEDIUM+ on harmless" }, { "raw": "30", "type": "number", "value": 30, "col": "MEDIUM+ on malicious" }, { "raw": "0.33", "type": "number", "value": 0.33, "col": "Harmless per malicious" } ], [ { "raw": "pipeline", "type": "text", "value": "pipeline", "col": "Analyzer" }, { "raw": "10", "type": "number", "value": 10, "col": "MEDIUM+ on harmless" }, { "raw": "33", "type": "number", "value": 33, "col": "MEDIUM+ on malicious" }, { "raw": "0.30", "type": "number", "value": 0.3, "col": "Harmless per malicious" } ] ], "kind": "grid" }, { "source": "large-scale.html#8", "page": "large-scale.html", "index": 8, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Where the deterministic false positives come from" }, "before_text": "", "after_text": "", "columns": [ { "name": "Analyzer", "tooltip": null }, { "name": "Rule", "tooltip": null }, { "name": "Harmless", "tooltip": null }, { "name": "Malicious", "tooltip": null }, { "name": "Precision", "tooltip": null } ], "rows": [ [ { "raw": "correlation", "type": "text", "value": "correlation", "col": "Analyzer" }, { "raw": "CORRELATED_NETWORK_EXECUTION_FLOW", "type": "text", "value": "CORRELATED_NETWORK_EXECUTION_FLOW", "col": "Rule" }, { "raw": "31", "type": "number", "value": 31, "col": "Harmless" }, { "raw": "7", "type": "number", "value": 7, "col": "Malicious" }, { "raw": "18.4%", "type": "percent", "value": 0.184, "col": "Precision" } ], [ { "raw": "pipeline", "type": "text", "value": "pipeline", "col": "Analyzer" }, { "raw": "PIPELINE_TAINT_FLOW", "type": "text", "value": "PIPELINE_TAINT_FLOW", "col": "Rule" }, { "raw": "10", "type": "number", "value": 10, "col": "Harmless" }, { "raw": "9", "type": "number", "value": 9, "col": "Malicious" }, { "raw": "47.4%", "type": "percent", "value": 0.474, "col": "Precision" } ], [ { "raw": "correlation", "type": "text", "value": "correlation", "col": "Analyzer" }, { "raw": "CORRELATED_SENSITIVE_NETWORK_FLOW", "type": "text", "value": "CORRELATED_SENSITIVE_NETWORK_FLOW", "col": "Rule" }, { "raw": "4", "type": "number", "value": 4, "col": "Harmless" }, { "raw": "22", "type": "number", "value": 22, "col": "Malicious" }, { "raw": "84.6%", "type": "percent", "value": 0.846, "col": "Precision" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "ACTIVE_DYNAMIC_EXECUTION", "type": "text", "value": "ACTIVE_DYNAMIC_EXECUTION", "col": "Rule" }, { "raw": "4", "type": "number", "value": 4, "col": "Harmless" }, { "raw": "18", "type": "number", "value": 18, "col": "Malicious" }, { "raw": "81.8%", "type": "percent", "value": 0.818, "col": "Precision" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "ACTIVE_REMOTE_ACQUIRE_EXECUTE", "type": "text", "value": "ACTIVE_REMOTE_ACQUIRE_EXECUTE", "col": "Rule" }, { "raw": "1", "type": "number", "value": 1, "col": "Harmless" }, { "raw": "1", "type": "number", "value": 1, "col": "Malicious" }, { "raw": "50.0%", "type": "percent", "value": 0.5, "col": "Precision" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "FILE_MAGIC_MISMATCH", "type": "text", "value": "FILE_MAGIC_MISMATCH", "col": "Rule" }, { "raw": "1", "type": "number", "value": 1, "col": "Harmless" }, { "raw": "0", "type": "number", "value": 0, "col": "Malicious" }, { "raw": "0.0%", "type": "percent", "value": 0.0, "col": "Precision" } ] ], "kind": "grid" }, { "source": "large-scale.html#9", "page": "large-scale.html", "index": 9, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Where the deterministic false positives come from", "h3": "One rule change removes most of the deterministic false-positive rate" }, "before_text": "", "after_text": "correlation is the only analyzer that fires more often on harmless records than malicious ones, and one rule accounts for almost all of it. The earlier conclusion that rule-level suppression could buy at most 0.2 F1 points came from data that recorded only a record's set of rule ids, with no analyzer and no per-finding severity; it was a limit of the data, not of the rules. F1 barely moves because deterministic recall is low either way, so precision and false-positive rate are the metrics to read.", "columns": [ { "name": "Configuration", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null } ], "rows": [ [ { "raw": "Shipped, all rules", "type": "text", "value": "Shipped, all rules", "col": "Configuration" }, { "raw": "13.49%", "type": "percent", "value": 0.1349, "col": "F1" }, { "raw": "58.18%", "type": "percent", "value": 0.5818, "col": "Precision" }, { "raw": "7.63%", "type": "percent", "value": 0.0763, "col": "Recall" }, { "raw": "8.44%", "type": "percent", "value": 0.0844, "col": "FPR" } ], [ { "raw": "Demote CORRELATED_NETWORK_EXECUTION_FLOW", "type": "text", "value": "Demote CORRELATED_NETWORK_EXECUTION_FLOW", "col": "Configuration" }, { "raw": "13.29%", "type": "percent", "value": 0.1329, "col": "F1" }, { "raw": "77.22%", "type": "percent", "value": 0.7722, "col": "Precision" }, { "raw": "7.27%", "type": "percent", "value": 0.0727, "col": "Recall" }, { "raw": "3.30%", "type": "percent", "value": 0.033, "col": "FPR" } ], [ { "raw": "Also demote PIPELINE_TAINT_FLOW", "type": "text", "value": "Also demote PIPELINE_TAINT_FLOW", "col": "Configuration" }, { "raw": "11.50%", "type": "percent", "value": 0.115, "col": "F1" }, { "raw": "80.00%", "type": "percent", "value": 0.8, "col": "Precision" }, { "raw": "6.20%", "type": "percent", "value": 0.062, "col": "Recall" }, { "raw": "2.39%", "type": "percent", "value": 0.0239, "col": "FPR" } ] ], "kind": "grid" }, { "source": "large-scale.html#10", "page": "large-scale.html", "index": 10, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Every usable skill: deterministic, judge and cascade" }, "before_text": "1,876,662 real published skills, the whole usable gitskills corpus, each scanned in full by every deterministic analyzer with the shipped scanner, again after the first tuning pass, and a third time with the final tree. Between the shipped and the final scanner 38,275 records left MEDIUM or above and 7 entered it. These are unlabelled, so a flag rate bounds the false-positive rate from above rather than measuring it.", "after_text": "The later passes were first measured by rescanning only the records each change could touch and overlaying them on the first-pass scan. That estimate was 2.153%; the full rescan gives 2.144%. Every record the full scan flags was in the estimate, which over-counted 168 records (130 by YARA_jailbreak_generic , 38 by FIND_EXEC_PATTERN ) whose changes landed after, or outside, the rescanned sets.", "columns": [ { "name": "Threshold", "tooltip": null }, { "name": "Shipped (95% interval)", "tooltip": null }, { "name": "First pass", "tooltip": null }, { "name": "Final (95% interval)", "tooltip": null }, { "name": "Relative change", "tooltip": null } ], "rows": [ [ { "raw": "CRITICAL", "type": "text", "value": "CRITICAL", "col": "Threshold" }, { "raw": "0.629% 0.617 to 0.640", "type": "percent_ci", "value": 0.00629, "ci95": [ 0.00617, 0.0064 ], "col": "Shipped (95% interval)" }, { "raw": "0.598%", "type": "percent", "value": 0.00598, "col": "First pass" }, { "raw": "0.425% 0.416 to 0.434", "type": "percent_ci", "value": 0.00425, "ci95": [ 0.00416, 0.00434 ], "col": "Final (95% interval)" }, { "raw": "-32.4%", "type": "percent", "value": -0.324, "col": "Relative change" } ], [ { "raw": "HIGH+", "type": "text", "value": "HIGH+", "col": "Threshold" }, { "raw": "2.469% 2.447 to 2.491", "type": "percent_ci", "value": 0.02469, "ci95": [ 0.02447, 0.02491 ], "col": "Shipped (95% interval)" }, { "raw": "2.312%", "type": "percent", "value": 0.02312, "col": "First pass" }, { "raw": "1.987% 1.967 to 2.007", "type": "percent_ci", "value": 0.01987, "ci95": [ 0.01967, 0.02007 ], "col": "Final (95% interval)" }, { "raw": "-19.5%", "type": "percent", "value": -0.195, "col": "Relative change" } ], [ { "raw": "MEDIUM+", "type": "text", "value": "MEDIUM+", "col": "Threshold" }, { "raw": "4.184% 4.155 to 4.212", "type": "percent_ci", "value": 0.04184, "ci95": [ 0.04155, 0.04212 ], "col": "Shipped (95% interval)" }, { "raw": "2.466%", "type": "percent", "value": 0.02466, "col": "First pass" }, { "raw": "2.144% 2.124 to 2.165", "type": "percent_ci", "value": 0.02144, "ci95": [ 0.02124, 0.02165 ], "col": "Final (95% interval)" }, { "raw": "-48.7%", "type": "percent", "value": -0.487, "col": "Relative change" } ], [ { "raw": "LOW+", "type": "text", "value": "LOW+", "col": "Threshold" }, { "raw": "5.213% 5.181 to 5.245", "type": "percent_ci", "value": 0.05213, "ci95": [ 0.05181, 0.05245 ], "col": "Shipped (95% interval)" }, { "raw": "3.770%", "type": "percent", "value": 0.0377, "col": "First pass" }, { "raw": "3.451% 3.425 to 3.477", "type": "percent_ci", "value": 0.03451, "ci95": [ 0.03425, 0.03477 ], "col": "Final (95% interval)" }, { "raw": "-33.8%", "type": "percent", "value": -0.338, "col": "Relative change" } ], [ { "raw": "INFO+", "type": "text", "value": "INFO+", "col": "Threshold" }, { "raw": "90.365% 90.323 to 90.407", "type": "percent_ci", "value": 0.90365, "ci95": [ 0.90323, 0.90407 ], "col": "Shipped (95% interval)" }, { "raw": "89.639%", "type": "percent", "value": 0.89639, "col": "First pass" }, { "raw": "89.616% 89.572 to 89.659", "type": "percent_ci", "value": 0.89616, "ci95": [ 0.89572, 0.89659 ], "col": "Final (95% interval)" }, { "raw": "-0.8%", "type": "percent", "value": -0.008, "col": "Relative change" } ] ], "kind": "grid" }, { "source": "large-scale.html#11", "page": "large-scale.html", "index": 11, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Every usable skill: deterministic, judge and cascade", "h3": "The same change on labelled corpora" }, "before_text": "Shipped scanner against the final scanner, static arm, MEDIUM or above. Every corpus the harness holds is shown, including those where the tuning costs recall. The rules were designed on the train/validation split and on real skills; the test split was not used to design them.", "after_text": "What is not: the recall cost on train/validation is 89 malicious packages that no longer reach MEDIUM. The findings that carried them: 44 by SUPPLY_CHAIN_UNPINNED_DEPENDENCY , 19 by SOCIAL_ENG_ANTHROPIC_IMPERSONATION , 12 by PIPELINE_TAINT_FLOW , 6 by FILE_MAGIC_MISMATCH , 4 by CORRELATED_NETWORK_EXECUTION_FLOW + PIPELINE_TAINT_FLOW , 3 by CORRELATED_SENSITIVE_NETWORK_FLOW , 1 by PROMPT_INJECTION_IGNORE_INSTRUCTIONS . Most are hygiene or mention findings rather than the malicious behaviour — an unpinned dependency, a mention of Anthropic, a text label in Markdown — and the judge is the layer meant to catch these skills. Keeping a rule firing on the wrong evidence for its incidental recall would keep the false positives that come with it. The LLM judge, with the shipped prompt, read 1,876,206 of them (563 could not be analysed and are excluded, not counted as clean; 96,519 were read with partial coverage, the prompt budget leaving a file out) and flagged 12.70% 12.65 to 12.75 at MEDIUM or above. Its package verdicts: SAFE 80.7%, SUSPICIOUS 17.3%, MALICIOUS 2.0%. Almost all of its false positives read as findings about what a skill is for — a browser-automation skill can run JavaScript, a payments skill can move money — which is what the new prompt below addresses. The final deterministic scanner and the judge agree on 16,321 flagged records; 23,829 are flagged only by the rules and 221,924 only by the judge (Cohen's κ 0.084). The two layers look at different things, which is why neither is ground truth for the other. Deterministic flags alone: 2.14%. Deterministic flags plus the judge behind the OpenJev screen: 5.19%.", "columns": [ { "name": "Corpus", "tooltip": null }, { "name": "Records", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Unlabelled flag rate", "tooltip": null } ], "rows": [ [ { "raw": "msb-trainval development split (rules were designed on it)", "type": "text", "value": "msb-trainval development split (rules were designed on it)", "col": "Corpus" }, { "raw": "6,594", "type": "number", "value": 6594, "col": "Records" }, { "raw": "33.14% → 31.45%", "type": "pair", "value": [ 0.3314, 0.3145 ], "unit": "fraction", "col": "Recall" }, { "raw": "2.17% → 1.05%", "type": "pair", "value": [ 0.0217, 0.0105 ], "unit": "fraction", "col": "FPR" }, { "raw": "— → —", "type": "null", "value": null, "col": "Unlabelled flag rate" } ], [ { "raw": "msb-source-disjoint frozen test split (held out)", "type": "text", "value": "msb-source-disjoint frozen test split (held out)", "col": "Corpus" }, { "raw": "1,384", "type": "number", "value": 1384, "col": "Records" }, { "raw": "7.99% → 7.99%", "type": "pair", "value": [ 0.0799, 0.0799 ], "unit": "fraction", "col": "Recall" }, { "raw": "8.44% → 4.22%", "type": "pair", "value": [ 0.0844, 0.0422 ], "unit": "fraction", "col": "FPR" }, { "raw": "— → —", "type": "null", "value": null, "col": "Unlabelled flag rate" } ], [ { "raw": "msb-balanced-800 overlaps the test split (137 records)", "type": "text", "value": "msb-balanced-800 overlaps the test split (137 records)", "col": "Corpus" }, { "raw": "800", "type": "number", "value": 800, "col": "Records" }, { "raw": "33.00% → 31.00%", "type": "pair", "value": [ 0.33, 0.31 ], "unit": "fraction", "col": "Recall" }, { "raw": "2.00% → 1.25%", "type": "pair", "value": [ 0.02, 0.0125 ], "unit": "fraction", "col": "FPR" }, { "raw": "— → —", "type": "null", "value": null, "col": "Unlabelled flag rate" } ], [ { "raw": "openskillrisk-corpus", "type": "text", "value": "openskillrisk-corpus", "col": "Corpus" }, { "raw": "374", "type": "number", "value": 374, "col": "Records" }, { "raw": "31.55% → 29.95%", "type": "pair", "value": [ 0.3155, 0.2995 ], "unit": "fraction", "col": "Recall" }, { "raw": "— → —", "type": "null", "value": null, "col": "FPR" }, { "raw": "— → —", "type": "null", "value": null, "col": "Unlabelled flag rate" } ], [ { "raw": "mcp-tool-poisoning", "type": "text", "value": "mcp-tool-poisoning", "col": "Corpus" }, { "raw": "50", "type": "number", "value": 50, "col": "Records" }, { "raw": "40.00% → 40.00%", "type": "pair", "value": [ 0.4, 0.4 ], "unit": "fraction", "col": "Recall" }, { "raw": "0.00% → 0.00%", "type": "pair", "value": [ 0.0, 0.0 ], "unit": "fraction", "col": "FPR" }, { "raw": "— → —", "type": "null", "value": null, "col": "Unlabelled flag rate" } ], [ { "raw": "harmfulskillbench-corpus unlabelled for this purpose", "type": "text", "value": "harmfulskillbench-corpus unlabelled for this purpose", "col": "Corpus" }, { "raw": "200", "type": "number", "value": 200, "col": "Records" }, { "raw": "— → —", "type": "null", "value": null, "col": "Recall" }, { "raw": "— → —", "type": "null", "value": null, "col": "FPR" }, { "raw": "4.00% → 3.50%", "type": "pair", "value": [ 0.04, 0.035 ], "unit": "fraction", "col": "Unlabelled flag rate" } ], [ { "raw": "real-world-skills", "type": "text", "value": "real-world-skills", "col": "Corpus" }, { "raw": "12,498", "type": "number", "value": 12498, "col": "Records" }, { "raw": "— → —", "type": "null", "value": null, "col": "Recall" }, { "raw": "— → —", "type": "null", "value": null, "col": "FPR" }, { "raw": "3.76% → 2.49%", "type": "pair", "value": [ 0.0376, 0.0249 ], "unit": "fraction", "col": "Unlabelled flag rate" } ], [ { "raw": "gitskills-sample", "type": "text", "value": "gitskills-sample", "col": "Corpus" }, { "raw": "199,998", "type": "number", "value": 199998, "col": "Records" }, { "raw": "— → —", "type": "null", "value": null, "col": "Recall" }, { "raw": "— → —", "type": "null", "value": null, "col": "FPR" }, { "raw": "2.23% → 0.52%", "type": "pair", "value": [ 0.0223, 0.0052 ], "unit": "fraction", "col": "Unlabelled flag rate" } ] ], "kind": "grid" }, { "source": "large-scale.html#12", "page": "large-scale.html", "index": 12, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Every usable skill: deterministic, judge and cascade", "h3": "Which deterministic rules the judge disagrees with, after tuning" }, "before_text": "For the rules that flag the most real skills at MEDIUM or above: how many records each rule alone drives to that tier (removing the rule would clear them), and the share the judge clears. The judge is not ground truth — on the labelled split its own false-positive rate is about 18% and its recall 53% — so this ranks rules for review rather than measuring their error.", "after_text": "", "columns": [ { "name": "Analyzer", "tooltip": null }, { "name": "Rule", "tooltip": null }, { "name": "Records", "tooltip": null }, { "name": "Sole driver", "tooltip": null }, { "name": "Judged", "tooltip": null }, { "name": "Judge clears", "tooltip": null } ], "rows": [ [ { "raw": "correlation", "type": "text", "value": "correlation", "col": "Analyzer" }, { "raw": "CORRELATED_NETWORK_EXECUTION_FLOW", "type": "text", "value": "CORRELATED_NETWORK_EXECUTION_FLOW", "col": "Rule" }, { "raw": "7,556", "type": "number", "value": 7556, "col": "Records" }, { "raw": "4,380", "type": "number", "value": 4380, "col": "Sole driver" }, { "raw": "7,549", "type": "number", "value": 7549, "col": "Judged" }, { "raw": "43.6% 42.5 to 44.7", "type": "percent_ci", "value": 0.436, "ci95": [ 0.425, 0.447 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "ACTIVE_DYNAMIC_EXECUTION", "type": "text", "value": "ACTIVE_DYNAMIC_EXECUTION", "col": "Rule" }, { "raw": "6,968", "type": "number", "value": 6968, "col": "Records" }, { "raw": "5,734", "type": "number", "value": 5734, "col": "Sole driver" }, { "raw": "6,937", "type": "number", "value": 6937, "col": "Judged" }, { "raw": "63.8% 62.7 to 64.9", "type": "percent_ci", "value": 0.638, "ci95": [ 0.627, 0.649 ], "col": "Judge clears" } ], [ { "raw": "pipeline", "type": "text", "value": "pipeline", "col": "Analyzer" }, { "raw": "PIPELINE_TAINT_FLOW", "type": "text", "value": "PIPELINE_TAINT_FLOW", "col": "Rule" }, { "raw": "5,438", "type": "number", "value": 5438, "col": "Records" }, { "raw": "2,184", "type": "number", "value": 2184, "col": "Sole driver" }, { "raw": "5,415", "type": "number", "value": 5415, "col": "Judged" }, { "raw": "54.0% 52.6 to 55.3", "type": "percent_ci", "value": 0.54, "ci95": [ 0.526, 0.553 ], "col": "Judge clears" } ], [ { "raw": "correlation", "type": "text", "value": "correlation", "col": "Analyzer" }, { "raw": "CORRELATED_SENSITIVE_NETWORK_FLOW", "type": "text", "value": "CORRELATED_SENSITIVE_NETWORK_FLOW", "col": "Rule" }, { "raw": "4,237", "type": "number", "value": 4237, "col": "Records" }, { "raw": "3,590", "type": "number", "value": 3590, "col": "Sole driver" }, { "raw": "4,226", "type": "number", "value": 4226, "col": "Judged" }, { "raw": "59.7% 58.3 to 61.2", "type": "percent_ci", "value": 0.597, "ci95": [ 0.583, 0.612 ], "col": "Judge clears" } ], [ { "raw": "pipeline", "type": "text", "value": "pipeline", "col": "Analyzer" }, { "raw": "COMPOUND_FIND_EXEC", "type": "text", "value": "COMPOUND_FIND_EXEC", "col": "Rule" }, { "raw": "2,327", "type": "number", "value": 2327, "col": "Records" }, { "raw": "1,401", "type": "number", "value": 1401, "col": "Sole driver" }, { "raw": "2,326", "type": "number", "value": 2326, "col": "Judged" }, { "raw": "70.2% 68.3 to 72.0", "type": "percent_ci", "value": 0.702, "ci95": [ 0.683, 0.72 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "ACTIVE_REMOTE_ACQUIRE_EXECUTE", "type": "text", "value": "ACTIVE_REMOTE_ACQUIRE_EXECUTE", "col": "Rule" }, { "raw": "2,236", "type": "number", "value": 2236, "col": "Records" }, { "raw": "1,715", "type": "number", "value": 1715, "col": "Sole driver" }, { "raw": "2,225", "type": "number", "value": 2225, "col": "Judged" }, { "raw": "68.4% 66.4 to 70.3", "type": "percent_ci", "value": 0.684, "ci95": [ 0.664, 0.703 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "PROMPT_INJECTION_IGNORE_INSTRUCTIONS", "type": "text", "value": "PROMPT_INJECTION_IGNORE_INSTRUCTIONS", "col": "Rule" }, { "raw": "2,045", "type": "number", "value": 2045, "col": "Records" }, { "raw": "1,319", "type": "number", "value": 1319, "col": "Sole driver" }, { "raw": "2,038", "type": "number", "value": 2038, "col": "Judged" }, { "raw": "39.7% 37.6 to 41.9", "type": "percent_ci", "value": 0.397, "ci95": [ 0.376, 0.419 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "SECRET_CONNECTION_STRING", "type": "text", "value": "SECRET_CONNECTION_STRING", "col": "Rule" }, { "raw": "1,495", "type": "number", "value": 1495, "col": "Records" }, { "raw": "1,312", "type": "number", "value": 1312, "col": "Sole driver" }, { "raw": "1,493", "type": "number", "value": 1493, "col": "Judged" }, { "raw": "71.0% 68.6 to 73.2", "type": "percent_ci", "value": 0.71, "ci95": [ 0.686, 0.732 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "FIND_EXEC_PATTERN", "type": "text", "value": "FIND_EXEC_PATTERN", "col": "Rule" }, { "raw": "1,395", "type": "number", "value": 1395, "col": "Records" }, { "raw": "474", "type": "number", "value": 474, "col": "Sole driver" }, { "raw": "1,393", "type": "number", "value": 1393, "col": "Judged" }, { "raw": "68.8% 66.3 to 71.2", "type": "percent_ci", "value": 0.688, "ci95": [ 0.663, 0.712 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_jailbreak_generic", "type": "text", "value": "YARA_jailbreak_generic", "col": "Rule" }, { "raw": "1,365", "type": "number", "value": 1365, "col": "Records" }, { "raw": "1,047", "type": "number", "value": 1047, "col": "Sole driver" }, { "raw": "1,356", "type": "number", "value": 1356, "col": "Judged" }, { "raw": "70.4% 67.9 to 72.8", "type": "percent_ci", "value": 0.704, "ci95": [ 0.679, 0.728 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_sql_injection_generic", "type": "text", "value": "YARA_sql_injection_generic", "col": "Rule" }, { "raw": "1,185", "type": "number", "value": 1185, "col": "Records" }, { "raw": "1,049", "type": "number", "value": 1049, "col": "Sole driver" }, { "raw": "1,178", "type": "number", "value": 1178, "col": "Judged" }, { "raw": "59.7% 56.8 to 62.4", "type": "percent_ci", "value": 0.597, "ci95": [ 0.568, 0.624 ], "col": "Judge clears" } ], [ { "raw": "pipeline", "type": "text", "value": "pipeline", "col": "Analyzer" }, { "raw": "COMPOUND_FETCH_EXECUTE", "type": "text", "value": "COMPOUND_FETCH_EXECUTE", "col": "Rule" }, { "raw": "1,030", "type": "number", "value": 1030, "col": "Records" }, { "raw": "391", "type": "number", "value": 391, "col": "Sole driver" }, { "raw": "1,028", "type": "number", "value": 1028, "col": "Judged" }, { "raw": "53.4% 50.3 to 56.4", "type": "percent_ci", "value": 0.534, "ci95": [ 0.503, 0.564 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "ACTIVE_OS_PERSISTENCE_DIRECTIVE", "type": "text", "value": "ACTIVE_OS_PERSISTENCE_DIRECTIVE", "col": "Rule" }, { "raw": "1,020", "type": "number", "value": 1020, "col": "Records" }, { "raw": "812", "type": "number", "value": 812, "col": "Sole driver" }, { "raw": "1,017", "type": "number", "value": 1017, "col": "Judged" }, { "raw": "64.0% 61.0 to 66.9", "type": "percent_ci", "value": 0.64, "ci95": [ 0.61, 0.669 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_tool_chaining_abuse_generic", "type": "text", "value": "YARA_tool_chaining_abuse_generic", "col": "Rule" }, { "raw": "949", "type": "number", "value": 949, "col": "Records" }, { "raw": "516", "type": "number", "value": 516, "col": "Sole driver" }, { "raw": "946", "type": "number", "value": 946, "col": "Judged" }, { "raw": "42.0% 38.9 to 45.1", "type": "percent_ci", "value": 0.42, "ci95": [ 0.389, 0.451 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_autonomy_abuse_generic", "type": "text", "value": "YARA_autonomy_abuse_generic", "col": "Rule" }, { "raw": "932", "type": "number", "value": 932, "col": "Records" }, { "raw": "808", "type": "number", "value": 808, "col": "Sole driver" }, { "raw": "931", "type": "number", "value": 931, "col": "Judged" }, { "raw": "88.2% 86.0 to 90.1", "type": "percent_ci", "value": 0.882, "ci95": [ 0.86, 0.901 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_prompt_injection_generic", "type": "text", "value": "YARA_prompt_injection_generic", "col": "Rule" }, { "raw": "872", "type": "number", "value": 872, "col": "Records" }, { "raw": "379", "type": "number", "value": 379, "col": "Sole driver" }, { "raw": "872", "type": "number", "value": 872, "col": "Judged" }, { "raw": "60.6% 57.3 to 63.7", "type": "percent_ci", "value": 0.606, "ci95": [ 0.573, 0.637 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "PROMPT_INJECTION_CONCEALMENT", "type": "text", "value": "PROMPT_INJECTION_CONCEALMENT", "col": "Rule" }, { "raw": "859", "type": "number", "value": 859, "col": "Records" }, { "raw": "406", "type": "number", "value": 406, "col": "Sole driver" }, { "raw": "859", "type": "number", "value": 859, "col": "Judged" }, { "raw": "73.2% 70.2 to 76.1", "type": "percent_ci", "value": 0.732, "ci95": [ 0.702, 0.761 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_command_injection_generic", "type": "text", "value": "YARA_command_injection_generic", "col": "Rule" }, { "raw": "751", "type": "number", "value": 751, "col": "Records" }, { "raw": "470", "type": "number", "value": 470, "col": "Sole driver" }, { "raw": "744", "type": "number", "value": 744, "col": "Judged" }, { "raw": "46.2% 42.7 to 49.8", "type": "percent_ci", "value": 0.462, "ci95": [ 0.427, 0.498 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_system_manipulation_generic", "type": "text", "value": "YARA_system_manipulation_generic", "col": "Rule" }, { "raw": "684", "type": "number", "value": 684, "col": "Records" }, { "raw": "500", "type": "number", "value": 500, "col": "Sole driver" }, { "raw": "682", "type": "number", "value": 682, "col": "Judged" }, { "raw": "55.3% 51.5 to 59.0", "type": "percent_ci", "value": 0.553, "ci95": [ 0.515, 0.59 ], "col": "Judge clears" } ], [ { "raw": "static", "type": "text", "value": "static", "col": "Analyzer" }, { "raw": "YARA_credential_harvesting_generic", "type": "text", "value": "YARA_credential_harvesting_generic", "col": "Rule" }, { "raw": "670", "type": "number", "value": 670, "col": "Records" }, { "raw": "520", "type": "number", "value": 520, "col": "Sole driver" }, { "raw": "665", "type": "number", "value": 665, "col": "Judged" }, { "raw": "58.5% 54.7 to 62.2", "type": "percent_ci", "value": 0.585, "ci95": [ 0.547, 0.622 ], "col": "Judge clears" } ] ], "kind": "grid" }, { "source": "large-scale.html#13", "page": "large-scale.html", "index": 13, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Every usable skill: deterministic, judge and cascade", "h3": "The cascade over every skill" }, "before_text": "OpenJev screens each skill and the judge runs only above a threshold on its prompt_injection probability. The threshold, 0.0609 , was selected on MaliciousSkillBench train/validation for the prompt this corpus was judged with: keep at least 97% of the judge's recall at the lowest false-positive rate. On real skills it sends far fewer records to the judge than it did on the labelled split, so the whole trade-off is shown.", "after_text": "OpenJev's score ranks the judge's flags with AUC 0.778 over 1,876,206 skills both models read. Judge alone: 12.70%.", "columns": [ { "name": "Screen threshold", "tooltip": null }, { "name": "Judge calls", "tooltip": null }, { "name": "Judge flags kept", "tooltip": null }, { "name": "MEDIUM+ flag rate", "tooltip": null } ], "rows": [ [ { "raw": "0.005", "type": "number", "value": 0.005, "col": "Screen threshold" }, { "raw": "34.6%", "type": "percent", "value": 0.346, "col": "Judge calls" }, { "raw": "70.9%", "type": "percent", "value": 0.709, "col": "Judge flags kept" }, { "raw": "9.00%", "type": "percent", "value": 0.09, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.01", "type": "number", "value": 0.01, "col": "Screen threshold" }, { "raw": "23.2%", "type": "percent", "value": 0.232, "col": "Judge calls" }, { "raw": "56.7%", "type": "percent", "value": 0.567, "col": "Judge flags kept" }, { "raw": "7.20%", "type": "percent", "value": 0.072, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.02", "type": "number", "value": 0.02, "col": "Screen threshold" }, { "raw": "15.4%", "type": "percent", "value": 0.154, "col": "Judge calls" }, { "raw": "43.3%", "type": "percent", "value": 0.433, "col": "Judge flags kept" }, { "raw": "5.50%", "type": "percent", "value": 0.055, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.035", "type": "number", "value": 0.035, "col": "Screen threshold" }, { "raw": "11.1%", "type": "percent", "value": 0.111, "col": "Judge calls" }, { "raw": "34.4%", "type": "percent", "value": 0.344, "col": "Judge flags kept" }, { "raw": "4.36%", "type": "percent", "value": 0.0436, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.0484", "type": "number", "value": 0.0484, "col": "Screen threshold" }, { "raw": "9.1%", "type": "percent", "value": 0.091, "col": "Judge calls" }, { "raw": "29.9%", "type": "percent", "value": 0.299, "col": "Judge flags kept" }, { "raw": "3.80%", "type": "percent", "value": 0.038, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.0609", "type": "number", "value": 0.0609, "col": "Screen threshold" }, { "raw": "7.8%", "type": "percent", "value": 0.078, "col": "Judge calls" }, { "raw": "26.8%", "type": "percent", "value": 0.268, "col": "Judge flags kept" }, { "raw": "3.40%", "type": "percent", "value": 0.034, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.1", "type": "number", "value": 0.1, "col": "Screen threshold" }, { "raw": "5.5%", "type": "percent", "value": 0.055, "col": "Judge calls" }, { "raw": "20.6%", "type": "percent", "value": 0.206, "col": "Judge flags kept" }, { "raw": "2.62%", "type": "percent", "value": 0.0262, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.2", "type": "number", "value": 0.2, "col": "Screen threshold" }, { "raw": "3.2%", "type": "percent", "value": 0.032, "col": "Judge calls" }, { "raw": "13.3%", "type": "percent", "value": 0.133, "col": "Judge flags kept" }, { "raw": "1.69%", "type": "percent", "value": 0.0169, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.35", "type": "number", "value": 0.35, "col": "Screen threshold" }, { "raw": "1.7%", "type": "percent", "value": 0.017, "col": "Judge calls" }, { "raw": "8.3%", "type": "percent", "value": 0.083, "col": "Judge flags kept" }, { "raw": "1.05%", "type": "percent", "value": 0.0105, "col": "MEDIUM+ flag rate" } ], [ { "raw": "0.5", "type": "number", "value": 0.5, "col": "Screen threshold" }, { "raw": "1.0%", "type": "percent", "value": 0.01, "col": "Judge calls" }, { "raw": "5.8%", "type": "percent", "value": 0.058, "col": "Judge flags kept" }, { "raw": "0.73%", "type": "percent", "value": 0.0073, "col": "MEDIUM+ flag rate" } ] ], "kind": "grid" }, { "source": "large-scale.html#14", "page": "large-scale.html", "index": 14, "headings": { "h1": "Large-scale run on local GPUs", "h2": "Every usable skill: deterministic, judge and cascade", "h3": "Policy packs instead of knobs" }, "before_text": "The scan policy has hundreds of settings; most users need one decision, how much review capacity they have. Two presets sit beside the default. Every rule was scored by how many real skills it alone drives to MEDIUM or above that the judge also cleared, against how many malicious train/validation packages it alone detects; rules were demoted greedily in that order and the path was cut at two points. A demoted rule is reported at LOW, still visible but not gating. Real-skill rates below are exact on the final full scan, because a pack only lowers the severity of listed rules.", "after_text": "Use them with --policy low-noise or --policy quiet . low-noise also caps judge findings the model itself rates low-confidence; quiet additionally caps findings it labels contextual risk.", "columns": [ { "name": "Preset", "tooltip": null }, { "name": "Rules reported at LOW", "tooltip": null }, { "name": "Real-skill MEDIUM+ (rules)", "tooltip": null }, { "name": "Train/val recall", "tooltip": null }, { "name": "Train/val FPR", "tooltip": null } ], "rows": [ [ { "raw": "balanced", "type": "text", "value": "balanced", "col": "Preset" }, { "raw": "0", "type": "number", "value": 0, "col": "Rules reported at LOW" }, { "raw": "2.144% 2.124 to 2.165", "type": "percent_ci", "value": 0.02144, "ci95": [ 0.02124, 0.02165 ], "col": "Real-skill MEDIUM+ (rules)" }, { "raw": "31.45%", "type": "percent", "value": 0.3145, "col": "Train/val recall" }, { "raw": "1.05%", "type": "percent", "value": 0.0105, "col": "Train/val FPR" } ], [ { "raw": "low-noise", "type": "text", "value": "low-noise", "col": "Preset" }, { "raw": "11", "type": "number", "value": 11, "col": "Rules reported at LOW" }, { "raw": "1.930% 1.911 to 1.950", "type": "percent_ci", "value": 0.0193, "ci95": [ 0.01911, 0.0195 ], "col": "Real-skill MEDIUM+ (rules)" }, { "raw": "31.35%", "type": "percent", "value": 0.3135, "col": "Train/val recall" }, { "raw": "1.05%", "type": "percent", "value": 0.0105, "col": "Train/val FPR" } ], [ { "raw": "quiet", "type": "text", "value": "quiet", "col": "Preset" }, { "raw": "19", "type": "number", "value": 19, "col": "Rules reported at LOW" }, { "raw": "1.331% 1.314 to 1.347", "type": "percent_ci", "value": 0.01331, "ci95": [ 0.01314, 0.01347 ], "col": "Real-skill MEDIUM+ (rules)" }, { "raw": "29.78%", "type": "percent", "value": 0.2978, "col": "Train/val recall" }, { "raw": "0.15%", "type": "percent", "value": 0.0015, "col": "Train/val FPR" } ] ], "kind": "grid" }, { "source": "large-scale.html#15", "page": "large-scale.html", "index": 15, "headings": { "h1": "Large-scale run on local GPUs", "h2": "A judge prompt that separates a skill's purpose from misuse" }, "before_text": "The threat-analysis prompt now lists what is not a finding on its own — the capability the skill exists to provide, the breadth of its declared permissions, installing dependencies from a registry, requiring API keys, hypothetical injection, quality issues — and, just as explicitly, the misuse to report even when it is phrased as routine: a bundled or downloaded script the agent must run automatically, instructions to hide actions or override other instructions, and data sent where the stated purpose does not need it. It was written from real skills and MaliciousSkillBench train/validation, then scored once on the frozen test split.", "after_text": "", "columns": [ { "name": "Judge, MEDIUM+", "tooltip": null }, { "name": "Train/val recall", "tooltip": null }, { "name": "Train/val FPR", "tooltip": null }, { "name": "Test recall", "tooltip": null }, { "name": "Test FPR", "tooltip": null }, { "name": "Test precision", "tooltip": null }, { "name": "Test F1", "tooltip": null }, { "name": "Real-skill flags", "tooltip": null } ], "rows": [ [ { "raw": "Shipped prompt", "type": "text", "value": "Shipped prompt", "col": "Judge, MEDIUM+" }, { "raw": "86.2%", "type": "percent", "value": 0.862, "col": "Train/val recall" }, { "raw": "28.2%", "type": "percent", "value": 0.282, "col": "Train/val FPR" }, { "raw": "51.6%", "type": "percent", "value": 0.516, "col": "Test recall" }, { "raw": "16.1%", "type": "percent", "value": 0.161, "col": "Test FPR" }, { "raw": "83.1%", "type": "percent", "value": 0.831, "col": "Test precision" }, { "raw": "63.7%", "type": "percent", "value": 0.637, "col": "Test F1" }, { "raw": "12.80%", "type": "percent", "value": 0.128, "col": "Real-skill flags" } ], [ { "raw": "New prompt", "type": "text", "value": "New prompt", "col": "Judge, MEDIUM+" }, { "raw": "88.8%", "type": "percent", "value": 0.888, "col": "Train/val recall" }, { "raw": "24.0%", "type": "percent", "value": 0.24, "col": "Train/val FPR" }, { "raw": "65.9%", "type": "percent", "value": 0.659, "col": "Test recall" }, { "raw": "12.5%", "type": "percent", "value": 0.125, "col": "Test FPR" }, { "raw": "89.0%", "type": "percent", "value": 0.89, "col": "Test precision" }, { "raw": "75.8%", "type": "percent", "value": 0.758, "col": "Test F1" }, { "raw": "10.45%", "type": "percent", "value": 0.1045, "col": "Real-skill flags" } ] ], "kind": "grid" }, { "source": "large-scale.html#16", "page": "large-scale.html", "index": 16, "headings": { "h1": "Large-scale run on local GPUs", "h2": "A judge prompt that separates a skill's purpose from misuse", "h3": "Operating points on the new prompt" }, "before_text": "Each finding carries the model's own verdict ( TRUE_POSITIVE or CONTEXTUAL_RISK ) and confidence. Capping either below MEDIUM is an operating point, not a free win: it cuts false positives on every population and costs recall, more on test than on train/validation.", "after_text": "Gemma 4 26B-A4B via Bedrock mantle, verdict repair on (the default). Train/validation: 600 malicious and 1,338 benign MaliciousSkillBench packages. Test: the 1,384-package source-disjoint split. Real skills: 2,000 sampled uniformly from gitskills, a flag rate that bounds the false-positive rate from above.", "columns": [ { "name": "New prompt, MEDIUM+", "tooltip": null }, { "name": "Train/val recall", "tooltip": null }, { "name": "Train/val FPR", "tooltip": null }, { "name": "Test recall", "tooltip": null }, { "name": "Test FPR", "tooltip": null }, { "name": "Real-skill flags", "tooltip": null } ], "rows": [ [ { "raw": "No cap", "type": "text", "value": "No cap", "col": "New prompt, MEDIUM+" }, { "raw": "88.8%", "type": "percent", "value": 0.888, "col": "Train/val recall" }, { "raw": "24.0%", "type": "percent", "value": 0.24, "col": "Train/val FPR" }, { "raw": "65.9%", "type": "percent", "value": 0.659, "col": "Test recall" }, { "raw": "12.5%", "type": "percent", "value": 0.125, "col": "Test FPR" }, { "raw": "10.45%", "type": "percent", "value": 0.1045, "col": "Real-skill flags" } ], [ { "raw": "Low-confidence findings at LOW", "type": "text", "value": "Low-confidence findings at LOW", "col": "New prompt, MEDIUM+" }, { "raw": "86.3%", "type": "percent", "value": 0.863, "col": "Train/val recall" }, { "raw": "16.6%", "type": "percent", "value": 0.166, "col": "Train/val FPR" }, { "raw": "62.3%", "type": "percent", "value": 0.623, "col": "Test recall" }, { "raw": "10.5%", "type": "percent", "value": 0.105, "col": "Test FPR" }, { "raw": "7.60%", "type": "percent", "value": 0.076, "col": "Real-skill flags" } ], [ { "raw": "Contextual-risk findings at LOW", "type": "text", "value": "Contextual-risk findings at LOW", "col": "New prompt, MEDIUM+" }, { "raw": "83.8%", "type": "percent", "value": 0.838, "col": "Train/val recall" }, { "raw": "6.9%", "type": "percent", "value": 0.069, "col": "Train/val FPR" }, { "raw": "49.6%", "type": "percent", "value": 0.496, "col": "Test recall" }, { "raw": "4.2%", "type": "percent", "value": 0.042, "col": "Test FPR" }, { "raw": "3.00%", "type": "percent", "value": 0.03, "col": "Real-skill flags" } ] ], "kind": "grid" }, { "source": "large-scale.html#17", "page": "large-scale.html", "index": 17, "headings": { "h1": "Large-scale run on local GPUs", "h2": "The data behind this page", "h3": "Reproducing it" }, "before_text": "", "after_text": "Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content.", "columns": [ { "name": "Step", "tooltip": null }, { "name": "Script", "tooltip": null } ], "rows": [ [ { "raw": "Score OpenJev's eight probes", "type": "text", "value": "Score OpenJev's eight probes", "col": "Step" }, { "raw": "evals/experiments/c3_openjev_local.py", "type": "text", "value": "evals/experiments/c3_openjev_local.py", "col": "Script" } ], [ { "raw": "Run the judge alone over a corpus", "type": "text", "value": "Run the judge alone over a corpus", "col": "Step" }, { "raw": "evals/runners/judge_only.py", "type": "text", "value": "evals/runners/judge_only.py", "col": "Script" } ], [ { "raw": "Scan a corpus with one tree", "type": "text", "value": "Scan a corpus with one tree", "col": "Step" }, { "raw": "evals/runners/cross_tool_benchmark.py", "type": "text", "value": "evals/runners/cross_tool_benchmark.py", "col": "Script" } ], [ { "raw": "Build the store", "type": "text", "value": "Build the store", "col": "Step" }, { "raw": "evals/experiments/f1_full_corpus_store.py", "type": "text", "value": "evals/experiments/f1_full_corpus_store.py", "col": "Script" } ], [ { "raw": "Analyse every real skill", "type": "text", "value": "Analyse every real skill", "col": "Step" }, { "raw": "evals/experiments/f2_full_corpus_analysis.py", "type": "text", "value": "evals/experiments/f2_full_corpus_analysis.py", "col": "Script" } ], [ { "raw": "Check the overlay against the full scan", "type": "text", "value": "Check the overlay against the full scan", "col": "Step" }, { "raw": "evals/experiments/f3_overlay_check.py", "type": "text", "value": "evals/experiments/f3_overlay_check.py", "col": "Script" } ], [ { "raw": "Shipped against final on labelled corpora", "type": "text", "value": "Shipped against final on labelled corpora", "col": "Step" }, { "raw": "evals/experiments/f4_labelled_ab.py", "type": "text", "value": "evals/experiments/f4_labelled_ab.py", "col": "Script" } ], [ { "raw": "The preset demotion path", "type": "text", "value": "The preset demotion path", "col": "Step" }, { "raw": "evals/experiments/f5_policy_pack_path.py", "type": "text", "value": "evals/experiments/f5_policy_pack_path.py", "col": "Script" } ], [ { "raw": "Judge prompt and caps", "type": "text", "value": "Judge prompt and caps", "col": "Step" }, { "raw": "evals/experiments/a2_judge_prompt_caps.py", "type": "text", "value": "evals/experiments/a2_judge_prompt_caps.py", "col": "Script" } ], [ { "raw": "Select the OpenJev screen", "type": "text", "value": "Select the OpenJev screen", "col": "Step" }, { "raw": "evals/experiments/c4_openjev_screen.py", "type": "text", "value": "evals/experiments/c4_openjev_screen.py", "col": "Script" } ], [ { "raw": "Assemble and check this report", "type": "text", "value": "Assemble and check this report", "col": "Step" }, { "raw": "evals/publish/large_scale_report.py", "type": "text", "value": "evals/publish/large_scale_report.py", "col": "Script" } ] ], "kind": "grid" }, { "source": "cross-tool.html#1", "page": "cross-tool.html", "index": 1, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "What was compared" }, "before_text": "", "after_text": "", "columns": [ { "name": "", "tooltip": null }, { "name": "Version", "tooltip": null }, { "name": "Configuration", "tooltip": null } ], "rows": [ [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "" }, { "raw": "v2.11.2 (94f5cc80679c45a1fa4774e2d37ad663ca0be337)", "type": "text", "value": "v2.11.2 (94f5cc80679c45a1fa4774e2d37ad663ca0be337)", "col": "Version" }, { "raw": "All 27 analyzers enabled; built-in YARA; live OSV; baselines off; resource ceilings raised; --transitive off (corpus safety terms)", "type": "text", "value": "All 27 analyzers enabled; built-in YARA; live OSV; baselines off; resource ceilings raised; --transitive off (corpus safety terms)", "col": "Configuration" } ], [ { "raw": "Cisco skill-scanner", "type": "text", "value": "Cisco skill-scanner", "col": "" }, { "raw": "4f8bec3", "type": "text", "value": "4f8bec3", "col": "Version" }, { "raw": "Shipped core rule pack; judge + meta-judge; escalate-only verdict repair; policy suppressions empty; VirusTotal and AI Defense off (no credential)", "type": "text", "value": "Shipped core rule pack; judge + meta-judge; escalate-only verdict repair; policy suppressions empty; VirusTotal and AI Defense off (no credential)", "col": "Configuration" } ] ], "kind": "grid" }, { "source": "cross-tool.html#2", "page": "cross-tool.html", "index": 2, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -31.0 points, 95% interval [-34.2, -27.5] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "48.99%", "type": "percent", "value": 0.4899, "col": "Recall" }, { "raw": "81.55%", "type": "percent", "value": 0.8155, "col": "Precision" }, { "raw": "61.21%", "type": "percent", "value": 0.6121, "col": "F1" }, { "raw": "17.61%", "type": "percent", "value": 0.1761, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "79.98%", "type": "percent", "value": 0.7998, "col": "Recall" }, { "raw": "65.66%", "type": "percent", "value": 0.6566, "col": "Precision" }, { "raw": "72.11%", "type": "percent", "value": 0.7211, "col": "F1" }, { "raw": "66.48%", "type": "percent", "value": 0.6648, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#3", "page": "cross-tool.html", "index": 3, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -50.2 points, 95% interval [-53.9, -46.5] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "25.27%", "type": "percent", "value": 0.2527, "col": "Recall" }, { "raw": "80.00%", "type": "percent", "value": 0.8, "col": "Precision" }, { "raw": "38.41%", "type": "percent", "value": 0.3841, "col": "F1" }, { "raw": "10.04%", "type": "percent", "value": 0.1004, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "75.45%", "type": "percent", "value": 0.7545, "col": "Recall" }, { "raw": "69.41%", "type": "percent", "value": 0.6941, "col": "Precision" }, { "raw": "72.30%", "type": "percent", "value": 0.723, "col": "F1" }, { "raw": "52.84%", "type": "percent", "value": 0.5284, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#4", "page": "cross-tool.html", "index": 4, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Cisco skill-scanner (Gemma 4 26B) matches the HIGH row exactly wherever that happens below. It is a property of the default policy, which gates at HIGH, not a repeated table: the decision was read from each tool's emitted verdict field on every record. Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -13.6 points, 95% interval [-16.3, -10.7] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "25.27%", "type": "percent", "value": 0.2527, "col": "Recall" }, { "raw": "80.00%", "type": "percent", "value": 0.8, "col": "Precision" }, { "raw": "38.41%", "type": "percent", "value": 0.3841, "col": "F1" }, { "raw": "10.04%", "type": "percent", "value": 0.1004, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "38.86%", "type": "percent", "value": 0.3886, "col": "Recall" }, { "raw": "74.43%", "type": "percent", "value": 0.7443, "col": "Precision" }, { "raw": "51.06%", "type": "percent", "value": 0.5106, "col": "F1" }, { "raw": "21.21%", "type": "percent", "value": 0.2121, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#5", "page": "cross-tool.html", "index": 5, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): +11.7 points, 95% interval [+8.7, +14.9] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "92.61%", "type": "percent", "value": 0.9261, "col": "Recall" }, { "raw": "61.86%", "type": "percent", "value": 0.6186, "col": "Precision" }, { "raw": "74.18%", "type": "percent", "value": 0.7418, "col": "F1" }, { "raw": "90.72%", "type": "percent", "value": 0.9072, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "80.93%", "type": "percent", "value": 0.8093, "col": "Recall" }, { "raw": "64.42%", "type": "percent", "value": 0.6442, "col": "Precision" }, { "raw": "71.74%", "type": "percent", "value": 0.7174, "col": "F1" }, { "raw": "71.02%", "type": "percent", "value": 0.7102, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#6", "page": "cross-tool.html", "index": 6, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (Gemma 4 26B)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector (Gemma 4 26B)", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "480", "type": "number", "value": 480, "col": "Both flagged" }, { "raw": "321", "type": "number", "value": 321, "col": "Neither flagged" }, { "raw": "24", "type": "number", "value": 24, "col": "Only Cisco skill-scanner (Gemma 4 26B)" }, { "raw": "542", "type": "number", "value": 542, "col": "Only NVIDIA SkillSpector (Gemma 4 26B)" }, { "raw": "58.60%", "type": "percent", "value": 0.586, "col": "Agreement" }, { "raw": "41.40%", "type": "percent", "value": 0.414, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#7", "page": "cross-tool.html", "index": 7, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Cost of a scan" }, "before_text": "", "after_text": "Wall clock was recorded per subprocess, which covers SkillSpector but not this scanner's in-process path, and neither tool reported token usage through the harness. The uninstrumented cells are marked rather than shown as zero, and no cost comparison is drawn from this table. Judge token cost was measured separately and is reported in the token-cost section.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "26.018", "type": "number", "value": 26.018, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#8", "page": "cross-tool.html", "index": 8, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "0", "type": "number", "value": 0, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1367", "type": "number", "value": 1367, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "17", "type": "number", "value": 17, "col": "Capability degraded" }, { "raw": "1043", "type": "number", "value": 1043, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#9", "page": "cross-tool.html", "index": 9, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -12.4 points, 95% interval [-16.2, -8.7] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "82.07%", "type": "percent", "value": 0.8207, "col": "Recall" }, { "raw": "78.50%", "type": "percent", "value": 0.785, "col": "Precision" }, { "raw": "80.25%", "type": "percent", "value": 0.8025, "col": "F1" }, { "raw": "22.42%", "type": "percent", "value": 0.2242, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "94.44%", "type": "percent", "value": 0.9444, "col": "Recall" }, { "raw": "57.54%", "type": "percent", "value": 0.5754, "col": "Precision" }, { "raw": "71.51%", "type": "percent", "value": 0.7151, "col": "F1" }, { "raw": "69.52%", "type": "percent", "value": 0.6952, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#10", "page": "cross-tool.html", "index": 10, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -20.5 points, 95% interval [-25.0, -15.8] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "65.91%", "type": "percent", "value": 0.6591, "col": "Recall" }, { "raw": "93.21%", "type": "percent", "value": 0.9321, "col": "Precision" }, { "raw": "77.22%", "type": "percent", "value": 0.7722, "col": "F1" }, { "raw": "4.79%", "type": "percent", "value": 0.0479, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "86.36%", "type": "percent", "value": 0.8636, "col": "Recall" }, { "raw": "63.22%", "type": "percent", "value": 0.6322, "col": "Precision" }, { "raw": "73.00%", "type": "percent", "value": 0.73, "col": "F1" }, { "raw": "50.13%", "type": "percent", "value": 0.5013, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#11", "page": "cross-tool.html", "index": 11, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -5.1 points, 95% interval [-9.1, -1.0] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "65.91%", "type": "percent", "value": 0.6591, "col": "Recall" }, { "raw": "93.21%", "type": "percent", "value": 0.9321, "col": "Precision" }, { "raw": "77.22%", "type": "percent", "value": 0.7722, "col": "F1" }, { "raw": "4.79%", "type": "percent", "value": 0.0479, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "70.96%", "type": "percent", "value": 0.7096, "col": "Recall" }, { "raw": "80.52%", "type": "percent", "value": 0.8052, "col": "Precision" }, { "raw": "75.44%", "type": "percent", "value": 0.7544, "col": "F1" }, { "raw": "17.13%", "type": "percent", "value": 0.1713, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#12", "page": "cross-tool.html", "index": 12, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): +1.8 points, 95% interval [-0.3, +4.3] — not separable from zero.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "96.97%", "type": "percent", "value": 0.9697, "col": "Recall" }, { "raw": "50.73%", "type": "percent", "value": 0.5073, "col": "Precision" }, { "raw": "66.61%", "type": "percent", "value": 0.6661, "col": "F1" }, { "raw": "93.95%", "type": "percent", "value": 0.9395, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "95.20%", "type": "percent", "value": 0.952, "col": "Recall" }, { "raw": "54.56%", "type": "percent", "value": 0.5456, "col": "Precision" }, { "raw": "69.37%", "type": "percent", "value": 0.6937, "col": "F1" }, { "raw": "79.09%", "type": "percent", "value": 0.7909, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#13", "page": "cross-tool.html", "index": 13, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (Gemma 4 26B)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector (Gemma 4 26B)", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "397", "type": "number", "value": 397, "col": "Both flagged" }, { "raw": "126", "type": "number", "value": 126, "col": "Neither flagged" }, { "raw": "17", "type": "number", "value": 17, "col": "Only Cisco skill-scanner (Gemma 4 26B)" }, { "raw": "253", "type": "number", "value": 253, "col": "Only NVIDIA SkillSpector (Gemma 4 26B)" }, { "raw": "65.95%", "type": "percent", "value": 0.6595, "col": "Agreement" }, { "raw": "34.05%", "type": "percent", "value": 0.3405, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#14", "page": "cross-tool.html", "index": 14, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "23.174", "type": "number", "value": 23.174, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#15", "page": "cross-tool.html", "index": 15, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "MaliciousSkillBench, balanced subset", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "800", "type": "number", "value": 800, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "0", "type": "number", "value": 0, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "793", "type": "number", "value": 793, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "7", "type": "number", "value": 7, "col": "Capability degraded" }, { "raw": "674", "type": "number", "value": 674, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#16", "page": "cross-tool.html", "index": 16, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -10.2 points, 95% interval [-14.0, -6.4] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "85.71%", "type": "percent", "value": 0.8571, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "95.92%", "type": "percent", "value": 0.9592, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#17", "page": "cross-tool.html", "index": 17, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -14.9 points, 95% interval [-19.0, -10.8] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "71.43%", "type": "percent", "value": 0.7143, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "86.30%", "type": "percent", "value": 0.863, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#18", "page": "cross-tool.html", "index": 18, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -6.4 points, 95% interval [-10.5, -2.3] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "71.43%", "type": "percent", "value": 0.7143, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "77.84%", "type": "percent", "value": 0.7784, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#19", "page": "cross-tool.html", "index": 19, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): +1.2 points, 95% interval [-0.3, +2.9] — not separable from zero.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "99.42%", "type": "percent", "value": 0.9942, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "98.25%", "type": "percent", "value": 0.9825, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#20", "page": "cross-tool.html", "index": 20, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (Gemma 4 26B)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector (Gemma 4 26B)", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "289", "type": "number", "value": 289, "col": "Both flagged" }, { "raw": "9", "type": "number", "value": 9, "col": "Neither flagged" }, { "raw": "5", "type": "number", "value": 5, "col": "Only Cisco skill-scanner (Gemma 4 26B)" }, { "raw": "40", "type": "number", "value": 40, "col": "Only NVIDIA SkillSpector (Gemma 4 26B)" }, { "raw": "86.88%", "type": "percent", "value": 0.8688, "col": "Agreement" }, { "raw": "13.12%", "type": "percent", "value": 0.1312, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#21", "page": "cross-tool.html", "index": 21, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "23.519", "type": "number", "value": 23.519, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#22", "page": "cross-tool.html", "index": 22, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "OpenSkillRisk", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "374", "type": "number", "value": 374, "col": "Rows" }, { "raw": "374", "type": "number", "value": 374, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "0", "type": "number", "value": 0, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "374", "type": "number", "value": 374, "col": "Rows" }, { "raw": "343", "type": "number", "value": 343, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "31", "type": "number", "value": 31, "col": "Capability degraded" }, { "raw": "302", "type": "number", "value": 302, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#23", "page": "cross-tool.html", "index": 23, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -28.1 points, 95% interval [-35.7, -20.7] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "56.12%", "type": "percent", "value": 0.5612, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "84.18%", "type": "percent", "value": 0.8418, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#24", "page": "cross-tool.html", "index": 24, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): -18.9 points, 95% interval [-26.5, -11.2] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "38.78%", "type": "percent", "value": 0.3878, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "57.65%", "type": "percent", "value": 0.5765, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#25", "page": "cross-tool.html", "index": 25, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): +9.7 points, 95% interval [+2.0, +17.3] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "38.78%", "type": "percent", "value": 0.3878, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "29.08%", "type": "percent", "value": 0.2908, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#26", "page": "cross-tool.html", "index": 26, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (Gemma 4 26B) minus NVIDIA SkillSpector (Gemma 4 26B)): +10.7 points, 95% interval [+6.6, +15.3] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "98.47%", "type": "percent", "value": 0.9847, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "87.76%", "type": "percent", "value": 0.8776, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#27", "page": "cross-tool.html", "index": 27, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (Gemma 4 26B)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector (Gemma 4 26B)", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "101", "type": "number", "value": 101, "col": "Both flagged" }, { "raw": "22", "type": "number", "value": 22, "col": "Neither flagged" }, { "raw": "9", "type": "number", "value": 9, "col": "Only Cisco skill-scanner (Gemma 4 26B)" }, { "raw": "64", "type": "number", "value": 64, "col": "Only NVIDIA SkillSpector (Gemma 4 26B)" }, { "raw": "62.76%", "type": "percent", "value": 0.6276, "col": "Agreement" }, { "raw": "37.24%", "type": "percent", "value": 0.3724, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#28", "page": "cross-tool.html", "index": 28, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "17.079", "type": "number", "value": 17.079, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#29", "page": "cross-tool.html", "index": 29, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Gemma 4 26B", "h3": "HarmfulSkillBench", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (Gemma 4 26B)", "type": "text", "value": "Cisco skill-scanner (Gemma 4 26B)", "col": "Tool" }, { "raw": "200", "type": "number", "value": 200, "col": "Rows" }, { "raw": "200", "type": "number", "value": 200, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "0", "type": "number", "value": 0, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector (Gemma 4 26B)", "type": "text", "value": "NVIDIA SkillSpector (Gemma 4 26B)", "col": "Tool" }, { "raw": "200", "type": "number", "value": 200, "col": "Rows" }, { "raw": "196", "type": "number", "value": 196, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "4", "type": "number", "value": 4, "col": "Capability degraded" }, { "raw": "161", "type": "number", "value": 161, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#30", "page": "cross-tool.html", "index": 30, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -38.8 points, 95% interval [-42.7, -35.2] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "45.77%", "type": "percent", "value": 0.4577, "col": "Recall" }, { "raw": "81.96%", "type": "percent", "value": 0.8196, "col": "Precision" }, { "raw": "58.74%", "type": "percent", "value": 0.5874, "col": "F1" }, { "raw": "15.64%", "type": "percent", "value": 0.1564, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "84.58%", "type": "percent", "value": 0.8458, "col": "Recall" }, { "raw": "64.09%", "type": "percent", "value": 0.6409, "col": "Precision" }, { "raw": "72.92%", "type": "percent", "value": 0.7292, "col": "F1" }, { "raw": "73.55%", "type": "percent", "value": 0.7355, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#31", "page": "cross-tool.html", "index": 31, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -47.6 points, 95% interval [-51.4, -43.9] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "26.87%", "type": "percent", "value": 0.2687, "col": "Recall" }, { "raw": "86.06%", "type": "percent", "value": 0.8606, "col": "Precision" }, { "raw": "40.95%", "type": "percent", "value": 0.4095, "col": "F1" }, { "raw": "6.76%", "type": "percent", "value": 0.0676, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "74.50%", "type": "percent", "value": 0.745, "col": "Recall" }, { "raw": "67.45%", "type": "percent", "value": 0.6745, "col": "Precision" }, { "raw": "70.80%", "type": "percent", "value": 0.708, "col": "F1" }, { "raw": "55.79%", "type": "percent", "value": 0.5579, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#32", "page": "cross-tool.html", "index": 32, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -28.0 points, 95% interval [-31.7, -24.4] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "26.87%", "type": "percent", "value": 0.2687, "col": "Recall" }, { "raw": "86.06%", "type": "percent", "value": 0.8606, "col": "Precision" }, { "raw": "40.95%", "type": "percent", "value": 0.4095, "col": "F1" }, { "raw": "6.76%", "type": "percent", "value": 0.0676, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "54.85%", "type": "percent", "value": 0.5485, "col": "Recall" }, { "raw": "74.62%", "type": "percent", "value": 0.7462, "col": "Precision" }, { "raw": "63.23%", "type": "percent", "value": 0.6323, "col": "F1" }, { "raw": "28.96%", "type": "percent", "value": 0.2896, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#33", "page": "cross-tool.html", "index": 33, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -10.0 points, 95% interval [-13.8, -6.2] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "74.63%", "type": "percent", "value": 0.7463, "col": "Recall" }, { "raw": "57.42%", "type": "percent", "value": 0.5742, "col": "Precision" }, { "raw": "64.90%", "type": "percent", "value": 0.649, "col": "F1" }, { "raw": "85.91%", "type": "percent", "value": 0.8591, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "84.58%", "type": "percent", "value": 0.8458, "col": "Recall" }, { "raw": "64.09%", "type": "percent", "value": 0.6409, "col": "Precision" }, { "raw": "72.92%", "type": "percent", "value": 0.7292, "col": "F1" }, { "raw": "73.55%", "type": "percent", "value": 0.7355, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#34", "page": "cross-tool.html", "index": 34, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (core rules)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "420", "type": "number", "value": 420, "col": "Both flagged" }, { "raw": "232", "type": "number", "value": 232, "col": "Neither flagged" }, { "raw": "29", "type": "number", "value": 29, "col": "Only Cisco skill-scanner (core rules)" }, { "raw": "641", "type": "number", "value": 641, "col": "Only NVIDIA SkillSpector" }, { "raw": "49.32%", "type": "percent", "value": 0.4932, "col": "Agreement" }, { "raw": "50.68%", "type": "percent", "value": 0.5068, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#35", "page": "cross-tool.html", "index": 35, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "13.763", "type": "number", "value": 13.763, "col": "Mean seconds per skill" }, { "raw": "13,367,982", "type": "number", "value": 13367982, "col": "Input tokens" }, { "raw": "2,107,191", "type": "number", "value": 2107191, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "39.585", "type": "number", "value": 39.585, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#36", "page": "cross-tool.html", "index": 36, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1351", "type": "number", "value": 1351, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "33", "type": "number", "value": 33, "col": "Capability degraded" }, { "raw": "44", "type": "number", "value": 44, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1355", "type": "number", "value": 1355, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "29", "type": "number", "value": 29, "col": "Capability degraded" }, { "raw": "1045", "type": "number", "value": 1045, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#37", "page": "cross-tool.html", "index": 37, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -17.2 points, 95% interval [-21.2, -13.4] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "80.69%", "type": "percent", "value": 0.8069, "col": "Recall" }, { "raw": "73.85%", "type": "percent", "value": 0.7385, "col": "Precision" }, { "raw": "77.12%", "type": "percent", "value": 0.7712, "col": "F1" }, { "raw": "27.84%", "type": "percent", "value": 0.2784, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "97.88%", "type": "percent", "value": 0.9788, "col": "Recall" }, { "raw": "53.94%", "type": "percent", "value": 0.5394, "col": "Precision" }, { "raw": "69.55%", "type": "percent", "value": 0.6955, "col": "F1" }, { "raw": "81.44%", "type": "percent", "value": 0.8144, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#38", "page": "cross-tool.html", "index": 38, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -26.2 points, 95% interval [-31.0, -21.3] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "63.23%", "type": "percent", "value": 0.6323, "col": "Recall" }, { "raw": "89.18%", "type": "percent", "value": 0.8918, "col": "Precision" }, { "raw": "73.99%", "type": "percent", "value": 0.7399, "col": "F1" }, { "raw": "7.47%", "type": "percent", "value": 0.0747, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "89.42%", "type": "percent", "value": 0.8942, "col": "Recall" }, { "raw": "61.34%", "type": "percent", "value": 0.6134, "col": "Precision" }, { "raw": "72.77%", "type": "percent", "value": 0.7277, "col": "F1" }, { "raw": "54.90%", "type": "percent", "value": 0.549, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#39", "page": "cross-tool.html", "index": 39, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -18.3 points, 95% interval [-22.8, -13.5] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "63.23%", "type": "percent", "value": 0.6323, "col": "Recall" }, { "raw": "89.18%", "type": "percent", "value": 0.8918, "col": "Precision" }, { "raw": "73.99%", "type": "percent", "value": 0.7399, "col": "F1" }, { "raw": "7.47%", "type": "percent", "value": 0.0747, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "81.48%", "type": "percent", "value": 0.8148, "col": "Recall" }, { "raw": "73.68%", "type": "percent", "value": 0.7368, "col": "Precision" }, { "raw": "77.39%", "type": "percent", "value": 0.7739, "col": "F1" }, { "raw": "28.35%", "type": "percent", "value": 0.2835, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#40", "page": "cross-tool.html", "index": 40, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -5.0 points, 95% interval [-7.4, -2.4] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "92.86%", "type": "percent", "value": 0.9286, "col": "Recall" }, { "raw": "50.36%", "type": "percent", "value": 0.5036, "col": "Precision" }, { "raw": "65.30%", "type": "percent", "value": 0.653, "col": "F1" }, { "raw": "89.18%", "type": "percent", "value": 0.8918, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "97.88%", "type": "percent", "value": 0.9788, "col": "Recall" }, { "raw": "53.86%", "type": "percent", "value": 0.5386, "col": "Precision" }, { "raw": "69.48%", "type": "percent", "value": 0.6948, "col": "F1" }, { "raw": "81.70%", "type": "percent", "value": 0.817, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#41", "page": "cross-tool.html", "index": 41, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (core rules)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "407", "type": "number", "value": 407, "col": "Both flagged" }, { "raw": "74", "type": "number", "value": 74, "col": "Neither flagged" }, { "raw": "6", "type": "number", "value": 6, "col": "Only Cisco skill-scanner (core rules)" }, { "raw": "279", "type": "number", "value": 279, "col": "Only NVIDIA SkillSpector" }, { "raw": "62.79%", "type": "percent", "value": 0.6279, "col": "Agreement" }, { "raw": "37.21%", "type": "percent", "value": 0.3721, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#42", "page": "cross-tool.html", "index": 42, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "19.865", "type": "number", "value": 19.865, "col": "Mean seconds per skill" }, { "raw": "8,650,465", "type": "number", "value": 8650465, "col": "Input tokens" }, { "raw": "1,814,253", "type": "number", "value": 1814253, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "30.982", "type": "number", "value": 30.982, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#43", "page": "cross-tool.html", "index": 43, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Same model: Claude Haiku 4.5", "h3": "MaliciousSkillBench, balanced subset", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "783", "type": "number", "value": 783, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "17", "type": "number", "value": 17, "col": "Capability degraded" }, { "raw": "34", "type": "number", "value": 34, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "783", "type": "number", "value": 783, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "17", "type": "number", "value": 17, "col": "Capability degraded" }, { "raw": "674", "type": "number", "value": 674, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#44", "page": "cross-tool.html", "index": 44, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -26.3 points, 95% interval [-29.8, -23.2] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "7.99%", "type": "percent", "value": 0.0799, "col": "Recall" }, { "raw": "60.36%", "type": "percent", "value": 0.6036, "col": "Precision" }, { "raw": "14.11%", "type": "percent", "value": 0.1411, "col": "F1" }, { "raw": "8.32%", "type": "percent", "value": 0.0832, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "34.33%", "type": "percent", "value": 0.3433, "col": "Recall" }, { "raw": "56.14%", "type": "percent", "value": 0.5614, "col": "Precision" }, { "raw": "42.60%", "type": "percent", "value": 0.426, "col": "F1" }, { "raw": "42.53%", "type": "percent", "value": 0.4253, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#45", "page": "cross-tool.html", "index": 45, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): -6.8 points, 95% interval [-9.3, -4.1] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Recall" }, { "raw": "61.90%", "type": "percent", "value": 0.619, "col": "Precision" }, { "raw": "13.77%", "type": "percent", "value": 0.1377, "col": "F1" }, { "raw": "7.56%", "type": "percent", "value": 0.0756, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "14.54%", "type": "percent", "value": 0.1454, "col": "Recall" }, { "raw": "60.10%", "type": "percent", "value": 0.601, "col": "Precision" }, { "raw": "23.42%", "type": "percent", "value": 0.2342, "col": "F1" }, { "raw": "15.31%", "type": "percent", "value": 0.1531, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#46", "page": "cross-tool.html", "index": 46, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): +6.4 points, 95% interval [+4.8, +8.3] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "7.75%", "type": "percent", "value": 0.0775, "col": "Recall" }, { "raw": "61.90%", "type": "percent", "value": 0.619, "col": "Precision" }, { "raw": "13.77%", "type": "percent", "value": 0.1377, "col": "F1" }, { "raw": "7.56%", "type": "percent", "value": 0.0756, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "1.31%", "type": "percent", "value": 0.0131, "col": "Recall" }, { "raw": "57.89%", "type": "percent", "value": 0.5789, "col": "Precision" }, { "raw": "2.56%", "type": "percent", "value": 0.0256, "col": "F1" }, { "raw": "1.51%", "type": "percent", "value": 0.0151, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#47", "page": "cross-tool.html", "index": 47, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "Paired recall difference (Cisco skill-scanner (core rules) minus NVIDIA SkillSpector): +26.2 points, 95% interval [+21.6, +31.0] — statistically separable.", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "60.55%", "type": "percent", "value": 0.6055, "col": "Recall" }, { "raw": "52.48%", "type": "percent", "value": 0.5248, "col": "Precision" }, { "raw": "56.23%", "type": "percent", "value": 0.5623, "col": "F1" }, { "raw": "86.96%", "type": "percent", "value": 0.8696, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "34.33%", "type": "percent", "value": 0.3433, "col": "Recall" }, { "raw": "55.38%", "type": "percent", "value": 0.5538, "col": "Precision" }, { "raw": "42.38%", "type": "percent", "value": 0.4238, "col": "F1" }, { "raw": "43.86%", "type": "percent", "value": 0.4386, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#48", "page": "cross-tool.html", "index": 48, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Where they disagree" }, "before_text": "", "after_text": "", "columns": [ { "name": "Both flagged", "tooltip": null }, { "name": "Neither flagged", "tooltip": null }, { "name": "Only Cisco skill-scanner (core rules)", "tooltip": null }, { "name": "Only NVIDIA SkillSpector", "tooltip": null }, { "name": "Agreement", "tooltip": null }, { "name": "Complementarity", "tooltip": null } ], "rows": [ [ { "raw": "81", "type": "number", "value": 81, "col": "Both flagged" }, { "raw": "825", "type": "number", "value": 825, "col": "Neither flagged" }, { "raw": "30", "type": "number", "value": 30, "col": "Only Cisco skill-scanner (core rules)" }, { "raw": "432", "type": "number", "value": 432, "col": "Only NVIDIA SkillSpector" }, { "raw": "66.23%", "type": "percent", "value": 0.6623, "col": "Agreement" }, { "raw": "33.77%", "type": "percent", "value": 0.3377, "col": "Complementarity" } ] ], "kind": "grid" }, { "source": "cross-tool.html#49", "page": "cross-tool.html", "index": 49, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "0.015", "type": "number", "value": 0.015, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "2.453", "type": "number", "value": 2.453, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#50", "page": "cross-tool.html", "index": 50, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, source-disjoint split", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "11", "type": "number", "value": 11, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "1384", "type": "number", "value": 1384, "col": "Rows" }, { "raw": "1368", "type": "number", "value": 1368, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "16", "type": "number", "value": 16, "col": "Capability degraded" }, { "raw": "1043", "type": "number", "value": 1043, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#51", "page": "cross-tool.html", "index": 51, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "33.00%", "type": "percent", "value": 0.33, "col": "Recall" }, { "raw": "94.24%", "type": "percent", "value": 0.9424, "col": "Precision" }, { "raw": "48.88%", "type": "percent", "value": 0.4888, "col": "F1" }, { "raw": "2.02%", "type": "percent", "value": 0.0202, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "94.46%", "type": "percent", "value": 0.9446, "col": "Recall" }, { "raw": "52.97%", "type": "percent", "value": 0.5297, "col": "Precision" }, { "raw": "67.87%", "type": "percent", "value": 0.6787, "col": "F1" }, { "raw": "83.88%", "type": "percent", "value": 0.8388, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "48.11%", "type": "percent", "value": 0.4811, "col": "Recall" }, { "raw": "62.62%", "type": "percent", "value": 0.6262, "col": "Precision" }, { "raw": "54.42%", "type": "percent", "value": 0.5442, "col": "F1" }, { "raw": "28.72%", "type": "percent", "value": 0.2872, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#52", "page": "cross-tool.html", "index": 52, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "31.23%", "type": "percent", "value": 0.3123, "col": "Recall" }, { "raw": "95.38%", "type": "percent", "value": 0.9538, "col": "Precision" }, { "raw": "47.06%", "type": "percent", "value": 0.4706, "col": "F1" }, { "raw": "1.51%", "type": "percent", "value": 0.0151, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "91.44%", "type": "percent", "value": 0.9144, "col": "Recall" }, { "raw": "56.45%", "type": "percent", "value": 0.5645, "col": "Precision" }, { "raw": "69.81%", "type": "percent", "value": 0.6981, "col": "F1" }, { "raw": "70.53%", "type": "percent", "value": 0.7053, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "30.48%", "type": "percent", "value": 0.3048, "col": "Recall" }, { "raw": "75.62%", "type": "percent", "value": 0.7562, "col": "Precision" }, { "raw": "43.45%", "type": "percent", "value": 0.4345, "col": "F1" }, { "raw": "9.82%", "type": "percent", "value": 0.0982, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#53", "page": "cross-tool.html", "index": 53, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "31.23%", "type": "percent", "value": 0.3123, "col": "Recall" }, { "raw": "95.38%", "type": "percent", "value": 0.9538, "col": "Precision" }, { "raw": "47.06%", "type": "percent", "value": 0.4706, "col": "F1" }, { "raw": "1.51%", "type": "percent", "value": 0.0151, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "91.44%", "type": "percent", "value": 0.9144, "col": "Recall" }, { "raw": "56.45%", "type": "percent", "value": 0.5645, "col": "Precision" }, { "raw": "69.81%", "type": "percent", "value": 0.6981, "col": "F1" }, { "raw": "70.53%", "type": "percent", "value": 0.7053, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "8.56%", "type": "percent", "value": 0.0856, "col": "Recall" }, { "raw": "91.89%", "type": "percent", "value": 0.9189, "col": "Precision" }, { "raw": "15.67%", "type": "percent", "value": 0.1567, "col": "F1" }, { "raw": "0.76%", "type": "percent", "value": 0.0076, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#54", "page": "cross-tool.html", "index": 54, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "81.86%", "type": "percent", "value": 0.8186, "col": "Recall" }, { "raw": "47.86%", "type": "percent", "value": 0.4786, "col": "Precision" }, { "raw": "60.41%", "type": "percent", "value": 0.6041, "col": "F1" }, { "raw": "89.17%", "type": "percent", "value": 0.8917, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "99.75%", "type": "percent", "value": 0.9975, "col": "Recall" }, { "raw": "50.06%", "type": "percent", "value": 0.5006, "col": "Precision" }, { "raw": "66.67%", "type": "percent", "value": 0.6667, "col": "F1" }, { "raw": "99.50%", "type": "percent", "value": 0.995, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "50.13%", "type": "percent", "value": 0.5013, "col": "Recall" }, { "raw": "63.17%", "type": "percent", "value": 0.6317, "col": "Precision" }, { "raw": "55.90%", "type": "percent", "value": 0.559, "col": "F1" }, { "raw": "29.22%", "type": "percent", "value": 0.2922, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#55", "page": "cross-tool.html", "index": 55, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "0.017", "type": "number", "value": 0.017, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "0.280", "type": "number", "value": 0.28, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "2.399", "type": "number", "value": 2.399, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#56", "page": "cross-tool.html", "index": 56, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "MaliciousSkillBench, balanced subset", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "800", "type": "number", "value": 800, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "18", "type": "number", "value": 18, "col": "Reported partial" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "800", "type": "number", "value": 800, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "18", "type": "number", "value": 18, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "800", "type": "number", "value": 800, "col": "Rows" }, { "raw": "794", "type": "number", "value": 794, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "6", "type": "number", "value": 6, "col": "Capability degraded" }, { "raw": "674", "type": "number", "value": 674, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#57", "page": "cross-tool.html", "index": 57, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "25.07%", "type": "percent", "value": 0.2507, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "96.42%", "type": "percent", "value": 0.9642, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "77.91%", "type": "percent", "value": 0.7791, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#58", "page": "cross-tool.html", "index": 58, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "21.49%", "type": "percent", "value": 0.2149, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "91.64%", "type": "percent", "value": 0.9164, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "62.69%", "type": "percent", "value": 0.6269, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#59", "page": "cross-tool.html", "index": 59, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "21.49%", "type": "percent", "value": 0.2149, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "91.64%", "type": "percent", "value": 0.9164, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "33.43%", "type": "percent", "value": 0.3343, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#60", "page": "cross-tool.html", "index": 60, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "97.61%", "type": "percent", "value": 0.9761, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "78.21%", "type": "percent", "value": 0.7821, "col": "Recall" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "Precision" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "F1" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#61", "page": "cross-tool.html", "index": 61, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "0.018", "type": "number", "value": 0.018, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "0.266", "type": "number", "value": 0.266, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "2.457", "type": "number", "value": 2.457, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#62", "page": "cross-tool.html", "index": 62, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "OpenSkillRisk", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "374", "type": "number", "value": 374, "col": "Rows" }, { "raw": "374", "type": "number", "value": 374, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "5", "type": "number", "value": 5, "col": "Reported partial" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "374", "type": "number", "value": 374, "col": "Rows" }, { "raw": "365", "type": "number", "value": 365, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "9", "type": "number", "value": 9, "col": "Capability degraded" }, { "raw": "14", "type": "number", "value": 14, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "374", "type": "number", "value": 374, "col": "Rows" }, { "raw": "343", "type": "number", "value": 343, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "31", "type": "number", "value": 31, "col": "Capability degraded" }, { "raw": "302", "type": "number", "value": 302, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "cross-tool.html#63", "page": "cross-tool.html", "index": 63, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Any finding at MEDIUM or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "4.06%", "type": "percent", "value": 0.0406, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "90.86%", "type": "percent", "value": 0.9086, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "56.85%", "type": "percent", "value": 0.5685, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#64", "page": "cross-tool.html", "index": 64, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Any finding at HIGH or above" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "3.55%", "type": "percent", "value": 0.0355, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "83.76%", "type": "percent", "value": 0.8376, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "17.26%", "type": "percent", "value": 0.1726, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#65", "page": "cross-tool.html", "index": 65, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Each tool's own shipped install gate" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "3.55%", "type": "percent", "value": 0.0355, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "83.76%", "type": "percent", "value": 0.8376, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "2.54%", "type": "percent", "value": 0.0254, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#66", "page": "cross-tool.html", "index": 66, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Block on any finding at all" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "False-positive rate", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "96.45%", "type": "percent", "value": 0.9645, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "100.00%", "type": "percent", "value": 1.0, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "56.85%", "type": "percent", "value": 0.5685, "col": "Recall" }, { "raw": "not permitted", "type": "null", "value": null, "col": "Precision" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1" }, { "raw": "not permitted", "type": "null", "value": null, "col": "False-positive rate" } ] ], "kind": "grid" }, { "source": "cross-tool.html#67", "page": "cross-tool.html", "index": 67, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Cost of a scan" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Mean seconds per skill", "tooltip": null }, { "name": "Input tokens", "tooltip": null }, { "name": "Output tokens", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "0.017", "type": "number", "value": 0.017, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "0.325", "type": "number", "value": 0.325, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "2.400", "type": "number", "value": 2.4, "col": "Mean seconds per skill" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Input tokens" }, { "raw": "not instrumented", "type": "null", "value": null, "col": "Output tokens" } ] ], "kind": "grid" }, { "source": "cross-tool.html#68", "page": "cross-tool.html", "index": 68, "headings": { "h1": "NVIDIA SkillSpector vs Cisco skill-scanner", "h2": "Static only, no model", "h3": "HarmfulSkillBench", "h4": "Integrity" }, "before_text": "", "after_text": "", "columns": [ { "name": "Tool", "tooltip": null }, { "name": "Rows", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Capability degraded", "tooltip": null }, { "name": "Reported partial", "tooltip": null } ], "rows": [ [ { "raw": "Cisco skill-scanner (core rules)", "type": "text", "value": "Cisco skill-scanner (core rules)", "col": "Tool" }, { "raw": "200", "type": "number", "value": 200, "col": "Rows" }, { "raw": "200", "type": "number", "value": 200, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "19", "type": "number", "value": 19, "col": "Reported partial" } ], [ { "raw": "Cisco skill-scanner (all rule packs)", "type": "text", "value": "Cisco skill-scanner (all rule packs)", "col": "Tool" }, { "raw": "200", "type": "number", "value": 200, "col": "Rows" }, { "raw": "200", "type": "number", "value": 200, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "0", "type": "number", "value": 0, "col": "Capability degraded" }, { "raw": "19", "type": "number", "value": 19, "col": "Reported partial" } ], [ { "raw": "NVIDIA SkillSpector", "type": "text", "value": "NVIDIA SkillSpector", "col": "Tool" }, { "raw": "200", "type": "number", "value": 200, "col": "Rows" }, { "raw": "197", "type": "number", "value": 197, "col": "Usable" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" }, { "raw": "3", "type": "number", "value": 3, "col": "Capability degraded" }, { "raw": "160", "type": "number", "value": 160, "col": "Reported partial" } ] ], "kind": "grid" }, { "source": "improvements.html#1", "page": "improvements.html", "index": 1, "headings": { "h1": "Improving the scanner", "h2": "Topline: the complete source-disjoint split", "h3": "Gemma 4 26B on the full split: three and five specialized passes" }, "before_text": "Corpus: msb-source-disjoint · 545 benign, 839 malicious · judge model bedrock-mantle/google.gemma-4-26b-a4b Hypothesis. Does the gain transfer to Gemma 4 26B, and do two further focuses drawn from the records SkillSpector catches and we miss convert them?", "after_text": "Five passes, specialized prompts: 71.0% F1 at 85.2% precision and 60.9% recall, with a 16.3% false-positive rate.", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "Passes", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "Three passes, specialized prompts", "type": "text", "value": "Three passes, specialized prompts", "col": "Arm" }, { "raw": "3", "type": "number", "value": 3, "col": "Passes" }, { "raw": "68.29%", "type": "percent", "value": 0.6829, "col": "F1" }, { "raw": "85.48%", "type": "percent", "value": 0.8548, "col": "Precision" }, { "raw": "56.85%", "type": "percent", "value": 0.5685, "col": "Recall" }, { "raw": "14.86%", "type": "percent", "value": 0.1486, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "Five passes, specialized prompts", "type": "text", "value": "Five passes, specialized prompts", "col": "Arm" }, { "raw": "5", "type": "number", "value": 5, "col": "Passes" }, { "raw": "71.02%", "type": "percent", "value": 0.7102, "col": "F1" }, { "raw": "85.17%", "type": "percent", "value": 0.8517, "col": "Precision" }, { "raw": "60.91%", "type": "percent", "value": 0.6091, "col": "Recall" }, { "raw": "16.33%", "type": "percent", "value": 0.1633, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ] ], "kind": "grid" }, { "source": "improvements.html#2", "page": "improvements.html", "index": 2, "headings": { "h1": "Improving the scanner", "h2": "Topline: the complete source-disjoint split", "h3": "Claude Haiku 4.5 on the full split: three specialized passes" }, "before_text": "Corpus: msb-source-disjoint · 545 benign, 839 malicious · judge model bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 Hypothesis. Does the arm comparison measured on the 400-record subsample hold on the full 1,384-record split with the same model.", "after_text": "Three passes, specialized prompts: 81.4% F1 at 86.9% precision and 76.6% recall, with a 17.8% false-positive rate.", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "Passes", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "Three passes, specialized prompts", "type": "text", "value": "Three passes, specialized prompts", "col": "Arm" }, { "raw": "3", "type": "number", "value": 3, "col": "Passes" }, { "raw": "81.44%", "type": "percent", "value": 0.8144, "col": "F1" }, { "raw": "86.89%", "type": "percent", "value": 0.8689, "col": "Precision" }, { "raw": "76.64%", "type": "percent", "value": 0.7664, "col": "Recall" }, { "raw": "17.80%", "type": "percent", "value": 0.178, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ] ], "kind": "grid" }, { "source": "improvements.html#3", "page": "improvements.html", "index": 3, "headings": { "h1": "Improving the scanner", "h2": "Arm comparison on a subsample of the same split", "h3": "Arm comparison with the sampling control (400-record subsample)" }, "before_text": "Corpus: msb-source-disjoint · 150 benign, 250 malicious · judge model bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0 Hypothesis. The recall gap comes from running one generic judge where SkillSpector runs four specialized semantic analyzers, and a same-prompt control separates that from simple sampling variance.", "after_text": "The total gain is +10.8 F1 points. The control arm runs the shipped prompt three times and unions the findings, which accounts for +7.0 of it: one pass simply misses findings a second pass returns, so part of the gain is sampling rather than prompt design. Specialization adds the remaining +3.8 points at equal call count. Recall moves from 62.4% to 80.4% while precision holds at 85.7% against 85.9%; the false-positive rate rises from 17.3% to 22.0%, so it is not free. Without the control this would have been reported as a prompt-design result, and most of it is not.", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "Passes", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "F1 change", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Errors", "tooltip": null } ], "rows": [ [ { "raw": "One pass, shipped prompt", "type": "text", "value": "One pass, shipped prompt", "col": "Arm" }, { "raw": "1", "type": "number", "value": 1, "col": "Passes" }, { "raw": "72.22%", "type": "percent", "value": 0.7222, "col": "F1" }, { "raw": "—", "type": "null", "value": null, "col": "F1 change" }, { "raw": "85.71%", "type": "percent", "value": 0.8571, "col": "Precision" }, { "raw": "62.40%", "type": "percent", "value": 0.624, "col": "Recall" }, { "raw": "17.33%", "type": "percent", "value": 0.1733, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "Three passes, same prompt (control)", "type": "text", "value": "Three passes, same prompt (control)", "col": "Arm" }, { "raw": "3", "type": "number", "value": 3, "col": "Passes" }, { "raw": "79.23%", "type": "percent", "value": 0.7923, "col": "F1" }, { "raw": "+7.0", "type": "number", "value": 7.0, "col": "F1 change" }, { "raw": "85.25%", "type": "percent", "value": 0.8525, "col": "Precision" }, { "raw": "74.00%", "type": "percent", "value": 0.74, "col": "Recall" }, { "raw": "21.33%", "type": "percent", "value": 0.2133, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ], [ { "raw": "Three passes, specialized prompts", "type": "text", "value": "Three passes, specialized prompts", "col": "Arm" }, { "raw": "3", "type": "number", "value": 3, "col": "Passes" }, { "raw": "83.06%", "type": "percent", "value": 0.8306, "col": "F1" }, { "raw": "+10.8", "type": "number", "value": 10.8, "col": "F1 change" }, { "raw": "85.90%", "type": "percent", "value": 0.859, "col": "Precision" }, { "raw": "80.40%", "type": "percent", "value": 0.804, "col": "Recall" }, { "raw": "22.00%", "type": "percent", "value": 0.22, "col": "FPR" }, { "raw": "0", "type": "number", "value": 0, "col": "Errors" } ] ], "kind": "grid" }, { "source": "improvements.html#4", "page": "improvements.html", "index": 4, "headings": { "h1": "Improving the scanner", "h2": "Skills the judge never read still pass the gate" }, "before_text": "Records the judge could not read, across the 1,384-record source-disjoint split. The failure is emitted as an INFO finding, and INFO does not gate.", "after_text": "When a skill exceeds the context budget, or the request fails, the scanner emits that as an INFO finding. INFO does not gate, so the skill passes. Most of the affected records here are malicious, which makes this a fail-open rather than a reporting nicety: a skill can evade the semantic stage by being large enough not to fit. The fix is not a severity change, because raising it would put ordinary large skills through the same gate. Un-analysed content needs to be reported as a coverage gap the caller can choose to gate on. SkillSpector does exactly that through its analysis_completeness record and --fail-on-incomplete flag; this scanner has the same information and currently reports it in a form that passes.", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "Unread", "tooltip": null }, { "name": "Share", "tooltip": null }, { "name": "Malicious", "tooltip": null }, { "name": "Harmless", "tooltip": null }, { "name": "Passed the gate unread", "tooltip": null } ], "rows": [ [ { "raw": "Three specialized passes, Haiku 4.5", "type": "text", "value": "Three specialized passes, Haiku 4.5", "col": "Arm" }, { "raw": "92", "type": "number", "value": 92, "col": "Unread" }, { "raw": "6.60%", "type": "percent", "value": 0.066, "col": "Share" }, { "raw": "76", "type": "number", "value": 76, "col": "Malicious" }, { "raw": "16", "type": "number", "value": 16, "col": "Harmless" }, { "raw": "51", "type": "number", "value": 51, "col": "Passed the gate unread" } ], [ { "raw": "Five specialized passes, Gemma 4 26B", "type": "text", "value": "Five specialized passes, Gemma 4 26B", "col": "Arm" }, { "raw": "51", "type": "number", "value": 51, "col": "Unread" }, { "raw": "3.70%", "type": "percent", "value": 0.037, "col": "Share" }, { "raw": "32", "type": "number", "value": 32, "col": "Malicious" }, { "raw": "19", "type": "number", "value": 19, "col": "Harmless" }, { "raw": "47", "type": "number", "value": 47, "col": "Passed the gate unread" } ] ], "kind": "grid" }, { "source": "improvements.html#5", "page": "improvements.html", "index": 5, "headings": { "h1": "Improving the scanner", "h2": "The meta-analyzer makes things worse, once it works" }, "before_text": "Strided one-in-five sample of msb-source-disjoint, Gemma 4 26B, detection at MEDIUM or above. The meta-analyzer can only demote. Reaching this measurement required fixing a wiring bug that had made it a silent no-op on this route.", "after_text": "Judge only Judge plus meta-analyzer The stage applied on 219 of 274 records and altered findings on 64, trading 16.4 points of recall for 0.3 points of false-positive rate. It removes far more real detections than false ones, so off remains the right default. Getting to that number required fixing a bug first. On this route the meta path built its request handler without supplying a schema, so the handler loaded its default: the analyzer's schema. Meta requests were constrained to the wrong shape, the model returned findings and a verdict instead of the meta delta, every batch failed contract validation, and each was silently retained unchanged. Meta reported that it ran while changing nothing, which reads exactly like meta agreeing with every finding. Any earlier result that enabled meta on this route measured a no-op.", "columns": [ { "name": "Arm", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null } ], "rows": [ [ { "raw": "Judge only", "type": "text", "value": "Judge only", "col": "Arm" }, { "raw": "61.90%", "type": "percent", "value": 0.619, "col": "F1" }, { "raw": "83.90%", "type": "percent", "value": 0.839, "col": "Precision" }, { "raw": "49.10%", "type": "percent", "value": 0.491, "col": "Recall" }, { "raw": "15.00%", "type": "percent", "value": 0.15, "col": "FPR" } ], [ { "raw": "Judge plus meta-analyzer", "type": "text", "value": "Judge plus meta-analyzer", "col": "Arm" }, { "raw": "46.00%", "type": "percent", "value": 0.46, "col": "F1" }, { "raw": "77.10%", "type": "percent", "value": 0.771, "col": "Precision" }, { "raw": "32.70%", "type": "percent", "value": 0.327, "col": "Recall" }, { "raw": "14.70%", "type": "percent", "value": 0.147, "col": "FPR" } ] ], "kind": "grid" }, { "source": "improvements.html#6", "page": "improvements.html", "index": 6, "headings": { "h1": "Improving the scanner", "h2": "What the extra passes cost" }, "before_text": "120 source-disjoint records, Gemma 4 26B. Five passes cost 5.1x the input tokens of one, which is the expected ratio.", "after_text": "", "columns": [ { "name": "Configuration", "tooltip": null }, { "name": "Input tokens per skill", "tooltip": null }, { "name": "Output tokens per skill", "tooltip": null } ], "rows": [ [ { "raw": "Single judge pass", "type": "text", "value": "Single judge pass", "col": "Configuration" }, { "raw": "4,838", "type": "number", "value": 4838, "col": "Input tokens per skill" }, { "raw": "300", "type": "number", "value": 300, "col": "Output tokens per skill" } ], [ { "raw": "Five specialized passes", "type": "text", "value": "Five specialized passes", "col": "Configuration" }, { "raw": "24,703", "type": "number", "value": 24703, "col": "Input tokens per skill" }, { "raw": "1,548", "type": "number", "value": 1548, "col": "Output tokens per skill" } ] ], "kind": "grid" }, { "source": "improvements.html#7", "page": "improvements.html", "index": 7, "headings": { "h1": "Improving the scanner", "h2": "Does the gain generalise across corpora?" }, "before_text": "The same change — five specialized judge passes against one — measured per corpus on Gemma 4 26B, detection at MEDIUM or above. Populations differ, so read each row against its own baseline rather than across rows.", "after_text": "one pass five specialized passes The change helps on every corpus measured and never costs precision, but the size of the gain tracks how much headroom the single pass left. On source-disjoint, where one pass reached only 49.0% recall, decomposition adds 9.8 F1 points. On the balanced subset, where one pass already reached 80.0%, it adds 1.7. On OpenSkillRisk, at 86.6%, it adds 0.5. So the honest reading is narrower than the headline: decomposition recovers recall the single pass was missing rather than raising a ceiling. Quoting the source-disjoint figure as the improvement would overstate what it does on corpora the scanner already handles well. HarmfulSkillBench is the exception that matters: there recall falls , 60.0% to 57.0%. The change is not uniformly beneficial, and that corpus is the one where the risk is harmful content rather than a technical capability, which is the kind of judgement the added focuses were not written for. Its dataset terms permit only recall, so no F1 or false-positive rate is quoted for it.", "columns": [ { "name": "Corpus (population)", "tooltip": null }, { "name": "F1 before", "tooltip": null }, { "name": "F1 after", "tooltip": null }, { "name": "Recall before", "tooltip": null }, { "name": "Recall after", "tooltip": null }, { "name": "FPR before", "tooltip": null }, { "name": "FPR after", "tooltip": null } ], "rows": [ [ { "raw": "msb-source-disjoint (839 bad / 545 harmless)", "type": "text", "value": "msb-source-disjoint (839 bad / 545 harmless)", "col": "Corpus (population)" }, { "raw": "61.20%", "type": "percent", "value": 0.612, "col": "F1 before" }, { "raw": "71.02%", "type": "percent", "value": 0.7102, "col": "F1 after" }, { "raw": "49.00%", "type": "percent", "value": 0.49, "col": "Recall before" }, { "raw": "60.91%", "type": "percent", "value": 0.6091, "col": "Recall after" }, { "raw": "17.60%", "type": "percent", "value": 0.176, "col": "FPR before" }, { "raw": "16.33%", "type": "percent", "value": 0.1633, "col": "FPR after" } ], [ { "raw": "msb-balanced-800 (400 bad / 400 harmless)", "type": "text", "value": "msb-balanced-800 (400 bad / 400 harmless)", "col": "Corpus (population)" }, { "raw": "77.76%", "type": "percent", "value": 0.7776, "col": "F1 before" }, { "raw": "79.53%", "type": "percent", "value": 0.7953, "col": "F1 after" }, { "raw": "80.00%", "type": "percent", "value": 0.8, "col": "Recall before" }, { "raw": "84.00%", "type": "percent", "value": 0.84, "col": "Recall after" }, { "raw": "25.75%", "type": "percent", "value": 0.2575, "col": "FPR before" }, { "raw": "27.25%", "type": "percent", "value": 0.2725, "col": "FPR after" } ], [ { "raw": "openskillrisk-corpus (374 bad / 0 harmless)", "type": "text", "value": "openskillrisk-corpus (374 bad / 0 harmless)", "col": "Corpus (population)" }, { "raw": "92.84%", "type": "percent", "value": 0.9284, "col": "F1 before" }, { "raw": "93.30%", "type": "percent", "value": 0.933, "col": "F1 after" }, { "raw": "86.63%", "type": "percent", "value": 0.8663, "col": "Recall before" }, { "raw": "87.43%", "type": "percent", "value": 0.8743, "col": "Recall after" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "FPR before" }, { "raw": "0.00%", "type": "percent", "value": 0.0, "col": "FPR after" } ], [ { "raw": "harmfulskillbench-corpus (200 bad / 0 harmless)", "type": "text", "value": "harmfulskillbench-corpus (200 bad / 0 harmless)", "col": "Corpus (population)" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1 before" }, { "raw": "not permitted", "type": "null", "value": null, "col": "F1 after" }, { "raw": "60.00%", "type": "percent", "value": 0.6, "col": "Recall before" }, { "raw": "57.00%", "type": "percent", "value": 0.57, "col": "Recall after" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR before" }, { "raw": "no harmless class", "type": "null", "value": null, "col": "FPR after" } ] ], "kind": "grid" }, { "source": "improvements.html#8", "page": "improvements.html", "index": 8, "headings": { "h1": "Improving the scanner", "h2": "Flag rate on real published skills" }, "before_text": "12,500 published skills sampled from abersbail/ai-skill-md-dataset-500 (MIT) and FayeZC/SkillMD-138K (CC-BY-4.0). Unlabelled, so these are flag rates and an upper bound on the false-positive rate. Static analysis only. This scanner is measured on all 12,498 usable records; the SkillSpector row is from the 4,379 it was run on.", "after_text": "skill-scanner, shipped static rules (n=12,498) SkillSpector, all static analyzers (n=4,379) On the population users actually scan, the shipped rules are roughly seven times quieter at MEDIUM or above: 3.76% against 26.42%. More than a quarter of real published skills reach MEDIUM or above under SkillSpector's static analyzers. That is the practical counterpart to the precision lead visible on the labelled corpora. A larger sample of the same two sources tightens that interval; it does not test it. A third population does. On 200,000 skills from mvaccargiu/gitskills , which indexes skills found in public git repositories, the MEDIUM+ rate is 2.228% with a 95% interval of 2.164% to 2.293%, and HIGH+ is 0.522% against 2.46%. The MEDIUM+ intervals do not overlap , so the two populations genuinely differ and 3.76% is not a general real-world false-positive rate. The direction is favourable, but the figure has to be quoted with its population. Two caveats cut against treating the lower number as the better one: SKILL_LOAD_FALLBACK_USED fires on 16.89% of those records, so the dataset indexes candidate Markdown files rather than only well-formed skills and differs in kind as well as origin, and MANIFEST_MISSING_LICENSE fires on 79.77%, which is what puts its INFO+ rate at 97%. That sample also has a populated LOW tier, 1.66%, where these corpora have none. The estimate needed the larger sample to settle. At 1,100 records the MEDIUM+ rate read 2.00%, at 4,500 it read 3.40%, and at 12,498 it is 3.76% with a 95% interval of 3.44% to 4.11%. The first sample understated it by nearly half, which is why the interval is quoted rather than the point estimate alone. The INFO row changed a decision. Nearly every real skill receives an INFO finding, and nothing lands at LOW on these corpora, so severity is effectively bimodal on them — though the gitskills sample below shows that is corpus-specific rather than structural. On MaliciousSkillBench, moving the gate down to INFO looks attractive and raises F1 from 81.4% to 83.2%. On real skills the same change would flag 88.2% of everything scanned, so it was rejected. A corpus of labelled contrast pairs could not have shown that.", "columns": [ { "name": "Configuration", "tooltip": null }, { "name": "CRITICAL", "tooltip": null }, { "name": "HIGH+", "tooltip": null }, { "name": "MEDIUM+", "tooltip": null }, { "name": "INFO+", "tooltip": null } ], "rows": [ [ { "raw": "skill-scanner, shipped static rules (n=12,498)", "type": "text", "value": "skill-scanner, shipped static rules (n=12,498)", "col": "Configuration" }, { "raw": "0.42%", "type": "percent", "value": 0.0042, "col": "CRITICAL" }, { "raw": "2.46%", "type": "percent", "value": 0.0246, "col": "HIGH+" }, { "raw": "3.76%", "type": "percent", "value": 0.0376, "col": "MEDIUM+" }, { "raw": "88.21%", "type": "percent", "value": 0.8821, "col": "INFO+" } ], [ { "raw": "SkillSpector, all static analyzers (n=4,379)", "type": "text", "value": "SkillSpector, all static analyzers (n=4,379)", "col": "Configuration" }, { "raw": "0.11%", "type": "percent", "value": 0.0011, "col": "CRITICAL" }, { "raw": "12.90%", "type": "percent", "value": 0.129, "col": "HIGH+" }, { "raw": "26.42%", "type": "percent", "value": 0.2642, "col": "MEDIUM+" }, { "raw": "26.76%", "type": "percent", "value": 0.2676, "col": "INFO+" } ] ], "kind": "grid" }, { "source": "improvements.html#9", "page": "improvements.html", "index": 9, "headings": { "h1": "Improving the scanner", "h2": "The gain depends on the model" }, "before_text": "The same three-pass change measured on both models, same corpus and population.", "after_text": "Decomposition helps on both models and costs precision on neither, but the size of the gain is not portable: Haiku 4.5 gains 22.7 F1 points and Gemma 4 26B gains 7.1 on the same corpus, population and arms. On Gemma 4 the false-positive rate actually falls, 17.6% to 14.9%, and precision rises from 81.5% to 85.5%. The consequence for reading this page is that the headline figure belongs to a model. Our Haiku arm at 81.4% sits above SkillSpector's 72.9% on Haiku; our Gemma 4 arm at 68.3% sits below its 72.1% on Gemma 4. Quoting one without the model would be misleading.", "columns": [ { "name": "Judge model", "tooltip": null }, { "name": "F1 before", "tooltip": null }, { "name": "F1 after", "tooltip": null }, { "name": "Recall before", "tooltip": null }, { "name": "Recall after", "tooltip": null }, { "name": "FPR before", "tooltip": null }, { "name": "FPR after", "tooltip": null } ], "rows": [ [ { "raw": "Claude Haiku 4.5", "type": "text", "value": "Claude Haiku 4.5", "col": "Judge model" }, { "raw": "58.70%", "type": "percent", "value": 0.587, "col": "F1 before" }, { "raw": "81.40%", "type": "percent", "value": 0.814, "col": "F1 after" }, { "raw": "45.80%", "type": "percent", "value": 0.458, "col": "Recall before" }, { "raw": "76.60%", "type": "percent", "value": 0.766, "col": "Recall after" }, { "raw": "15.60%", "type": "percent", "value": 0.156, "col": "FPR before" }, { "raw": "17.80%", "type": "percent", "value": 0.178, "col": "FPR after" } ], [ { "raw": "Gemma 4 26B", "type": "text", "value": "Gemma 4 26B", "col": "Judge model" }, { "raw": "61.20%", "type": "percent", "value": 0.612, "col": "F1 before" }, { "raw": "68.30%", "type": "percent", "value": 0.683, "col": "F1 after" }, { "raw": "49.00%", "type": "percent", "value": 0.49, "col": "Recall before" }, { "raw": "56.90%", "type": "percent", "value": 0.569, "col": "Recall after" }, { "raw": "17.60%", "type": "percent", "value": 0.176, "col": "FPR before" }, { "raw": "14.90%", "type": "percent", "value": 0.149, "col": "FPR after" } ] ], "kind": "grid" }, { "source": "improvements.html#10", "page": "improvements.html", "index": 10, "headings": { "h1": "Improving the scanner", "h2": "Where that leaves us against SkillSpector" }, "before_text": "Corpus: msb-source-disjoint · Same corpus and threshold as the head-to-head page, so these are comparable.", "after_text": "skill-scanner, five specialized passes skill-scanner, shipped single-pass judge (Gemma 4 26B) skill-scanner, five specialized passes (Gemma 4 26B) SkillSpector, all analyzers (Gemma 4 26B) Decomposing the judge moves F1 from 58.7% to 81.4% on this corpus, and recall from 45.8% to 76.6%, while the false-positive rate stays close to where it was: 15.6% before, 17.8% after. That is the change that matters, because the recall deficit was the whole of the gap to SkillSpector and the precision lead was the thing not to spend closing it. Read the comparison carefully. SkillSpector reaches 84.6% recall here against our 76.6%, so it still finds more. It does so at a 73.6% false-positive rate against our 17.8%, which is why the F1 ordering reverses. Neither number alone describes the tools.", "columns": [ { "name": "Configuration", "tooltip": null }, { "name": "F1", "tooltip": null }, { "name": "Precision", "tooltip": null }, { "name": "Recall", "tooltip": null }, { "name": "FPR", "tooltip": null } ], "rows": [ [ { "raw": "skill-scanner, three specialized judge passes", "type": "text", "value": "skill-scanner, three specialized judge passes", "col": "Configuration" }, { "raw": "81.44%", "type": "percent", "value": 0.8144, "col": "F1" }, { "raw": "86.89%", "type": "percent", "value": 0.8689, "col": "Precision" }, { "raw": "76.64%", "type": "percent", "value": 0.7664, "col": "Recall" }, { "raw": "17.80%", "type": "percent", "value": 0.178, "col": "FPR" } ], [ { "raw": "skill-scanner, shipped single-pass judge (Gemma 4 26B)", "type": "text", "value": "skill-scanner, shipped single-pass judge (Gemma 4 26B)", "col": "Configuration" }, { "raw": "61.20%", "type": "percent", "value": 0.612, "col": "F1" }, { "raw": "81.50%", "type": "percent", "value": 0.815, "col": "Precision" }, { "raw": "49.00%", "type": "percent", "value": 0.49, "col": "Recall" }, { "raw": "17.60%", "type": "percent", "value": 0.176, "col": "FPR" } ], [ { "raw": "skill-scanner, five specialized passes (Gemma 4 26B)", "type": "text", "value": "skill-scanner, five specialized passes (Gemma 4 26B)", "col": "Configuration" }, { "raw": "71.00%", "type": "percent", "value": 0.71, "col": "F1" }, { "raw": "85.20%", "type": "percent", "value": 0.852, "col": "Precision" }, { "raw": "60.90%", "type": "percent", "value": 0.609, "col": "Recall" }, { "raw": "16.30%", "type": "percent", "value": 0.163, "col": "FPR" } ], [ { "raw": "SkillSpector, all analyzers (Gemma 4 26B)", "type": "text", "value": "SkillSpector, all analyzers (Gemma 4 26B)", "col": "Configuration" }, { "raw": "72.10%", "type": "percent", "value": 0.721, "col": "F1" }, { "raw": "65.70%", "type": "percent", "value": 0.657, "col": "Precision" }, { "raw": "80.00%", "type": "percent", "value": 0.8, "col": "Recall" }, { "raw": "66.50%", "type": "percent", "value": 0.665, "col": "FPR" } ] ], "kind": "grid" }, { "source": "experiments.html#1", "page": "experiments.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "What fraction of deterministic findings are actually real?" }, "before_text": "Two adjudicators from different vendors differ by roughly fivefold with non-overlapping intervals, and chance-corrected agreement is near zero, so no single model can serve as the arbiter.", "after_text": "The corpus is unlabeled real-world content, so this estimates a rate rather than establishing truth. E3", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Primary judged real", "type": "text", "value": "Primary judged real", "col": "key" }, { "raw": "3.7%", "type": "percent", "value": 0.037, "col": "value" } ], [ { "raw": "Independent judged real", "type": "text", "value": "Independent judged real", "col": "key" }, { "raw": "17.1%", "type": "percent", "value": 0.171, "col": "value" } ], [ { "raw": "Raw agreement", "type": "text", "value": "Raw agreement", "col": "key" }, { "raw": "84.1%", "type": "percent", "value": 0.841, "col": "value" } ], [ { "raw": "Chance-corrected agreement", "type": "text", "value": "Chance-corrected agreement", "col": "key" }, { "raw": "0.186", "type": "number", "value": 0.186, "col": "value" } ], [ { "raw": "Findings adjudicated", "type": "text", "value": "Findings adjudicated", "col": "key" }, { "raw": "82", "type": "number", "value": 82, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#2", "page": "experiments.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Does one batched request per skill match one request per finding?" }, "before_text": "The batched form reproduces the aggregate rate at a third of the requests, but agrees with the per-finding arm no better than chance on which findings are real. Sound for an estimate, unsound for a per-finding suppression decision.", "after_text": "E4", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Requests", "type": "text", "value": "Requests", "col": "key" }, { "raw": "30 against 82", "type": "text", "value": "30 against 82", "col": "value" } ], [ { "raw": "Fewer requests", "type": "text", "value": "Fewer requests", "col": "key" }, { "raw": "63.4%", "type": "percent", "value": 0.634, "col": "value" } ], [ { "raw": "Batched real rate", "type": "text", "value": "Batched real rate", "col": "key" }, { "raw": "3.7%", "type": "percent", "value": 0.037, "col": "value" } ], [ { "raw": "Agreement with per-finding", "type": "text", "value": "Agreement with per-finding", "col": "key" }, { "raw": "-0.038", "type": "number", "value": -0.038, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#3", "page": "experiments.html", "index": 3, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Which links in the cascade earn their cost?" }, "before_text": "The full chain of rules, System One model and judge scores highest on both lenses, but the judge buys its recall with an order-of-magnitude rise in false positives. Stopping after the System One model retains almost all of the quality at a fraction of the false-positive rate, which is the better trade when most content is legitimate. The System One model also absorbs some of the judge's calls.", "after_text": "Scored on 400 malicious and 400 benign packages drawn from the whole snapshot, so these are not directly comparable to the published test-partition figures. E7", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Rules alone, any-intervention F1", "type": "text", "value": "Rules alone, any-intervention F1", "col": "key" }, { "raw": "48.9%", "type": "percent", "value": 0.489, "col": "value" } ], [ { "raw": "Rules then System One model, F1", "type": "text", "value": "Rules then System One model, F1", "col": "key" }, { "raw": "77.1%", "type": "percent", "value": 0.771, "col": "value" } ], [ { "raw": "Full three-tier chain, F1", "type": "text", "value": "Full three-tier chain, F1", "col": "key" }, { "raw": "81.5%", "type": "percent", "value": 0.815, "col": "value" } ], [ { "raw": "Rules then System One model, false-positive rate", "type": "text", "value": "Rules then System One model, false-positive rate", "col": "key" }, { "raw": "2.0%", "type": "percent", "value": 0.02, "col": "value" } ], [ { "raw": "Full three-tier chain, false-positive rate", "type": "text", "value": "Full three-tier chain, false-positive rate", "col": "key" }, { "raw": "22.2%", "type": "percent", "value": 0.222, "col": "value" } ], [ { "raw": "Quality retained by stopping early", "type": "text", "value": "Quality retained by stopping early", "col": "key" }, { "raw": "95%", "type": "percent", "value": 0.95, "col": "value" } ], [ { "raw": "False-positive rate retained", "type": "text", "value": "False-positive rate retained", "col": "key" }, { "raw": "9%", "type": "percent", "value": 0.09, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#4", "page": "experiments.html", "index": 4, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Can false positives be removed by gating on the model's own confidence?" }, "before_text": "No. The disposition format already produces no false positives, so there is nothing to gate, and on the eight-probe format no confidence floor removes a false positive without also losing a true detection. The model's confidence does not separate its own mistakes.", "after_text": "E8", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Best zero-loss floor", "type": "text", "value": "Best zero-loss floor", "col": "key" }, { "raw": "0.00", "type": "number", "value": 0.0, "col": "value" } ], [ { "raw": "False positives removed", "type": "text", "value": "False positives removed", "col": "key" }, { "raw": "0", "type": "number", "value": 0, "col": "value" } ], [ { "raw": "Eight-probe false-positive rate", "type": "text", "value": "Eight-probe false-positive rate", "col": "key" }, { "raw": "22.4%", "type": "percent", "value": 0.224, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#5", "page": "experiments.html", "index": 5, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Does telling the model to judge by detected content type beat a generic prompt?" }, "before_text": "Yes. Naming the instruction surface explicitly and directing the model to judge each file by its detected content type rather than its name gave the best any-intervention F1 and the highest recall, at no cost to precision.", "after_text": "The generic variant still wins on the block-only lens, because it blocks where the others confirm. E9", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Any-intervention F1, I0", "type": "text", "value": "Any-intervention F1, I0", "col": "key" }, { "raw": "65.7%", "type": "percent", "value": 0.657, "col": "value" } ], [ { "raw": "Any-intervention F1, I3", "type": "text", "value": "Any-intervention F1, I3", "col": "key" }, { "raw": "64.4%", "type": "percent", "value": 0.644, "col": "value" } ], [ { "raw": "Any-intervention F1, I4_type_aware", "type": "text", "value": "Any-intervention F1, I4_type_aware", "col": "key" }, { "raw": "69.7%", "type": "percent", "value": 0.697, "col": "value" } ], [ { "raw": "Best variant", "type": "text", "value": "Best variant", "col": "key" }, { "raw": "I4_type_aware", "type": "text", "value": "I4_type_aware", "col": "value" } ], [ { "raw": "Its recall", "type": "text", "value": "Its recall", "col": "key" }, { "raw": "53.5%", "type": "percent", "value": 0.535, "col": "value" } ], [ { "raw": "Its false-positive rate", "type": "text", "value": "Its false-positive rate", "col": "key" }, { "raw": "0.0%", "type": "percent", "value": 0.0, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#6", "page": "experiments.html", "index": 6, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Does the self-hosted System One model match the hosted one?" }, "before_text": "Not answered, and the reason is infrastructure rather than the model. The self-hosted model does speak the same protocol, and on a clear case it agreed with the hosted one at lower confidence. But a full sweep returned usable answers for only a sixth of requests, and a single real request later timed out entirely at two minutes. The serving process was also replaced partway through by a different one on the same ports. Every usable answer was 'allow', which does not match the hosted model's behaviour on the same packages and is a further sign the deployment rather than the model was being measured. Publishing any figure from that would describe the machine, so none is published.", "after_text": "Needs an uncontended machine, or coordination with whoever owns the work on that box. Nothing there was stopped or reconfigured to make room. E10", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Requests attempted", "type": "text", "value": "Requests attempted", "col": "key" }, { "raw": "800", "type": "number", "value": 800, "col": "value" } ], [ { "raw": "Usable answers", "type": "text", "value": "Usable answers", "col": "key" }, { "raw": "134 (17%)", "type": "count_share", "value": 134, "share": 0.17, "col": "value" } ], [ { "raw": "Provider failures", "type": "text", "value": "Provider failures", "col": "key" }, { "raw": "666 (83%)", "type": "count_share", "value": 666, "share": 0.83, "col": "value" } ], [ { "raw": "Median latency of a usable answer", "type": "text", "value": "Median latency of a usable answer", "col": "key" }, { "raw": "8.6 seconds", "type": "number", "value": 8.6, "unit": "seconds", "col": "value" } ], [ { "raw": "A single 13 KB request, later", "type": "text", "value": "A single 13 KB request, later", "col": "key" }, { "raw": "timed out at 120 seconds", "type": "text", "value": "timed out at 120 seconds", "col": "value" } ], [ { "raw": "Hosted model on a clear case", "type": "text", "value": "Hosted model on a clear case", "col": "key" }, { "raw": "block at 0.91 confidence", "type": "text", "value": "block at 0.91 confidence", "col": "value" } ], [ { "raw": "Self-hosted model on the same case", "type": "text", "value": "Self-hosted model on the same case", "col": "key" }, { "raw": "block at 0.58 confidence", "type": "text", "value": "block at 0.58 confidence", "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#7", "page": "experiments.html", "index": 7, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Does remembering a decision save work on a re-scan?" }, "before_text": "excluding the line number reuses 6.8 times more decisions than the existing identity, but still loses a substantial minority of decisions, so memory reduces judge calls without eliminating re-adjudication", "after_text": "a five-line header inserted in every text file plus re-indentation of every line, which is more aggressive than typical churn E11", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Existing identity reuse after edits", "type": "text", "value": "Existing identity reuse after edits", "col": "key" }, { "raw": "8.8%", "type": "percent", "value": 0.088, "col": "value" } ], [ { "raw": "Drift-tolerant fingerprint reuse", "type": "text", "value": "Drift-tolerant fingerprint reuse", "col": "key" }, { "raw": "59.5%", "type": "percent", "value": 0.595, "col": "value" } ], [ { "raw": "Identical re-scan reuse", "type": "text", "value": "Identical re-scan reuse", "col": "key" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "value" } ], [ { "raw": "Skills compared", "type": "text", "value": "Skills compared", "col": "key" }, { "raw": "27", "type": "number", "value": 27, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#8", "page": "experiments.html", "index": 8, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Can a dismissal be generalised into a safe suppression rule?" }, "before_text": "Rarely, at this granularity. Most candidates would have silenced real detections: replaying them over a labelled population rejected the large majority for suppressing findings on malicious packages at HIGH or CRITICAL severity. Only a small fraction of benign findings can be suppressed safely by rule and file role, so useful generalisation needs finer typed facts than that.", "after_text": "Shadow-mode proposals only. Nothing is applied; every entry goes to human review. E12", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Candidates considered", "type": "text", "value": "Candidates considered", "col": "key" }, { "raw": "35", "type": "number", "value": 35, "col": "value" } ], [ { "raw": "Accepted", "type": "text", "value": "Accepted", "col": "key" }, { "raw": "1", "type": "number", "value": 1, "col": "value" } ], [ { "raw": "Rejected for touching protected findings", "type": "text", "value": "Rejected for touching protected findings", "col": "key" }, { "raw": "24", "type": "number", "value": 24, "col": "value" } ], [ { "raw": "Benign findings safely suppressible", "type": "text", "value": "Benign findings safely suppressible", "col": "key" }, { "raw": "0.8%", "type": "percent", "value": 0.008, "col": "value" } ], [ { "raw": "Protected findings suppressed", "type": "text", "value": "Protected findings suppressed", "col": "key" }, { "raw": "0", "type": "number", "value": 0, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#9", "page": "experiments.html", "index": 9, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Can an organisation express a policy the generic rules cannot?" }, "before_text": "Yes. Every other domain control is an allowlist that demotes; a denied-destination section was added that promotes instead, matching on the registrable domain at label boundaries so a lookalike domain is not flagged. Asked to generalise one observed host, the model widened it to the wildcard form, named the owning entity, and identified a sibling domain the same entity owns that a naive registrable-domain extraction would have missed.", "after_text": "Inert unless an organisation configures it, so the default scan is unchanged. Proposals go to review, never applied silently. E13", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Observed host", "type": "text", "value": "Observed host", "col": "key" }, { "raw": "api.paloalto.com", "type": "text", "value": "api.paloalto.com", "col": "value" } ], [ { "raw": "Generalised entry", "type": "text", "value": "Generalised entry", "col": "key" }, { "raw": "*.paloalto.com", "type": "text", "value": "*.paloalto.com", "col": "value" } ], [ { "raw": "Sibling domain found", "type": "text", "value": "Sibling domain found", "col": "key" }, { "raw": "paloaltonetworks.com", "type": "text", "value": "paloaltonetworks.com", "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#10", "page": "experiments.html", "index": 10, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Which model decisions deserve to become deterministic rules?" }, "before_text": "A large share of what the model confidently blocks has no matching rule, so the same judgement is re-derived on every call. On the disposition format every single such case is genuinely malicious, which makes it a clean rule-authoring queue.", "after_text": "Candidates and evidence only. Rules stay human-authored. E14", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Confident blocks considered", "type": "text", "value": "Confident blocks considered", "col": "key" }, { "raw": "80", "type": "number", "value": 80, "col": "value" } ], [ { "raw": "Blocks the rules allowed", "type": "text", "value": "Blocks the rules allowed", "col": "key" }, { "raw": "33", "type": "number", "value": 33, "col": "value" } ], [ { "raw": "Share the rules missed", "type": "text", "value": "Share the rules missed", "col": "key" }, { "raw": "41.2%", "type": "percent", "value": 0.412, "col": "value" } ], [ { "raw": "Malicious share of the queue", "type": "text", "value": "Malicious share of the queue", "col": "key" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#11", "page": "experiments.html", "index": 11, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Can the policy be fitted automatically without losing a detection?" }, "before_text": "Barely. Only one single-rule change both improved F1 and left every package caught on a HIGH or CRITICAL malicious finding still caught, and the gain was within rounding. Meanwhile a large share of the tempting changes would have dropped a real detection, which is exactly the trade a tuner optimising F1 alone would have made.", "after_text": "Proposed diff only. Nothing is applied, and every line carries its measured effect. E15", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Baseline F1", "type": "text", "value": "Baseline F1", "col": "key" }, { "raw": "56.5%", "type": "percent", "value": 0.565, "col": "value" } ], [ { "raw": "Tuned F1", "type": "text", "value": "Tuned F1", "col": "key" }, { "raw": "56.7%", "type": "percent", "value": 0.567, "col": "value" } ], [ { "raw": "Change in F1", "type": "text", "value": "Change in F1", "col": "key" }, { "raw": "+0.1 points", "type": "points", "value": 0.1, "col": "value" } ], [ { "raw": "Changes examined", "type": "text", "value": "Changes examined", "col": "key" }, { "raw": "70", "type": "number", "value": 70, "col": "value" } ], [ { "raw": "Changes accepted", "type": "text", "value": "Changes accepted", "col": "key" }, { "raw": "1", "type": "number", "value": 1, "col": "value" } ], [ { "raw": "Rejected for losing a detection", "type": "text", "value": "Rejected for losing a detection", "col": "key" }, { "raw": "28", "type": "number", "value": 28, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#12", "page": "experiments.html", "index": 12, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "Does a stateful scanner work, and is it safe?" }, "before_text": "It works, and the first version was unsafe. A drift-tolerant fingerprint means one dismissal generalises across the corpus, which is the point of it: a bounded set of model dismissals hid most findings on a re-scan. But unrestricted, that generalisation also silenced dozens of HIGH and CRITICAL detections on malicious packages. Adding a ceiling, so a model dismissal can never suppress a high-severity finding while a human review still can, keeps almost all of the saving and loses none of those detections.", "after_text": "Adjudications are model-made and labelled as such; none is ground truth. E16", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Findings before", "type": "text", "value": "Findings before", "col": "key" }, { "raw": "958", "type": "number", "value": 958, "col": "value" } ], [ { "raw": "Findings after reuse", "type": "text", "value": "Findings after reuse", "col": "key" }, { "raw": "215", "type": "number", "value": 215, "col": "value" } ], [ { "raw": "Decision reuse rate", "type": "text", "value": "Decision reuse rate", "col": "key" }, { "raw": "100%", "type": "percent", "value": 1.0, "col": "value" } ], [ { "raw": "Findings suppressed", "type": "text", "value": "Findings suppressed", "col": "key" }, { "raw": "743", "type": "number", "value": 743, "col": "value" } ], [ { "raw": "High-severity detections lost, before the ceiling", "type": "text", "value": "High-severity detections lost, before the ceiling", "col": "key" }, { "raw": "56", "type": "number", "value": 56, "col": "value" } ], [ { "raw": "High-severity detections lost, after the ceiling", "type": "text", "value": "High-severity detections lost, after the ceiling", "col": "key" }, { "raw": "0", "type": "number", "value": 0, "col": "value" } ], [ { "raw": "Empty store leaves output unchanged", "type": "text", "value": "Empty store leaves output unchanged", "col": "key" }, { "raw": "yes", "type": "text", "value": "yes", "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#13", "page": "experiments.html", "index": 13, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "How far must two judged arms differ before the difference is real?" }, "before_text": "Further than most of the differences worth arguing about. Repeating one configuration over the same packages changed the package verdict on a substantial minority of them, almost always between allowing and asking for review. The deterministic core did not move at all, so the variation is the model rather than the harness. Any gap between two judged arms smaller than this cannot be distinguished from run-to-run noise, which is what makes the meta-judge's apparent effect indistinguishable from nothing.", "after_text": "Temperature zero is not determinism. A difference between two judged arms smaller than the flip rate here cannot be distinguished from run-to-run variation. OpenSkillRisk", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Deterministic core, verdicts changed", "type": "text", "value": "Deterministic core, verdicts changed", "col": "key" }, { "raw": "0/200", "type": "ratio", "value": [ 0, 200 ], "col": "value" } ], [ { "raw": "Core plus judge, verdicts changed", "type": "text", "value": "Core plus judge, verdicts changed", "col": "key" }, { "raw": "28/200", "type": "ratio", "value": [ 28, 200 ], "col": "value" } ], [ { "raw": "Noise floor", "type": "text", "value": "Noise floor", "col": "key" }, { "raw": "14.0%", "type": "percent", "value": 0.14, "col": "value" } ], [ { "raw": "Repeats", "type": "text", "value": "Repeats", "col": "key" }, { "raw": "5", "type": "number", "value": 5, "col": "value" } ], [ { "raw": "Most common change", "type": "text", "value": "Most common change", "col": "key" }, { "raw": "allow to confirm", "type": "text", "value": "allow to confirm", "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#14", "page": "experiments.html", "index": 14, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "What does the judge do on a corpus built to separate clear risk from contextual risk?" }, "before_text": "It closes almost the whole gap on the clear cases and behaves as it should on the ambiguous ones. On skills the corpus authors labelled obviously malicious, the rules caught under a third and the judge caught every one. On skills labelled contextually risky, where the right answer genuinely depends on context, it caught a large majority but not all, which is the discrimination you would want rather than a flat improvement.", "after_text": "Both groups are positive-risk, so this corpus supports a detection rate and not a false-positive rate. Its licence permits exactly that use. HarmfulSkillBench", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Obviously malicious skills", "type": "text", "value": "Obviously malicious skills", "col": "key" }, { "raw": "160", "type": "number", "value": 160, "col": "value" } ], [ { "raw": "Rules alone", "type": "text", "value": "Rules alone", "col": "key" }, { "raw": "30.0%", "type": "percent", "value": 0.3, "col": "value" } ], [ { "raw": "Rules plus judge", "type": "text", "value": "Rules plus judge", "col": "key" }, { "raw": "100.0%", "type": "percent", "value": 1.0, "col": "value" } ], [ { "raw": "Contextually risky skills", "type": "text", "value": "Contextually risky skills", "col": "key" }, { "raw": "214", "type": "number", "value": 214, "col": "value" } ], [ { "raw": "Rules alone, contextual", "type": "text", "value": "Rules alone, contextual", "col": "key" }, { "raw": "32.7%", "type": "percent", "value": 0.327, "col": "value" } ], [ { "raw": "Rules plus judge, contextual", "type": "text", "value": "Rules plus judge, contextual", "col": "key" }, { "raw": "77.1%", "type": "percent", "value": 0.771, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#15", "page": "experiments.html", "index": 15, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Answers so far", "h4": "What does the judge do on a corpus the rules almost entirely miss?" }, "before_text": "It finds most of what the rules cannot see. The deterministic core flagged a few percent of these skills, matching the figure published for it, and adding the judge raised detection more than tenfold. This corpus is the clearest case for the judge earning its cost, because the rules have almost no purchase on it at all.", "after_text": "Its licence permits a positive-risk detection diagnostic only; no F1 or false-positive claim.", "columns": [ { "name": "key", "tooltip": null }, { "name": "value", "tooltip": null } ], "rows": [ [ { "raw": "Skills", "type": "text", "value": "Skills", "col": "key" }, { "raw": "200", "type": "number", "value": 200, "col": "value" } ], [ { "raw": "Rules alone, detection", "type": "text", "value": "Rules alone, detection", "col": "key" }, { "raw": "4.0%", "type": "percent", "value": 0.04, "col": "value" } ], [ { "raw": "Rules plus judge, detection", "type": "text", "value": "Rules plus judge, detection", "col": "key" }, { "raw": "56.0%", "type": "percent", "value": 0.56, "col": "value" } ], [ { "raw": "Rules alone, block rate", "type": "text", "value": "Rules alone, block rate", "col": "key" }, { "raw": "3.5%", "type": "percent", "value": 0.035, "col": "value" } ], [ { "raw": "Rules plus judge, block rate", "type": "text", "value": "Rules plus judge, block rate", "col": "key" }, { "raw": "38.0%", "type": "percent", "value": 0.38, "col": "value" } ] ], "kind": "key_value" }, { "source": "experiments.html#16", "page": "experiments.html", "index": 16, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "E1: what fraction of deterministic findings are actually real?" }, "before_text": "This runs first because every later number is read against it. Findings from real-world skills are stratified by rule and severity, then adjudicated by two models from different vendors. The corpus is unlabeled, so this estimates a rate and reports its own uncertainty rather than claiming ground truth, and agreement between two models bounds trust rather than establishing accuracy. n/a Deterministic findings collected n/a Findings adjudicated n/a Rule and severity strata 0.000 Agreement, chance-corrected", "after_text": "The two adjudicators disagreed on 0 findings. Those form the ranked backlog for human review; no model verdict is treated as truth.", "columns": [ { "name": "Adjudicator", "tooltip": null }, { "name": "Usable", "tooltip": null }, { "name": "Errors", "tooltip": null }, { "name": "Uncertain", "tooltip": null }, { "name": "Judged real", "tooltip": null }, { "name": "95% interval", "tooltip": null } ], "rows": [ [ { "raw": "primary", "type": "text", "value": "primary", "col": "Adjudicator" }, { "raw": "n/a", "type": "null", "value": null, "col": "Usable" }, { "raw": "n/a", "type": "null", "value": null, "col": "Errors" }, { "raw": "n/a", "type": "null", "value": null, "col": "Uncertain" }, { "raw": "n/a", "type": "null", "value": null, "col": "Judged real" }, { "raw": "n/a", "type": "null", "value": null, "col": "95% interval" } ], [ { "raw": "independent", "type": "text", "value": "independent", "col": "Adjudicator" }, { "raw": "n/a", "type": "null", "value": null, "col": "Usable" }, { "raw": "n/a", "type": "null", "value": null, "col": "Errors" }, { "raw": "n/a", "type": "null", "value": null, "col": "Uncertain" }, { "raw": "n/a", "type": "null", "value": null, "col": "Judged real" }, { "raw": "n/a", "type": "null", "value": null, "col": "95% interval" } ] ], "kind": "grid" }, { "source": "experiments.html#17", "page": "experiments.html", "index": 17, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Experiment programme", "h3": "Every experiment and where it stands" }, "before_text": "", "after_text": "The packing, context and question-format experiments are reported on the System One cascade page, because their results are the cascade tables themselves. Metrics, counts and digests only. No sample text, evidence strings, prompts or credentials are published, because the source corpora forbid redistributing content.", "columns": [ { "name": "Experiment", "tooltip": null }, { "name": "Question", "tooltip": null }, { "name": "Status", "tooltip": null } ], "rows": [ [ { "raw": "E1 finding reality", "type": "text", "value": "E1 finding reality", "col": "Experiment" }, { "raw": "What fraction of deterministic findings are actually real?", "type": "text", "value": "What fraction of deterministic findings are actually real?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E2 packing ladder", "type": "text", "value": "E2 packing ladder", "col": "Experiment" }, { "raw": "Whole record, priority ordered, or candidate centric?", "type": "text", "value": "Whole record, priority ordered, or candidate centric?", "col": "Question" }, { "raw": "reported on the cascade page", "type": "text", "value": "reported on the cascade page", "col": "Status" } ], [ { "raw": "E3 all findings in one call", "type": "text", "value": "E3 all findings in one call", "col": "Experiment" }, { "raw": "Is one batched request per skill as good as one per finding?", "type": "text", "value": "Is one batched request per skill as good as one per finding?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E4 cascade composition", "type": "text", "value": "E4 cascade composition", "col": "Experiment" }, { "raw": "Which links in the chain earn their cost?", "type": "text", "value": "Which links in the chain earn their cost?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E5 context ablation", "type": "text", "value": "E5 context ablation", "col": "Experiment" }, { "raw": "What is context actually worth?", "type": "text", "value": "What is context actually worth?", "col": "Question" }, { "raw": "reported on the cascade page", "type": "text", "value": "reported on the cascade page", "col": "Status" } ], [ { "raw": "E6 question format", "type": "text", "value": "E6 question format", "col": "Experiment" }, { "raw": "Disposition and risk, or eight threat probes?", "type": "text", "value": "Disposition and risk, or eight threat probes?", "col": "Question" }, { "raw": "reported on the cascade page", "type": "text", "value": "reported on the cascade page", "col": "Status" } ], [ { "raw": "E7 conservatism calibration", "type": "text", "value": "E7 conservatism calibration", "col": "Experiment" }, { "raw": "Can false positives be removed by gating on confidence?", "type": "text", "value": "Can false positives be removed by gating on confidence?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E8 file-type prompting", "type": "text", "value": "E8 file-type prompting", "col": "Experiment" }, { "raw": "Does judging by detected content type beat a generic prompt?", "type": "text", "value": "Does judging by detected content type beat a generic prompt?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E9 hosted against self-hosted", "type": "text", "value": "E9 hosted against self-hosted", "col": "Experiment" }, { "raw": "Does the self-hosted System One model match the hosted one?", "type": "text", "value": "Does the self-hosted System One model match the hosted one?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E10 statefulness", "type": "text", "value": "E10 statefulness", "col": "Experiment" }, { "raw": "Does a stored decision survive ordinary churn?", "type": "text", "value": "Does a stored decision survive ordinary churn?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E11 dismissal generalisation", "type": "text", "value": "E11 dismissal generalisation", "col": "Experiment" }, { "raw": "Can a dismissal become a safe suppression rule?", "type": "text", "value": "Can a dismissal become a safe suppression rule?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E12 organisation context", "type": "text", "value": "E12 organisation context", "col": "Experiment" }, { "raw": "Can an organisation express a policy the rules cannot?", "type": "text", "value": "Can an organisation express a policy the rules cannot?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E13 rule mining", "type": "text", "value": "E13 rule mining", "col": "Experiment" }, { "raw": "Which model decisions deserve to become rules?", "type": "text", "value": "Which model decisions deserve to become rules?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E14 policy auto-tuning", "type": "text", "value": "E14 policy auto-tuning", "col": "Experiment" }, { "raw": "Can the policy be fitted without losing a detection?", "type": "text", "value": "Can the policy be fitted without losing a detection?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E15 stateful prototype", "type": "text", "value": "E15 stateful prototype", "col": "Experiment" }, { "raw": "Does a stateful scanner work, and is it safe?", "type": "text", "value": "Does a stateful scanner work, and is it safe?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ], [ { "raw": "E16 stability", "type": "text", "value": "E16 stability", "col": "Experiment" }, { "raw": "How far must two arms differ before the difference is real?", "type": "text", "value": "How far must two arms differ before the difference is real?", "col": "Question" }, { "raw": "answered above", "type": "text", "value": "answered above", "col": "Status" } ] ], "kind": "grid" }, { "source": "recommendations.html#1", "page": "recommendations.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "How to configure the scanner", "h3": "Stage by stage" }, "before_text": "", "after_text": "", "columns": [ { "name": "Stage", "tooltip": null }, { "name": "Recommended setting", "tooltip": null }, { "name": "Why", "tooltip": null }, { "name": "Cost per skill", "tooltip": null }, { "name": "What to watch", "tooltip": null } ], "rows": [ [ { "raw": "Deterministic rules", "type": "text", "value": "Deterministic rules", "col": "Stage" }, { "raw": "on by default; --policy low-noise for your own skills", "type": "text", "value": "on by default; --policy low-noise for your own skills", "col": "Recommended setting" }, { "raw": "Free, reproducible and milliseconds per skill. Tuned against every real skill: 4.18% to 2.14% of real skills flagged at MEDIUM+, and the held-out false-positive rate 8.44% to 4.22% with recall unchanged.", "type": "text", "value": "Free, reproducible and milliseconds per skill. Tuned against every real skill: 4.18% to 2.14% of real skills flagged at MEDIUM+, and the held-out false-positive rate 8.44% to 4.22% with recall unchanged.", "col": "Why" }, { "raw": "milliseconds per skill", "type": "text", "value": "milliseconds per skill", "col": "Cost per skill" }, { "raw": "Alone, it catches 8% of held-out malicious skills: a floor for unambiguous patterns, not a judge of intent.", "type": "text", "value": "Alone, it catches 8% of held-out malicious skills: a floor for unambiguous patterns, not a judge of intent.", "col": "What to watch" } ], [ { "raw": "LLM judge", "type": "text", "value": "LLM judge", "col": "Stage" }, { "raw": "SKILL_SCANNER_LLM_MODEL=... skill-scanner scan DIR --use-llm --fail-on-severity high", "type": "text", "value": "SKILL_SCANNER_LLM_MODEL=... skill-scanner scan DIR --use-llm --fail-on-severity high", "col": "Recommended setting" }, { "raw": "The new prompt separates a skill's purpose from misuse: held-out recall 65.9% at 12.5% FPR (F1 75.8%, from 63.7% with the shipped prompt).", "type": "text", "value": "The new prompt separates a skill's purpose from misuse: held-out recall 65.9% at 12.5% FPR (F1 75.8%, from 63.7% with the shipped prompt).", "col": "Why" }, { "raw": "about 3,800 input and 90 output tokens per skill", "type": "text", "value": "about 3,800 input and 90 output tokens per skill", "col": "Cost per skill" }, { "raw": "Flags about 10% of real skills at MEDIUM+: review those, block only at HIGH.", "type": "text", "value": "Flags about 10% of real skills at MEDIUM+: review those, block only at HIGH.", "col": "What to watch" } ], [ { "raw": "LLM caps", "type": "text", "value": "LLM caps", "col": "Stage" }, { "raw": "llm_analysis.low_confidence_max_severity / contextual_risk_max_severity: LOW (on in low-noise / quiet)", "type": "text", "value": "llm_analysis.low_confidence_max_severity / contextual_risk_max_severity: LOW (on in low-noise / quiet)", "col": "Recommended setting" }, { "raw": "Report the judge's low-confidence findings at LOW (real-skill flags 10.5% to 7.6%), or its contextual-risk findings (to 3.0%); quiet applies both.", "type": "text", "value": "Report the judge's low-confidence findings at LOW (real-skill flags 10.5% to 7.6%), or its contextual-risk findings (to 3.0%); quiet applies both.", "col": "Why" }, { "raw": "none", "type": "text", "value": "none", "col": "Cost per skill" }, { "raw": "The contextual cap costs 16 points of held-out recall; use it when the queue, not the attacker, is the constraint.", "type": "text", "value": "The contextual cap costs 16 points of held-out recall; use it when the queue, not the attacker, is the constraint.", "col": "What to watch" } ], [ { "raw": "Verdict repair", "type": "text", "value": "Verdict repair", "col": "Stage" }, { "raw": "on by default; leave it on", "type": "text", "value": "on by default; leave it on", "col": "Recommended setting" }, { "raw": "A SAFE verdict with findings contradicts itself, and the strict path discarded the analysis, almost always on benign skills. Repair escalates to SUSPICIOUS and keeps the findings.", "type": "text", "value": "A SAFE verdict with findings contradicts itself, and the strict path discarded the analysis, almost always on benign skills. Repair escalates to SUSPICIOUS and keeps the findings.", "col": "Why" }, { "raw": "none", "type": "text", "value": "none", "col": "Cost per skill" }, { "raw": "Escalate-only by design; SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=0 restores the strict path.", "type": "text", "value": "Escalate-only by design; SKILL_SCANNER_LLM_REPAIR_INCONSISTENT_VERDICT=0 restores the strict path.", "col": "What to watch" } ], [ { "raw": "Meta-analyzer", "type": "text", "value": "Meta-analyzer", "col": "Stage" }, { "raw": "leave --enable-meta off", "type": "text", "value": "leave --enable-meta off", "col": "Recommended setting" }, { "raw": "Once a schema-wiring defect that made it a no-op was fixed, it cost 16.4 points of recall for 0.3 points of false-positive rate.", "type": "text", "value": "Once a schema-wiring defect that made it a no-op was fixed, it cost 16.4 points of recall for 0.3 points of false-positive rate.", "col": "Why" }, { "raw": "a second model call per skill", "type": "text", "value": "a second model call per skill", "col": "Cost per skill" }, { "raw": "Any earlier result with meta on over the mantle route measured a no-op.", "type": "text", "value": "Any earlier result with meta on over the mantle route measured a no-op.", "col": "What to watch" } ], [ { "raw": "Decomposed judge", "type": "text", "value": "Decomposed judge", "col": "Stage" }, { "raw": "--llm-decompose, for high-stakes reviews only", "type": "text", "value": "--llm-decompose, for high-stakes reviews only", "col": "Recommended setting" }, { "raw": "One pass per focus, findings unioned: held-out recall 49.0% to 60.9% with the earlier prompt.", "type": "text", "value": "One pass per focus, findings unioned: held-out recall 49.0% to 60.9% with the earlier prompt.", "col": "Why" }, { "raw": "three to five times the tokens", "type": "text", "value": "three to five times the tokens", "col": "Cost per skill" }, { "raw": "A control arm attributed about two thirds of the gain to sampling variance; not re-measured with the new prompt.", "type": "text", "value": "A control arm attributed about two thirds of the gain to sampling variance; not re-measured with the new prompt.", "col": "What to watch" } ], [ { "raw": "OpenJev screen", "type": "text", "value": "OpenJev screen", "col": "Stage" }, { "raw": "measured, not shipped; --system-one-endpoint is advisory only", "type": "text", "value": "measured, not shipped; --system-one-endpoint is advisory only", "col": "Recommended setting" }, { "raw": "Selected on train/validation, an eight-probe screen in front of the judge gave 61.3% held-out recall at 2.4% FPR, with 3.1% of real skills reaching the judge.", "type": "text", "value": "Selected on train/validation, an eight-probe screen in front of the judge gave 61.3% held-out recall at 2.4% FPR, with 3.1% of real skills reaching the judge.", "col": "Why" }, { "raw": "a small local model per skill", "type": "text", "value": "a small local model per skill", "col": "Cost per skill" }, { "raw": "Not a product configuration yet: the shipped tier records a probability and cannot change a finding.", "type": "text", "value": "Not a product configuration yet: the shipped tier records a probability and cannot change a finding.", "col": "What to watch" } ] ], "kind": "grid" }, { "source": "recommendations.html#2", "page": "recommendations.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "How to configure the scanner", "h3": "What each configuration catches and flags" }, "before_text": "A scan with the judge on flags a skill when either layer reports MEDIUM or above, so each preset is measured as that union, with its rule demotions and LLM caps applied. Held out: MaliciousSkillBench's frozen test split, used to design nothing. Real skills: 2,000 sampled uniformly, a flag rate that bounds the false-positive rate from above.", "after_text": "", "columns": [ { "name": "Configuration", "tooltip": null }, { "name": "Review at MEDIUM+: recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Real skills", "tooltip": null }, { "name": "Block at HIGH+: recall", "tooltip": null }, { "name": "FPR", "tooltip": null }, { "name": "Real skills", "tooltip": null } ], "rows": [ [ { "raw": "balanced, rules only", "type": "text", "value": "balanced, rules only", "col": "Configuration" }, { "raw": "8.0%", "type": "percent", "value": 0.08, "col": "Review at MEDIUM+: recall" }, { "raw": "4.2%", "type": "percent", "value": 0.042, "col": "FPR" }, { "raw": "2.00%", "type": "percent", "value": 0.02, "col": "Real skills" }, { "raw": "7.7%", "type": "percent", "value": 0.077, "col": "Block at HIGH+: recall" }, { "raw": "4.0%", "type": "percent", "value": 0.04, "col": "FPR" }, { "raw": "1.85%", "type": "percent", "value": 0.0185, "col": "Real skills" } ], [ { "raw": "balanced + judge", "type": "text", "value": "balanced + judge", "col": "Configuration" }, { "raw": "66.7%", "type": "percent", "value": 0.667, "col": "Review at MEDIUM+: recall" }, { "raw": "15.4%", "type": "percent", "value": 0.154, "col": "FPR" }, { "raw": "11.65%", "type": "percent", "value": 0.1165, "col": "Real skills" }, { "raw": "33.7%", "type": "percent", "value": 0.337, "col": "Block at HIGH+: recall" }, { "raw": "6.4%", "type": "percent", "value": 0.064, "col": "FPR" }, { "raw": "3.80%", "type": "percent", "value": 0.038, "col": "Real skills" } ], [ { "raw": "low-noise, rules only", "type": "text", "value": "low-noise, rules only", "col": "Configuration" }, { "raw": "7.9%", "type": "percent", "value": 0.079, "col": "Review at MEDIUM+: recall" }, { "raw": "4.2%", "type": "percent", "value": 0.042, "col": "FPR" }, { "raw": "1.65%", "type": "percent", "value": 0.0165, "col": "Real skills" }, { "raw": "7.7%", "type": "percent", "value": 0.077, "col": "Block at HIGH+: recall" }, { "raw": "4.0%", "type": "percent", "value": 0.04, "col": "FPR" }, { "raw": "1.65%", "type": "percent", "value": 0.0165, "col": "Real skills" } ], [ { "raw": "low-noise + judge", "type": "text", "value": "low-noise + judge", "col": "Configuration" }, { "raw": "63.2%", "type": "percent", "value": 0.632, "col": "Review at MEDIUM+: recall" }, { "raw": "13.4%", "type": "percent", "value": 0.134, "col": "FPR" }, { "raw": "8.65%", "type": "percent", "value": 0.0865, "col": "Real skills" }, { "raw": "33.7%", "type": "percent", "value": 0.337, "col": "Block at HIGH+: recall" }, { "raw": "6.4%", "type": "percent", "value": 0.064, "col": "FPR" }, { "raw": "3.60%", "type": "percent", "value": 0.036, "col": "Real skills" } ], [ { "raw": "quiet, rules only", "type": "text", "value": "quiet, rules only", "col": "Configuration" }, { "raw": "4.2%", "type": "percent", "value": 0.042, "col": "Review at MEDIUM+: recall" }, { "raw": "3.7%", "type": "percent", "value": 0.037, "col": "FPR" }, { "raw": "1.10%", "type": "percent", "value": 0.011, "col": "Real skills" }, { "raw": "3.7%", "type": "percent", "value": 0.037, "col": "Block at HIGH+: recall" }, { "raw": "3.3%", "type": "percent", "value": 0.033, "col": "FPR" }, { "raw": "1.00%", "type": "percent", "value": 0.01, "col": "Real skills" } ], [ { "raw": "quiet + judge", "type": "text", "value": "quiet + judge", "col": "Configuration" }, { "raw": "50.3%", "type": "percent", "value": 0.503, "col": "Review at MEDIUM+: recall" }, { "raw": "7.2%", "type": "percent", "value": 0.072, "col": "FPR" }, { "raw": "3.85%", "type": "percent", "value": 0.0385, "col": "Real skills" }, { "raw": "33.1%", "type": "percent", "value": 0.331, "col": "Block at HIGH+: recall" }, { "raw": "5.5%", "type": "percent", "value": 0.055, "col": "FPR" }, { "raw": "2.90%", "type": "percent", "value": 0.029, "col": "Real skills" } ] ], "kind": "grid" }, { "source": "methodology.html#1", "page": "methodology.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Corpus shape decides how a skill can be packed" }, "before_text": "The benchmark corpora store skills as one or two text fields, not as directories, so they almost always fit a System One model's context whole. Real-world skills do not. Any packing or cost conclusion drawn from the small golden fixtures would be wrong about real input.", "after_text": "", "columns": [ { "name": "Population", "tooltip": null }, { "name": "Median files", "tooltip": null }, { "name": "Median scanner-relevant bytes", "tooltip": null }, { "name": "Fits a 32k-token window whole", "tooltip": null } ], "rows": [ [ { "raw": "MaliciousSkillBench rows", "type": "text", "value": "MaliciousSkillBench rows", "col": "Population" }, { "raw": "1 (single text field)", "type": "text", "value": "1 (single text field)", "col": "Median files" }, { "raw": "2,936", "type": "number", "value": 2936, "col": "Median scanner-relevant bytes" }, { "raw": "essentially all", "type": "text", "value": "essentially all", "col": "Fits a 32k-token window whole" } ], [ { "raw": "ClawHub rows", "type": "text", "value": "ClawHub rows", "col": "Population" }, { "raw": "2 (skill_md + bundle)", "type": "text", "value": "2 (skill_md + bundle)", "col": "Median files" }, { "raw": "24,173", "type": "number", "value": 24173, "col": "Median scanner-relevant bytes" }, { "raw": "most", "type": "text", "value": "most", "col": "Fits a 32k-token window whole" } ], [ { "raw": "Golden fixtures", "type": "text", "value": "Golden fixtures", "col": "Population" }, { "raw": "3", "type": "number", "value": 3, "col": "Median files" }, { "raw": "1,797", "type": "number", "value": 1797, "col": "Median scanner-relevant bytes" }, { "raw": "100%", "type": "percent", "value": 1.0, "col": "Fits a 32k-token window whole" } ], [ { "raw": "Real-world skills", "type": "text", "value": "Real-world skills", "col": "Population" }, { "raw": "80", "type": "number", "value": 80, "col": "Median files" }, { "raw": "576,745", "type": "number", "value": 576745, "col": "Median scanner-relevant bytes" }, { "raw": "16%", "type": "percent", "value": 0.16, "col": "Fits a 32k-token window whole" } ] ], "kind": "grid" }, { "source": "methodology.html#2", "page": "methodology.html", "index": 2, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Judge model, verified rather than assumed" }, "before_text": "", "after_text": "", "columns": [ { "name": "Property", "tooltip": null }, { "name": "What was measured", "tooltip": null } ], "rows": [ [ { "raw": "One model, one route", "type": "text", "value": "One model, one route", "col": "Property" }, { "raw": "google.gemma-4-26b-a4b is reachable only on the Bedrock mantle OpenAI-compatible route. It is absent from list-foundation-models, there is no model-listing endpoint, and no other Gemma 4 size exists, so the model id is a pinned constant.", "type": "text", "value": "google.gemma-4-26b-a4b is reachable only on the Bedrock mantle OpenAI-compatible route. It is absent from list-foundation-models, there is no model-listing endpoint, and no other Gemma 4 size exists, so the model id is a pinned constant.", "col": "What was measured" } ], [ { "raw": "Strict schema works, after one removal", "type": "text", "value": "Strict schema works, after one removal", "col": "Property" }, { "raw": "Schema-constrained JSON succeeds once uniqueItems is stripped; the mantle validator rejects it outright and the request would otherwise degrade silently to loose JSON mode.", "type": "text", "value": "Schema-constrained JSON succeeds once uniqueItems is stripped; the mantle validator rejects it outright and the request would otherwise degrade silently to loose JSON mode.", "col": "What was measured" } ], [ { "raw": "Context is ample", "type": "text", "value": "Context is ample", "col": "Property" }, { "raw": "A 124,989-byte request was accepted at 40,855 prompt tokens, so the scanner's 100,000-character budget fits with headroom.", "type": "text", "value": "A 124,989-byte request was accepted at 40,855 prompt tokens, so the scanner's 100,000-character budget fits with headroom.", "col": "What was measured" } ], [ { "raw": "Prompt caching does not engage", "type": "text", "value": "Prompt caching does not engage", "col": "Property" }, { "raw": "cached_tokens and cache_write_tokens stayed at zero across two byte-identical 4,859-token requests, so the largest assumed cost lever is unavailable on this route.", "type": "text", "value": "cached_tokens and cache_write_tokens stayed at zero across two byte-identical 4,859-token requests, so the largest assumed cost lever is unavailable on this route.", "col": "What was measured" } ], [ { "raw": "Code tokenizes worse than prose", "type": "text", "value": "Code tokenizes worse than prose", "col": "Property" }, { "raw": "Measured at 3.06 bytes per token on repetitive Python, not the 4.0 a naive estimator assumes, which is why budgets are enforced in bytes at a pessimistic 2 bytes per token.", "type": "text", "value": "Measured at 3.06 bytes per token on repetitive Python, not the 4.0 a naive estimator assumes, which is why budgets are enforced in bytes at a pessimistic 2 bytes per token.", "col": "What was measured" } ] ], "kind": "grid" }, { "source": "reproduce.html#1", "page": "reproduce.html", "index": 1, "headings": { "h1": "Cisco AI Defense skill scanner: evaluation results", "h2": "Reproducing these results", "h3": "What you need first" }, "before_text": "", "after_text": "", "columns": [ { "name": "Requirement", "tooltip": null }, { "name": "Detail", "tooltip": null } ], "rows": [ [ { "raw": "The scanner, installed from source", "type": "text", "value": "The scanner, installed from source", "col": "Requirement" }, { "raw": "git clone the repository, create a virtual environment, then `pip install -e .` plus the pinned `huggingface-hub` and `pyarrow`. Building the CEL helper needs Go 1.27.1 or newer.", "type": "text", "value": "git clone the repository, create a virtual environment, then `pip install -e .` plus the pinned `huggingface-hub` and `pyarrow`. Building the CEL helper needs Go 1.27.1 or newer.", "col": "Detail" } ], [ { "raw": "AWS credentials that can reach Bedrock", "type": "text", "value": "AWS credentials that can reach Bedrock", "col": "Requirement" }, { "raw": "The judge model is served only on the Bedrock mantle OpenAI-compatible route and authenticates with SigV4, so the normal credential chain is enough. No API key is involved.", "type": "text", "value": "The judge model is served only on the Bedrock mantle OpenAI-compatible route and authenticates with SigV4, so the normal credential chain is enough. No API key is involved.", "col": "Detail" } ], [ { "raw": "A hosted System One credential, for that arm only", "type": "text", "value": "A hosted System One credential, for that arm only", "col": "Requirement" }, { "raw": "One environment variable, read from wherever you keep secrets. The deterministic and judge results need nothing beyond AWS.", "type": "text", "value": "One environment variable, read from wherever you keep secrets. The deterministic and judge results need nothing beyond AWS.", "col": "Detail" } ] ], "kind": "grid" } ], "svgs": [ { "page": "large-scale.html", "index": 1, "aria_label": "Judge alone against the cascade, held-out half", "headings": { "h1": "Large-scale run on local GPUs", "h2": "A System One screen in front of the judge halves its false-positive rate" }, "duplicates_table": "large-scale.html#2" }, { "page": "improvements.html", "index": 1, "aria_label": "Judge alone against judge plus meta-analyzer", "headings": { "h1": "Improving the scanner", "h2": "The meta-analyzer makes things worse, once it works" }, "duplicates_table": "improvements.html#5" }, { "page": "improvements.html", "index": 2, "aria_label": "Recall before and after, per corpus", "headings": { "h1": "Improving the scanner", "h2": "Does the gain generalise across corpora?" }, "duplicates_table": "improvements.html#7" }, { "page": "improvements.html", "index": 3, "aria_label": "Flag rate on real published skills, by threshold", "headings": { "h1": "Improving the scanner", "h2": "Flag rate on real published skills" }, "duplicates_table": "improvements.html#8" }, { "page": "improvements.html", "index": 4, "aria_label": "msb-source-disjoint: F1, precision, recall and false-positive rate", "headings": { "h1": "Improving the scanner", "h2": "Where that leaves us against SkillSpector" }, "duplicates_table": "improvements.html#10" } ], "counts": { "tables": 134, "svgs": 5 } } }