Judge and meta-judge

Enabling the judge and the meta-judge

Four arms over one population: the deterministic core alone, the core plus the LLM analyzer, the core plus both the analyzer and the meta-judge, and the core plus the meta-judge with the analyzer left off. The core-only arm exists to prove the harness before any judged number is read.

This track is explicitly non-blocking. The release path asserts that the analyzer factory keeps the judge off by default, and that assertion still holds; the judge is enabled per-arm by the evaluation runner alone.

Arm	F1	Precision	Recall	FPR	Meta invoked
`core_only` (baseline)	13.74%	60.75%	7.75%	7.71%	n/a
`core_judge`	37.68%	78.49%	24.79%	10.46%	0.00%
`core_judge_meta`	37.24%	78.24%	24.43%	10.46%	0.00%
`core_meta`	13.74%	60.75%	7.75%	7.71%	0.00%

What enabling the judge actually did

On the same locked population, enabling the LLM analyzer took F1 from 13.74% to 37.68% and signal recall from 8.46% to 85.58%. Precision rose as well, from 60.75% to 78.49%, which is the opposite of what the premise predicted: at a precision of 99% on the development split there was no room to improve, but on this source-disjoint split the rules were missing far more than they were over-flagging. True positives went from 65 to 208 while false positives went from 42 to 57.

The cost is visible on the stricter lens: the share of benign packages carrying any actionable finding roughly doubled, from 8.44% to 18.90%. So the judge is a clear win at the block threshold and a noticeable source of noise below it.

What the meta-judge did

It ran on 0.00% of packages and changed nothing measurable. It suppressed 0 findings and added 0, and F1 moved by -0.44 points against the judge-only arm, which is inside the run-to-run variation the judge itself shows. On the remaining packages it declined to run: 186 had no ambiguous finding and 115 had no findings at all.

Reading that rate correctly took a fix. The first version of this instrumentation keyed on the routing reason text rather than the routing decision, and so reported that meta had never run at all. It had run on most packages. The conclusion is the same either way, but it would have been published for the wrong reason.

LLM judge: every metric, corpus by corpus

Scored on the any-intervention lens, so a finding raised for review counts as a catch. Errors are requests the stage could not complete; they are excluded rather than counted as clean.

Corpus	Population	Recall	Precision	F1	FPR
HarmfulSkillBench	200 bad, 0 harmless	56.00%	no harmless class	no harmless class	no harmless class
MaliciousSkillBench balanced subset	400 bad, 400 harmless	81.75%	78.61%	80.15%	22.25%
MaliciousSkillBench source-disjoint (the published test split)	839 bad, 545 harmless	48.99%	80.91%	61.02%	17.80%
OpenSkillRisk: contextually risky	214 bad, 0 harmless	77.10%	no harmless class	no harmless class	no harmless class
OpenSkillRisk: obviously malicious	160 bad, 0 harmless	100.00%	no harmless class	no harmless class	no harmless class

Every member is positive-risk, so there is no harmless class and precision is undefined. Its licence also forbids any false-positive-rate claim. Every member is positive-risk, so there is no harmless class and precision is undefined.

The judge against the rules, detection only

The locked corpus above is one population. These are the others, each with the figure its licence permits. Detection is the share of skills flagged at all.

Corpus	Population	Rules	Judge	Change
HarmfulSkillBench	200 skills	4.00%	56.00%	+52.0 points
MaliciousSkillBench balanced subset	400 malicious, 400 benign	33.00%	81.75%	+48.8 points
OpenSkillRisk: contextually risky	214 skills	32.71%	77.10%	+44.4 points
OpenSkillRisk: obviously malicious	160 skills	30.00%	100.00%	+70.0 points

What is good: The judge improves detection on every corpus, and the gain is largest exactly where the rules are weakest. On HarmfulSkillBench the rules found 4% and the judge found 56%.

What is not: It is not free. On the locked corpus the share of harmless skills carrying an actionable finding roughly doubled, so this belongs behind a review queue rather than an automatic block.

Why the meta invocation rate is worth checking

The meta-judge has a routing gate that returns early when no finding is ambiguous, so "meta enabled" can mean "meta never ran". An arm reporting a near-zero invocation rate is indistinguishable from the core arm and must not be read as a measurement of the meta-judge. Any arm in that state is flagged in the report rather than presented as a result.

Flagged arms

core_judge_meta: meta was enabled but never invoked; this arm does not measure the meta-judge. routing reasons: {'ambiguous_finding_context': 1083, 'clear_deterministic_findings': 186, 'no_findings': 115}
core_meta: meta was enabled but never invoked; this arm does not measure the meta-judge. routing reasons: {'ambiguous_finding_context': 52, 'clear_deterministic_findings': 929, 'no_findings': 403}

Model fitness, measured rather than assumed

The judge model returns a package verdict of SAFE while also listing findings on a measurable fraction of packages, which is self-contradictory and causes the strict contract to discard the whole analysis. Measured on the benchmark corpus the contradiction appeared on 22.5% of benign packages and 0% of malicious ones. That asymmetry matters: discarding the analysis only on benign packages suppresses the judge exactly where it would have produced false positives, which flatters its measured precision. These arms therefore run with an opt-in, escalate-only verdict repair that rewrites the summary verdict and never the findings, applied 430 times. The repair is off by default in the product.