How the judge cross-examines a finding

A design review has two failure modes. It can miss a real problem, and it can drown a real problem in false positives until someone mutes the tool. The deterministic and ML layers are tuned to miss little; the cost is that some of what they raise is context-dependent. The judge is the layer that reads that context and says, with a confidence and a reason, whether a finding deserves a human's attention.

Crucially, it is a verifier, not a generator. It never invents findings. It only rules on findings the earlier layers already produced, which keeps the model on a narrow, checkable job.

The judge does not decide what is true. It decides what is worth reading, and it shows its work. The deterministic layer remains the source of truth.

The input

A finding, with its context attached

For each finding, the judge receives the rule that fired, the component it fired on, and the surrounding graph: what the component is, how untrusted input could reach it, which trust boundaries it spans, and what the earlier layers concluded. That context is the difference between "an MCP server uses non-TLS transport" as an abstract rule and the same finding when the endpoint resolves to 127.0.0.1. One is a real exposure; the other is a dev loopback where TLS buys nothing.

The mechanism

A three-panelist majority, each a different lens

The judge does not ask one model once. It convenes a small panel. Each panelist reviews the same finding through a different lens, exploitability, blast radius, whether the flagged condition even reproduces, and returns its own vote. The majority verdict wins, the confidence is the mean of the winning votes, and one winner's reasoning is kept as the record. A single confident model can be confidently wrong; a panel that has to agree is harder to fool.

The output

Three verdicts, a confidence, and a reason

Every finding leaves the judge with one of three verdicts and a schema-constrained answer, so a stray sentence from the model can never become a malformed record.

• confirmed: the finding holds up under the panel's scrutiny.
• false_positive: the flagged condition is not a real risk in this context.
• needs_review: genuinely ambiguous; a human should look.

# the model must answer in this shape, nothing else
{
  "verdict": "false_positive",
  "confidence": 0.83,
  "reasoning": "endpoint resolves to 127.0.0.1 loopback;
               TLS adds no confidentiality on the local host."
}

The verdict is recorded on the finding and carried into the terminal output, the evidence chain, and SARIF. A finding never silently changes state; you see judge: confirmed 0.94 next to it, and the reasoning travels with it.

The move that saves time

A confident false positive is waived, never deleted

When the panel returns false_positive above a confidence threshold, and only then, Attestral can auto-suppress it: the finding becomes a machine-generated waiver, stamped with the judge's model and reasoning. It drops out of your must-fix list without dropping out of the record. This is the same discipline the whole tool runs on, a waived finding is marked, not erased, so the evidence chain still verifies and an auditor can see exactly what was set aside and why.

In the scan

Where it runs

The judge runs last, after the deterministic and ML layers have produced findings, and only when you pass --judge with an API key set. It reviews what exists and records verdicts; it does not gate the earlier layers. Everything it emits is tagged origin=llm.

# attestral scan examples/vulnerable-agent --judge --judge-suppress
cross-examining 16 findings with the judge...
  ATL-202  lethal trifecta        judge: confirmed 0.94
  ATL-101  non-TLS transport      judge: false_positive 0.83  auto-waived
  ATL-ML-001  injection text      judge: needs_review 0.55
16 findings · 13 confirmed · 1 auto-waived FP · 2 need review
           origin=llm  verdicts recorded in the chain

Honest limits

Why this layer is optional, and last

That trade is exactly why the judge sits on top of a deterministic floor rather than replacing it. The reproducible layer is what an audit rests on; the judge is a triage assistant that makes the pile of findings smaller and better-explained. It is powerful and it is clearly fenced, and the output tells you, on every finding, which layer spoke.

A model in the loop is fine when it is a verifier that shows its reasoning and cannot silently delete evidence. It is not fine as the thing an auditor has to trust on faith.

Keep reading

The layers underneath

The judge only rules on findings the earlier layers raise: the deterministic layer that scores structure, and the DeBERTa classifier that scores language. All three feed one evidence chain, each tagged by origin.

References