LLM-as-Judge for Drift Detection, and When the Judge Itself Decays

A drift regression test needs a scorer, and deterministic scorers, exact match, schema validation, invariant checks, only cover part of what an agent produces. The rest gets scored by another language model acting as a judge, because human review does not scale to a suite replayed on every deployment or every K interactions. That choice imports a second system into the measurement pipeline, one with its own reliability profile and its own capacity to change between one replay and the next. A composite score like the Agent Stability Index is only as trustworthy as the judge computing its response-consistency sub-metrics.

This is a distinct failure surface from the ones already covered here. It is not context rot, because the judge model typically scores one short, self-contained comparison at a time rather than a long accumulated window. It is not the agent under test drifting; it is the measurement instrument itself carrying bias, instability, or an unannounced version change. The drift-regression-tests pitfall list already names the symptom directly: "if scoring uses an LLM judge that shares state with the system under test, the judge drifts too and scores stay flat while quality falls." This article is about the mechanism behind that line, and about a second, separate risk: a judge that changes even when it does not share state with anything.

Mechanism

Two problems compound, and they have different root causes.

The first is that an LLM judge is unreliable even on the day it is deployed, before any drift enters the picture. A large-scale evaluation of 21 judges from 9 providers, across MT-Bench, JudgeBench, and RewardBench, generating roughly 541,000 individual judgments over 118 runs, found that every judge's raw exact-match agreement overstated its chance-corrected agreement (Cohen's kappa) by 33.8 to 41.2 percentage points, a cohort mean gap of 38.6 points (Duan et al., arXiv:2606.19544). Raw agreement, the number most dashboards would show by default, systematically flatters judge reliability. The same study found judge rankings were unstable across benchmarks: 11 of 21 models shifted rank by four or more positions between benchmarks, with one model moving from 5th to 20th. And it surfaced what the authors call a consistency-bias paradox: two production judges scored above 0.95 on test-retest reliability while simultaneously showing position bias above 0.10, meaning a judge can be highly repeatable and still systematically favor whichever response it sees first or second. High self-consistency is not evidence of low bias; they are separate axes.

Self-preference compounds this for any pipeline where the judge model is related to the system it scores, a common setup when the same vendor's model both generates and grades. A quantification framework built around equal-quality response pairs, isolating bias from genuine quality differences, found self-preference bias spanning from strongly positive to strongly negative across models, and a counterintuitive pattern where stronger, more capable models often showed larger self-preference, not smaller (Zhu et al., arXiv:2604.22891). Generation quality and evaluative objectivity were largely decoupled: a better model at producing answers was not a more neutral judge of other models' answers. Verbosity bias, once a defining weakness of early LLM judges, has narrowed but not vanished; the same reliability study measured it below 0.011 across all 21 judges on MT-Bench, an order of magnitude improvement over 2023-era findings, which is worth knowing so verbosity is not assumed to be the dominant bias in a modern judge without checking. Beyond bias, judges are also directly gameable: a "null model" that ignores the actual input and returns a fixed, generic response achieved an 86.5% length-controlled win rate on AlpacaEval 2.0, an 83.0 score on Arena-Hard-Auto, and 9.55 on MT-Bench, with the cheating responses transferring across benchmarks via adversarial search rather than requiring benchmark-specific tuning (Zheng et al., arXiv:2410.07137). A judge does not need to be drifting to be fooled; it can be reliably wrong from day one.

The second problem is that a judge which was well-behaved at deployment can change without anyone editing the harness. Hosted judge endpoints are updated by their providers, sometimes silently, and a scoring prompt can be edited for an unrelated reason. Both move the judge's output distribution, and from outside the pipeline a moved judge looks identical to a system that regressed: scores drop either way. A framework built specifically for this ambiguity uses a fixed, human-labeled anchor set that the current judge rescores at intervals, paired with a statistical monitor of the judge-versus-human gap, to attribute each alarm to {none, system, judge} rather than leaving it ambiguous (Xu et al., arXiv:2606.15474). In its evaluation, the method correctly attributed 60 of 60 runs to judge drift after a silent version bump, with zero false system attributions, and correctly attributed 110 of 120 runs (92%) to a contaminating prompt change at its tested guard width; a standard rolling z-test baseline, by comparison, false-alarmed on 75% of drift-free streams. The paper's own framing is the useful takeaway: "a silent version bump or scoring-prompt update changes how a judge scores, so every drift alarm is ambiguous between a worse product and a changed judge" until something separates the two.

How to measure

• Report chance-corrected agreement, not raw exact match. The 33.8 to 41.2 point kappa gap above means raw agreement alone will make an unreliable judge look reliable.
• Run a position-swap check on every comparative judgment (A-vs-B and B-vs-A) and track the flip rate as a standing metric, since the consistency-bias paradox shows high test-retest reliability does not rule out position bias.
• Test self-preference directly if the judge and the system under test share a vendor or lineage: construct equal-quality response pairs from third-party models and check whether the judge's own family wins disproportionately more often than a null baseline would predict.
• Maintain a fixed, human-labeled anchor set and rescore it with the live judge on a regular cadence, per the attribution framework above; a divergence in the judge-versus-human gap, isolated from the system's own scores, is judge drift, not system drift.
• Version-pin and log the judge model identity with every score. An unexplained score-distribution shift that coincides with an unlogged provider-side update is a judge-drift candidate before it is treated as a regression.
• Periodically probe with a known-cheat input, a null or off-topic response, and confirm the judge scores it low; a passing null-model probe is a standing indicator that the judge is currently gameable.

Mitigation

Keep the judge stateless and isolated from the system under test, as the drift-regression-tests pitfall already states; a judge that accumulates context from the same session it is scoring inherits that session's own drift and stops being an independent instrument.

Replace single holistic scores with structured, pairwise, per-dimension forced choice where the budget allows. The self-preference study's mitigation, splitting a holistic verdict into forced choices across explicit dimensions such as relevance, accuracy, depth, logic, and clarity, achieved an average 31.5% reduction in measured self-preference bias while preserving the judge's ability to discriminate real quality differences.

Treat the judge as a versioned dependency, not a stable constant. Pin the model version where the provider allows it, log the version and prompt hash with every score, and re-run the anchor set whenever either changes, rather than only reacting after scores move.

Before declaring a system regression, run the attribution check: rescore the anchor set with the current judge and compare the judge-versus-human gap against its own baseline. Only treat the movement as behavioral state decay in the system under test once the judge itself is confirmed stable across that check.

Where stakes are high enough to justify the cost, use a small panel of judges from different model families rather than one, and flag disagreement itself as a signal; a single self-preferring judge is a silent bias, but the same bias shows up as measurable disagreement the moment a second, independent judge is added.

Related terms

• Drift regression tests: the suite this article's judges typically score; its own pitfall list first named judge decay.
• Agent Stability Index: a composite score whose response-consistency category is a common consumer of LLM-judge output.
• Behavioral state decay: the system-side phenomenon that a drifted or biased judge can be mistaken for, or can mask.
• State externalization: the same durable-anchor-set pattern used here for judge attribution applies to any recalled fact a pipeline treats as ground truth.

FAQ

Does using an LLM judge introduce behavioral state decay into the measurement pipeline? Not by the definition used on this site, which describes runtime degradation from accumulated agent state. A stateless, freshly-instantiated judge does not accumulate state across scoring calls. But it can still be unreliable from a fixed baseline (bias, gameability) or change discontinuously (provider-side version updates, prompt edits), both of which corrupt a drift signal without being state decay themselves.

If a judge has low measured bias, is it safe to trust its drift alarms? Low bias on the metrics you checked is not the same as low bias overall, and it says nothing about stability over time. The consistency-bias paradox shows a judge can pass a test-retest check while still carrying meaningful position bias, and any judge is subject to silent provider-side updates regardless of its bias profile at initial evaluation. Periodic re-validation against a human anchor set is what closes that gap, not a one-time bias audit.

Can deterministic scorers replace LLM judges entirely to avoid this? Only for tasks with a checkable structure: exact match, schema validation, invariant checks. Open-ended quality judgments, tone, reasoning soundness, appropriateness of a free-text response, do not reduce to a deterministic check, which is why judges get used at all. The realistic goal is bounding where a judge is used and monitoring it as its own dependency, not eliminating it.

How often should an anchor set be rescored? The attribution research above demonstrates the method working at a regular interval fine-grained enough to catch a single version bump or prompt edit before many downstream replays run on a moved judge; the specific cadence should match how often your judge's provider or your own scoring prompt can change, not an arbitrary calendar default.