Designing Drift Regression Tests for Long-Running Agents

Standard evaluation answers the question "can this model do the task?" It runs a benchmark against a fresh context and reports a score. But behavioral state decay is a function of accumulated state, so an agent that scores well on every fresh-state eval can still degrade steadily in production as its context, memory, and plans age. The failure is invisible precisely where most teams are looking. Drift regression tests close this gap: they hold task difficulty constant and vary the one thing normal evals hold constant, the age and condition of the agent's state.

The method

A drift regression test replays a fixed suite of golden tasks against the agent's current accumulated state at regular intervals across a session or deployment lifetime. Holding the tasks fixed removes task-mix variation, but not inference randomness, backend model updates, tool or judge changes, or environment shifts: pair each aged-state replay with a contemporaneous clean-state control, pin dependencies and fixtures, and compare replicated runs with uncertainty intervals before attributing movement to state. The output is a decay curve: score as a function of interaction count or session age, per task category.

Design the suite around the drift types in the agent drift taxonomy:

• Instruction adherence probes: tasks that quietly violate a constraint from the original brief if the constraint has been lost, targeting context rot.
• Goal probes: tasks whose correct answer differs depending on whether the agent is serving the original objective or a drifted one, targeting goal drift.
• Trace probes: standardized tasks with defined mandatory trace invariants (required verification steps, ordering constraints), so behavioral drift shows up as violated invariants before output failure. Exact-sequence matching is usually too brittle, since multiple valid strategies can exist.
• Staleness probes: tasks referencing an environment detail that has changed mid-session, targeting plan decay.

Concrete procedure

• Step 1: freeze and version a golden suite specification (10 to 30 tasks is an illustrative size, not a sourced rule) with deterministic scoring where possible (exact match, schema validation, invariant checks) and rubric scoring only where necessary. Freezing the specification is compatible with held-out parameterized variants of each task; it is the spec and scorers that stay fixed, not every literal string.
• Step 2: capture a baseline by running the suite against fresh state N times and recording mean and variance per task.
• Step 3: schedule replays every K interactions (or at session milestones). For each replay: snapshot the live state, run the suite in a sandboxed copy of that state so the test itself does not pollute production context, score, and append results to a ledger keyed by state age.
• Step 4: compare each replay to baseline. In pseudocode: for each task, drift_signal = baseline_mean minus current_score; flag when the suite-level mean signal exceeds a threshold for two consecutive replays, or any single task falls below its hard floor.
• Step 5: wire the flag to a corrective action, typically re-anchoring or a fresh session seeded from externalized state, and record the intervention in the same ledger so you can measure recovery.

Suite scores can also feed the consistency dimensions of an Agent Stability Index style composite, giving you both a controlled signal (replays) and an observational one (live telemetry). The underlying drift study's simulation projects, under its assumptions, detectable drift after a median of 73 interactions (Rath, arXiv:2601.04170); treat that as a description of the simulation only, and choose your replay cadence from acceptable detection delay, replay cost, and the decay you actually observe in production.

Pitfalls

• Testing fresh state by accident. If the replay harness starts a clean session, you are re-running a model eval and will never see decay. The aged state snapshot is the entire point.
• Contaminating production state. Replays run inside the live session both pollute the context they measure and leak golden answers into it. Always run against a sandboxed copy.
• Suite memorization. If golden tasks recur verbatim in the visible history, later replays measure recall of earlier replays. Sandboxing prevents this; parameterized task variants add safety.
• Judge decay. If scoring uses an LLM judge that shares state with the system under test, the judge drifts too and scores stay flat while quality falls. Keep judges stateless and fresh per replay.
• Flakiness misread as drift. Single-replay dips are usually variance. Require sustained movement across consecutive replays, the same discipline the ASI threshold rule uses.
• Curve without a response. A decay curve nobody acts on is dashboard decoration. Every threshold needs a wired intervention and an owner.

Checklist

• Golden suite frozen, versioned, and covering all four probe types.
• Deterministic scorers preferred; any LLM judge stateless and isolated from the system under test.
• Baseline distribution (mean and variance per task) recorded on validated fresh state.
• Replays run against sandboxed snapshots of aged state on a fixed cadence.
• Results appended to a durable ledger keyed by state age.
• Sustained-degradation alert rule with a wired corrective action.
• Recovery after intervention measured with the same suite.