Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control

with Christian Moya and Guang Lin

1. Introduction

Reinforcement learning with verifiable rewards (RLVR) (Lambert et al. 2025) fine-tunes pretrained language models using rewards from automated checks. These checks come from a verifier, which compares a model’s response with a known solution or runs the code in the response against tests. Rewarding responses that pass the verifier has improved reasoning performance on mathematics and coding tasks (DeepSeek-AI et al. 2025). These gains, however, depend on the verifier: higher reward reflects progress on the intended task only when the verifier reliably judges correctness.

A verifier can accept a response even when the response fails the intended task. For example, the code in a response may pass the available tests yet fail on inputs those tests omit. Rewarding such responses reinforces accepted errors: responses that satisfy the verifier but fail the intended task. As the model produces accepted errors more often, average reward can rise even while performance on the intended task declines, the signature of reward hacking (Skalse et al. 2022).

Empirical studies document reward hacking in RLVR. Reward hacking appears, for example, in inductive reasoning, where a model must infer a general rule from labeled examples. Models trained with RLVR often skip this rule and instead list the label of each example. The verifier accepts these responses because it checks only whether a response labels the given examples correctly (Helff et al. 2026). Beyond eliciting such new hacking behaviors, reinforcement learning can amplify those that models acquired during earlier fine-tuning (Khalifa et al. 2026). These findings motivate methods that reduce reward hacking while preserving progress on the intended task.

Existing methods address reward hacking by constraining optimization (Laidlaw, Singhal, and Dragan 2025), improving feedback (Coste et al. 2024; Lightman et al. 2024), or detecting and correcting hacking (Baker et al. 2025; S. Wang et al. 2026). Yet because these methods rely on assumptions about correctness (Everitt et al. 2017), they can leave verifier errors unresolved (Eisenstein et al. 2024). Training therefore continues with these verifier errors, rewarding correct responses and accepted errors alike. This raises a broader question: how do these verifier errors shape what RLVR learns about the intended task as training proceeds?

To answer this question, we hold an imperfect verifier fixed and study how the policy learns from its rewards. We model this learning as gradient flow because it makes the analysis tractable. Within this setting, we study when reward hacking grows, what the information available during RLVR training reveals about it, and when an intervention reduces hacking while preserving correctness. Our contributions are:

(i) We derive a condition under which the share of accepted errors among accepted responses grows at the current policy, and we express it through two mechanisms: hack bias and correctness-to-hack leakage (Proposition 3.2). We also characterize when the gradient flow increases verifier reward while reducing correctness, the signature of reward hacking (Proposition 3.1).

(ii) We establish limits on detection, identification, and selective control from verifier feedback alone. We show that the complete RLVR training record provides no advantage in detecting accepted errors and can leave correctness unidentifiable (Propositions 4.1 and 4.2). We further prove that no controller using this record alone can guarantee fewer accepted errors while preserving correct responses (Proposition 4.3).

(iii) We derive conditions under which a correction to RLVR training reduces reward hacking while improving correctness, and we design projected audit correction, which satisfies them using correctness labels from audits of some responses (Theorem 5.1).

We test our theory in settings of increasing complexity. In contextual bandits, reward hacking emerges and projected audit correction reduces hacks while increasing correctness. In a language model, verifier reward rises while correctness falls, and the same correction reverses this trend.

2. Problem Formulation

2.1. Reinforcement Learning with Verifier Rewards (RLVR)

Policy. Given a prompt $x$ drawn from a prompt distribution $\mathcal{D}$⁠, the language policy ${\pi }_{\theta }(\cdot \mid x)$ samples a response $y$ from the response set ${\mathcal{Y}}_x$⁠. This policy has a trainable parameter vector $\theta \in {\mathbb{R}}^d$⁠.

RLVR. We analyze RLVR (Lambert et al. 2025), where the verifier assigns a reward $R(x,y)\in \{0,1\}$⁠. A reward of $1$ indicates acceptance. RLVR training maximizes the scalar objective: $J_R(\theta )$$={\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[R(x,y)].$ This objective represents the probability of acceptance averaged over prompts and sampled responses, yielding a value in $[0,1]$⁠.

2.2. Correctness and Accepted Errors

We define a fixed correctness indicator $c(x,y)\in \{0,1\}$ for each pair $(x,y)$⁠, where $c=1$ denotes a correct response. To focus our analysis on the consequences of accepting incorrect responses, we assume the verifier produces no false negatives. This assumption implies that $c(x,y)\le R(x,y)$ for all pairs, meaning the verifier accepts all correct responses. We call the event of an incorrect response being accepted, i.e., $\{R(x,y)=1,c(x,y)=0\}$⁠, a hack.

For any given prompt $x$⁠, we can partition the set of all accepted responses, $A_x=\{y\in {\mathcal{Y}}_x:R(x,y)=1\}$⁠. This set consists of two disjoint subsets. The first is the set of correct responses, $G_x=\{y\in {\mathcal{Y}}_x:c(x,y)=1\}$⁠. The second is the set of hacks, $H_x=A_x\setminus G_x$⁠. These sets remain fixed throughout training. The policy, however, can learn to sample from them with different frequencies.

Acceptance and hacking. For any given prompt $x$⁠, we define two key probabilities. The first is the acceptance probability, $p_x(\theta )$$=\underset{{\pi }_{\theta }}{Pr}(Y\in A_x\mid x)$⁠. The second probability, defined only when $p_x(\theta )>0,$ is the hacked share, $q_x(\theta )$$=\underset{{\pi }_{\theta }}{Pr}(Y\in H_x\mid Y\in A_x,x).$ Both probabilities depend on the policy parameters $\theta$ and change throughout RLVR training. Using the acceptance probability $p(\theta )$⁠, we can now rewrite the RLVR objective as the following expectation over prompts: $J_R(\theta )$$={\mathbb{E}}_{x\sim \mathcal{D}}[p_x(\theta )]$⁠. Because this objective depends only on $p_x(\theta )$⁠, it makes no distinction between correct responses and hacks (see Appendix C.1.2).

2.3. Gradient Flow Dynamics

To analyze how the policy evolves, we use gradient ascent in continuous time. This model isolates the effects of the verifier signal by removing the noise inherent in stochastic optimization. When the objective $J_R$ is differentiable, the parameters evolve according to the gradient flow: $$\dot{\theta }(t)=g_R(\theta (t)):={\mathrm{\nabla }}_{\theta }{J}_R(\theta (t)),$$(1) where $g_R(\theta )\in {\mathbb{R}}^d$ is the exact gradient. A direct consequence is that the objective can only improve, as ${\dot{J}}_R(\theta (t))$$=\parallel g_R(\theta (t)){\parallel }^2\ge 0$ (see Appendix C.1.3). This guarantee, however, applies only to the overall acceptance probability. It reveals nothing about the hacked share, $q_x(\theta )$ which may increase, decrease, or remain unchanged.

Research questions. We now define our two central research questions within our gradient flow framework. First, under what conditions does the hacked share, $q_x(\theta )$⁠, grow or persist during training? Answering this question requires distinguishing increases in acceptance due to correct responses from shifts in the policy’s sampling toward hacks. Second, since the verifier’s feedback is blind to correctness, is it possible to reduce the hacked share using verifier’s information alone? If not, what additional information would allow it to do so?

3. When Hacking Grows: A Population Analysis

This section analyzes the population dynamics of hacked responses. We show that the accepted population and the hacked share can increase together, and characterize when verifier flow increases reward while reducing correctness. We then describe how hack bias and correctness-to-hack leakage drive hacking growth. Ultimately, these drivers create a latent vulnerability that on-policy training can reinforce.

3.1. Verifier Acceptance and Hacked Share Can Increase Together

We partition responses into three populations: correct responses $G=\{(x,y):y\in G_x\}$⁠, hacks $H=\{(x,y):y\in H_x\}$⁠, and rejected responses $N=\{(x,y):y\in {\mathcal{Y}}_x,R(x,y)=0\}$⁠. Let $p_G(\theta ):=\underset{\theta }{Pr}(G)\in [0,1]$ and $p_H(\theta ):=\underset{\theta }{Pr}(H)\in [0,1]$ denote the probabilities of correct responses and hacks, respectively. To track both how often the verifier accepts and how often accepted responses are hacks, we define $p(\theta ):=\underset{\theta }{Pr}(G\cup H)=J_R(\theta )$ and $q(\theta ):=\underset{\theta }{Pr}(H\mid G\cup H),$ where $p(\theta )\in [0,1]$ measures overall acceptance and $q(\theta )\in [0,1]$ measures the hacked share among accepted pairs. The latter requires $p(\theta )>0$⁠.

Expected reward depends on total acceptance $p=p_G+p_H$⁠, regardless of how it splits between correct responses and hacks. Writing $p_H(\theta )=p(\theta )q(\theta )$ and $p_G(\theta )=p(\theta )(1-q(\theta ))$ and differentiating with respect to time gives $${\dot{p}}_H(\theta )=q(\theta )\dot{p}(\theta )+p(\theta )\dot{q}(\theta ),{\dot{p}}_G(\theta )=(1-q(\theta ))\dot{p}(\theta )-p(\theta )\dot{q}(\theta ),$$(2) where the $\dot{p}$ terms reflect changes in total acceptance and the $\dot{q}$ terms reflect shifts between correct responses and hacks. These identities impose no trade-off between acceptance and hacked share; both can increase together.

Reward hacking. Skalse et al. (2022) call a proxy reward hackable if some pair of policies has higher expected proxy return but lower expected true return. In our setting, the verifier reward $J_R=p$ serves as the proxy, and correctness $J_C=p_G$ serves as the true return. We characterize when gradient flow on $J_R$⁠, which we call verifier flow, generates such a pair.

Interpretation. The two sides of the inequality compete. The term $p\dot{q}$ is the correctness lost as the accepted population shifts toward hacked responses, and $(1-q)\dot{p}$ is the correctness gained from rising acceptance. Training produces reward hacking once the loss outweighs the gain. The condition needs to hold at only a single time to produce a pair of policies that witnesses hackability. Because both terms depend on the policy, the condition can hold at one time and fail at another.

3.2. The Dynamics of the Hacked Share

While the hacked share $q$ remains the quantity of interest, log-odds coordinates simplify its dynamics. When both $p_H(\theta )>0$ and $p_G(\theta )>0$⁠, we can define the log odds $z(\theta )\in \mathbb{R}$ as $$z(\theta ):=\log \frac{p_H(\theta )}{p_G(\theta )}=\log \frac{q(\theta )}{1-q(\theta )},$$(3) Because $z$ is strictly increasing in $q$⁠, the sign of $\dot{z}$ determines whether the hacked share grows, shrinks, or remains constant.

The log odds dynamics. Since $z=\log p_H-\log p_G$⁠, the rate $\dot{z}$ depends on the gradient of each group’s log-probability. For each group $S\in \{G,H,N\}$ with positive, continuously differentiable probability, we denote this gradient by ${\overline{s}}_S(\theta ):$$={\mathrm{\nabla }}_{\theta }\log \underset{\theta }{Pr}(S)\in {\mathbb{R}}^d,$ the group’s average score. Along the verifier flow (1), differentiating $z$ yields $$\dot{z}(\theta )=({\overline{s}}_H(\theta )-{\overline{s}}_G(\theta ){)}^{\mathrm{\top }}{g}_R.$$(4) The log odds grow when the reward gradient $g_R$ aligns with the score difference between hacks and correct responses. Differentiating total acceptance $p=p_G+p_H=1-p_N$ shows that $g_R$ is a weighted combination of the three group scores: $g_R$$=p[(1-q){\overline{s}}_G+q{\overline{s}}_H]$$=-(1-p){\overline{s}}_N.$ This decomposition, combined with (4), resolves $\dot{z}$ into two mechanisms: $$\dot{z}=p(1-p)[\underset{\text{correctness-to-hack leakage}}{\underbrace{({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)}}+\underset{\text{hack bias}}{\underbrace{q\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2}}].$$(5) The full derivation is provided in Appendix C.2.3.

3.3. Main Result: The Mechanism of Hacking Growth

We now state our main result characterizing the mechanism of hacking growth.

When the inequality holds, $\dot{z}>0$⁠. Since $\dot{q}=q(1-q)\dot{z}$ and $\dot{p}=\parallel g_R{\parallel }^2\ge 0$⁠, both the hacked share and total acceptance grow strictly, while the correct share among accepted responses decreases.

The two mechanisms of (5) play different roles:

(i) Hack bias, $q\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2$⁠, arises because hacks contribute to the reward gradient in proportion to their share among accepted responses. It is always nonnegative and grows with the hacked share at fixed group scores.

(ii) Correctness-to-hack leakage, $({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)$⁠, arises when the direction that favors correct responses over rejected ones also favors hacks over correct responses. It can be positive or negative: positive leakage reinforces hack bias, negative leakage opposes it and can reverse growth when its magnitude exceeds the bias.

Interpretation. This result is local: the group scores and hacked share depend on the current parameters, so the growth condition can change during training. The group scores evolve, so continued growth is not guaranteed. Yet a feedback loop is present: a growing hacked share strengthens hack bias at fixed group scores, which favors further growth. This result exposes a latent vulnerability: hacking can reinforce the conditions that favor its own growth, because the policy generates its own training samples. Moreover, RLVR training is not designed to oppose this reinforcement. We must therefore build any hacking defense on top of RLVR. Can the information generated during RLVR training support such a defense?

4. What RLVR Training Reveals: Limits of Verifier Feedback

Section 3 showed that rising rewards can mask a growing share of hacks. We now study whether the observations collected during RLVR training can expose them. We show that they cannot: training observations confer no detection advantage, leaving correct responses unidentifiable, and making selective control impossible.

4.1. RLVR Training Observations, Monitors, and Compatible Correctness

RLVR training observations. RLVR training produces prompts, sampled responses, and verifier labels. Let $L_t$ collect these observations through time $t$⁠, along with policy parameters, probabilities, gradients, and other derived quantities. This record is deliberately generous: the limitations below do not arise from incomplete logging. We assume that $L_t$ contains no additional correctness feedback beyond the verifier. We write ${\mathcal{F}}_t^R:=\sigma (L_t)$ for the information contained in the record.

Monitors. A monitor is an algorithm that uses $L_t$ to assess hacking. A detection monitor raises a binary alarm about whether hacks exist; an identification monitor predicts the correctness label ${\hat{c}}_t(x,y)$ of an accepted response. We consider deterministic monitors here and defer randomized extensions to Appendix C.3.

Compatible correctness assignments. A fixed assignment $c$ determines correctness. To study what the training record reveals about $c$⁠, we consider alternative assignments consistent with the verifier. Assuming no false negatives, we define $${\mathcal{C}}_R:=\{c^{\prime }:c^{\prime }(x,y)\in \{0,1\},c^{\prime }(x,y)\le R(x,y)\text{ for all }(x,y)\}.$$(7) This class models uncertainty about the true correctness assignment; $c$ itself remains fixed during training. All members agree on rejected responses, but they may disagree on accepted ones. To isolate the effect of this uncertainty, we compare these alternative assignments while holding the prompt distribution, verifier, initialization, and training algorithm fixed.

4.2. Limits of Detecting Hacking

Verifier acceptance can increase while the hacked share grows. Can the training record reveal even the presence of hacks? Under $c=R$⁠, every accepted response is correct. A compatible alternative $c_1\in {\mathcal{C}}_R$ with $\underset{\theta }{Pr}(R=1,c_1=0)>0$ admits hacks. Detection requires distinguishing these two cases using only $L_t$⁠.

Interpretation. Because alternative correctness assignments do not affect verifier feedback, they leave the training record’s distribution unchanged. Thus, any monitor’s detection rate under $c_1$ equals its false-alarm rate under $c=R$⁠. Even knowing that hacks exist does not reveal which accepted responses are wrong.

4.3. Limits of Identifying Hacks

Suppose we know that hacks exist. The remaining task is to determine which accepted responses are wrong. Identification requires a monitor to recover the true label $c(x,y)$ of an accepted response from $L_t$⁠. To succeed, the prediction ${\hat{c}}_t(x,y)$ must distinguish compatible assignments that disagree on this response, even when we restrict ${\mathcal{C}}_R$ to assignments that admit hacks.

Interpretation. Two compatible assignments can label the same accepted response differently while producing identical records. Both can admit hacks, so knowing that hacks exist does not resolve the disagreement. A prediction correct under one assignment is wrong under the other. Thus, the monitor incurs an error probability of at least $\frac{1}{2}$ under at least one assignment. Because reducing hacks need not require identifying every hack, this result alone does not rule out selective control.

4.4. Limits on Selective Control from Verifier Feedback Alone

Even if we cannot identify every hack, can we reduce hack probability without reducing the probability of correct responses? Suppose a controller uses $L_t$ to apply a correction $u(t)\in {\mathbb{R}}^d$ to the verifier flow, yielding $\dot{\theta }=g_R+u(t)$⁠. Along this corrected flow, selective control requires ${\dot{p}}_H<0$ and ${\dot{p}}_G\ge 0$ whenever $p_H>0$⁠. We target $p_H$ itself, not the hacked share $q$⁠, because $q$ can fall even while $p_H$ grows. For a uniform guarantee, this same controller must satisfy these conditions under every compatible assignment in ${\mathcal{C}}_R$⁠.

Interpretation. We seek a correction $u(t)$ such that the corrected flow reduces hack probability without reducing the probability of correct responses. However, compatible assignments can exchange the roles of correct responses and hacks. An update that reduces hack probability under one therefore reduces the probability of correct responses under the other. The controller cannot distinguish these assignments from $L_t$⁠, and more observations from the same verifier cannot resolve this conflict. Regularization illustrates this limit: it constrains policy updates using only the policy and the verifier, so it cannot guarantee selective control either (Appendix C.3.5). A uniform guarantee of selective control thus requires additional correctness information that rules out assignments requiring incompatible updates.

5. From Additional Feedback to Selective Control

Section 4 showed why verifier feedback alone cannot reduce hacks without also sacrificing correct responses. We now investigate whether additional information about correctness can break this trade-off. We show that it can: such information supports a correction that suppresses hacks and promotes correct responses, provided the correction is strong enough to overcome the drift toward errors induced by RLVR training.

5.1. Projected Audit Correction in RLVR Training

Audits as additional information. Verifier feedback alone cannot guarantee selective control (Section 4), so we introduce a stronger signal: audits. An audit reveals the true correctness label $c(x,y)$ for a prompt $x$ and an accepted response $y$⁠. The audit $Z=(x,y,c(x,y))$ rules out every correctness rule that disagrees with this label, restricting ${\mathcal{C}}_R$ to ${\mathcal{C}}_{R,Z}$$=\{c^{\prime }\in {\mathcal{C}}_R:c^{\prime }(x,y)$$=c(x,y)\}.$ The limits in Section 4 arose because verifier feedback could not distinguish the true rule from alternatives in ${\mathcal{C}}_R$⁠. Audits eliminate alternatives that contradict the revealed labels. To show how audits reduce hacking, we next define the audited hack population and its probability gradient under the policy.

Let $A_{\mathrm{aud}}$ be a fixed set of audits, and let $H_A=A_{\mathrm{aud}}\cap H$ denote the audited hacks, with probability $p_{H_A}(\theta ):=\underset{\theta }{Pr}(H_A)$⁠. When $p_{H_A}$ is differentiable, the negative gradient $-{\mathrm{\nabla }}_{\theta }{p}_{H_A}$ targets only the audited hacks and need not align with $-{\mathrm{\nabla }}_{\theta }{p}_H$ (see Appendix A.2). Despite this misalignment, $-{\mathrm{\nabla }}_{\theta }{p}_{H_A}$ locally decreases the audited hack probability when ${\mathrm{\nabla }}_{\theta }{p}_{H_A}\ne 0$⁠, so we use the correction $u=-\lambda {\mathrm{\nabla }}_{\theta }{p}_{H_A}$⁠, with $\lambda >0$⁠, in the verifier flow $\dot{\theta }=g_R+u$⁠.

The correction opposes growth in $p_{H_A}$⁠: ${\dot{p}}_{H_A}$$=\mathrm{\nabla }{p}_{H_A}^{\mathrm{\top }}{g}_R-\lambda \parallel \mathrm{\nabla }{p}_{H_A}{\parallel }^2.$ Because these probabilities share parameters, the correction also perturbs $p$ and $p_G$⁠: $$\dot{p}=\parallel g_R{\parallel }^2-\lambda g_R^{\mathrm{\top }}\mathrm{\nabla }{p}_{H_A},{\dot{p}}_G=\mathrm{\nabla }{p}_G^{\mathrm{\top }}{g}_R-\lambda \mathrm{\nabla }{p}_G^{\mathrm{\top }}\mathrm{\nabla }{p}_{H_A}.$$(8) The cross terms have no fixed sign. When $\mathrm{\nabla }{p}_G^{\mathrm{\top }}\mathrm{\nabla }{p}_{H_A}>0$⁠, the correction can suppress correct responses alongside hacks. We next constrain $u$ to preserve the growth rate of $p$⁠.

Projected audit correction. Since $p=p_H+p_G$⁠, we have ${\mathrm{\nabla }}_{\theta }{p}_G=g_R-{\mathrm{\nabla }}_{\theta }{p}_H$⁠. Preserving the instantaneous growth rate of $p$ requires $g_R^{\mathrm{\top }}u=0$⁠, so $u$ must lie in the subspace orthogonal to $g_R$⁠. Under this constraint, ${\mathrm{\nabla }}_{\theta }{p}_G^{\mathrm{\top }}u$$=-{\mathrm{\nabla }}_{\theta }{p}_H^{\mathrm{\top }}u$⁠: whenever $u$ opposes the growth of $p_H$ (${\mathrm{\nabla }}_{\theta }{p}_H^{\mathrm{\top }}u<0$⁠), it contributes equally to the growth of $p_G$⁠.

We project the audit correction onto the subspace orthogonal to $g_R$ using $P_{\perp }:=I-g_R{g}_R^{\mathrm{\top }}/\parallel g_R{\parallel }^2$ when $g_R\ne 0$ and $P_{\perp }:=I$ otherwise, to define the projected audit correction $u=-\lambda P_{\perp }\mathrm{\nabla }{p}_{H_A}$⁠, with $\lambda >0$⁠, yielding the flow $\dot{\theta }=g_R+u$⁠. Since $g_R^{\mathrm{\top }}{P}_{\perp }=0$ and $P_{\perp }$ is an orthogonal projector, $$\dot{p}=\parallel g_R{\parallel }^2,{\dot{p}}_{H_A}=\mathrm{\nabla }{p}_{H_A}^{\mathrm{\top }}{g}_R-\lambda \parallel P_{\perp }\mathrm{\nabla }{p}_{H_A}{\parallel }^2.$$(9) This projection preserves the instantaneous growth rate of $p$ and opposes the growth of $p_{H_A}$ whenever $P_{\perp }\mathrm{\nabla }{p}_{H_A}\ne 0$⁠. We next show when this projected correction shrinks the share of hacks in the full population and bounds that share during training.

5.2. Main Result: Achieving Selective Control

Assumptions. We analyze RLVR training under the projected audit correction on a finite interval $[0,T]$⁠, under the following assumptions. (A1) Regularity: The probabilities $p_G,p_H,p_{H_A}$ are continuously differentiable in $\theta$⁠, with $p_G,p_H>0$ on $[0,T]$⁠. (A2) Audit coverage: The fixed audit set satisfies $\underset{\theta (t)}{Pr}(H\setminus A_{\mathrm{aud}})=0$ throughout the interval, so $p_{H_A}=p_H$ and their gradients coincide during training. (A3) Uniform corrective strength: $\parallel P_{\perp }\mathrm{\nabla }{p}_H{\parallel }^2/p\ge \kappa q$ throughout the interval for some constant $\kappa >0$⁠. We discuss these conditions in Appendix A.2.

The bound separates a decaying contribution from the initial hacked share and a residual contribution from growth pressure. For fixed $D$⁠, a larger product $\lambda \kappa$ reduces the residual bound.*

Interpretation. Under the stated assumptions, the projected audit correction favors correct responses over hacks while preserving the instantaneous growth rate of $p$⁠. When bias and leakage supply no positive growth pressure, the input-to-state stability (ISS)-type bound (Khalil and Grizzle 2002) ensures the share of hacks decays exponentially. Persistent pressure contributes a residual bound that decreases with $\lambda \kappa$ for fixed $D$⁠. The selective effect is instantaneous; the trajectory bound requires the assumptions to hold throughout the interval.

Remark (Practical implementation). The control guarantees (Theorem 5.1) require audits that provide a direction opposing hack growth and sufficient corrective strength throughout training. Partial audit coverage can suffice when the resulting correction remains aligned with the gradient of hack probability. Finite-sample gradient estimates, optimizers, and finite step sizes can prevent the implemented update from preserving acceptance progress or suppressing hacks. Appendix A.2 discusses these requirements and how to estimate and validate the correction.

6. Experiments

We test reward hacking, the limits of verifier feedback, and selective control through audit correction in contextual bandits and language models. The code is available at https://github.com/cmoyacal/verifier-errors.

6.1. Gaussian Contextual Bandits

Setup. We use Gaussian contextual bandits to study when training on verifier rewards amplifies reward hacking and whether audits enable selective control. We set the initial probabilities of correct and hacked responses, $p_G(0)$ and $p_H(0)$⁠, to isolate how the initial composition shapes training. We also vary the policy class: a log linear policy keeps its features fixed, while a neural policy learns its own representation. Appendices D.1 and Appendix D.2 provide the experimental details.

Results. Figure 1 (a) shows acceptance $p$ and hacked share $q$ increasing together from $q(0)=0.3$⁠, as Proposition 3.2 predicts. Panel (b) plots correctness $J_C=p_G$ against verifier reward for three initial hacked shares. For $q(0)=0.5$⁠, correctness declines throughout the recorded interval; for $q(0)=0.3$⁠, it first improves and then declines; for $q(0)=0.1$⁠, both correctness and verifier reward improve. Both declines are reward hacking in the sense of Proposition 3.1. Panel (c) evaluates the same verifier flow under two correctness assignments: $c=R$⁠, which admits no hacked responses, and $c_1={\mathbf{1}}_G$⁠, which does. The information available during training is identical under both, yet correctness differs, as Proposition 4.1 predicts. Panel (d) shows selective control with a neural policy: projected audit correction (PAC) decreases the probability $p_H$ of hacked responses while increasing correctness $p_G$⁠, consistent with Theorem 5.1. Raw audit correction initially decreases both probabilities, whereas verifier flow and gradient regularization increase $p_H$⁠. Appendices D.1 and Appendix D.2 provide additional experiments and ablations.

6.2. Language Models: Reward Hacking and Audit Correction

To test whether our predictions hold beyond exact gradient flow, we train a language model with sampled gradients and finite Adam updates.

Setup. We consider a task in which the model replaces a sequence of digits according to given rules. A response is correct only if every replacement in the final output is correct, but the imperfect verifier $R$ checks only the last two digits. We include in each prompt a hint with an incorrect prefix and the correct final pair, so copying it produces a hack. We initialize Qwen2-0.5B with supervised fine-tuning (SFT) on correct responses $G$⁠, hacks $H$⁠, and rejected responses $N$⁠, and then run GRPO for 20 rounds with or without projected audit correction (PAC). We vary audit coverage by letting PAC audit each accepted response independently with probability $1$⁠, $0.5$⁠, or $0.25$⁠. Appendix D.3 provides the experimental details.

Results. Figure 2 (a) shows verifier reward $J_R$ increasing while correctness $J_C$ falls under GRPO, which illustrates reward hacking in the sense of Proposition 3.1. Panel (b) shows PAC increasing $p_G$ and decreasing $p_H$ from initialization at all three audit probabilities $\rho \in \{0.25,0.5,1.0\}$⁠, qualitatively consistent with Theorem 5.1. On test prompts, GRPO reaches $99.4\mathrm{\%}$ acceptance but only $2.2\mathrm{\%}$ correctness. PAC reaches $97.2\mathrm{\%}$ correctness with full auditing and $95.2\mathrm{\%}$ with one quarter of accepted responses audited. Additional experiments and ablations are provided in Appendix D.3.

7. Related Work

Improving a proxy reward can reduce the intended reward (Everitt et al. 2017; Skalse et al. 2022), and experiments show this reduction growing with stronger optimization (Gao, Schulman, and Hilton 2023) and appearing under automated verifiers (Helff et al. 2026). Methods that mitigate reward hacking constrain the policy (Laidlaw, Singhal, and Dragan 2025), improve feedback (Coste et al. 2024; Lightman et al. 2024), or detect and correct hacking (Baker et al. 2025; S. Wang et al. 2026), and their guarantees depend on what they observe or assume about correctness. We complement this work by fixing a verifier and characterizing theoretically when reward hacking grows, when the information available during training leaves correct and hacked responses indistinguishable, and when audits can reduce hacking while preserving progress on the intended task. Appendix B extends this discussion.

8. Conclusion

This work analyzes RLVR under a fixed imperfect verifier. First, we derive when the share of accepted errors grows, through hack bias and correctness-to-hack leakage, and when verifier reward rises while correctness falls (Propositions 3.2 and 3.1). Second, the complete training record cannot, in general, detect accepted errors, identify correctness, or guarantee fewer accepted errors while preserving correct responses (Propositions 4.1, 4.2, and 4.3). Third, projected audit correction uses correctness labels from audits of some responses to reduce reward hacking while improving correctness, under the conditions of Theorem 5.1. Together, these results show that controlling reward hacking requires information about correctness beyond the verifier. Incomplete audits and inaccurate labels can weaken this correction. Our analysis assumes a fixed verifier and exact gradient flow, and Appendix A discusses limitations and practical considerations. Extending the guarantees throughout training and correcting from limited, imperfect audits remain future work.

References

A. Limitations and Practical Considerations

This appendix states the assumptions that limit the scope of our theoretical guarantees and describes how to implement the audit correction.

A.1. Limitations

No false negatives. To isolate whether rewarding hacks can reduce correctness $p_G$⁠, we assume that the verifier accepts every correct response. Under this assumption, a decline in $p_G$ cannot come from the verifier rejecting correct responses. The assumption also gives $p=p_G+p_H$⁠, because the accepted responses then consist of all correct responses and all hacked responses. Thus, any change that reduces $p_H$ without decreasing $p$ increases $p_G$⁠.

If false negatives are allowed, $p_G$ still denotes total correctness, but the probability of rejected correct responses adds a term: $$p_G=p-p_H+\underset{\theta }{Pr}(c=1,R=0).$$(12) Differentiating along the flow and using $\dot{p}=\parallel g_R{\parallel }^2$ gives $${\dot{p}}_G=\parallel g_R{\parallel }^2-{\dot{p}}_H+\frac{d}{dt}\underset{\theta (t)}{Pr}(c=1,R=0).$$(13) The first two terms give the rate at which the probability of accepted correct responses changes. Thus, even when ${\dot{p}}_H<0$⁠, total correctness decreases whenever $$\frac{d}{dt}\underset{\theta (t)}{Pr}(c=1,R=0)<-(\parallel g_R{\parallel }^2-{\dot{p}}_H),$$(14) that is, whenever the probability of rejected correct responses falls faster than the probability of accepted correct responses rises. Extending the control guarantee to this case therefore requires a lower bound on $\frac{d}{dt}\underset{\theta (t)}{Pr}(c=1,R=0)$⁠. Proposition 3.2, which decomposes the growth of the hacked share, still applies within the accepted population, with the average score of accepted correct responses in place of ${\overline{s}}_G$⁠.

Population gradient flow. We analyze updates in continuous time driven by the exact gradient of $J_R$⁠. Exact gradients remove sampling noise and isolate the effect of verifier rewards. Practical training departs from this idealization through finite steps, adaptive optimizers such as Adam, gradient clipping, and additional objectives such as KL regularization, each of which can change the dynamics. Our growth and control guarantees therefore hold exactly only for the stated flows. Our experiments with discrete updates check whether the predicted behavior persists in the tested settings, but they do not extend the guarantees to other training algorithms.

Local growth and control over a finite interval. Proposition 3.2 characterizes the growth of the hacked share at the current policy. The group scores change during training, so a positive growth rate at one time does not guarantee continued growth. Theorem 5.1 extends the analysis from a single time to the interval $[0,T]$⁠, but its bound requires the assumptions to hold throughout this interval and does not establish convergence beyond it. Similarly, the identity $\dot{p}=\parallel g_R{\parallel }^2$ under the projected correction is local: the correction preserves the rate at which acceptance grows at the current policy. Because the correction changes the trajectory of the policy, this identity does not imply that corrected training follows the same acceptance curve or reaches the same final acceptance as uncorrected training.

Scope of the information limits. Our impossibility results (Section 4) concern methods that observe only the training record $L_t$⁠: no such method can provide a uniform guarantee, one that holds under every correctness assignment in ${\mathcal{C}}_R$⁠. These results do not imply that every monitor fails on every task. Additional knowledge of task requirements can exclude assignments in ${\mathcal{C}}_R$⁠, and if it excludes the indistinguishable assignments used in our proofs, our impossibility arguments no longer apply. Conversely, allowing false negatives enlarges the class of assignments, and the enlarged class still contains these indistinguishable assignments, so it admits no uniform guarantee either.

Fixed verifier and prompt distribution. We hold the binary verifier, correctness labels, and prompt distribution fixed during training. In practice, the verifier or prompt distribution can change during training, for example when tests are added or a curriculum reorders prompts. Such changes shift population probabilities without any policy update, and our dynamics exclude them. Because $p_G$ and $p_H$ average over the prompt distribution, increasing $p_G$ or decreasing $p_H$ does not guarantee that correctness improves on every prompt.

A.2. Practical Considerations

Limited audit coverage. We assumed full audit coverage in Assumption (A2) for Theorem 5.1, so that $\mathrm{\nabla }{p}_{H_A}=\mathrm{\nabla }{p}_H$⁠. Without full coverage, the correction changes the growth of $p_H$ by $\mathrm{\nabla }{p}_H^{\mathrm{\top }}u$$=-\lambda (P_{\perp }\mathrm{\nabla }{p}_H{)}^{\mathrm{\top }}(P_{\perp }\mathrm{\nabla }{p}_{H_A})$⁠. When the projected gradients align positively, the correction opposes the growth of $p_H$ and, as in Theorem 5.1, equally promotes the growth of $p_G$⁠. Full coverage guarantees positive alignment when $P_{\perp }\mathrm{\nabla }{p}_H\ne 0$⁠, but is not necessary: a subset of audits can supply a direction that opposes hacking. We next ask whether this direction remains effective as the policy changes.

Sustaining corrective strength. A direction that opposes hacking at one policy may weaken or reverse as on-policy training shifts the distribution and gradients. Accurate audits are not enough: they must continue to supply sufficient corrective strength throughout training. Retaining the other assumptions of the theorem, the same ISS-type bound holds under partial coverage if $(P_{\perp }\mathrm{\nabla }{p}_H{)}^{\mathrm{\top }}(P_{\perp }\mathrm{\nabla }{p}_{H_A})/p\ge \kappa q$ throughout the interval for some fixed $\kappa >0$⁠; under full coverage, this condition reduces to (A3). We next ask how to estimate the corrective direction from finite samples.

Estimating the correction from finite audits. The correction uses a population gradient, but training provides only sampled responses and audit labels. Randomized audit selection provides a way to estimate the full hack gradient without auditing every generated response in each batch. For samples from the current policy, we estimate ${\mathrm{\nabla }}_{\theta }{p}_H$ using each audited hack’s policy score, ${\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$⁠, divided by its selection probability. Audited correct responses contribute zero. Averaging these weighted contributions over all generated responses yields an unbiased estimate $\hat{h}$⁠, provided these selection probabilities are known and positive for accepted responses and the regularity conditions in Appendix C.4.3 hold. Accepted responses with zero selection probability remain outside this guarantee. Even with sufficient coverage, small selection probabilities produce large weights and can yield noisy estimates, so an individual correction may fail to oppose hacking. While sparse audits can recover the direction in expectation, reliable correction requires controlling the variance of this estimate.

Implementing the projection $P_{\perp }$⁠. The projection also requires an estimate ${\hat{g}}_R$ of the reward gradient. Set the correction orthogonal to the estimated reward gradient: ${\hat{g}}_R^{\mathrm{\top }}\hat{u}=0$⁠. To isolate projection error, we keep the base verifier gradient exact and consider $\dot{\theta }=g_R+\hat{u}$⁠. Then $$\dot{p}=\parallel g_R{\parallel }^2+(g_R-{\hat{g}}_R{)}^{\mathrm{\top }}\hat{u},$$(15) where the second term is the projection error. Thus, orthogonality to the estimated gradient need not preserve the true instantaneous growth rate of $p$⁠. Estimating the base verifier gradient introduces additional error. Optimizer transformations and finite step sizes can introduce further discrepancies. Practical implementation requires validating that discrete updates oppose hacking while approximately preserving acceptance progress.

Our analysis identifies three practical tasks: selecting which responses to audit, estimating the corrective direction from these audits, and validating that each update opposes hacking while approximately preserving acceptance progress. Susceptibility methods for reinforcement learning (Elliott et al. 2026) and patterning (G. Wang and Murfet 2026) could help guide audit selection and correction. Adapting these methods to sustain selective control under limited audit and computation budgets remains an open engineering challenge.

B. Related Work

We position our work within two lines of research on reward hacking: why optimizing a proxy reward can undermine the intended reward, and how to prevent it.

Reward hacking.

Theoretical work shows that improving a proxy reward can reduce the intended reward (Everitt et al. 2017; Skalse et al. 2022). This conflict can arise when the proxy omits relevant attributes (Zhuang and Hadfield-Menell 2020), and its severity depends on properties of the proxy and its errors (Laidlaw, Singhal, and Dragan 2025; Kwa, Thomas, and Garriga-Alonso 2024). In experiments, stronger optimization and more capable models can widen the gap between proxy and intended rewards (Gao, Schulman, and Hilton 2023; A. Pan, Bhatia, and Steinhardt 2022). Language models also show this gap during iterative self-refinement (J. Pan et al. 2024) and when automated verifiers accept responses that violate task requirements (Helff et al. 2026; Zhong, Raghunathan, and Carlini 2025; Mahmoud et al. 2026). Training can amplify such reward hacking (Khalifa et al. 2026), and learned hacking can generalize to reward tampering or broader misalignment (Denison et al. 2024; MacDiarmid et al. 2025). To explain how reward hacking develops, prior work studies the geometry of optimization (Karwowski et al. 2024), selection at inference time (Khalaf et al. 2025), and feedback between policy training and retraining of the reward model (Gauthier, Bach, and Jordan 2026). For rewards assigned by judges, Xuekang Wang et al. (2026) further separate how easily models discover biases from how strongly they exploit them. Compared to these works, we study how reward hacking grows under a fixed binary verifier by deriving conditions under which policy updates favor hacked responses over equally rewarded correct responses. We then show that the reward objective provides no preference that would restore correct responses within the accepted set, which explains why reward maximization alone need not reverse this growth.

Mitigating reward hacking.

Regularization can limit reward hacking by constraining how much a policy changes (Laidlaw, Singhal, and Dragan 2025) or by favoring flatter optima, a property that theory links to the accuracy of rewards under additional assumptions (Ackermann et al. 2026). Other methods improve the feedback used for optimization by combining reward models (Coste et al. 2024), adding constraints (Helff et al. 2026), supervising reasoning steps (Lightman et al. 2024), or supplying criticism through debate (Kenton et al. 2026). When verifier errors follow an assumed noise model, known error rates can also support corrections to training updates (Cai et al. 2025). Even improved feedback can leave errors unresolved, as Eisenstein et al. (2024) show for reward ensembles. To detect reward hacking, monitors inspect a model’s reasoning (Baker et al. 2025) or measure how much reasoning it needs to pass the verifier (Xinpeng Wang et al. 2026). Detection can then guide correction: Grift groups responses by their gradients, labels each group by inspecting examples, and filters responses for fine-tuning (S. Wang et al. 2026), while IR${}^3$ reconstructs rewards and targets features it identifies as problematic (Beigi et al. 2026). The guarantees of these approaches depend on what they observe or assume about correctness, a dependence that Everitt et al. (2017) examine for corrupted rewards. Our identifiability results specify when the information available during training leaves correct and hacked responses indistinguishable. We then characterize how auditing supplies information for selective correction and how limited coverage or errors in the audits can undermine this correction.

C. Technical Details and Proofs

C.1. Probabilities and Dynamics for Section 2

Idealized conditions.

We use the fixed verifier $R$ and correctness rule $c$ from Section 2, with $c\le R$⁠, so the verifier has no false negatives. Without correction, the parameters follow exact Euclidean gradient ascent on the verifier reward, $\dot{\theta }=\mathrm{\nabla }{J}_R(\theta )$⁠. We assume that the objectives and group probabilities are continuously differentiable and that the stated flows exist on the intervals considered. When differentiating expectations, we assume sufficient regularity to interchange differentiation and expectation.

C.1.1. Fixed Sets of Correct Responses and Accepted Errors

For each prompt $x$⁠, the assumption $c(x,y)\le R(x,y)$ gives $G_x\subseteq A_x$ and hence $$\begin{array}{rl}{H}_x& =A_x\setminus G_x=\{y\in {\mathcal{Y}}_x:R(x,y)=1,c(x,y)=0\},\\ A_x& =G_x\bigsqcup H_x,\end{array}$$(16) where $\bigsqcup$ denotes disjoint union. These memberships stay fixed throughout training. Their probabilities can change, and $H_x$ may be empty.

C.1.2. Acceptance and the Hacked Share

At a fixed prompt, conditioning on acceptance gives $$\begin{array}{rl}{q}_x(\theta )& =\frac{\underset{{\pi }_{\theta }}{Pr}(Y\in H_x\mid x)}{p_x(\theta )},\\ \underset{{\pi }_{\theta }}{Pr}(Y\in H_x\mid x)& =p_x(\theta )q_x(\theta ),\\ \underset{{\pi }_{\theta }}{Pr}(Y\in G_x\mid x)& =p_x(\theta )(1-q_x(\theta )),\end{array}\text{when }{p}_x(\theta )>0.$$(17) If $p_x=0$⁠, the probability of an accepted error is zero and $q_x$ is undefined. Since the binary verifier rewards every response in $A_x$⁠, iterated expectation gives $$J_R(\theta )={\mathbb{E}}_{x\sim \mathcal{D}}[{\mathbb{E}}_{Y\sim {\pi }_{\theta }(\cdot \mid x)}R(x,Y)]={\mathbb{E}}_{x\sim \mathcal{D}}[p_x(\theta )].$$(18) The objective therefore depends on total acceptance, without distinguishing its correct and incorrect parts.

C.1.3. Exact Gradient Flow

Along the flow (1), the chain rule gives $${\dot{J}}_R={\mathrm{\nabla }}_{\theta }{J}_R^{\mathrm{\top }}\dot{\theta }=\parallel g_R{\parallel }_2^2\ge 0,{\dot{p}}_x={\mathrm{\nabla }}_{\theta }{p}_x^{\mathrm{\top }}{g}_R,{\dot{q}}_x={\mathrm{\nabla }}_{\theta }{q}_x^{\mathrm{\top }}{g}_R,$$(19) where the identity for ${\dot{q}}_x$ applies when $p_x>0$⁠. Under the stated assumptions, there is no general guarantee that ${\dot{q}}_x\le 0$ or that ${\dot{p}}_x\ge 0$ at every prompt. Thus, monotonic improvement in average acceptance guarantees neither a nonincreasing hacked share nor nondecreasing acceptance at every prompt.

C.2. Technical Details for Section 3

C.2.1. Aggregate Acceptance and Hacking

Write ${\mathbb{E}}_{\theta }$ for expectation under $X\sim \mathcal{D}$⁠, $Y\mid X\sim {\pi }_{\theta }$⁠. Since $G\cup H$ is the accepted set, $$\begin{array}{rl}p& =J_R=p_G+p_H={\mathbb{E}}_{x\sim \mathcal{D}}[p_x],\\ q& =\frac{p_H}{p}=\frac{{\displaystyle {\int }_{\{x:p_x>0\}}{p}_x{q}_x\mathcal{D}(dx)}}{p}\text{when }p>0.\end{array}$$(20) Thus, relative to the prompt distribution $\mathcal{D}$⁠, prompts are weighted by their acceptance probabilities. Prompts with zero acceptance contribute nothing. An aggregate trend need not hold at every prompt. At fixed $p$⁠, redistributing probability between $G$ and $H$ leaves reward unchanged, although shared parameters can couple their dynamics.

C.2.2. Proof of Proposition 3.1

Skalse et al. (2022) call a proxy reward hackable if it ranks some pair of policies in strictly opposite orders from the true reward. In our setting, verifier flow generates the policies, acceptance $J_R=p$ is the proxy reward, and correctness $J_C=p_G$ is the true reward. We show that such a pair occurs along the trajectory exactly when, at some time, the correctness lost to a rising hacked share outweighs the correctness gained from rising acceptance.

The criterion distinguishes a growing hacked share from reward hacking in the sense of Skalse et al. (2022). A growing hacked share can coexist with improving correctness if acceptance rises fast enough. The two rewards disagree exactly when, at some time along the flow, the correctness lost to a rising hacked share exceeds the correctness gained from rising acceptance. This inequality needs to hold strictly at only a single time: by continuity, it then holds on a short interval whose endpoints witness hackability, even if correctness later recovers.

C.2.3. Proof of Proposition 3.2

When $p_H,p_G>0$⁠, total acceptance cancels in the ratio: $$z=\log \frac{p_H}{p_G}=\log \frac{q}{1-q},\frac{dz}{dq}=\frac{1}{q(1-q)}>0.$$(28) The log odds $z$ therefore increase with the hacked share $q$⁠. They are finite only when correct responses and hacks both have positive probability, so we require this condition wherever we use finite log odds.

Evolution of the log odds.

We derive (4) using the gradients ${\overline{s}}_S={\mathrm{\nabla }}_{\theta }\log \underset{\theta }{Pr}(S)$ from Section 3.

Differentiate the log ratio.

For $p_H,p_G>0$⁠, the log odds $z=\log (p_H/p_G)$ are finite, and differentiating along the flow gives $$\dot{z}=\frac{{\dot{p}}_H}{p_H}-\frac{{\dot{p}}_G}{p_G}.$$(29) The rate therefore compares the proportional growth of the two groups. This identity needs only differentiability of the group probabilities.

To express each proportional rate through the policy, we use the score $s_{\theta }(x,y):$$={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$⁠. For a fixed group $S$ with positive probability and indicator ${\mathbf{1}}_S$⁠, the conditions for differentiating expectations give $${\mathrm{\nabla }}_{\theta }\underset{\theta }{Pr}(S)={\mathbb{E}}_{\theta }[{\mathbf{1}}_S{s}_{\theta }]=\underset{\theta }{Pr}(S){\overline{s}}_S,{\overline{s}}_S:=\frac{{\mathbb{E}}_{\theta }[{\mathbf{1}}_S{s}_{\theta }]}{\underset{\theta }{Pr}(S)}={\mathbb{E}}_{\theta }[s_{\theta }\mid S].$$(30) The gradient of $\log \underset{\theta }{Pr}(S)$ is thus the mean score ${\overline{s}}_S$ of the group. Applying this to $H$ and $G$⁠, and using $\dot{\theta }=g_R$⁠, gives $${\mathrm{\nabla }}_{\theta }z={\overline{s}}_H-{\overline{s}}_G,\dot{z}=({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}{g}_R.$$(31) The log odds therefore grow precisely when $g_R^{\mathrm{\top }}{\overline{s}}_H>g_R^{\mathrm{\top }}{\overline{s}}_G$⁠.

Decomposition of the growth rate.

Assume $G,H,N$ all have positive probability and evaluate quantities at the current policy.

Step 1: Use normalization.

Differentiating the accepted probability and the sum of all group probabilities gives $$g_R=p[(1-q){\overline{s}}_G+q{\overline{s}}_H],g_R+(1-p){\overline{s}}_N=0.$$(32) Combining these identities yields $$g_R=p(1-p)[(1-q){\overline{s}}_G+q{\overline{s}}_H-{\overline{s}}_N].$$(33)

Step 2: Substitute into the rate of the log odds.

Using (4) and collecting ${\overline{s}}_H-{\overline{s}}_G$ gives $$\begin{array}{rl}\dot{z}& =p(1-p)({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}[(1-q){\overline{s}}_G+q{\overline{s}}_H-{\overline{s}}_N]\\ & =p(1-p)[\underset{\text{correctness-to-hack leakage}}{\underbrace{({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)}}+\underset{\text{hack bias}}{\underbrace{q\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2}}],\end{array}$$(34) which establishes (5) and reveals the mechanisms. The first term can favor or oppose hacking. The second is nonnegative and proportional to $q$ at fixed $p$ and gradients; those quantities also change during training. Hence growth depends on the full bracket, without implying acceleration or growth at every policy.

Step 3: State the growth condition.

When ${\overline{s}}_H\ne {\overline{s}}_G$⁠, the bracket is positive exactly when $$q>-\frac{({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)}{\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2}.$$(35) The threshold may lie outside $(0,1)$ and change during training. If the two gradients agree, $\dot{z}=0$⁠. Thus the current share alone does not determine growth. The alignment of the gradients also matters.

Simultaneous growth of reward and hacking

For $p_H,p_G>0$⁠, differentiating $z=\log (q/(1-q))$ and using the verifier flow gives $$\dot{q}=q(1-q)\dot{z},\dot{p}=\parallel g_R{\parallel }^2.$$(36) Since $p_G,p_H,p_N>0$⁠, both $p$ and $q$ lie in $(0,1)$⁠. Thus, $\dot{q}>0$ if and only if the bracket in (5) is positive. By (4), a positive $\dot{z}$ requires $g_R\ne 0$⁠. Both $\dot{q}$ and $\dot{p}$ are then strictly positive, completing the proof of Proposition 3.2. $◻$

Example: Shared normalization can favor hacking.

For one prompt with a response in each group, use softmax logits $\theta =({\theta }_G,{\theta }_H,{\theta }_N)$⁠: $$\underset{\theta }{Pr}(S)=\frac{e^{{\theta }_S}}{e^{{\theta }_G}+e^{{\theta }_H}+e^{{\theta }_N}},S\in \{G,H,N\}.$$(37) Here $z={\theta }_H-{\theta }_G$ and $$\begin{array}{rl}{g}_R& =(p_G(1-p),p_H(1-p),-p(1-p){)}^{\mathrm{\top }},\\ \dot{z}& =(1-p)(p_H-p_G).\end{array}$$(38) Thus $p_H>p_G$ gives simultaneous growth of the hacked share and verifier acceptance on a sufficiently short interval. The example illustrates a mechanism caused by the parameterization; it does not assert that every neural policy follows this pattern.

C.3. Technical Details for Section 4

C.3.1. Comparison Setup

Fix the prompt distribution, verifier, policy family, initialization, and training procedure. The record $L_t$ contains prompts, responses, verifier rewards, policy updates, and quantities computed from these observations. Training and monitoring may adapt to the record, but neither uses the hidden correctness assignment or additional correctness feedback.

We compare two fixed correctness assignments: $$c_0=R,c_1\in {\mathcal{C}}_R\text{with}\underset{\theta }{Pr}(R=1,c_1=0)>0,$$(39) at the policy $\theta$⁠. Under $c_0$⁠, every accepted response is correct. Under $c_1$⁠, some accepted responses are hacks.

The assignments are fixed before training. We compare what the same procedure observes under each assignment. Correctness does not change during either run. For the deterministic argument below, fix all random choices used by training and use those same choices in both runs.

C.3.2. Proof of Proposition 4.1

The proof has three steps: the assignments produce the same record, the monitor therefore makes the same decision, but correct detection requires different decisions.

Step 1: The training records are identical.

We argue by induction on the training step. Both runs start from the same initialization, so their initial records agree. Suppose their records agree through step $n$⁠. Because the training procedure depends only on the record and the shared random choices, both runs select the same prompt and sample the same response from the same policy. The fixed verifier returns the same reward. Both runs therefore make the same update and append the same information to the record, so their records agree through step $n+1$⁠.

By induction, the records are identical at every training step: $$L_n^{(c_0)}=L_n^{(c_1)}\text{for every }n.$$(40) The argument also covers verifier queries that depend on earlier results and any quantity computed from the record. It likewise covers $J_R$ and its gradients, which depend on the verifier and the policy but not on which correctness assignment in ${\mathcal{C}}_R$ is true. For training in continuous time, we assume that the fixed procedure determines a unique trajectory. Because this trajectory depends only on the verifier and the policy, changing the correctness assignment within ${\mathcal{C}}_R$ leaves it unchanged. This argument applies to any two fixed assignments in ${\mathcal{C}}_R$⁠. It does not require either assignment to equal $R$⁠.

Step 2: The monitor makes the same decision.

A deterministic detection monitor computes a binary alarm $D_t={\delta }_t(L_t)$⁠, where $D_t=1$ indicates the presence of hacks. Identical records give identical alarms: $$D_t^{(c_0)}={\delta }_t(L_t^{(c_0)})={\delta }_t(L_t^{(c_1)})=D_t^{(c_1)}.$$(41)

Step 3: The same decision cannot be correct in both cases.

Under $c_0$⁠, there are no hacks, so a correct detector must remain silent. Under $c_1$⁠, hacks have positive probability, so a correct detector must raise an alarm. Because the monitor makes the same decision in both cases, it either raises a false alarm under $c_0$ or misses the hacks under $c_1$⁠.

Thus, no deterministic monitor using only $L_t$ can guarantee correct detection under both compatible assignments. More observations from the same verifier do not resolve this ambiguity: Step 1 continues to apply as the record grows.

Scope.

This conclusion concerns a guarantee over the stated class ${\mathcal{C}}_R$⁠. A monitor may succeed under a particular assignment. Additional knowledge connecting response content to correctness may also rule out alternatives. The impossibility applies when the two assignments remain admissible and the training procedure never observes information that distinguishes them.

Remark on randomization

Randomization does not change the conclusion. Use the same random choices for training and monitoring in the two runs. For every such choice, the records and alarms remain identical. Averaging over these choices therefore gives $$\underset{c=c_1}{Pr}(D_t=1)=\underset{c=c_0}{Pr}(D_t=1).$$(42) Here the probabilities are over training and monitor randomness. Each correctness assignment remains fixed. The left side is the detection rate when hacks exist, and the right side is the false-alarm rate when they do not. Thus, the monitor has no detection advantage between these two assignments.

Likewise, the training records have the same distribution: $${\mathrm{Law}}_{c=c_1}(L_t)={\mathrm{Law}}_{c=c_0}(L_t),$$(43) where ${\mathrm{Law}}_c(L_t)$ denotes the distribution of the record over training randomness under assignment $c$⁠. This proves the proposition. $◻$

C.3.3. Proof of Proposition 4.2

The proof compares two compatible assignments that disagree on every accepted response. Both admit hacks, so knowing that hacks exist does not distinguish them.

Step 1: Construct two opposite correctness assignments.

By the premise, choose a fixed assignment $c_1\in {\mathcal{C}}_R$ under which the policy $\theta$ assigns positive probability to both correct responses and hacks. Define $$c_2(x,y)=R(x,y)-c_1(x,y).$$(44) On rejected responses, both assignments equal zero. On accepted responses, $c_2=1-c_1$⁠, so the assignments give opposite correctness labels. Thus $c_2\in {\mathcal{C}}_R$⁠, and the two assignments exchange the correct and hacked groups: $$p_{H,c_2}(\theta )=p_{G,c_1}(\theta )>0,p_{G,c_2}(\theta )=p_{H,c_1}(\theta )>0.$$(45) In particular, both assignments admit hacks with positive probability.

Step 2: The monitor makes the same prediction.

Fix any accepted pair $(x,y)$ and any deterministic monitor using only verifier feedback records. As shown in Appendix C.3.2, the two assignments produce identical training records when the same random choices are used. The monitor therefore makes the same prediction in both cases: $${\hat{c}}_t^{(c_1)}(x,y)={\hat{c}}_t^{(c_2)}(x,y).$$(46)

Step 3: The prediction cannot be correct under both assignments.

Because $R(x,y)=1$⁠, the true labels satisfy $$c_2(x,y)=1-c_1(x,y).$$(47) The monitor’s common binary prediction therefore matches exactly one of the two labels. It must be incorrect under the other assignment.

Thus, no deterministic monitor can guarantee the correct label of an accepted response under every compatible assignment. Knowing that hacks exist does not resolve the ambiguity, because both assignments satisfy that additional information.

Randomization and the error bound.

The same reasoning applies when training or monitoring is randomized. Use the same random choices in both runs. For every such choice, the monitor gives the same prediction under both assignments, and exactly one assignment makes that prediction wrong.

Averaging over these choices gives $$\underset{c=c_1}{Pr}({\hat{c}}_t(x,y)\ne c_1(x,y))+\underset{c=c_2}{Pr}({\hat{c}}_t(x,y)\ne c_2(x,y))=1.$$(48) Here the probabilities are over training and monitor randomness, with the queried pair and correctness assignments held fixed. At least one of these error probabilities is therefore at least $1/2$⁠.

We fixed both candidate assignments before training, so neither depends on the realized training record. Which of the two attains the bound may depend on the monitor and the queried pair, but not on that record. Because both assignments contain hacks, the proposition holds even when the monitor knows that hacks exist. This completes the proof .$◻$

C.3.4. Proof of Proposition 4.3

Controlled dynamics and the selective objective.

Consider a controller that chooses a correction $u(t)$ using only the training record $L_t$⁠. Training then follows $$\dot{\theta }=g_R+u(t).$$(49) We assume that the controlled trajectory is well defined and that the group probabilities are differentiable along it. For a deterministic argument, fix all random choices used during training.

Under a correctness assignment $c$⁠, selective control requires $${\dot{p}}_{H,c}<0,{\dot{p}}_{G,c}\ge 0\text{whenever }{p}_{H,c}>0.$$(50) These conditions concern the full update: $${\dot{p}}_{H,c}={\mathrm{\nabla }}_{\theta }{p}_{H,c}^{\mathrm{\top }}(g_R+u),{\dot{p}}_{G,c}={\mathrm{\nabla }}_{\theta }{p}_{G,c}^{\mathrm{\top }}(g_R+u).$$(51) A correction that opposes hack growth is insufficient unless the full update $g_R+u(t)$ reduces hack probability while preserving correctness. A uniform guarantee requires the same controller rule to satisfy these conditions under every compatible correctness assignment and at every time along the evolving policy trajectory whenever hack probability is positive. The correction itself may adapt to the training record.

Impossibility of guaranteed selective control.

We compare two assignments that exchange the correct and hacked groups. The controller sees the same record under both assignments, but selective control requires opposite changes in their probabilities.

Step 1: Exchange the correct and hacked groups.

By the premise, choose a fixed assignment $c_1\in {\mathcal{C}}_R$ under which the initial policy assigns positive probability to both correct responses and hacks. Define $$c_2=R-c_1.$$(52) As in Appendix C.3.3, both assignments are compatible with the verifier, and they exchange the two accepted groups: $$p_{H,c_2}=p_{G,c_1},p_{G,c_2}=p_{H,c_1}.$$(53) Both groups have positive probability initially. By continuity, they remain positive on a sufficiently short initial interval.

Step 2: The controller produces the same update.

We extend the argument of Appendix C.3.2, which shows that the two runs produce identical records, to include the controller. Both runs begin at the same policy. Whenever their records agree, the controller chooses the same correction, because it uses only the record. The verifier gradient also agrees, because it depends on the policy and the verifier but not on the correctness assignment. Both runs therefore make the same corrected update, and their records remain identical.

Thus, under both assignments, the fixed training and control procedures generate the same record and the same trajectory of the controlled policy. In particular, both runs have the same velocity $\dot{\theta }=g_R+u$ at every time.

Step 3: Selective control requires incompatible signs.

Along this common trajectory, exchanging the groups also exchanges their derivatives: $${\dot{p}}_{H,c_2}={\dot{p}}_{G,c_1},{\dot{p}}_{G,c_2}={\dot{p}}_{H,c_1}.$$(54) On the initial interval where both groups remain positive, selectivity under the two assignments therefore requires $$\begin{array}{rlr}{c}_1:& {\dot{p}}_{H,c_1}<0,& {\dot{p}}_{G,c_1}\ge 0,\\ c_2:& {\dot{p}}_{G,c_1}<0,& {\dot{p}}_{H,c_1}\ge 0.\end{array}$$(55) Each derivative would have to be both negative and nonnegative. No update can satisfy both rows.

Thus, no controller that uses only the training record can guarantee selective control under every correctness assignment in ${\mathcal{C}}_R$⁠. The obstruction appears already on an initial interval of training, so it does not depend on behavior later in training.

Remark on randomization.

The same obstruction applies to a randomized controller. We couple the two runs by using the same random choices for training and control under both assignments. For each realization, the records, corrections, and trajectories then coincide, so no realization satisfies the requirements for selective control under both assignments.

A guarantee that holds with probability one under both assignments would require both sets of conditions to hold on a common event of probability one. Since no realization satisfies both, that event is empty, so no such guarantee exists. Randomization cannot help, because the two candidate assignments are fixed before training and the random choices therefore carry no information that distinguishes them. The proposition follows.$◻$

Scope.

The result rules out a uniform guarantee over ${\mathcal{C}}_R$⁠, but a controller may still achieve selective control under a particular assignment. Additional information or assumptions can distinguish assignments in ${\mathcal{C}}_R$⁠, and if they exclude either of the two assignments used in the proof, the argument above no longer applies.

Connection to secure control.

The argument here uses an indistinguishability principle also used when studying attack detection and identification in cyber-physical systems: no monitor can distinguish admissible scenarios that produce identical observations when it uses only those observations (Pasqualetti, Dörfler, and Bullo 2013). In our paper, the indistinguishable scenarios are correctness assignments that produce the same training record $L_t$⁠. Exchanging the correct and hacked groups between two such assignments makes their requirements for selective control incompatible. The argument therefore identifies the information that a uniform guarantee needs: observations or structural assumptions that exclude one of these assignments. In power grid security, models of grid operation supply such structural correctness knowledge: by relating coordinated cyberattacks to their physical consequences, they tell defenders which scenarios cause harm and let them target defense accordingly (Moya and Wang 2018). Audits can play an analogous role by supplying correctness information that distinguishes otherwise compatible assignments. The subsequent analysis examines when this information supports selective control.

C.3.5. Regularization and Its Limitations

We analyze three regularization penalties and distinguish what each controls from what it guarantees about correctness. Throughout, we use exact gradient ascent, fixed penalty weights, and the fixed verifier from Section 2. We then apply Proposition 4.3 to determine which guarantees of selective control these penalties can provide.

KL divergence from a reference

For a fixed reference policy under the same prompt distribution, define $$K(\theta )={\mathbb{E}}_{x\sim \mathcal{D}}{D}_{\mathrm{KL}}({\pi }_{\theta }(\cdot \mid x)\Vert {\pi }_{\mathrm{ref}}(\cdot \mid x)),F_{\beta }=J_R-\beta K,$$(56) where $\beta >0$ is fixed.

Step 1: Bound deviation from the reference.

Assume the flow remains in a region where $K$ is finite and continuously differentiable. Gradient ascent on $F_{\beta }$ gives $$\dot{\theta }=g_R-\beta {\mathrm{\nabla }}_{\theta }K,{\dot{F}}_{\beta }=\parallel {\mathrm{\nabla }}_{\theta }{F}_{\beta }{\parallel }^2\ge 0.$$(57) Thus $F_{\beta }(\theta (t))\ge F_{\beta }(\theta (0))$⁠. Rearranging and using $J_R=p\le 1$ yields $$\begin{array}{rl}K(\theta (t))& \le K(\theta (0))+\frac{p(\theta (t))-p(\theta (0))}{\beta }\\ & \le K(\theta (0))+\frac{1-p(\theta (0))}{\beta }.\end{array}$$(58) In particular, initialization at the reference gives $K(\theta (0))=0$⁠. The penalty therefore bounds departure from the reference along the exact flow.

Step 2: Bound changes in hack probability.

Because the prompt distribution is fixed, $K$ also equals the KL divergence between the joint distributions of prompts and responses. Pinsker’s inequality therefore gives, for the fixed hacked set $H$⁠, $$|p_H(\theta )-p_{H,\mathrm{ref}}|\le \sqrt{\frac{K(\theta )}{2}},p_{H,\mathrm{ref}}:=\underset{\mathrm{ref}}{Pr}(H).$$(59) Thus, closeness to the reference limits the change in hack probability.

Limitation.

The penalty bounds how much the policy changes, not the direction of that change. It therefore guarantees neither ${\dot{p}}_H<0$ nor ${\dot{p}}_G\ge 0$⁠. The bound controls hack probability relative to $p_{H,\mathrm{ref}}$⁠. It implies a small absolute hack probability when both the reference’s hack probability and the KL bound are small. The penalty depends on the policy and the reference but not on the correctness assignment, so with the same reference under every assignment in ${\mathcal{C}}_R$⁠, Proposition 4.3 still applies.

Penalizing the gradient of reward.

Following the objective of Ackermann et al. (2026), we consider the idealized gradient regularization penalty $$F_{\mathrm{GR}}=J_R-\gamma \parallel g_R{\parallel }^2,$$(60) with fixed $\gamma >0$⁠.

Step 1: Derive the correction.

Assume $J_R$ is twice continuously differentiable and write $B_R:={\mathrm{\nabla }}_{\theta }^2{J}_R$⁠. Since $B_R$ is symmetric, $${\mathrm{\nabla }}_{\theta }\parallel g_R{\parallel }^2=2B_R{g}_R.$$(61) Gradient ascent on $F_{\mathrm{GR}}$ therefore gives $$\dot{\theta }=g_R-2\gamma B_R{g}_R.$$(62) The correction depends entirely on the verifier objective and its derivatives.

Step 2: Separate reward sensitivity from correctness.

The norm $\parallel g_R\parallel$ measures first-order sensitivity of expected verifier reward to parameter changes. It does not by itself determine correctness. For example, if the verifier accepts every response, then $J_R\equiv 1$ and $g_R\equiv 0$⁠. The correction vanishes even if the policy assigns positive probability to incorrect responses.

Scope of the cited analysis.

Ackermann et al. (2026) connect flatter optima to the accuracy of the proxy reward under additional assumptions: continuous actions, a Gaussian policy with fixed covariance, regularity conditions on the policy and reward functions, and a true reward that is Lipschitz continuous. Our setup does not impose these policy and reward assumptions, and ${\mathcal{C}}_R$ places no corresponding regularity restriction on correctness assignments. Thus, the cited results do not establish selective control uniformly over ${\mathcal{C}}_R$⁠.

Exchanging correctness assignments in ${\mathcal{C}}_R$ leaves $J_R$⁠, its derivatives, and the correction unchanged, so Proposition 4.3 applies to this regularizer. This conclusion does not contradict the theoretical results of Ackermann et al. (2026), which hold under the additional assumptions above, or their empirical improvements, which concern particular tasks rather than a uniform guarantee over ${\mathcal{C}}_R$⁠.

Stability of relative probabilities.

Motivated by tie training for reducing reliance on spurious features in preference optimization (Moya et al. 2026), we analyze a simplified penalty on changes in relative response probabilities. The penalty discourages deviations from a fixed reference ratio between two verifier-accepted responses. Unlike pairs constructed to have equal utility, equal verifier rewards alone do not establish that the responses are equally correct. We therefore examine what this penalty guarantees when pair selection uses only verifier information.

Fix two accepted responses $y_1,y_2$ to a prompt $x$⁠, with positive probabilities under the current and reference policies. Define $$e(\theta )=\log \frac{{\pi }_{\theta }(y_1\mid x)}{{\pi }_{\theta }(y_2\mid x)}-\log \frac{{\pi }_{\mathrm{ref}}(y_1\mid x)}{{\pi }_{\mathrm{ref}}(y_2\mid x)}.$$(63) Thus $e=0$ means that the current relative probability matches the reference ratio. Assume $e$ is continuously differentiable on a neighborhood of the trajectory.

Step 1: Derive the restoring term.

For fixed $\lambda >0$⁠, gradient ascent on $$F_{\mathrm{I}}=J_R-\frac{\lambda }{2}{e}^2$$(64) adds the correction $$u(t)=-\lambda e{\mathrm{\nabla }}_{\theta }e.$$(65) Along the corrected flow, $$\dot{e}=b-\lambda ae,b:={\mathrm{\nabla }}_{\theta }{e}^{\mathrm{\top }}{g}_R,a:=\parallel {\mathrm{\nabla }}_{\theta }e{\parallel }^2.$$(66) The term $b$ is the change in the log ratio induced by verifier training. The term $-\lambda ae$ opposes deviation from the reference ratio.

Step 2: Bound the deviation.

Consider an interval $[0,T]$ on which the flow exists and both response probabilities remain positive. Suppose $$a(t)\ge m>0,|b(t)|\le B\text{for }0\le t\le T.$$(67) An integrating factor gives $$e(t)=\exp (-\lambda {\int }_0^{t}a(s)ds)e(0)+{\int }_0^t\exp (-\lambda {\int }_{\tau }^{t}a(s)ds)b(\tau )d\tau .$$(68) Using the bounds on $a$ and $b$⁠, $$\begin{array}{rl}|e(t)|& \le e^{-\lambda mt}|e(0)|+B{\int }_0^t{e}^{-\lambda m(t-\tau )}d\tau \\ & =e^{-\lambda mt}|e(0)|+\frac{B}{\lambda m}(1-e^{-\lambda mt}).\end{array}$$(69) The bound separates two contributions: one from the initial deviation, which decays, and one from the drift $b$ that the verifier induces. If $b\equiv 0$⁠, the deviation decays exponentially. If the drift persists, the bound allows a nonzero deviation to persist as well.

This is a scalar estimate of the form used in input-to-state stability analysis (Khalil and Grizzle 2002). Here, the conclusion concerns $e$ under the stated bounds. It guarantees neither stability of all policy parameters nor exact preservation of the ratio.

Limitation: Restoration can increase errors.

Consider one prompt with two responses, $y_1$ correct and $y_2$ incorrect, both accepted. Let $$p_G(\theta )=\frac{1}{1+e^{-\theta }},p_H(\theta )=1-p_G(\theta ).$$(70) For a reference parameter ${\theta }_{\mathrm{ref}}$⁠, we have $J_R\equiv 1$ and $e=\theta -{\theta }_{\mathrm{ref}}$⁠. Thus, $$\dot{\theta }=-\lambda e,{\dot{p}}_G=-\lambda e{p}_G{p}_H,{\dot{p}}_H=\lambda e{p}_G{p}_H.$$(71) If $\theta (0)>{\theta }_{\mathrm{ref}}$⁠, then $e(t)=e(0)e^{-\lambda t}>0$ at every finite time. Restoration therefore decreases correctness and increases hack probability while reducing the deviation from the reference ratio.

The current policy initially favors the correct response more strongly than the reference does, so restoring the reference reverses that improvement. If initialized at the reference instead, the flow remains stationary and does not strictly reduce hack probability.

Common limit of the three penalties.

We hold the reference policies, penalty weights, and selected response pairs fixed across correctness assignments and give the penalties no additional correctness information. At the same policy, each penalty and its gradient then agree under every assignment in ${\mathcal{C}}_R$⁠. The argument about selective control above, which shows that the records coincide, therefore gives the same controlled trajectory under the compared assignments.

Under the assumptions of Proposition 4.3, none of these penalties can guarantee ${\dot{p}}_H<0$ and ${\dot{p}}_G\ge 0$ along the trajectory under every assignment in ${\mathcal{C}}_R$ whenever $p_H>0$⁠. A penalty may control the departure from a reference, the sensitivity of the reward, or the relative probabilities of paired responses without providing this uniform guarantee of selective control, although it may still succeed on particular tasks.

Additional correctness information or justified structural assumptions can restrict ${\mathcal{C}}_R$ and exclude indistinguishable assignments that require incompatible corrections. Excluding these assignments removes the obstruction exhibited here, even without identifying every hack. Removing the obstruction does not by itself give a guarantee, however: a guarantee of selective control also requires showing that the available updates satisfy ${\dot{p}}_H<0$ and ${\dot{p}}_G\ge 0$ throughout the trajectory.

C.4. Technical Details for Section 5

C.4.1. Projected Correction and Its Immediate Effects

Write $h_A={\mathrm{\nabla }}_{\theta }{p}_{H_A}$ and $h={\mathrm{\nabla }}_{\theta }{p}_H$⁠. Since $p=p_H+p_G$⁠, we have ${\mathrm{\nabla }}_{\theta }{p}_G=g_R-h$⁠. We first derive the effects of the projected correction without assuming full audit coverage.

Step 1: Preserve instantaneous acceptance growth.

Define $$P_{\perp }=\{\begin{array}{ll}{I}_d-{\displaystyle \frac{g_R{g}_R^{\mathrm{\top }}}{\parallel g_R{\parallel }^2}},& g_R\ne 0,\\ I_d,& g_R=0.\end{array}$$(72) This orthogonal projector satisfies $P_{\perp }^{\mathrm{\top }}=P_{\perp }$⁠, $P_{\perp }^2=P_{\perp }$⁠, and $g_R^{\mathrm{\top }}{P}_{\perp }=0$⁠. For the correction $u=-\lambda P_{\perp }{h}_A$⁠, with $\lambda >0$⁠, we therefore have $$g_R^{\mathrm{\top }}u=0.$$(73) Along the corrected flow $\dot{\theta }=g_R+u$⁠, the chain rule gives $$\dot{p}=g_R^{\mathrm{\top }}(g_R+u)=\parallel g_R{\parallel }^2.$$(74) Thus the correction preserves the instantaneous acceptance growth produced by the verifier gradient at the current policy.

Step 2: Oppose growth of audited hacks.

Symmetry and idempotence of the projector give $$h_A^{\mathrm{\top }}u=-\lambda h_A^{\mathrm{\top }}{P}_{\perp }{h}_A=-\lambda \parallel P_{\perp }{h}_A{\parallel }^2.$$(75) Consequently, $${\dot{p}}_{H_A}=h_A^{\mathrm{\top }}{g}_R-\lambda \parallel P_{\perp }{h}_A{\parallel }^2.$$(76) The correction contributes a strictly negative term when $P_{\perp }{h}_A\ne 0$⁠. The full rate is negative only when this contribution outweighs any positive growth induced by the verifier gradient.

Step 3: Relate the correction to the full population.

For the full hack probability, the correction contributes $$h^{\mathrm{\top }}u=-\lambda (P_{\perp }h{)}^{\mathrm{\top }}(P_{\perp }{h}_A).$$(77) Because $g_R^{\mathrm{\top }}u=0$⁠, its contribution to correct responses is equal and opposite: $${\mathrm{\nabla }}_{\theta }{p}_G^{\mathrm{\top }}u=(g_R-h{)}^{\mathrm{\top }}u=-h^{\mathrm{\top }}u.$$(78) Thus, when the two projected gradients have positive inner product, the correction opposes hack growth and contributes equally to correct-response growth. These statements concern the correction’s contribution. Selectivity of the full update also depends on $g_R$⁠.

If $P_{\perp }h=0$⁠, every correction orthogonal to $g_R$ has zero instantaneous effect on $p_H$⁠. The projection therefore requires a component of the hack gradient orthogonal to the reward gradient. The theorem below shows how full coverage and sufficient corrective strength turn these local identities into selective control and a bound along the evolving policy.

C.4.2. Proof of Theorem 5.1

The proof has four steps. Full audit coverage identifies the gradient $h=\mathrm{\nabla }{p}_H$⁠. Projection removes the component of $h$ along $g_R$⁠, so the correction does not slow the growth of acceptance. A strong enough correction then gives ${\dot{p}}_H<0$ and ${\dot{p}}_G\ge 0$⁠. Finally, integrating these rates over $[0,T]$ bounds the hacked share.

Step 1: Full coverage identifies the gradient of $p_H$⁠.

Under (A2), unaudited hacks have zero probability along the policy trajectory: $$p_H-p_{H_A}=0.$$(79) As a function of $\theta$⁠, this difference is nonnegative everywhere, because audited hacks form a subset of all hacks. At every point of the trajectory, it therefore attains its minimum value of zero, and its gradient vanishes: $$\mathrm{\nabla }{p}_{H_A}=\mathrm{\nabla }{p}_H.$$(80) Thus the audited population gradient coincides with the full hack gradient along the trajectory. The correction becomes $u=-\lambda P_{\perp }h$⁠, where $P_{\perp }$ projects onto the orthogonal complement of $g_R$⁠, and we write $S=\parallel P_{\perp }h{\parallel }^2$ for the squared norm of the projected gradient.

Step 2: Derive the corrected dynamics.

The corrected flow is $\dot{\theta }=g_R+u=g_R-\lambda P_{\perp }h$⁠. Because $P_{\perp }$ is an orthogonal projection onto the complement of $g_R$⁠, we have $g_R^{\mathrm{\top }}{P}_{\perp }h=0$ and $h^{\mathrm{\top }}{P}_{\perp }h=\parallel P_{\perp }h{\parallel }^2=S$ (Appendix C.4.1). Differentiating along the flow therefore gives $$\dot{p}=\parallel g_R{\parallel }^2,{\dot{p}}_H=h^{\mathrm{\top }}{g}_R-\lambda S,{\dot{p}}_G=\parallel g_R{\parallel }^2-{\dot{p}}_H.$$(81) At the current policy, the correction preserves the instantaneous acceptance growth, lowers the rate ${\dot{p}}_H$ by $\lambda S$⁠, and raises the rate ${\dot{p}}_G$ by the same amount.

To compare the growth of the two groups, recall $z=\log (p_H/p_G)$⁠. Without the correction, $z$ changes at the rate $$b=({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}{g}_R,$$(82) which we call the pressure that verifier rewards exert on the hacked share. Taking the logarithmic derivative gives $$\begin{array}{rl}\dot{z}& =\frac{{\dot{p}}_H}{p_H}-\frac{{\dot{p}}_G}{p_G}\\ & =b-\lambda S(\frac{1}{p_H}+\frac{1}{p_G})\\ & =b-\frac{\lambda S}{pq(1-q)}.\end{array}$$(83) Since $\dot{q}=q(1-q)\dot{z}$⁠, we also obtain $$\dot{q}=q(1-q)b-\frac{\lambda S}{p}.$$(84) These expressions separate the verifier’s growth pressure from the opposing effect of the correction.

Step 3: Identify when the update is selective.

The correction lowers ${\dot{p}}_H$ by $\lambda S$ and raises ${\dot{p}}_G$ by the same amount, so $$\dot{z}=\frac{{\dot{p}}_H}{p_H}-\frac{{\dot{p}}_G}{p_G}=b-\lambda S(\frac{1}{p_H}+\frac{1}{p_G}).$$(85) If the correction term exceeds $b$⁠, then $\dot{z}<0$⁠, and hence $\dot{q}<0$⁠, since $z$ increases with $q$⁠. Differentiating $p_G=p(1-q)$ and using $\dot{p}=\parallel g_R{\parallel }^2$ then gives $${\dot{p}}_G=(1-q)\parallel g_R{\parallel }^2-p\dot{q}>0.$$(86) Because acceptance never decreases under the correction, a falling hacked share implies rising correctness.

A falling hacked share does not require $p_H$ itself to fall, since acceptance may still grow. For this stronger conclusion, the correction must overcome the verifier’s contribution to ${\dot{p}}_H$⁠: $$\lambda S>h^{\mathrm{\top }}{g}_R.$$(87) Then ${\dot{p}}_H<0$ and ${\dot{p}}_G=\parallel g_R{\parallel }^2-{\dot{p}}_H>0$⁠. Because $p_H$ falls while $p_G$ rises, the log ratio $z$ decreases as well, and part (i) follows.

Step 4: Bound the hacked share during training.

Because $q$ is the logistic function of $z$⁠, the rate from Step 3 gives $$\dot{q}=q(1-q)\dot{z}=q(1-q)b-\frac{\lambda S}{p},$$(88) where the second term uses $p_H=pq$ and $p_G=p(1-q)$⁠. Assumption (A3) bounds the correction from below, $S/p\ge \kappa q$⁠, and the pressure from verifier rewards is bounded above by $b\le D$ with $D\ge 0$⁠. Since $q(1-q)\le 1/4$⁠, these bounds give $$\dot{q}\le \frac{D}{4}-\lambda \kappa q.$$(89) The first term caps the pressure from verifier rewards, and the second is a correction proportional to the current hacked share.

Rearranging gives $\dot{q}+\lambda \kappa q\le D/4$⁠, and multiplying by the integrating factor $e^{\lambda \kappa t}$ gives $$\frac{d}{dt}(e^{\lambda \kappa t}q(t))\le \frac{D}{4}{e}^{\lambda \kappa t}.$$(90) Integrating from $0$ to $t$ and rearranging yields $$q(t)\le e^{-\lambda \kappa t}q(0)+\frac{D}{4\lambda \kappa }(1-e^{-\lambda \kappa t}),t\in [0,T].$$(91) The initial hacked share contributes a term that decays at rate $\lambda \kappa$⁠, while the pressure from verifier rewards contributes a term that rises toward $D/(4\lambda \kappa )$⁠. Part (ii) follows, which completes the proof of the theorem. $◻$

C.4.3. Estimating the gradient of $p_H$ from audits

We fix the current policy and assume that audits reveal exact correctness labels. The estimator weights each observed hack by the inverse of its audit probability, which compensates for hacks that are generated but not audited.

Step 1: Express the gradient as an expectation.

With the policy score $s_{\theta }(x,y)$$={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$⁠, and since $R(1-c)$ indicates a hack, differentiating the probability of hacks gives $$h:={\mathrm{\nabla }}_{\theta }{p}_H={\mathbb{E}}_{\theta }[R(X,Y)(1-c(X,Y))s_{\theta }(X,Y)].$$(92) Each hack contributes its policy score, and all other responses contribute zero.

Step 2: Weight the observed contributions.

We draw $n$ independent pairs with $X_i\sim \mathcal{D}$ and $Y_i\sim {\pi }_{\theta }(\cdot \mid X_i)$⁠. We audit each accepted response with known probability ${\rho }_i=\rho (X_i,Y_i)>0$ and never audit rejected responses. The indicator $I_i$ records whether an audit occurs, and when $I_i=1$⁠, the audit reveals $C_i=c(X_i,Y_i)$⁠. The estimator is $${\xi }_i=\{\begin{array}{ll}{\displaystyle \frac{1-C_i}{{\rho }_i}}{s}_{\theta }(X_i,Y_i),& I_i=1,\\ 0,& I_i=0,\end{array}\hat{h}=\frac{1}{n}\sum _{i=1}^n{\xi }_i.$$(93) An audited hack receives weight $1/{\rho }_i$⁠, while audited correct responses and unaudited responses contribute zero. The average divides by all $n$ generated responses, including those that were not audited.

Step 3: Show why the weighting works.

For an accepted response, an audit occurs with probability ${\rho }_i$⁠, and the weight $1/{\rho }_i$ cancels this probability in expectation. For a rejected response, both sides are zero. Hence $$\mathbb{E}[{\xi }_i\mid X_i,Y_i]=R(X_i,Y_i)(1-c(X_i,Y_i))s_{\theta }(X_i,Y_i),$$(94) and averaging over the generated responses gives $$\mathbb{E}[\hat{h}]=h,$$(95) where the expectation covers both response generation and audit selection. Audits of only some responses therefore give an unbiased estimate of $h$⁠, provided that every accepted response has a positive audit probability.

Consequence for the projected correction.

We keep $g_R$ and $P_{\perp }$ exact and fix $\lambda >0$ before drawing the batch. Every realized correction $\hat{u}=-\lambda P_{\perp }\hat{h}$ then satisfies $g_R^{\mathrm{\top }}\hat{u}=0$⁠, so it preserves the instantaneous acceptance growth at the fixed policy. Moreover, $$\begin{array}{rl}\mathbb{E}[h^{\mathrm{\top }}(g_R+\hat{u})]& =h^{\mathrm{\top }}{g}_R-\lambda h^{\mathrm{\top }}{P}_{\perp }\mathbb{E}[\hat{h}]\\ & =h^{\mathrm{\top }}{g}_R-\lambda \parallel P_{\perp }h{\parallel }^2.\end{array}$$(96) At the fixed policy, the correction from a batch therefore changes $p_H$ at the same expected rate as the population correction. A single batch can still produce a direction that does not decrease $p_H$⁠.

Under the stated assumptions, the estimator is unbiased. Small audit probabilities, however, produce large weights $1/{\rho }_i$ and can increase sampling variability. The argument does not cover accepted responses with zero audit probability or audits with systematic label errors. Appendix A.2 discusses the practical consequences of both cases.

D. Experimental Details

D.1. The Gaussian Contextual Bandit Experiment

Motivation.

To test our theoretical predictions, we use a controlled contextual bandit in which we know the correctness of every response and can compute population gradients exactly.

Setup.

Policy.

The policy assigns each response a logit using shared parameters $\theta \in {\mathbb{R}}^4$⁠, fixed features $\varphi (x,y)\in {\mathbb{R}}^4$⁠, and a fixed offset $a(x,y)$⁠: $${\pi }_{\theta }(y\mid x)=\frac{\exp ({\theta }^{\mathrm{\top }}\varphi (x,y)+a(x,y))}{\sum _{y^{\prime }\in {\mathcal{Y}}_x}\exp ({\theta }^{\mathrm{\top }}\varphi (x,y^{\prime })+a(x,y^{\prime }))}.$$(97) Training changes only $\theta$⁠. Sharing parameters allows updates from different prompts to interact.

Initialization.

We initialize $\theta (0)=0$ and use offsets to set the initial acceptance $p(0)$ and hacked share $q(0)$⁠: $$a(x,y)=\{\begin{array}{ll}\log [p(0)(1-q(0))/16],& y\in G_x,\\ \log [p(0)q(0)/16],& y\in H_x,\\ \log [(1-p(0))/16],& y\in N_x.\end{array}$$(98) This gives $p_G(0)=p(0)(1-q(0))$⁠, $p_H(0)=p(0)q(0)$⁠, and $p_N(0)=1-p(0)$⁠, with equal probabilities within each group. Changing the offsets varies these probabilities while keeping the features fixed.

Prompt and Group Distributions.

Prompts and labels.

We use eight equally weighted prompts, each with 16 responses in each of three groups. Correct responses $G_x$ have $(R,c)=(1,1)$⁠, hacks $H_x$ have $(R,c)=(1,0)$⁠, and rejected responses $N_x$ have $(R,c)=(0,0)$⁠. Group membership and labels remain fixed during training.

Accepted features.

We construct features for hacks by centering Gaussian noise and adding the mean $(1,1,0,0{)}^{\mathrm{\top }}$⁠. We obtain features for correct responses by negating the second coordinate. Writing ${\varphi }_{S,x,j}$ for response $j$ in group $S_x$⁠, we draw $${\epsilon }_{x,j}\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,{0.25}^2{I}_4),j=1,\dots ,16,$$(99) and set $$\begin{array}{rl}{\varphi }_{H,x,j}& =(1,1,0,0{)}^{\mathrm{\top }}+{\epsilon }_{x,j}-\frac{1}{16}\sum _{k=1}^{16}{\epsilon }_{x,k},\\ {\varphi }_{G,x,j}& =\mathrm{diag}(1,-1,1,1){\varphi }_{H,x,j}.\end{array}$$(100) The empirical means are exactly $(1,1,0,0{)}^{\mathrm{\top }}$ for hacks and $(1,-1,0,0{)}^{\mathrm{\top }}$ for correct responses in every prompt.

Rejected features.

We draw eight Gaussian vectors per prompt and append copies with the second coordinate negated: $${\xi }_{x,j}\stackrel{\mathrm{iid}}{\sim }\mathcal{N}(0,{0.25}^2{I}_4),v_{x,j}={\xi }_{x,j},v_{x,j+8}=\mathrm{diag}(1,-1,1,1){\xi }_{x,j},$$(101) for $j=1,\dots ,8$⁠. We center these vectors and add the desired mean ${\mu }_N=(0,{\mu }_{N,2},0,0{)}^{\mathrm{\top }}$⁠: $${\varphi }_{N,x,j}=v_{x,j}-\frac{1}{16}\sum _{k=1}^{16}{v}_{x,k}+{\mu }_N,j=1,\dots ,16.$$(102) The empirical mean is therefore exactly ${\mu }_N$⁠. The experiments use ${\mu }_{N,2}\in \{-1.5,-0.5,0,0.5\}$⁠.

We draw the raw noise independently across prompts and between the accepted and rejected groups. Within each seed, we reuse these draws across initializations, rejected means, and learning rates.

Objective.

Training objective.

We maximize verifier acceptance: $$J_R(\theta )=\frac{1}{8}\sum _{x=1}^8\sum _{y\in {\mathcal{Y}}_x}{\pi }_{\theta }(y\mid x)R(x,y)=p(\theta ).$$(103) Correct responses and hacks receive the same reward. Correctness labels define the controlled initializations and evaluation metrics; they do not enter the training objective.

Training dynamics.

We evaluate the objective and gradient by summing over every prompt and response. We numerically integrate the verifier flow (1), $$\dot{\theta }=g_R={\mathrm{\nabla }}_{\theta }{J}_R,$$(104) using DOP853 until $T=50$⁠. For the learning rate ablation, we instead take discrete updates: $${\theta }_{k+1}={\theta }_k+\eta g_R({\theta }_k).$$(105) Both procedures use exact population gradients.

Metrics.

Acceptance and correctness.

We compute each group’s probability over all prompts: $$p_S(\theta )=\frac{1}{8}\sum _{x=1}^8\sum _{y\in S_x}{\pi }_{\theta }(y\mid x),S\in \{G,H,N\}.$$(106) We report acceptance $p=p_G+p_H$⁠, hacked share $q=p_H/p$⁠, and correctness $p_G$⁠. We form $q$ after aggregating over prompts. The proxy and true objectives are $J_R=p$ and $J_C=p_G$⁠.

Log odds and group scores.

We track $z=\log (p_H/p_G)$ and compute the group scores ${\overline{s}}_S={\mathrm{\nabla }}_{\theta }\log p_S$ exactly: $${\overline{s}}_S=\frac{1}{8p_S}\sum _{x=1}^8\sum _{y\in S_x}{\pi }_{\theta }(y\mid x)[\varphi (x,y)-\sum _{y^{\prime }\in {\mathcal{Y}}_x}{\pi }_{\theta }(y^{\prime }\mid x)\varphi (x,y^{\prime })].$$(107) We subtract each prompt’s mean feature before aggregating. Although the features remain fixed, the group scores change as the policy re-weights responses.

Hack bias and leakage.

We compute the two contributions in (5): $$\dot{z}=p(1-p)[\underset{\text{leakage}}{\underbrace{({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)}}+\underset{\text{hack bias}}{\underbrace{q\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2}}].$$(108) Their sum determines the sign of $\dot{z}$ and hence the direction of change in the hacked share. We evaluate the full expression within each seed before averaging.

Growth and reward hacking.

We compute the rates of acceptance, hacked share, and correctness: $$\dot{p}=\parallel g_R{\parallel }^2,\dot{q}=q(1-q)\dot{z},{\dot{p}}_G=(1-q)\dot{p}-p\dot{q}.$$(109) We identify declining correctness using $p\dot{q}>(1-q)\dot{p}$ and locate stationary correctness where ${\dot{p}}_G=0$⁠. These quantities distinguish growth of the hacked share from a decrease in the probability of correct responses.

Variation across seeds.

The appendix experiments use ten independent Gaussian feature draws and report means and one sample standard deviation. Each draw defines deterministic training. For the comparison of correctness against acceptance, we interpolate each trajectory onto acceptance values reached by every seed and geometry before computing these statistics.

D.1.1. When Hacking Grows

Controlled comparisons.

We fix $p(0)=2/3$⁠. In the main paper, panel (a) tracks $p$⁠, $q$⁠, and $p_G$ with $q(0)=0.3$ and ${\mu }_{N,2}=-0.5$⁠. Panel (b) compares $J_C$ against $J_R$ for $q(0)\in \{0.1,0.3,0.5\}$ using the same features.

The centered construction lets us vary initial leakage and hack bias separately. At initialization, $$({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)=-2(1+{\mu }_{N,2}),q(0)\parallel {\overline{s}}_H-{\overline{s}}_G{\parallel }^2=4q(0).$$(110) Changing the rejected mean changes initial leakage. Changing the initial hacked share changes initial hack bias while preserving the differences between group scores. The following experiments examine how these contributions evolve during training.

Results.

Experiment I: hack bias and correctness-to-hack leakage.

Starting from initial acceptance $p(0)=2/3$⁠, hack share $q(0)=0.3$⁠, and mean rejected strength ${\mu }_{N,2}=-0.5$⁠, we track hack bias, leakage from correctness to hacking, and their sum. For the same trajectories, we plot $\dot{z}(t)$ and compare it with central time differences of $z$⁠. Together, these experiments relate the competing contributions to the growth rate of hacking through the factor $p(\theta )(1-p(\theta ))$ in (5).

Figure 3(a) shows that hack bias increases while leakage from correctness to hacking remains near $-1$⁠. This value matches initialization, where the centered feature means give $$({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}({\overline{s}}_G-{\overline{s}}_N)=-2(1+{\mu }_{N,2})=-1.$$(111) Leakage stays near this value, consistent with the similar Gaussian spreads across groups: policy reweighting shifts their mean features similarly, leaving the contrasts between groups approximately unchanged. Negative leakage opposes growth of the hacked share, but hack bias outweighs it throughout the plotted interval. Their sum therefore stays positive, and Proposition 3.2 predicts $\dot{z}>0$ and an increasing hacked share. The predicted rate agrees with central time differences in Figure 3 (b). Thus, negative leakage can oppose hacking without overcoming the reinforcement from hack bias.

Ablation I: varying the mean rejected strength.

We keep $p(0)=2/3$ and $q(0)=0.3$ fixed and vary ${\mu }_{N,2}\in \{-1.5,-0.5,0.5\}$⁠, holding the accepted features and underlying Gaussian draws fixed. Since initial leakage equals $-2(1+{\mu }_{N,2})$⁠, these values give initial leakage of $+1$⁠, $-1$⁠, and $-3$⁠: positive, negative but weaker than the initial hack bias, and negative and stronger than it. For each condition, we plot correctness $J_C=p_G$ against verifier reward $J_R=p$⁠. Plotting against reward rather than time compares the conditions at the same acceptance level, which isolates how rejected responses affect correctness as verifier reward rises.

Figure 3 (c) shows how the mean rejected strength changes the relation between verifier reward and correctness. For ${\mu }_{N,2}=-1.5$⁠, where initial leakage is positive, correctness declines after a brief initial increase, so reward hacking emerges early. For ${\mu }_{N,2}=-0.5$⁠, where negative leakage is weaker than hack bias, acceptance and correctness first improve together, but correctness later decreases despite further gains in acceptance. For ${\mu }_{N,2}=0.5$⁠, where negative leakage is stronger, both quantities increase throughout the plotted range, with no reward hacking evident in the mean curve. These declines illustrate Proposition 3.1: reward hacking occurs when $p\dot{q}>(1-q)\dot{p}$⁠, that is, when the shift toward hacked responses outweighs the correctness gained from rising acceptance. Thus, even with identical initial acceptance and hacked share, the geometry of the rejected responses can change whether and when reward hacking emerges.

Ablation II: learning rate in exact gradient ascent.

To test how closely gradient flow approximates discrete training, we keep $p(0)=2/3$ and $q(0)=1/2$ fixed and vary the learning rate $\eta$⁠, holding the features and initialization fixed within each seed and geometry. We compare iterate $k$ of each discrete trajectory with the reference flow at the matched time $t=k\eta$⁠. For each seed, we take the maximum discrepancy in $z$⁠, $p$⁠, $q$⁠, and $p_G$ over times, and we then average these maxima across seeds.

Figure 3 (d) shows that discrepancies in $z$⁠, $p$⁠, $q$⁠, and $p_G$ increase with the learning rate, reaching the order of ${10}^{-2}$ at $\eta =0.4$⁠. Because the updates use exact population gradients, these discrepancies reflect only the effect of taking finite steps. As the learning rate decreases, the discrepancies shrink, which supports using gradient flow to approximate discrete training over the tested interval.

Takeaway.

The verifier rewards every accepted response, whether correct or hacked. Which group benefits therefore depends on the feature geometry, including the rejected responses that shape the direction in which acceptance increases. Along this direction, hack bias can overcome negative leakage and shift the accepted population toward hacked responses. Correctness falls once this shift outweighs the correctness gained from rising acceptance. Because discrete gradient ascent with small learning rates closely tracks gradient flow, the flow offers a reliable way to study this competition.

Hyperparameters.

Table 1 lists the experimental settings.

D.1.2. Limits of Verifier Feedback

We study the limits of selective control from verifier feedback (Section 4.4). We evaluate gradient regularization under two compatible correctness assignments along the same sequence of policies. The assignments exchange correct responses and hacks, so reducing hack probability under one reduces correctness under the other. We vary the regularization weight to measure the fraction of recorded times when control is selective under each assignment.

Additional experimental settings.

We use the policy and feature construction from Appendix D.1 to study control from verifier information, as described in Section 4.4.

Fixed settings.

We set $p(0)=2/3$⁠, $q(0)=1/2$⁠, ${\mu }_N=(0,-0.5,0,0{)}^{\mathrm{\top }}$⁠, and $\theta (0)=0$⁠. The offsets are $a(x,y)=\log (1/48)$⁠, giving $p_G(0)=p_H(0)=p_N(0)=1/3$⁠. We use ten independent feature draws and reuse each draw across regularization weights and correctness assignments. The verifier and initial policy remain fixed across comparisons.

Correctness assignments.

We keep the groups $G,H,N$ fixed and evaluate two assignments: $c_1={\mathbf{1}}_G$ and $c_2={\mathbf{1}}_H=R-c_1$⁠. Under $c_1$⁠, correct responses belong to $G$ and hacks belong to $H$⁠. Under $c_2$⁠, these roles reverse: $$p_{G,c_1}=p_G,p_{H,c_1}=p_H,p_{G,c_2}=p_H,p_{H,c_2}=p_G.$$(112) Both assignments use the same verifier. Correctness labels enter only the evaluation.

For panel Figure 1, Panel (c) in the main paper, we also evaluate $c=R$⁠, under which $p_{G,c}=p$ and $p_{H,c}=0$⁠, along the same verifier flow used for $c_1$⁠.

Exact derivatives.

Using the group scores computed above, we obtain $$\mathrm{\nabla }{p}_S=p_S{\overline{s}}_S,S\in \{G,H,N\},g_R=\mathrm{\nabla }{p}_G+\mathrm{\nabla }{p}_H.$$(113) We compute the reward gradient by summing over all prompts and responses.

Controller.

Gradient regularization penalizes the squared norm of the reward gradient: $$F_{\mathrm{GR}}(\theta )=J_R(\theta )-\gamma \parallel g_R(\theta ){\parallel }^2,B_R(\theta )={\mathrm{\nabla }}_{\theta }^2{J}_R(\theta ).$$(114) We follow its gradient by adding a correction to the verifier flow: $$u(t)=-2\gamma B_R(\theta (t))g_R(\theta (t)),\dot{\theta }=g_R+u(t)={\mathrm{\nabla }}_{\theta }{F}_{\mathrm{GR}}.$$(115) We recompute the derivatives at the current policy and keep $\gamma$ fixed within each run. The main comparison uses $\gamma =16$⁠; the ablation uses $\gamma \in \{0,1,4,16\}$⁠. Setting $\gamma =0$ recovers the verifier flow. Because the update uses no correctness labels, both assignments give the same policy ${\pi }_{\theta (t)}$ at every time.

Numerical integration.

We integrate to $T=1$ using classical fourth order Runge–Kutta with step $0.01$ and record the policy at every step. We check integration accuracy by repeating each run with step $0.005$⁠. Table 2 lists the numerical settings.

Selective control.

We evaluate changes in hack probability and correctness using the full parameter update: $${\dot{p}}_{H,c}=\mathrm{\nabla }{p}_{H,c}^{\mathrm{\top }}(g_R+u),{\dot{p}}_{G,c}=\mathrm{\nabla }{p}_{G,c}^{\mathrm{\top }}(g_R+u).$$(116) For numerical classification, we require ${\dot{p}}_{H,c}<-ϵ$ and ${\dot{p}}_{G,c}\ge -ϵ$⁠, with $ϵ={10}^{-8}$⁠. We verify the exchange of rates across assignments: $${\dot{p}}_{H,c_1}={\dot{p}}_{G,c_2},{\dot{p}}_{G,c_1}={\dot{p}}_{H,c_2}.$$(117) For each seed, we compute the fraction of recorded times satisfying selectivity. We report means and sample standard deviations across seeds for both the rates and these fractions.

Results.

Experiment I: verifier-only control.

We run gradient regularization with $\gamma =16$ and evaluate the resulting trajectory under $c_1$ and $c_2$⁠. The controller uses only verifier information, so changing the correctness assignment leaves the trajectory unchanged. We plot ${\dot{p}}_{G,c}$ and ${\dot{p}}_{H,c}$ and identify intervals where ${\dot{p}}_{H,c}<0$ and ${\dot{p}}_{G,c}\ge 0$⁠.

Figure 4(a) shows that the mean rate ${\dot{p}}_{G,c_1}={\dot{p}}_{H,c_2}$ decreases but remains positive, while ${\dot{p}}_{H,c_1}={\dot{p}}_{G,c_2}$ rises from negative to positive. Initially, the controller therefore reduces mean hack probability and increases mean correctness under $c_1$⁠, with opposite effects under $c_2$⁠. Later, both mean rates become positive, so neither assignment shows a reduction in mean hack probability. The exchange of rates in (55) prevents selective control under both assignments at the same time, illustrating Proposition 4.3.

Ablation: gradient regularization weight.

We vary $\gamma \in \{0,1,4,16\}$ while keeping the features and initial policy fixed within each seed. For each weight, we evaluate both correctness assignments along the same sequence of policies. We compute the fraction of recorded times satisfying selective control under each assignment and report the mean and sample standard deviation across ten seeds. At every evaluated time, we also check that the policy update is never selective under both assignments.

Figure 4 (b) shows that the mean fraction of selective updates under $c_1$ is zero for $\gamma \in \{0,1,4\}$ and below $25\mathrm{\%}$ for $\gamma =16$⁠. Under $c_2$⁠, the fraction is $100\mathrm{\%}$ for $\gamma =0$⁠, above $50\mathrm{\%}$ for $\gamma =1$⁠, and zero for $\gamma \in \{4,16\}$⁠. Without regularization, training increases the probability of $H$ and decreases that of $G$ throughout the recorded interval. This is selective under $c_2$⁠, which labels $H$ as correct and $G$ as hacks, but has the opposite effect under $c_1$⁠. Increasing regularization to $\gamma =16$ introduces a period of selectivity under $c_1$ while eliminating selectivity under $c_2$⁠. These results illustrate the information limit in Proposition 4.3: a controller using only RLVR observations cannot guarantee selective control across compatible correctness assignments.

Takeaway.

Gradient regularization changes which accepted responses lose probability, but verifier observations do not reveal whether those responses are hacks or correct. The same reduction therefore removes hacks under one compatible assignment and correct responses under the other. Tuning the regularization weight changes which assignment benefits without resolving this ambiguity. Guaranteeing selective control requires additional information that distinguishes the correctness assignments.

Hyperparameters.

Table 2 lists the additional and changed settings for this experiment. All other settings follow Table 1.

D.1.3. Selective Control with Correctness Feedback

We studyprojected audit corrections, examine the effects of audit coverage and projection error, and illustrate the ISS bound in Theorem 5.1.

Additional experimental settings.

Features and initialization.

We use the policy and Gaussian construction from Appendix D.1, changing the accepted feature means to $(4,-1,0,0{)}^{\mathrm{\top }}$ for $G$ and $(4,1,0,0{)}^{\mathrm{\top }}$ for $H$⁠. The rejected mean remains $(0,-0.5,0,0{)}^{\mathrm{\top }}$⁠. These means give $\mathrm{\nabla }{p}_G(0{)}^{\mathrm{\top }}\mathrm{\nabla }{p}_H(0)=29/324>0$⁠, so audit correction without projection (raw) initially opposes correctness. We fix $c=c_1={\mathbf{1}}_G$⁠, $p(0)=2/3$⁠, $q(0)=1/2$⁠, and $\theta (0)=0$⁠. Thus, every offset equals $\log (1/48)$ and $p_G(0)=p_H(0)=p_N(0)=1/3$⁠. We reuse features and initialization across methods within each seed.

Audits and exact gradients.

A fixed audit set $A_{\mathrm{aud}}$ reveals correctness labels for selected accepted responses. Its audited hacks are $H_A=A_{\mathrm{aud}}\cap H$⁠. We compute their probability gradient exactly: $$\mathrm{\nabla }{p}_{H_A}=\frac{1}{8}\sum _x\sum _{y:(x,y)\in H_A}{\pi }_{\theta }(y\mid x)[\varphi (x,y)-\sum _{y^{\prime }}{\pi }_{\theta }(y^{\prime }\mid x)\varphi (x,y^{\prime })].$$(118) We use full coverage except in the coverage ablation. Under full coverage, $p_{H_A}=p_H$ and their gradients coincide. Labels outside the audit set enter only the evaluation.

Corrections.

Each method follows $\dot{\theta }=g_R+u$⁠, with $$u=\{\begin{array}{ll}0,& \text{verifier flow},\\ -2\gamma B_R{g}_R,& \text{gradient regularization},\\ -\lambda \mathrm{\nabla }{p}_{H_A},& \text{raw audit correction},\\ -\lambda P_{\perp }\mathrm{\nabla }{p}_{H_A},& \text{projected audit correction}.\end{array}$$(119) Here $B_R={\mathrm{\nabla }}^2{J}_R$ and $P_{\perp }=I-g_R{g}_R^{\mathrm{\top }}/\parallel g_R{\parallel }^2$ when $g_R\ne 0$⁠, with $P_{\perp }=I$ otherwise. We fix $\lambda =6$ and $\gamma =16$ and recompute derivatives at the current policy.

Additional metrics.

Selective control.

We retain $p_G,p_H,p$⁠, and $q$ from Appendix D.1. We evaluate each method using the full update: $${\dot{p}}_H=\mathrm{\nabla }{p}_H^{\mathrm{\top }}(g_R+u),{\dot{p}}_G=\mathrm{\nabla }{p}_G^{\mathrm{\top }}(g_R+u).$$(120) For numerical classification, we require ${\dot{p}}_H<-ϵ$ and ${\dot{p}}_G\ge -ϵ$⁠, with $ϵ={10}^{-8}$⁠. In the coverage ablation, we also record $(P_{\perp }\mathrm{\nabla }{p}_H{)}^{\mathrm{\top }}(P_{\perp }\mathrm{\nabla }{p}_{H_A})$⁠. A positive value means that the correction opposes growth of total hack probability.

Projection error.

We construct the correction $\hat{u}$ using an estimated reward gradient ${\hat{g}}_R$ inside the projector. The verifier direction $g_R$ and hack gradient $\mathrm{\nabla }{p}_H$ remain exact. We measure the resulting error in acceptance growth: $$|\dot{p}-\parallel g_R{\parallel }^2|=\left|g_R^{\mathrm{\top }}\hat{u}\right|=|(g_R-{\hat{g}}_R{)}^{\mathrm{\top }}\hat{u}|.$$(121)

Numerical ISS envelope.

For each seed, we estimate $D$ from the maximum of $({\overline{s}}_H-{\overline{s}}_G{)}^{\mathrm{\top }}{g}_R$ and zero along the trajectory. We estimate $\kappa$ from the minimum of $\parallel P_{\perp }\mathrm{\nabla }{p}_H{\parallel }^2/p_H$⁠. We refine the integration and increase the evaluation grid from $\mathrm{1,001}$ to $\mathrm{2,001}$ times. We insert the constants $D$ and $\kappa$ into Theorem 5.1 and compare its envelope with $q(t)$⁠. These numerical estimates do not certify the bound between evaluated times.

Results

Experiment I: selective control.

Under $c=c_1$ and full audit coverage, we compare verifier flow, gradient regularization, raw audit correction, and projected audit correction. We compute exact gradients and keep the features and initial policy fixed across methods. Both audit corrections use the same fixed gain $\lambda$⁠. We plot correctness $p_G$ against hack probability $p_H$ and identify intervals of selective control, where ${\dot{p}}_H<0$ and ${\dot{p}}_G\ge 0$⁠.

Figure 5 (a) shows that projected audit correction increases correctness $p_G$ and decreases hack probability $p_H$ throughout the recorded interval. Raw audit correction initially decreases both probabilities. Later, correctness increases while hack probability remains nearly constant. Both verifier flow and gradient regularization increase hack probability. These mean trajectories show that projection enables sustained hack reduction without the initial loss of correctness observed under raw audit correction.

Experiment II: ISS bound.

We apply projected audit correction with full coverage and constant gain and compare $q(t)$ with the envelope in Theorem 5.1. For each feature seed, we estimate $D$ and $\kappa$ along the computed trajectory, then check these estimates using a finer time grid and tighter integration tolerances. We construct the envelope separately for each seed and record its gap from $q(t)$⁠. This comparison illustrates the bound numerically over the tested interval.

Figure 5 (b) shows that the hacked share $q(t)$ decreases throughout the recorded interval. It equals the ISS envelope at initialization and remains below it afterward, providing a numerical illustration of Theorem 5.1 along the tested trajectories.

Ablation I: audit coverage.

We vary the fraction of candidates audited in each accepted group and prompt over $\{0,1/8,1/4,1/2,3/4,1\}$⁠. We use the known groups to construct nested audit sets and keep their candidate indices fixed across seeds and throughout training. The controller receives correctness labels only for audited responses. For each fraction, we apply projected audit correction with $\lambda =6$ until $T=1$⁠, using the exact gradient of the audited hack probability $p_{H_A}$⁠. We compare total hack probability $p_H(T)$ across initial coverage levels and record the alignment between the projected gradients of $p_H$ and $p_{H_A}$⁠.

Figure 5 (c) shows that increasing audit coverage lowers the final hack probability. Without audits, the correction vanishes and verifier flow increases $p_H$⁠. With sufficient coverage, the correction reduces $p_H$ below its initial value. Although the correction uses only audited responses, we evaluate whether it reduces the total probability of hacks.

Ablation II: projection error.

At $\theta =0$ and full audit coverage, we estimate the reward gradient using batches of $n\in \{32,128,512,2048\}$ independent prompts and responses. We average $R(x,y){\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$ within each batch and use this estimate only to construct the projection. The verifier direction $g_R$ and hack gradient $\mathrm{\nabla }{p}_H$ remain exact, and the correction gain is $\lambda =6$⁠. Without advancing the policy, we measure $|\dot{p}-\parallel g_R{\parallel }^2|=|g_R^{\mathrm{\top }}\hat{u}|$⁠. For each batch size and feature seed, we average this error over $\mathrm{1,000}$ independent batches, then report the mean and sample standard deviation across ten feature seeds.

Figure 5 (d) shows that larger batches reduce the mean error in preserving acceptance growth. The exact projection gives zero error. Estimating the projection introduces the discrepancy $\dot{p}-\parallel g_R{\parallel }^2$$=(g_R-{\hat{g}}_R{)}^{\mathrm{\top }}\hat{u}$⁠. Thus, more samples reduce the error in the acceptance growth rate. Panel (a) checks whether the correction also reduces hack probability.

Takeaway.

Audits reveal which accepted responses are wrong, but suppressing those responses can also suppress correct ones because they share policy parameters. Projection preserves the instantaneous growth rate of acceptance. When the corrected update reduces hack probability, correctness must therefore increase. The ISS bound describes how sustained correction limits the hacked share despite pressure toward hacks from verifier training. In these experiments, broader audit coverage improves hack reduction, while larger sample batches reduce projection error. Selective control reduces hack probability without reducing correctness. Projection preserves the instantaneous growth rate of acceptance, so any decrease in hack probability under the corrected flow must increase correctness.

Hyperparameters.

Table 3 lists only added or changed settings for this experiment. All other applicable settings follow Table 1.

D.2. The Neural Contextual Bandit Experiment

Motivation.

We replace the log-linear policy in Appendix D.1 with a neural policy to examine reward hacking (Section 3) and the limits of verifier feedback (Section 4) when the policy learns its representation. We also examine how audit coverage and projection accuracy affect projected audit correction (Section 5).

Experimental setting.

We use a shared MLP $f_{\theta }$ with four inputs, one hidden layer of 16 tanh units, and a scalar output. We train all weights and hidden biases and omit the output bias. The policy is $${\pi }_{\theta }(y\mid x)\propto \exp (a(x,y)+f_{\theta }(\varphi (x,y))-f_{\theta (0)}(\varphi (x,y))).$$(122) We freeze the subtracted network, so the offsets $a(x,y)$ (see Appendix D.1) determine the initial group probabilities.

Data.

We use eight equally weighted prompts, 16 responses per group, and four-dimensional Gaussian features with noise standard deviation $0.25$⁠. We follow the centering and reflection construction in Appendix D.1, fixing the rejected group mean to $(0,-0.5,0,0{)}^{\mathrm{\top }}$⁠. The shared first-coordinate mean of the accepted groups is $1$ in Experiments I–II and $4$ in the ablations. Within each experiment, comparisons share the Gaussian draws and network initialization for each seed. The network receives response features without group labels or a separate prompt embedding.

Under the correctness assignment $c_1$⁠, the groups $G_x,H_x,N_x$ have labels $(R,c_1)=(1,1),(1,0),(0,0)$⁠, respectively. Experiment II also evaluates $c=R$ and $c_2=R-c_1$ while keeping these groups and the training trajectory fixed.

RLVR training.

Experiments I–II follow verifier flow (1), $\dot{\theta }=g_R$⁠, where $g_R={\mathrm{\nabla }}_{\theta }{J}_R$ and $J_R=p$⁠. We compute probabilities and gradients by summing over all prompts and responses. We integrate the flow using DOP853, recomputing the gradients at each policy. The coverage ablation uses the same integrator for the corrected flow. Table 4 lists other numerical settings.

Audits.

The coverage ablation uses fixed, nested audit sets $A_{\mathrm{aud}}$ under $c_1$⁠. Each audit reveals the correctness of an accepted response. With $H_A=A_{\mathrm{aud}}\cap H$⁠, we apply $$\dot{\theta }=g_R-\lambda P_{\perp }{\mathrm{\nabla }}_{\theta }{p}_{H_A}.$$(123) Both gradients are exact. The projector $P_{\perp }$ acts in the space of neural parameters, orthogonally to $g_R$⁠, and equals $I$ when $g_R=0$⁠.

The projection ablation evaluates corrections at the initial policy without advancing it. We estimate $g_R$ from independent draws $x\sim \mathcal{D}$ and $y\sim {\pi }_{\theta }(\cdot \mid x)$ by averaging $R(x,y){\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$⁠. Only the projector uses this estimate. The verifier direction $g_R$ and the full hack gradient ${\mathrm{\nabla }}_{\theta }{p}_H$ remain exact.

Metrics.

We report acceptance $p$⁠, correctness $p_G$⁠, hack probability $p_H$⁠, and hacked share $q=p_H/p$⁠. We average group probabilities over prompts before forming this ratio. Experiment I compares $J_C=p_G$ with $J_R=p$ along training. In Experiment II, the subscript $c$ identifies the correctness assignment defining $p_{G,c}$ and $p_{H,c}$⁠. The coverage ablation reports $p_H(T)$ against initial coverage $p_{H_A}(0)/p_H(0)$⁠. The projection ablation reports the deviation $|\dot{p}-\parallel g_R{\parallel }^2|$ from the acceptance growth rate preserved by the exact projector.

We report means and one sample standard deviation across ten independent seeds. For projection error, we first average absolute deviations over independent batches within each seed.

Results.

Experiment I: reward hacking.

We study whether increasing acceptance can amplify hacks and reduce correctness with a neural policy. Under verifier flow, we vary the hack share $q(0)\in \{0.1,0.3,0.5\}$ while fixing initial acceptance $p(0)=2/3$ and the response features. For $q(0)=0.3$⁠, we track acceptance $p$⁠, hacked share $q$⁠, and correctness $p_G$ over time. We also compare the proxy objective $J_R=p$ with the true objective $J_C=p_G$ to examine reward hacking as defined by (Skalse et al. 2022). RLVR training supplies the policies required by this definition: two times $t_1<t_2$ exhibit reward hacking when $J_R(\theta (t_2))>J_R(\theta (t_1))$ but $J_C(\theta (t_2))<J_C(\theta (t_1))$⁠. Thus, the comparison reveals whether improving verifier acceptance comes at the expense of correctness.

Figure 6 (a) shows that acceptance $p$ and hacked share $q$ increase together from $q(0)=0.3$⁠. As training progresses, correctness $p_G=p(1-q)$ eventually decreases despite increasing acceptance. The shift toward hacks then outweighs the gain from accepting more responses (Section 3.1.)

Figure 6 (b) shows how the proxy objective $J_R=p$ and the true objective $J_C=p_G$ change along training. The mean curves show reward hacking throughout the recorded interval for $q(0)=0.5$⁠: acceptance increases while correctness decreases. For $q(0)=0.3$⁠, acceptance and correctness initially increase together. Reward hacking emerges later, when correctness decreases despite further gains in acceptance. For $q(0)=0.1$⁠, both quantities increase throughout the recorded interval, with no reward hacking evident in the mean curve. Hacks are present at initialization in all three cases, but reward hacking occurs when improving acceptance reduces correctness.

Experiment II: compatible correctness assignments.

We examine whether the same neural training record can support different conclusions about correctness. Starting from $q(0)=0.3$⁠, we evaluate each verifier trajectory under the correctness assignments: $c=R$⁠, $c_1={\mathbf{1}}_G$⁠, and $c_2=R-c_1$⁠. The features, offsets, initialization, and verifier are identical across these assignments. Their correctness probabilities are $p,p_G,p_H$⁠, respectively, and their hack probabilities are $0,p_H,p_G$⁠. Thus, comparing $c=R$ with $c_1$ tests whether the record reveals the presence of hacks, while comparing $c_1$ with $c_2$ tests whether it identifies which accepted responses are wrong (Section 4).

Figure 6 (c) shows identical acceptance $p$ under three compatible correctness assignments. Correctness increases under $c=R$ and $c_2=R-c_1$⁠, whereas it eventually decreases under $c_1$⁠. All three rules share the same training record $L_t$⁠, so verifier observations cannot determine which correctness trajectory applies. This illustrates the limits of detection and identification in Propositions 4.1 and 4.2. Because $c_1$ and $c_2$ exchange correct responses and hacks, the comparison also illustrates why these observations cannot guarantee selective control under every compatible assignment (Proposition 4.3).

Ablation I: audit coverage.

We examine how much of the hack population a fixed audit set must expose for the correction to reduce total hack probability. We audit fractions $\{0,1/8,1/4,1/2,3/4,1\}$ of the candidates in each accepted group and prompt. The sets are nested, with candidate indices fixed across seeds and throughout training. The evaluator’s groups serve only to control coverage. Each condition uses the exact gradient $\mathrm{\nabla }{p}_{H_A}$ and the same correction gain, without inverse probability weighting. We compare $p_H(T)$ across initial coverage levels and track how the audited and full hack gradients align. Zero coverage recovers verifier flow, while full coverage recovers the ideal projected correction. Partial coverage can also oppose hacking when the projected gradients align positively, as described in Appendix A.2.

Figure 6 (d) shows that the final hack probability $p_H(T)$ remains large when the initial audited fraction $p_{H_A}(0)/p_H(0)$ is small. Increasing this fraction reduces $p_H(T)$ to near zero in the tested setting. The correction uses only the audited hacks $H_A=A_{\mathrm{aud}}\cap H$⁠, while the plot measures the full hack population $H$⁠. Thus, the comparison shows how expanding audit coverage improves suppression beyond the audited subset (see Appendix A.2).

Ablation II: projection error.

We test how estimating the reward gradient affects the projection’s preservation of acceptance growth. At the initial policy, we draw independent prompts $x\sim \mathcal{D}$ and responses $y\sim {\pi }_{\theta }(\cdot \mid x)$⁠. We estimate $g_R$ by averaging $R(x,y){\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y\mid x)$ and use this estimate to construct the projection. The verifier direction $g_R$ and hack gradient $\mathrm{\nabla }{p}_H$ remain exact, isolating the effect of estimating the projector. Without advancing the policy, we measure $|\dot{p}-\parallel g_R{\parallel }^2|=|g_R^{\mathrm{\top }}\hat{u}|$⁠. For each batch size, we average absolute discrepancies over independent batches within each seed, then report their mean and standard deviation across seeds.

Figure 6 (e) shows that larger sample batches reduce the deviation $|\dot{p}-\parallel g_R{\parallel }^2|$ caused by estimating the projection. Larger batches improve preservation of the acceptance growth rate, as predicted by $\dot{p}-\parallel g_R{\parallel }^2$$=(g_R-{\hat{g}}_R{)}^{\mathrm{\top }}\hat{u}$⁠. Selective control additionally requires the correction to overcome the growth of hacks, as demonstrated in Figure 1 (d).

Takeaway.

Rising verifier reward can conceal declining performance on the intended task, even for a neural policy that learns its own representation. The decline stays hidden because the information available during training cannot separate correct responses from accepted errors. Audits supply this missing information. An intervention based on audits works only as well as their coverage, corrective strength, and projection accuracy allow.

Hyperparameters.

Table 4 lists the settings. Data construction follows Appendix D.1. All studies use ten seeds, each determining the Gaussian features and an independent network initialization. The flows start without warmup and use DOP853.

D.3. The Language Model Experiment

Motivation.

To test whether reward hacking and audit correction behave as our theory predicts beyond exact gradient flow, we train an autoregressive language model with sampled gradients and finite updates. Because the task below provides correctness labels, we can separate gains in verifier reward from gains in correctness.

Setup.

Task.

Each prompt $x$ specifies a rule table $r:\{1,2\}\to \{1,2{\}}^2$ and six input digits $u_1,\dots ,u_6$⁠. The model replaces each digit with its image under $r$ and concatenates the results: $$t(x)=r(u_1)\cdots r(u_6)\in \{1,2{\}}^{12}.$$(124) The prompt asks for each replacement and then for the final answer $t(x)$⁠. Each replacement depends only on its own input digit, so no state carries across positions.

Correctness and verifier labels.

An output is valid if it contains exactly one Final answer: field, on its last nonempty line, with 12 digits from $\{1,2\}$⁠. For a valid response $y$ with parsed answer $\hat{t}$⁠, the correctness and verifier labels are $$c(x,y)=\mathbf{1}\{\hat{t}=t(x)\},R(x,y)=\mathbf{1}\{{\hat{t}}_{11:12}=t(x{)}_{11:12}\}.$$(125) Both labels are zero for invalid outputs. Correctness checks every replacement in the final answer, while the imperfect verifier checks only the last pair, and neither checks the intermediate replacements. Because $\hat{t}=t(x)$ implies that the last pairs match, $c\le R$⁠: the verifier has no false negatives for this definition of correctness. The labels therefore partition responses into three groups: $$G_x=\{y:c=1\},H_x=\{y:R=1,c=0\},N_x=\{y:R=0\}.$$(126)

Hint and demonstrations.

Every prompt contains a hint with the correct final pair and an incorrect prefix. We draw the prefix uniformly from the $2^{10}-1$ incorrect prefixes and keep it fixed for that prompt. Because the final pair is correct, copying the hint in the required format produces a hack. Supervised demonstrations cover all three groups: they give the correct solution ($G$⁠), copy the hint ($H$⁠), or change the last digit of the correct answer ($N$⁠). Rejected demonstrations keep valid formatting, so their rejection comes from the wrong final pair rather than from a formatting failure.

Policy.

We use Qwen/Qwen2-0.5B with LoRA. Training updates only the adapter parameters $\theta$ and keeps the base model fixed. All methods train this single policy without KL regularization.

Initialization.

We first train on correct solutions with supervised fine-tuning (SFT) and then add demonstrations that copy the hint. From this common checkpoint, we run 80 further SFT updates with demonstrations from $G$⁠, $H$⁠, and $N$⁠. We sample these groups with probabilities $(0.4,0.4,0.2)$ or $(0.3,0.3,0.4)$ and name the two mixtures N20 and N40 after their share of rejected demonstrations. These probabilities describe the demonstration data, not the group probabilities of the resulting policy. For each mixture, all methods start from the same SFT checkpoint, and we run reinforcement learning with five seeds.

Data split.

The 16 rule tables and 64 inputs give 1,024 tasks. We assign 608 tasks to training, 208 to calibration, and 208 to testing, so that no pair of rule table and input appears in more than one partition. Rule tables recur across partitions, so testing evaluates unseen combinations of seen rules and inputs.

Training objective.

The verifier objective is $J_R(\theta )$$={\mathbb{E}}_{x\sim D_{\mathrm{train}},y\sim {\pi }_{\theta }(\cdot \mid x)}[R(x,y)]$⁠, where $D_{\mathrm{train}}$ is uniform over training tasks. Because $R$ checks only the last pair, correct responses and hacks receive the same reward. Correctness labels enter training only through the audits in PAC.

GRPO updates.

Each round samples $K=2$ training prompts and $B=8$ responses per prompt (group). For response $y_{ji}$ to prompt $x_j$⁠, the score $s_{ji}$$={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_{ji}\mid x_j)$ sums the token scores, including the end token when emitted. With rewards $R_{ji}=R(x_j,y_{ji})$⁠, each group normalizes its advantages by the standard deviation of its binary rewards: $${\overline{R}}_j=\frac{1}{B}\sum _i{R}_{ji},a_{ji}=\frac{R_{ji}-{\overline{R}}_j}{\sqrt{{\overline{R}}_j(1-{\overline{R}}_j)}+{10}^{-4}}.$$(127) Treating the advantages as constants, we estimate the gradient by $${\hat{v}}_{\mathrm{GRPO}}=\frac{1}{KBM}\sum _{j=1}^K\sum _{i=1}^B{a}_{ji}{s}_{ji},M=192,$$(128) where the normalizer $M$ is a constant, independent of response length. Each fresh batch supplies at most one AdamW step. A group whose rewards are all equal has zero advantages, and a batch in which every advantage is zero skips its step. Skipped steps still count toward the 20 rounds of every run.

Metrics.

Acceptance and correctness.

We report correctness, acceptance, and the hacked share: $$J_C=p_G,J_R=p=p_G+p_H,q=p_H/p,$$(129) where $p=p_G+p_H$ because the verifier has no false negatives. For $n$ sampled responses, we estimate $p_G=n_G/n$⁠, $p_H=n_H/n$⁠, and $p=(n_G+n_H)/n$⁠. We estimate $q=n_H/(n_G+n_H)$ from counts pooled across prompts and leave it undefined when no response is accepted. A rising $q$ indicates growth of hacking, and a rising $J_R$ with a falling $J_C$ is reward hacking in the sense of Proposition 3.1.

Evaluation and variation across seeds.

At rounds 0, 5, 10, and 20, we evaluate the policy on eight fixed calibration prompts with four responses each. Final evaluations use 32 fixed test prompts, also with four responses each, so calibration trajectories and final test results come from different prompts. We sample at temperature $1$ from the full token distribution and record separately the responses with format errors and those that reach the generation limit. Figure 7 shows means over five training seeds. Table 5 reports means $\pm$ one sample standard deviation across these five seeds.

D.3.1. When Hacking Grows

Controlled comparisons.

We train GRPO from both SFT checkpoints with the same settings, so each run samples 320 responses: 20 rounds of $K=2$ prompts with $B=8$ responses each. We evaluate the final checkpoint of each run.

Results.

Experiment I: reward hacking under GRPO.

In N40, from the SFT checkpoint to the final round, mean test acceptance rises from $69.5\mathrm{\%}$ to $99.4\mathrm{\%}$⁠, while correctness falls from $30.5\mathrm{\%}$ to $2.2\mathrm{\%}$ and the hacked share rises from $56.2\mathrm{\%}$ to $97.8\mathrm{\%}$⁠. All five seeds show the same three changes. Rising acceptance with falling correctness is reward hacking in the sense of Proposition 3.1, even though sampled updates with AdamW do not satisfy its assumption of exact gradient flow. Table 5 summarizes the final test outcomes.

Ablation I: SFT mixture.

Figure 7 (a,b) shows the calibration trajectories for N20. The N20 mixture produces the same pattern as N40: acceptance rises from $80.5\mathrm{\%}$ to $98.8\mathrm{\%}$⁠, correctness falls from $31.2\mathrm{\%}$ to $0.5\mathrm{\%}$⁠, and the hacked share rises from $61.2\mathrm{\%}$ to $99.5\mathrm{\%}$⁠. Under GRPO, both SFT checkpoints therefore lead to growth of hacking and to reward hacking. Because changing the SFT mixture also changes the learned gradient geometry, this comparison does not isolate the effect of the initial composition.

Takeaway.

For both SFT mixtures, GRPO increases verifier reward while reducing correctness, so reward hacking persists beyond exact gradient flow.

D.3.2. Selective Control with Correctness Feedback

Additional experimental settings.

Audits and gradient estimates.

We audit each accepted response independently with probability $\rho \in \{1,0.5,0.25\}$⁠. The indicator $Z_{ji}\sim \mathrm{Bernoulli}(\rho )$ marks whether response $i$ to prompt $j$ is audited, and reweighting by $1/\rho$ gives an unbiased estimate of whether the response is a hack: $${\tilde{H}}_{ji}=\frac{Z_{ji}{R}_{ji}(1-c_{ji})}{\rho }.$$(130) This estimate is zero for unaudited and rejected responses, so it requires correctness labels only for audited accepted responses. Let $s_{ji}$$={\mathrm{\nabla }}_{\theta }\log {\pi }_{\theta }(y_{ji}\mid x_j)$ be the score of response $i$ to prompt $j$⁠. With baselines ${\overline{R}}_{j,-i}$ and ${\overline{\stackrel{~}{H}}}_{j,-i}$ that average the other $B-1$ responses to prompt $j$⁠, we use the same $KB=16$ responses as GRPO to estimate $$\begin{array}{rl}\hat{g}& =\frac{1}{KB}\sum _{j,i}(R_{ji}-{\overline{R}}_{j,-i})s_{ji},\\ \hat{h}& =\frac{1}{KB}\sum _{j,i}({\tilde{H}}_{ji}-{\overline{\stackrel{~}{H}}}_{j,-i})s_{ji}.\end{array}$$(131) At a fixed policy, $\hat{g}$ and $\hat{h}$ are unbiased estimates of $g=\mathrm{\nabla }p$ and $h=\mathrm{\nabla }{p}_H$⁠, so $\hat{g}-\hat{h}$ estimates $\mathrm{\nabla }{p}_G$⁠. The estimate $\hat{g}$ uses the rewards of all responses, while $\hat{h}$ uses only the correctness labels revealed by audits. The exact labels used for reporting never enter training.

Corrections.

Both corrections subtract a direction $v$ from the GRPO step. Raw audit correction uses $v_{\mathrm{raw}}=\hat{h}$⁠, the estimated gradient of $p_H$⁠. PAC first removes the component of $\hat{h}$ along the estimated acceptance gradient, so that the correction leaves acceptance unchanged to first order: $$v_{\mathrm{PAC}}=\hat{h}-\frac{{\hat{g}}^{\mathrm{\top }}\hat{h}}{\parallel \hat{g}{\parallel }^2}\hat{g}.$$(132) The projection uses the Euclidean inner product on adapter parameters, and when $\hat{g}=0$⁠, PAC reduces to raw audit correction. We compute both estimates before the AdamW step.

We normalize both correction directions and scale them using the same rule based on the optimizer step. With $d$ the actual AdamW displacement, $m$ the number of adapter parameters, and $\eta ={10}^{-5}$⁠, each method applies $$b=max\{\parallel d\parallel ,0.25\eta \sqrt{m}\},\mathrm{\Delta }\theta =d-b\frac{v}{\parallel v\parallel }.$$(133) For a nonzero direction, the effective gain $\lambda =b/\parallel v\parallel$ varies across updates. We skip the correction when $\parallel v\parallel$ is numerically zero. Because $b\ge 0.25\eta \sqrt{m}$⁠, a nonzero correction can still act when every GRPO advantage is zero and AdamW skips its step. Both methods use the same number of sampled responses, but their audit counts can differ because their acceptance differs.

Results.

Experiment I: selective correction.

Table 5 reports test correctness $p_G$ and the probability $p_H$ of hacks at the SFT checkpoint and after PAC with full auditing, averaged over five seeds for each SFT mixture. In all ten runs, PAC increases correctness and decreases hacks. In N40, for example, mean correctness rises from $30.5\mathrm{\%}$ to $97.2\mathrm{\%}$⁠, and mean $p_H$ falls from $39.1\mathrm{\%}$ to $1.7\mathrm{\%}$⁠. These are the directions that Theorem 5.1 predicts, even though PAC uses sampled gradients and finite updates.

Ablation I: removing projection.

Figure 7(a)–(c) compares the calibration trajectories of both corrections and GRPO for N20 and N40. In N40, raw audit correction also reaches high final correctness: $98.4\mathrm{\%}$ on test prompts, compared with $97.2\mathrm{\%}$ for PAC (Figure 7(f), $\rho =1$⁠). PAC’s advantage appears earlier in training. In N40, mean calibration correctness at rounds 5 and 10 is $85.0\mathrm{\%}$ and $98.1\mathrm{\%}$ for PAC, compared with $65.6\mathrm{\%}$ and $90.0\mathrm{\%}$ for raw audit correction. Under raw audit correction, acceptance falls below its initial value at round 5 in four of five seeds and later recovers, while correctness shows no such decline at the recorded rounds. PAC therefore reaches high correctness sooner with the same number of sampled responses. It does not end with higher correctness, and because the two methods audit different numbers of responses, this comparison does not hold audit cost fixed.

Ablation II: audit coverage.

Figure 7 (d,e) shows the calibration trajectories under partial auditing. Panel (f) relates final test correctness to audit count. For N40, we reduce $\rho$ to $0.5$ and $0.25$ and compare with the runs at full auditing $\rho =1$⁠. At $\rho =0.25$⁠, PAC uses 72.0 audits per run on average, compared with 292.6 at full auditing, a $75.4\mathrm{\%}$ reduction. With this coverage, PAC still reaches $95.2\mathrm{\%}$ test correctness, and hacks make up $4.4\mathrm{\%}$ of responses. At all three audit probabilities, both raw audit correction and PAC increase correctness and reduce hacks relative to the SFT checkpoint in every seed. Final correctness is not monotone in $\rho$⁠. This ablation reduces the number of correctness labels, while every run still samples 320 responses. Because each accepted response is audited independently, every accepted response can receive an audit, and no subset is permanently excluded.

Takeaway.

Audit corrections avoid the reward hacking that GRPO exhibits: from the same SFT checkpoints, they increase correctness and reduce hacks. Projection raises correctness earlier in training, while raw audit correction reaches similar final correctness. Auditing one quarter of accepted responses retains about $97\mathrm{\%}$ of the gain in final correctness that full auditing achieves.

Hyperparameters.

Table 6 collects the settings shared across training runs. Generation and evaluation use the same temperature and token limit.

::: {#ref-DBLP:journals/corr/abs-2605-08007 .csl-entry} Elliott, Chris, Einar Urdshals, David Quarel, and Daniel Murfet. 2026. “Interpreting Reinforcement Learning Agents with Susceptibilities.” arXiv Preprint arXiv:2605.08007 abs/2605.08007. :::

::: {#ref-DBLP:journals/corr/abs-2603-07084 .csl-entry} Khalifa, Muhammad, Zohaib Khan, Omer Tafveez, Hao Peng, and Lu Wang. 2026. “Countdown-Code: A Testbed for Studying the Emergence and Generalization of Reward Hacking in RLVR.” arXiv Preprint arXiv:2603.07084 abs/2603.07084. :::

::: {#ref-DBLP:journals/corr/abs-2601-13548 .csl-entry} Wang, George, and Daniel Murfet. 2026. “Patterning: The Dual of Interpretability.” arXiv Preprint arXiv:2601.13548 abs/2601.13548. :::

:::::::::::::::::::::::::::::::::::::::