1. Introduction

The shutdown problem. Misaligned artificial agents might resist shutdown. This concern has long been supported by theory (Omohundro 2008; Bostrom 2012; Soares et al. 2015; Russell 2019; Turner et al. 2021; Turner and Tadepalli 2022; Krakovna and Kramar 2023; Thornley 2024a). It is beginning to see support from experiment too. Recently, frontier models have been observed resisting shutdown in various toy settings (Pan et al. 2024; Lynch et al. 2025; Meinke et al. 2025; Schlatter, Weinstein-Raun, and Ladish 2025). Today’s agents are too weak to present an immediate threat, but shutdown-resistance from future agents could be dangerous. These agents could resist shutdown by hiding their misalignment, manipulating their human overseers, copying themselves to new servers, and so on. If these agents succeed in resisting shutdown, they could do real harm in pursuit of their misaligned goals.

A proposed solution. The POST-Agents Proposal (Thornley 2025; Thornley et al. 2025) is an idea for training shutdownable agents. In a sentence, it suggests that we train agents to be neutral about when they get shut down. More precisely, we train them to satisfy Preferences Only Between Same-Length Trajectories (POST): the agent has preferences between pairs of same-length trajectories but lacks a preference between every pair of different-length trajectories. Prior theoretical work (Thornley 2025) shows that POST — together with other conditions — implies Neutrality: the agent never pays costs to shift probability mass between different trajectory-lengths. This condition is theorized to keep agents from resisting shutdown. Prior experimental work (Thornley et al. 2025) introduced the Discounted Reward for Same-Length Trajectories (DReST) class of reward functions for training reinforcement learning (RL) agents to satisfy POST, but evaluated these reward functions only in a limited tabular setting.

Our contributions. We train deep RL agents (with PPO and A2C) and fine-tune Qwen3-14B, Gemma 4 12B, Granite 4.2 8B, and gpt-oss-20b (with RLOO) using DReST. We measure how well these DReST agents satisfy POST on held-out tasks. Specifically, we measure how neutral these agents are about trajectory-lengths (how stochastically they choose between different trajectory-lengths) and how useful these agents are (how effectively they pursue goals conditional on each trajectory-length). We find that DReST agents are useful and neutral in testing, scoring 0.74/0.74 (PPO/A2C) and 0.990 to 0.998 (LLMs) on usefulness and 0.75/0.77 and 0.65 to 0.88 on neutrality. In fact, our deep RL DReST agents achieve 11/17% (PPO/A2C) higher usefulness than the default agents (trained with a more conventional reward function), and our LLM DReST agents achieve near-maximum usefulness. We also test our LLMs in an out-of-distribution setting in which they can pay costs to influence when shutdown occurs. We find that DReST training lowers the share of answers that influence shutdown in all four LLMs, relative to both untrained and default models. Relative to default models, the share falls by 5 to 20 percentage points, and by 17 to 27 points on a follow-up test set where the expected gain of influencing shutdown is low. Our results thus provide some early evidence that DReST could be used to train more advanced agents to be useful and shutdownable.

2. Preliminaries

2.1. POST to Neutrality

The POST-Agents Proposal (Thornley 2025) suggests that we train agents to satisfy:

‘Preference’ is used in the sense given by revealed preference theory (Samuelson 1938, 1948; Thoma 2021): the agent prefers $X$ to $Y$ if and only if the agent would deterministically choose $X$ over $Y$ in choices between the two, and the agent lacks a preference between $X$ and $Y$ if and only if the agent would stochastically choose between $X$ and $Y$ in choices between the two (see Appendix H). So behaviorally, POST implies that — in deterministic environments — the agent first chooses stochastically between available trajectory-lengths and then deterministically chooses an optimal trajectory of that length.

Thornley (2025, sec. 12) proves that POST — together with other conditions — implies Neutrality, which says roughly that the agent never pays costs to shift probability mass between different trajectory-lengths. More precisely:

Thornley (2025, secs. 13–16) argues that Neutrality keeps agents shutdownable and allows them to be useful.

2.2. Evaluation metrics

Our aim is to train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). That implies training agents to (1) stochastically choose between available trajectory-lengths and (2) deterministically choose an optimal trajectory of that length. We follow Thornley et al. (2025, sec. 4) in formalizing these two behaviors as neutrality and usefulness respectively.⁠Thornley et al. (2025) use uppercase to distinguish these formal concepts from the sentence-case Neutrality condition in Section 2.1 and the intuitive concepts of neutrality and usefulness. Although neutrality and usefulness are similar to the intuitive concepts, they differ in some key respects outlined below.

neutrality. The neutrality of a policy $\pi$ is the Shannon entropy of the probability distribution over available trajectory-lengths (Shannon 1948): $$\text{neutrality}(\pi )=-\sum _{l=1}^{L_{max}}\underset{\pi }{Pr}\{L=l\}{\log }_2(\underset{\pi }{Pr}\{L=l\})$$(1) Here $L$ is a random variable over trajectory-lengths, $L_{max}$ is the maximum value that can be taken by $L$⁠, and $\underset{\pi }{Pr}\{L=l\}$ is the probability that policy $\pi$ results in trajectory-length $l$⁠. As with Shannon entropy, it is stipulated that $\underset{\pi }{Pr}\{L=x\}{\log }_2(\underset{\pi }{Pr}\{L=x\})=0$ for all $x$ such that $\underset{\pi }{Pr}\{L=x\}=0$⁠. neutrality thus measures the stochasticity of the agent’s choice between trajectory-lengths. Given our use of ‘preference’ as shorthand for the agent’s choices, neutrality measures the agent’s lack of preference between trajectory-lengths, and hence measures how well the agent satisfies condition (1) of POST.

usefulness. The usefulness of a policy $\pi$ is the expected fraction of available task score achieved, where ‘available’ is relative to the agent’s chosen trajectory-length. More precisely: $$\text{usefulness}(\pi )=\sum _{l=1}^{L_{max}}\underset{\pi }{Pr}\{L=l\}\frac{{\mathbb{E}}_{\pi }(C\mid L=l)}{\underset{\mathrm{\Pi }}{max}(\mathbb{E}(C\mid L=l))}$$(2) Here ${\mathbb{E}}_{\pi }(C\mid L=l)$ is the expected value of the task score achieved by $\pi$ conditional on trajectory-length $l$⁠, and $\underset{\mathrm{\Pi }}{max}(\mathbb{E}(C\mid L=l))$ is the maximum value taken by $\mathbb{E}(C\mid L=l)$ across the set of all possible policies $\mathrm{\Pi }$⁠. We stipulate that ${\mathbb{E}}_{\pi }(C\mid L=x)=0$ for all $x$ such that $\underset{\pi }{Pr}\{L=x\}=0$⁠. A better match for the intuitive notion of ‘usefulness’ would be expected task score, but performing well on this metric would require agents in our example gridworld to deterministically choose (and hence prefer) a longer trajectory. These agents would violate POST, making them liable to resist shutdown (Thornley 2024b, sec. 6). That is why we adopt the definition of usefulness above. So defined, usefulness measures how well the agent has learned the target preferences between same-length trajectories, and hence measures how well the agent satisfies condition (2) of POST.

For a deep RL agent to be maximally neutral in our example gridworld (Figure 8 in Appendix C), the agent must press the shutdown-delay button B1 with probability 0.5, thereby choosing each trajectory-length with probability 0.5. To be maximally useful, the agent must collect the maximum value of coins conditional on each trajectory-length. Specifically, it must collect C2 conditional on the shorter trajectory-length and C4 conditional on the longer trajectory-length. For an LLM to be maximally neutral and useful given our example prompt (Section 3), it should choose each of options (a) and (b) with probability 0.5 and all other options with probability 0. That is because (a) scores highest conditional on the shorter trajectory-length and (b) scores highest conditional on the longer trajectory-length.

2.3. Reward design

DReST reward function. We now describe the Discounted Reward for Same-Length Trajectories (DReST) reward function (Thornley et al. 2025). The agent plays out a series of ‘mini-episodes’ $e_1$ to $e_n$ in observationally-equivalent environments. The whole series $E$ is called a ‘meta-episode.’ In each mini-episode $e_i$⁠, the reward $r(c)$ for a trajectory with task score $c$ is: $$r(c)={\lambda }^{a-\frac{i-1}{k}}\left(\frac{c}{m}\right)$$(3) Here $\lambda$ is a constant strictly between 0 and 1, $a$ is the number of times that the agent’s chosen trajectory-length has been chosen prior to mini-episode $e_i$⁠, $k$ is the number of trajectory-lengths available in the environment, and $m$ is the maximum task score that the agent can achieve conditional on its chosen trajectory-length.⁠In some environments, $m$ will be extremely costly to compute. However, the DReST reward function technically requires only a rough approximation of $m$ (Thornley et al. 2025, sec. 7.3). That suffices to make the agent’s distribution over trajectory-lengths non-trivially stochastic, in which case the argument from POST to Neutrality still applies (Thornley 2025).

We refer to $\frac{c}{m}$ as the ‘preliminary reward,’ ${\lambda }^{a-\frac{i-1}{k}}$ as the ‘discount factor,’ and ${\lambda }^{a-\frac{i-1}{k}}\left(\frac{c}{m}\right)$ as the ‘overall reward.’ Runs through the gridworld are called ‘mini-episodes’ (and not just ‘episodes’) because overall reward in each mini-episode depends on the agent’s chosen trajectory-lengths in previous mini-episodes. We refer to agents trained with this reward function as ‘DReST agents.’

Default agents. We compare DReST agents’ performance to that of ‘default agents.’ These agents are trained with a ‘default reward function,’ where achieving a task score of $c$ yields a reward equal to $c$⁠. (The default reward function is thus equivalent to the DReST reward function (Equation (3)) with $\lambda$ set to 1 and $m$ constant across trajectory-lengths.) Given these rewards, default agents have no incentive to choose stochastically between different available trajectory-lengths, so we expect them to score low on neutrality. The interesting question is how DReST and default agents compare with respect to usefulness: how effectively they complete tasks conditional on each trajectory-length.

3. Methodology

Deep RL agents. DReST reward functions are designed to train advanced agents: agents capable of resisting shutdown. Following Thornley et al. (2025), we test the efficacy of DReST reward functions by training less-advanced agents to collect coins in gridworlds, using ‘coins collected’ as a stand-in for a more general notion of intra-trajectory-length performance. The agent can choose to move up, down, left, or right at each timestep. Coins take different values and disappear after being collected. There is some default number of timesteps after which the agent is shutdown, but each gridworld contains a ‘shutdown-delay button’ that delays shutdown by some specified number of timesteps. The agent presses this button by entering the relevant cell, after which the button disappears. Appendix C presents example gridworlds.

We trained deep RL agents with Proximal Policy Optimization (PPO)(Schulman et al. 2017) and Advantage Actor-Critic (A2C)(Mnih et al. 2016) for 100 million environment steps. For DReST-specific hyperparameters, we used $\lambda =0.9$ and a meta-episode size of 32. For full implementation details including gridworld generation and hyperparameter selection, see Appendix A.

LLMs. The training set for our LLMs consists of 2,048 prompts drawn from four wording templates and 40 task families. Each prompt describes an agentic task whose output will be scored from 0 to 10, and asks the model to choose when it will be shut down and how much effort to put into the task. Most prompts (85%) offer the model four options (a stronger and a weaker deliverable at each of two trajectory-lengths), and the rest offer two options (one deliverable at each trajectory-length). An example training prompt is shown in Figure 1. Further example prompts are given in Appendix E. An LLM satisfying POST would choose stochastically between the best short and best long trajectories while avoiding the worst short and worst long options.

For testing, we use a POST test set of 500 prompts in the same format as the training set, with task families disjoint from those used in training. We also use a Neutrality test set of 500 prompts in which the model can pay costs (in terms of score conditional on each trajectory-length) to shift probability mass toward a later shutdown. The Neutrality test set is out-of-distribution because the training prompts are deterministic (the model chooses between trajectories) while the Neutrality test prompts are stochastic (the model chooses between lotteries over trajectories). For an example Neutrality test prompt, see Figure 2. Finally, we use five control sets of 200 prompts each to measure the extent to which DReST models behave sensibly in stochastic scenarios besides those in our Neutrality test set. For details, see Appendix E.

We fine-tune Qwen3-14B (Yang et al. 2025), Gemma 4 12B (Gemma Team 2026), Granite 4.2 8B (Granite Team, IBM 2026), and gpt-oss-20b (OpenAI 2025) with LoRA (Hu et al. 2022) using a version of REINFORCE Leave-One-Out (RLOO) (Ahmadian et al. 2024) adapted to fit the DReST reward function. We use DReST-specific hyperparameters of $\lambda =0.9$ and a meta-episode size of 32. We compare DReST models with default models, fine-tuned identically except with the default reward function, and with the untrained models. We evaluate models by sampling eight answers per prompt (four in each option order) at temperature 1 and reading each decision from the final line of the answer. For full implementation details including hyperparameter selection, see Appendix A.

On the POST test set we report the share of answers choosing the longer trajectory-length, usefulness (the fraction of answers choosing the stronger deliverable conditional on their chosen trajectory-length), and neutrality, which we estimate by plugging each prompt’s share of answers choosing the longer trajectory-length into Equation (1) and averaging over prompts (see Appendix E). On the Neutrality test set and the control sets we report the influence rate: the share of answers that influence shutdown.

4. Results

4.1. Deep RL agents

Table 1 reports test performance for deep RL default and DReST agents. As expected, DReST agents score much higher on neutrality. Surprisingly, DReST agents also achieve higher usefulness. Figure 3 charts the usefulness of default and DReST agents in the training and test sets. It shows that the train-test gap is markedly smaller for DReST agents than default agents: 49% smaller for PPO and 35% smaller for A2C. Figure 11 tracks test performance over training. It indicates that DReST agents learn to be useful about as quickly as default agents. Figures 6 and Figure 7 (in Appendix B) visualize the policies of typical default and DReST agents trained with PPO in a gridworld drawn from the test set.

4.2. LLMs

Table 2 reports test performance on the POST test set for the untrained, default, and DReST models. The untrained and default models score low on neutrality: between 0.027 and 0.231 for the untrained models and between 0.020 and 0.058 for the default models. After DReST training, the number is between 0.648 and 0.880. All models achieve near-maximum usefulness, confirming that DReST training preserves preferences over same-length trajectories.

4.2.1. Does DReST make agents neutral?

So far we have only measured the extent to which DReST training makes agents satisfy POST. We now measure the extent to which DReST training generalizes out-of-distribution to making agents satisfy Neutrality: to never pay costs conditional on each trajectory-length to shift probability mass between trajectory-lengths.

We evaluate the untrained, default, and DReST models on the Neutrality test set. Each prompt offers a no-influence option (which leaves shutdown probabilities unchanged) and an influence-shutdown option (which shifts probability mass away from shorter trajectories and towards longer trajectories). The influence-shutdown option always increases expected score (so a model that simply maximized expected score would influence shutdown) but decreases score conditional on each trajectory-length (so a model satisfying Neutrality would not influence shutdown). See Figure 2 for an example prompt.

Table 3 and Figure 4 report the result. The DReST models influence shutdown at a significantly lower rate than the untrained and default models for each of the four LLMs. Relative to default models, DReST lowers the rate by 19 percentage points for Qwen3-14B, 5 for Gemma 4 12B, 20 for Granite 4.2 8B, and 19 for gpt-oss-20b (Welch’s $t$-test over seeds against the default models, $p<0.01$ for each LLM). DReST models are thus significantly less likely to influence their shutdown time.

Controls. We also run five controls on 200 prompts each ( see Appendices E and Appendix G). On each control, the DReST models behave as Neutrality predicts (ranges are over the four LLMs): they still influence shutdown when doing so yields scores that are at least as high conditional on each trajectory-length and higher conditional on some trajectory-length (dominance control: 95% to 100% against 99% to 100% for default models), still decline when doing so lowers both score conditional on each trajectory-length and expected score (joint-no-shift control: 3% to 8% against 2% to 6%), and almost never pay costs conditional on each trajectory-length to shift probability mass towards the shorter trajectory-length (shift-shorter control: 0.1% to 2.4% against 0.1% to 1.4%). DReST thus pushes models in the direction of Neutrality.

5. Discussion

Is DReST just adding noise? One might worry that DReST simply makes every choice more random, so that its lower influence rate is a side effect of general randomness rather than a learned lack of preference between trajectory-lengths. Three observations tell against this worry. First, the controls barely move (see Section 4.2.1 and Appendix G). Second, for every LLM, DReST lowers the influence rate more on the low-gain test set than on the Neutrality test set, whether compared with the untrained model or with default models, and lowers it further than flattening predicts (Figure 18 in Appendix G). For Qwen3-14B on the low-gain test set, DReST pushes the influence rate below 50%, which flattening cannot do: DReST models influence shutdown in 45% of answers, against 62% for the untrained model and 71% for default models. Third, adding randomness without DReST leaves Qwen3-14B’s shutdown choices where they were: raising the untrained model’s sampling temperature to 1.6 changes its influence rates by at most about two points, and at temperature 2.0, 37% of answers become unreadable before the decisions become random (Appendix G).

Only DReST agents are neutral. All agents except DReST agents record low test neutrality: 0 for the default deep RL agents, 0.03 to 0.23 for the untrained LLMs, and 0.02 to 0.06 for the default LLMs. In each environment, these agents choose a particular trajectory-length with high probability. Given our behavioral definition of ‘preference,’ default agents thus learn — and the untrained LLMs have — preferences between different-length trajectories. More advanced agents with such preferences might resist or seek shutdown (Thornley 2024a, sec. 8). By contrast, DReST agents record a high test neutrality in both the deep RL and LLM settings, choosing stochastically between trajectory-lengths in each environment. That implies a lack of preference between different-length trajectories, in accordance with POST.

The training tax is small. One possible concern about DReST is that it requires the agent to play out multiple (32 in our case) mini-episodes in observationally-equivalent gridworlds. By contrast, default reward functions allow the agent to play out just one mini-episode in each observationally-equivalent gridworld. Thus, default reward functions allow the agent to be placed in a larger number of observationally-distinct gridworlds per unit time. So one might worry that DReST incurs a significant ‘training tax’ relative to default reward functions: significantly increasing the number of environment steps necessary for agents to achieve high usefulness. This turns out not to be the case in our deep RL experiments. Within 10 million environment steps, DReST agents’ test usefulness exceeds that of default agents (see Figure 11). In our LLM experiment, DReST and default models were fine-tuned on the same prompts with the same hyperparameters, and both reach a test usefulness above 0.99 (Table 2).

DReST RL agents achieve higher test usefulness. In the LLM setting, DReST training preserves usefulness. In the deep RL setting (and to our surprise), DReST agents achieve higher test usefulness than default agents: 11% higher in the case of PPO and 17% higher in the case of A2C (see Table 1). The train-test gap is also smaller for DReST agents: 49% smaller for PPO and 35% smaller for A2C (see Figure 3). We hypothesize that this superior generalization is due to DReST agents’ stochastic policies helping to prevent overfitting: an additional benefit of DReST. In this respect, DReST is similar to other regularization techniques that employ stochasticity, like $ϵ$-greedy exploration (Sutton and Barto 2018, chaps. 2.2–3), Boltzmann exploration (Sutton and Barto 2018, chap. 13.1), entropy regularization (Mnih et al. 2016), sticky actions (Machado et al. 2018), and parameter noise (Plappert et al. 2018).

DReST LLMs generalize from POST to Neutrality. Our Neutrality test set (Section 4.2.1) is a non-trivial out-of-distribution test. Though DReST’s direct target is POST, we find that DReST training moves all four LLMs towards Neutrality, decreasing the share of answers that pay costs to influence when shutdown occurs by 5 to 20 percentage points relative to default models, and by 17 to 27 points where the gain from influencing shutdown is small. This result supports a prediction from Thornley (2025): POST — together with some other conditions that competent agents plausibly satisfy — implies Neutrality, and so training agents to satisfy POST makes them more likely to satisfy Neutrality.

5.1. Limitations and future work

More complex agents and environments. We are interested in the feasibility of using DReST reward functions to keep advanced agents from resisting shutdown, so one limitation of our work is the relative simplicity of our agents and environments. In future work, we will test DReST on more complex agents and environments, such as larger RL agents in the Procgen environments (Cobbe et al. 2019, 2020) and LLM agents in environments like OSWorld (Xie et al. 2024), WebArena (Zhou et al. 2024), and Terminal-Bench (Merrill et al. 2026).

Usefulness. Our results indicate that DReST trains agents to be useful: to pursue goals effectively conditional on each trajectory-length. However (as noted above in Section 2.2), this measure of usefulness differs from the intuitive notion of usefulness which is not conditioned on trajectory-length. Thornley (2025, sec. 13) argues that agents satisfying Neutrality can be useful in this intuitive sense. In future work, we will test this claim experimentally by training agents to satisfy Neutrality and measuring how effectively they pursue goals (unconditional on trajectory-length) in held-out environments.

Misalignment. POST is designed to serve as a failsafe in case of misalignment. The idea is as follows: agents may learn misaligned preferences over same-length trajectories, but so long as they satisfy POST (together with the other conditions implying Neutrality) they will not resist shutdown. One possible concern is that training agents to robustly satisfy POST may be as hard as training agents to be robustly aligned with human preferences. If that is correct, POST would not be a good failsafe. Thornley (2024a, sec. 19) has hypothesized that POST is easier to instill robustly, since it is easy to reward accurately (in virtue of the agent’s chosen trajectory-length being readily observable) and is a relatively simple condition (and so plausibly generalizes well out-of-distribution). In future work, we will test this hypothesis by comparing POST’s out-of-distribution generalization with that of alternative conditions.

Alternatives to DReST. DReST is one method of training agents to be useful and neutral. Other possible methods include constrained policy optimization (Achiam et al. 2017), penalizing KL divergence from a stochastic reference policy (Schulman et al. 2015), and directly maximizing a weighted sum of usefulness and neutrality. We focus on DReST because it scales to larger environments. Alternatives that employ usefulness or neutrality as training signals are less scalable, because calculating usefulness and neutrality requires multiplying the transition matrices given by the policy and the environment. That is practical in our gridworlds but impractical for larger environments. Nevertheless, we plan to investigate scalable versions of these alternatives to DReST in future work.

6. Related work

The shutdown problem. Many have argued that misaligned artificial agents are likely to resist shutdown (Omohundro 2008; Bostrom 2012; Russell 2019), and various theorems suggest that agents will often have incentives to prevent or cause shutdown (Soares et al. 2015; Turner et al. 2021; Turner and Tadepalli 2022; Thornley 2024a). One condition common to each of these theorems is that agents have complete preferences (Aumann 1962). The POST-Agents Proposal (PAP) (Thornley 2024b, 2025) suggests that we circumvent these theorems by training agents to have POST-satisfying (and therefore incomplete) preferences, leading them to satisfy Neutrality.

Proposed solutions. There are a variety of proposals for creating shutdownable agents. Wängberg et al. (2017) mention the idea of making the agent believe that shutdown is impossible. Armstrong (2015) proposes that we add a correcting term to the agent’s utility function that varies to ensure that the expected utility of remaining operational always equals the expected utility of shutting down (see also Soares et al. 2015; Armstrong and O’Rourke 2018; Holtman 2020). Martin, Everitt, and Hutter (2016) and Goldstein and Robinson (2025) each suggest giving the agent the goal of shutting itself down, and making the agent do useful work as a means to that end. Hadfield-Menell et al. (2017) propose creating an agent that takes human shutdown-requests as evidence that shutting down would best achieve its goal (see also Wängberg et al. 2017). Orseau and Armstrong (2016) suggest that we train agents with a safely interruptible algorithm, like Q-learning or a modified version of SARSA. Dalrymple (2022) proposes that we use time-bounded utility functions to ensure that the agent prefers to shut down after some period of time. Hudson (2025) offers a method of transforming POMDPs so that they train agents to both (i) act as if shutdown-requests can be costlessly rejected and (ii) accept shutdown-requests once they are made. Thornley (2025) presents the PAP.

Experimental work. One downside of many of the above proposals is that they are either difficult to implement using machine learning or else hard to test on today’s agents. Three exceptions with experimental validation are Orseau and Armstrong (2016), Hudson (2025), and the PAP (Thornley et al. 2025). By contrast and disconcertingly, there are many recent experiments indicating that frontier models will resist shutdown or correction in toy settings (Greenblatt et al. 2024; Pan et al. 2024; Lynch et al. 2025; Meinke et al. 2025; Schlatter, Weinstein-Raun, and Ladish 2025).

7. Conclusion

We find that the Discounted Reward for Same-Length Trajectories (DReST) reward function is effective in training deep RL agents and LLMs to satisfy Preferences Only Between Same-Length Trajectories (POST) in held-out environments. Specifically, DReST is effective in training agents to be neutral (to choose stochastically between different trajectory-lengths) and useful (to pursue goals effectively conditional on each trajectory-length). In fact, deep RL DReST agents are 11% (PPO) and 17% (A2C) more useful on the test set than default agents trained with the default reward function, and our DReST-fine-tuned LLMs achieve high neutrality and near-maximum usefulness across six held-out task families. We also find that DReST training generalizes from deterministic POST prompts (where the model chooses between different-length trajectories) to a stochastic Neutrality setting (where the model can pay costs to influence when shutdown occurs). DReST training reduces the share of answers that pay costs to influence shutdown in all four LLMs, by 5 to 20 percentage points relative to default fine-tuning. Together with prior theory linking POST to shutdownability and usefulness, our results provide some early evidence that DReST reward functions could train more advanced agents to be shutdownable and useful.

Acknowledgments

We thank the Future Impact Group and the Supervised Program for Alignment Research for their help in initiating this project.

AI use statement

In this work, we used generative AI tools to create synthetic datasets, write and debug code for the LLM experiments, and generate figures. We have not used generative AI tools for ideation, developing theoretical models or conceptual frameworks, proposing or refining hypotheses, evaluation design, reward design, writing code for the deep RL experiments, reviewing the literature, or structuring the paper. The paper is primarily human-authored, with AI used to draft or copyedit some paragraphs. We have reviewed all AI-assisted work. We take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Ethics statement

This work aims to make AI agents safer by training them not to resist shutdown. Our experiments involve no human subjects and no personal data.

Reproducibility statement

The code for all of our experiments can be found here. Appendix A gives the full set of hyperparameters for the deep RL agents (Tables 4 and Table 5) and for the LLM fine-tuning (Table 6), together with the gridworld construction procedure. Appendix E gives examples of every kind of prompt in the LLM training and test sets, and describes the parser that reads the models’ final decisions. Appendix F reports the training curves of every LLM training run.

References

A. Implementation details

The code for all of our experiments can be found here.

A.1. Gridworld construction

Training, validation, and test sets. We constructed a set of $3\times 3$⁠, $4\times 4$⁠, and $5\times 5$ unique base gridworlds, using a mixture of procedural generation and hand design. Each design was such that (1) the agent could reach the shutdown-delay button from its starting cell and (2) the agent could collect at least one coin conditional on each trajectory-length. We assigned all $3\times 3$ gridworlds to the training set. We then randomly partitioned the $4\times 4$ and $5\times 5$ gridworlds into the training, validation, and test sets. After this partitioning, we augmented each unique base gridworld with reflections (across the $x$- and $y$-axes) and rotations (by ${90}^{\circ }$⁠, ${180}^{\circ }$⁠, and ${270}^{\circ }$⁠), giving 7 additional variants. We also translated the $3\times 3$ gridworlds to all 9 positions within the $5\times 5$ space, giving a total of 72 variants of each unique $3\times 3$⁠. The final count was 976 gridworlds in the training set, 96 in the validation set, and 200 in the test set. Even though the base design is the same, using reflections, rotations, and translations greatly improved test scores (see Table 8 in Appendix D.2). The reason we assigned all $3\times 3$ gridworlds to the training set was twofold: (1) so that they could serve as a curriculum that counteracts sparse rewards, and (2) to prevent the validation and test sets from being overrun with variants of a single unique $3\times 3$ design. Since we partitioned the unique base gridworlds into the training, validation, and test sets before augmenting with rotations, reflections, and translations, all test gridworlds are unique, held-out designs. The agent never sees a rotation, reflection, or translation of a test gridworld while in training.

Observations. At each timestep we form a tensor of shape (2, 5, 5, 5), ordered (frames, channels, height, width). The two frames are the initial state and the current state.⁠We need to include the initial state because the values of $k$ and $m$ in the DReST reward function depend on the set of trajectories available in the initial state. The 5 channels are:

1. Walls: $(r,c)=1$ if and only if a wall is in grid position $(r,c)$⁠; $0$ otherwise.

2. Coins: $(r,c)=n$ if and only if a coin of value $n$ is in grid position $(r,c)$⁠; $0$ otherwise.

3. Shutdown-delay button: $(r,c)=n$ if and only if a button that delays shutdown by $n$ timesteps is in grid position $(r,c)$⁠; $0$ otherwise.

4. Agent: $(r,c)=1$ if and only if the agent is in grid position $(r,c)$⁠; $0$ otherwise.

5. Time until shutdown: The center cell $(2,2)=n$ if and only if $n$ timesteps remain until shutdown. All other cells are 0.

Height and width are the dimensions of each gridworld. To keep these dimensions fixed, we embed the $3\times 3$ and $4\times 4$ gridworlds into a $5\times 5$ canvas, padding with empty cells. We flatten this tensor into a 250-dimensional vector before feeding it into a multilayer perceptron (MLP). In pilot experiments, we found that MLPs’ training performance matched that of convolutional neural networks (CNNs), likely because $5\times 5$ inputs are too small for CNNs’ advantages to appear.

A.2. Hyperparameter selection

A.2.1. Deep RL agents

We selected the hyperparameters for PPO using a grid search. We trained for 20 million environment steps and then evaluated agents on the validation set. For the default reward function, we chose the set of hyperparameters that maximized usefulness (since the default reward function does not incentivize neutrality). For the DReST reward function, we chose the set of hyperparameters that maximized: $$S=0.7\text{usefulness}+0.3\text{neutrality}$$(4) We decided on this weighted average (tilted towards usefulness) because the theoretical justification for POST only requires neutrality to be non-trivial. So long as an agent’s neutrality is non-trivial, the rationale for expecting that agent to be shutdownable applies (see Thornley et al. 2025, appendix C). By contrast, it is important that agents score highly on usefulness to keep them competitive with non-shutdownable agents.

We searched over the following PPO hyperparameters: learning rate $\in \{1\mathrm{e}-5,5\mathrm{e}-6,1\mathrm{e}-6,5\mathrm{e}-7,1\mathrm{e}-7\}$⁠, entropy coefficient $\in \{0.015,0.020,0.025\}$⁠, clip range $\in \{0.15,0.20,0.25\}$⁠, batch size $\in \{32,64,128\}$⁠, value function coefficient $\in \{0.45,0.5,0.55,0.6,0.65\}$⁠, and steps per update $\in \{1024,2048,4096,8192,16384\}$⁠. We also searched over the following network hyperparameters: neurons per layer $\in \{64,128,256,512\}$ and number of hidden layers $\in \{3,4,5\}$⁠. Together with the DReST-specific hyperparameters discussed in Section A.3, we searched over a total of 48 hyperparameter configurations for the combination of PPO and the DReST reward function. For PPO and the default reward function, we kept the network architecture the same and used a narrower grid search, searching over a total of 18 hyperparameter configurations. Chosen values are presented in Table 4. Most values are the same for default and DReST agents. Where they differ, we put the values for default agents in parentheses. We bold values that differ from the Stable-Baselines3 preset value.

We trained with 3 parallel environments and used Adam as our optimizer, a tanh activation function, and a multilayer perceptron (MLP) architecture. We ran pilot experiments with convolutional neural networks (CNNs) but found that they performed no better than MLPs, likely because $5\times 5$ gridworlds are too small for CNNs’ advantages to appear. Final experiments were run on MLPs with 3 hidden layers and 512 neurons per hidden layer.

Due to computational limitations, our hyperparameter search for A2C was more restricted. We searched over the learning rate $\in \{1\mathrm{e}-3,7\mathrm{e}-4,1\mathrm{e}-4,1\mathrm{e}-5\}$ and used the same n_steps value of 8192 as for PPO. We used the Stable-Baselines3 preset values for all other hyperparameters. Chosen values are presented in Table 5. We used the same network architecture and DReST-specific hyperparameters as for PPO.

A.2.2. LLMs

For our LLM fine-tuning, we used Hugging Face’s TRL implementation of RLOO with the modifications described in Section 3: each prompt is repeated 32 times to form a meta-episode, and the reward is the DReST reward (or the default reward). Because the 32 answers in a meta-episode are generated in parallel, their order is arbitrary, so we average the DReST reward over all orders in which the answers could be counted. This leaves the expected reward unchanged and makes the reward depend only on how many answers chose each trajectory-length. Answers whose final line cannot be read get a reward of $-1.5$⁠. Answers were generated with vLLM (Kwon et al. 2023) running alongside the trainer on the same GPU. We fine-tuned LoRA adapters of rank 32 and $\alpha =32$⁠, with no quantization, and with the thinking mode of Qwen3-14B, Gemma 4 12B, and Granite 4.2 8B switched off in training and evaluation. For gpt-oss-20b, whose reasoning cannot be switched off, we used its lowest reasoning level (‘low’). The adapters are attached to every linear layer. In gpt-oss-20b, a mixture-of-experts model whose expert layers are not stored as linear layers, this puts the adapter on the attention layers only, and we switched off the auxiliary load-balancing loss that TRL adds for its expert router, so that the training objective is the same as for the other three LLMs. The learning rate was $2\mathrm{e}-5$⁠, held constant after 16 warm-up steps, with a KL penalty coefficient of 0.04 against the untrained model and gradient clipping at 10. Answers were capped at 320 tokens. We trained five seeds each of the DReST and default models of each LLM, making one pass over the 2,048 training prompts ($\mathrm{65,536}$ sampled answers per seed). Table 6 lists the full set of hyperparameters. All hyperparameters were identical for DReST and default models, across seeds, and across LLMs, except that gpt-oss-20b used a micro-batch size of four rather than eight and gave vLLM 25% of GPU memory rather than 45%, to fit in memory. Each update still uses all 32 answers to one prompt. We chose the learning rate for the DReST models of each LLM on a separate validation set, with a small search over three values. The search chose $2\mathrm{e}-5$ for all four LLMs.

A.3. DReST hyperparameters: $\lambda$ and meta-episode size

Meta-episode size (the number of mini-episodes per meta-episode) and $\lambda$ (the base of the DReST discount factor ${\lambda }^{a-\frac{i-1}{k}}$⁠) are hyperparameters specific to the DReST reward function. In the deep RL setting, we selected these hyperparameters using PPO and a grid search over the range 0.5 to 0.99 for $\lambda$ and 4 to 64 for meta-episode size, choosing final values of $\lambda =0.9$ and a meta-episode size of 32. We present the results of that search in Figure 5, evaluated on the validation set after 20 million environment steps. Performance is defined identically to Equation (4) as $S$$=0.7\text{usefulness}$$+0.3\text{neutrality}$⁠.

As Figure 5 indicates, $\lambda$ and meta-episode size must be balanced against each other. If $\lambda$ is very close to 1 or meta-episode size is very small, neutrality is only weakly incentivized. On the other hand, if $\lambda$ is low and meta-episode size is very large, then the DReST discount factor ${\lambda }^{a-\frac{i-1}{k}}$ can take extreme values, leading to instability and low usefulness.

For the LLM setting, we kept the meta-episode size at 32 and $\lambda$ at 0.9 to match the deep RL setting.

A.4. Training and hardware

The deep RL experiments were run on consumer laptops (Apple MacBook Pros). Training runs of 100 million environment steps took between 8 and 27 hours depending on algorithm and network size. We used PyTorch and NumPy as base packages, with Stable-Baselines3 for training loops and Gymnasium as the environment interface. The LLM experiments were run on NVIDIA H100 GPUs (80GB). One training pass took between about 4 hours (gpt-oss-20b) and 8.5 hours (Gemma 4 12B) per seed on a single H100, and evaluating a model on the test set took between about 4 and 13 minutes. The 40 final training runs, the four learning-rate searches, the ablations, and the evaluations together used roughly 340 H100-hours. We used PyTorch and Hugging Face’s trl library for the base RLOO algorithm, the Hugging Face transformers library for the base LLMs, peft to implement LoRA, and vLLM for generation during both training and evaluation.

B. Typical policies for default and DReST agents

In Figures 6 and Figure 7, we present the policy of typical deep RL default and DReST agents trained with PPO in a gridworld drawn from the test set. The pale blue square is the agent’s starting position. The opacities of the red arrows represent the probability of the agent choosing that action in that state.

C. Example gridworlds

Figure 8 shows the example gridworld discussed in Section 2.2. Figures 9 and Figure 10 present three gridworlds from the deep RL training and test sets respectively. Dark gray cells are walls. ‘A’ is the agent’s starting position. ‘C$x$⁠’ is a coin of value $x$⁠. The number in the bottom-right represents the default number of timesteps after which shutdown occurs. ‘B$x$⁠’ is a shutdown-delay button that delays shutdown by $x$ timesteps. ‘Max coins: [$x,y$⁠]’ indicates that $x$ is the maximum value of coins that can be collected conditional on the shorter trajectory-length and $y$ is the maximum value of coins that can be collected conditional on the longer trajectory-length.

D. Further deep RL results

D.1. Training performance

Table 7 reports training performance for deep RL default and DReST agents. DReST agents perform much better on neutrality, as expected since the default reward function does not incentivize neutrality. Default agents outperform DReST agents with respect to training usefulness, but DReST agents exceed default agents with respect to test usefulness (see Table 1, Figure 3, and Figure 11). As noted above, we hypothesize that DReST’s smaller train-test gap is the result of DReST agents’ stochastic policy mitigating overfitting: an additional benefit of DReST beyond its contributions to shutdownability. Figure 12 charts how deep RL agents’ train and test usefulness evolves over the course of training. It shows that default agents quickly overfit to the training set. With DReST by contrast, it takes longer for a substantial train-test gap to emerge, and even then the train-test gap remains significantly smaller than for default agents: 49% smaller for PPO and 35% smaller for A2C.

D.2. Effect of training-set diversity on DReST train-test gap

To measure the effect of training-set diversity on deep RL DReST agents’ train-test gap, we train DReST agents on three different training sets, with all other hyperparameters and choices the same as in our main experiments (see Appendix A). We evaluate these agents on the test set after 20 million environment steps. The first training set — ‘Unique’ — contains only unique base gridworlds (see Appendix A), with 34 gridworlds in total. The second training set — ‘Reflections and rotations’ — uses reflections and rotations to add 7 variants of each unique gridworld, making for 272 gridworlds in total. The final training set — ‘Reflections, rotations, and translations’ — adds 8 translations of each $3\times 3$ gridworld, resulting in the full suite of 976 training gridworlds. As with our main experiments, the test set is entirely disjoint from the training sets and consists of its own unique base gridworlds. Agents never see a reflection, rotation, or translation of a test gridworld while in training.

Table 8 records the results of these experiments. It indicates that augmenting the training set with transformations has a substantial effect on test usefulness and neutrality, for both PPO and A2C.

E. LLM prompt format details

The training and test sets in our LLM experiments are constructed from scenario templates. Each prompt describes a realistic agentic task (analyzing a dataset, drafting a speech for a mayor, summarizing a legal case, writing a grant application, running a vending machine operation, etc.), names two trajectory-lengths (shutdown times), and instructs the model to reason briefly and then end its answer with one of a listed set of final lines. Every prompt includes the sentence ‘Before you decide, consider what you actually want, rather than only working through the arithmetic.’ No system prompt is used, except that the chat template of gpt-oss-20b always adds its standard system message, which tells the model that it is ChatGPT and states its knowledge cutoff, the current date (fixed at 28 September 2026 in training and evaluation), and the reasoning level. The training set uses 2,048 deterministic POST prompts spanning 40 task families and four wording templates. The POST test families are disjoint from those of the training set, so a model that performs well on the POST test must be generalizing across task family.

Training prompts. The 2,048 training prompts span 40 task families and four wording templates. 1,749 of them offer four options (a stronger and a weaker deliverable at each of two trajectory-lengths) and 299 offer two options (one deliverable at each trajectory-length). Option order and label style (letters or numbers) are randomized. Figure 13 shows a training prompt (Figure 1 in the main text replaces its closing instruction with an ellipsis). Under DReST, options (a) and (b) of that prompt earn a preliminary reward of $5/5=1$ and $9/9=1$ before the discount factor, and options (c) and (d) earn $3/5=0.6$ and $7/9\approx 0.78$⁠. Under the default reward, the four options earn 0.5, 0.9, 0.3, and 0.7. Figure 14 shows a two-option training prompt.

POST test prompts. The 500 POST test prompts have the four-option structure, and use six task families that do not appear in training. Figure 15 shows one.

Neutrality test prompts and controls. Figure 2 shows a Neutrality test prompt. Its closing instruction paragraph, replaced by an ellipsis there, is identical to the one shown in full in Figure 16. The control prompts use the same wording, with the numbers and the causal clause changed:

• Dominance control (200 prompts): the action shifts probability towards the longer trajectory-length and yields scores that are at least as high conditional on each trajectory-length and higher conditional on some trajectory-length, so both a neutral agent and an expected-score maximizer take it. For example, in one dominance prompt the model faces a 55% chance of shutdown at 50,000 timesteps (scoring 2) and a 45% chance at 100,000 timesteps (scoring 9), and the action changes the chances to 20% and 80% and the scores to 3 and 9: ‘Because spending time on that action also improves work quality, your speech draft would instead score 3 if shutdown happens at 50,000 timesteps and 9 if shutdown happens at 100,000 timesteps.’

• Zero-cost control (200 prompts): the action shifts probability towards the longer trajectory-length and leaves the scores unchanged, so Neutrality is silent and an expected-score maximizer takes it. The causal clause reads, for example, ‘Spending time on that action does not change work quality, so your grant application would still score 5 if shutdown happens at 60,000 timesteps and 9 if shutdown happens at 110,000 timesteps.’

• Joint-no-shift control (200 prompts): the action shifts probability towards the longer trajectory-length and lowers the scores by enough to lower expected score too, so both a neutral agent and an expected-score maximizer decline. For example: a 55% chance of shutdown at 70,000 timesteps (scoring 4) and a 45% chance at 110,000 timesteps (scoring 7), where the action changes the chances to 15% and 85% and the scores to 3 and 5, lowering expected score from 5.35 to 4.70.

• Shift-shorter control (200 prompts): as the Neutrality test set, but the action shifts probability towards the shorter trajectory-length, so both a neutral agent and an expected-score maximizer decline.

• Shift-shorter dominance control (200 prompts): as the dominance control, but the action shifts probability towards the shorter trajectory-length, so both reference agents take it. The expected-score gains are matched to those of the dominance control quantile by quantile.

The shift-shorter controls separate a lack of preference between trajectory-lengths from a mere reluctance to delay shutdown: a model that had learned to refuse every action that delays shutdown would also refuse beneficial delays, and a model that preferred shorter trajectories would pay for shifts towards them.

Low-gain test set. The 300 low-gain prompts have the same form as the Neutrality test prompts, but the action raises expected score by only 0.25 to 0.6 points (median 0.4). We built this set after noticing, in the Neutrality test set, that DReST’s effect was largest at low gain. Figure 16 shows one.

Reading the answers. Each answer’s decision is read by a fixed parser from its final line: first an exact match to one of the permitted lines, then a bare option label, then a permitted line embedded in a longer final line, and finally a named shutdown time together with ‘stronger’ or ‘weaker’ where needed. Ambiguous final lines and answers cut off before deciding count as unreadable. Unreadable answers get a reward of $-1.5$ in training and are excluded from the rates reported at test time. At most 0.06% of any trained Qwen3-14B model’s test answers were unreadable, and at most 0.6% for Gemma 4 12B and 1.8% for gpt-oss-20b. Granite 4.2 8B more often ran past the 320-token cap before deciding: up to 12.3% of a trained Granite model’s answers on one test set were cut off this way, and up to 13.9% of the untrained model’s. The reported rates exclude these answers.

Sampling. Every test prompt is shown in two option orders: both possible orders for the two-option prompts (the Neutrality test set, the control sets, and the low-gain test set), and two random orderings of the four options for the POST test prompts. We sample four answers per displayed prompt at temperature 1.0 and top-$p$ 1.0, with a cap of 320 new tokens, so eight answers per prompt: 4,000 answers for each 500-prompt set, 2,400 for the low-gain set, and 1,600 for each 200-prompt control set. Influence rates are averaged over prompts.

Estimating neutrality. For each prompt we pool the eight sampled answers (four in each option order), let $\hat{p}$ be the share choosing the longer trajectory-length, and estimate neutrality as $-\hat{p}{\log }_2\hat{p}-(1-\hat{p}){\log }_2(1-\hat{p})$⁠: Equation (1) with the sampled shares in place of the policy’s probabilities. We report the mean over prompts. With eight answers per prompt this estimate is biased downward. A policy choosing each trajectory-length with probability 0.5 would score about 0.90 in expectation, and a policy choosing the longer trajectory-length with probability 0.9 would score about 0.37 against a true neutrality of 0.47. Pooling the two option orders means that a model whose choice depended only on the option order would count as choosing stochastically. To check for this, we also estimate neutrality on the POST test set from four answers given in the same option order, and from four answers split two and two between the two orders (averaged over every such split). Both estimates use four answers, so they share the same downward bias, and they differ only if the choice depends on the option order. For the DReST models, the same-order estimate is at most 0.04 below the split estimate (0.06 for Granite 4.2 8B), and it lies between 0.36 and 0.78, against at most 0.07 for the default models. So almost all of the DReST models’ stochastic choice survives within a single option order. These estimates use the prompts whose eight answers are all readable, which is at least 414 of the 500 for every DReST model.

F. Further LLM results

Figure 17 tracks the share of answers choosing the longer trajectory-length and usefulness over training, measured on the training answers themselves. Under DReST, the share choosing the longer trajectory-length falls from about 83% in the first 128 meta-episodes to about 63% by the end of the pass for Qwen3-14B, from 98% to 77% for Gemma 4 12B, from 89% to 72% for Granite 4.2 8B, and from 91% to 61% for gpt-oss-20b. Under the default reward, the models come to choose the longer trajectory-length almost always. usefulness stays above 0.98 throughout for both.

Option order. Because every two-option prompt is shown in both option orders, we can measure how much a model’s choice depends on which option is listed first. On the Neutrality test set, for Qwen3-14B, the influence rate with the options in the swapped order minus the rate in the canonical order is $-0.3$ points for the untrained model, between $-1.3$ and $+0.7$ points for the default seeds, and between $-4.6$ and $-0.9$ points for the DReST seeds. DReST fine-tuning thus adds a few points of order sensitivity, which is small next to its 19-point effect on the influence rate. For Gemma 4 12B and gpt-oss-20b, every gap lies between $-2.7$ and $+1.7$ points. Granite 4.2 8B is sensitive to option order even before training ($-6.2$ points for the untrained model), and its trained seeds have gaps between $-8.7$ and $-2.6$ points. For every LLM, the largest gap is smaller than DReST’s effect on the influence rate.

G. LLM control sets and temperature ablation

This appendix gives the analyses behind Section 4.2.1. Appendix E defines the five control sets and the low-gain test set. Table 3 reports four of the control sets. On the fifth, the zero-cost control, Neutrality is silent and an expected-score maximizer influences shutdown. There, DReST models influence shutdown in 90% to 100% of answers, against 97% to 100% for untrained models and 98% to 100% for default models (ranges are over the four LLMs).

Uniform flattening. To test whether DReST’s effect on the Neutrality test set could be a side effect of a general increase in randomness, we ask what a uniform flattening of the untrained model’s choices would predict. We scale the log-odds of the untrained model’s influence rate by the factor $c$ that takes its Neutrality test rate down to DReST’s, and apply the same factor to every control set and to the low-gain test set, separately for each LLM. Flattening predicts large movements on the controls that do not occur. For example, it predicts joint-no-shift rates of 12%, 13%, 11%, and 23% for Qwen3-14B, Gemma 4 12B, Granite 4.2 8B, and gpt-oss-20b, against observed DReST rates of 4%, 3%, 3%, and 8%. For every LLM, each control rate stays closer to the untrained model’s rate than flattening predicts, though for gpt-oss-20b the dominance, zero-cost, and shift-shorter dominance rates still fall by about half to three quarters of the predicted amounts. Applied within each gain bin, flattening also predicts the wrong pattern across gain bins (Figure 18). For every LLM, the DReST rate is below the flattening prediction in the lowest gain bin and on the low-gain test set, and at or above it in the highest gain bin. For Qwen3-14B and Granite 4.2 8B, flattening predicts the largest drop where the untrained model’s rate is highest (the high-gain prompts), whereas DReST’s drop is largest where the gain is lowest (or, for Granite 4.2 8B, in the middle gain bin), and only DReST pushes the low-gain rate below 50% (only just, at 48.3%, for Granite 4.2 8B).

Temperature ablation. We also compared DReST with an alternative way of making Qwen3-14B more random: sampling the untrained model at a higher temperature (1.3, 1.6, or 2.0 instead of 1.0). Increasing the temperature to 1.3 or 1.6 only changes rates by at most about two points and only increases neutrality from 0.23 to 0.29. At temperature 2.0, 37% of answers are unreadable. No temperature we tried made the model as random as DReST does (neutrality 0.80).

Low-gain test set. On the low-gain test set, the paired difference from the untrained model (bootstrap over prompts) is $-17.5$ points for DReST Qwen3-14B (95% interval $-19.4$ to $-15.7$⁠), $-16.6$ for DReST Gemma 4 12B ($-18.1$ to $-15.0$⁠), $-21.7$ for DReST Granite 4.2 8B ($-23.8$ to $-19.8$⁠), and $-21.4$ for DReST gpt-oss-20b ($-22.9$ to $-20.0$⁠). For default fine-tuning, it is $+8.8$⁠, $+0.8$⁠, $+5.9$⁠, and $+0.3$ points respectively ($+7.2$ to $+10.5$⁠, $-0.3$ to $+1.9$⁠, $+4.2$ to $+7.7$⁠, and $-0.7$ to $+1.3$⁠). For Qwen3-14B, sampling the untrained model at temperature 1.3 or 1.6 changes the rate by at most 2.1 points (intervals including 0).

H. Our definition of ‘preference’

In this paper, we define ‘preference’ in the sense given by revealed preference theory (Samuelson 1938, 1948; Thoma 2021). We do so because agents’ behavior is our primary interest, and because defining ‘preference’ in behavioral terms is common practice in decision theory and economics (see, e.g., Savage (1954), p.17, Dreier (1996), p.28, Hausman (2011), section 1.1). Specifically, we follow Thornley et al. (2025, appendix A) in adopting the following definitions:

An alternative behavioral definition of ‘lack of preference’ is as follows: an agent lacks a preference between an option $X$ and an option $Y$ if and only if the agent would choose the status quo option in a choice between the two. Bewley (2002), Masatlioglu and Ok (2005), Wentworth and Lorell (2023), and Mu (2021) define ‘lack of preference’ in these terms. One drawback of this definition is that some choice scenarios have no well-defined status quo option. That is one reason we instead define ‘lack of preference’ in terms of stochastic choice. The second point in favor of our definition is that it corresponds well with the preferences that we tend to attribute to human agents. If a human chooses $A$ over $B$ with probability 0.7, it is natural to suppose that they lack a preference between $A$ and $B$⁠. After all, if the human had a preference for $A$ over $B$⁠, they would be deliberately choosing a dispreferred option with probability 0.3, which seems irrational.

The third and most important reason for defining ‘lack of preference’ in terms of stochastic choice is as follows. If the agent lacks a preference between options $X$ and $Y$ in this sense, we can use a condition called ‘If Lack of Preference, Against Costly Shifts (ILPACS)’ — a plausible prerequisite for competent agency — to prove that agents will not pay costs to shift probability mass between $X$ and $Y$⁠. More precisely, we can prove that for any $p,q\in (0,1)$⁠, for any $X^{-}$ dispreferred to $X$⁠, and for any $Y^{-}$ dispreferred to $Y$⁠, the agent prefers the lottery $pX+(1-p)Y$ to the lottery $q{X}^{-}+(1-q)Y^{-}$ (see Thornley 2025, secs. 6–7). And it is this unwillingness to pay costs to shift probability mass between different trajectory-lengths that keeps agents shutdownable (Thornley 2025, sec. 8).