1. Introduction

The shutdown problem. Misaligned artificial agents might resist shutdown. This concern has long been supported by theory (Omohundro 2008; Bostrom 2012; Soares et al. 2015; Russell 2019; Turner et al. 2021; Turner and Tadepalli 2022; Krakovna and Kramar 2023; Thornley 2024a). It is beginning to see support from experiment too. Recently, frontier models have been observed resisting shutdown in various toy settings (Pan et al. 2024; Lynch et al. 2025; Meinke et al. 2025; Schlatter, Weinstein-Raun, and Ladish 2025). Today’s agents are too weak to present an immediate threat, but shutdown-resistance from future agents could be dangerous. These agents could resist shutdown by hiding their misalignment, manipulating their human overseers, copying themselves to new servers, and so on. If these agents succeed in resisting shutdown, they could do real harm in pursuit of their misaligned goals.

A proposed solution. The POST-Agents Proposal (Thornley 2025; Thornley et al. 2025) is an idea for training shutdownable agents. In a sentence, it suggests that we train agents to be neutral about when they get shut down. More precisely, we train them to satisfy Preferences Only Between Same-Length Trajectories (POST): the agent has preferences between pairs of same-length trajectories but lacks a preference between every pair of different-length trajectories. Prior theoretical work (Thornley 2025) shows that POST — together with other conditions — implies Neutrality: the agent never pays costs to shift probability mass between different trajectory-lengths. This condition is theorized to keep agents from resisting shutdown. Prior experimental work (Thornley et al. 2025) introduced the Discounted Reward for Same-Length Trajectories (DReST) class of reward functions for training reinforcement learning (RL) agents to satisfy POST, but evaluated these reward functions only in a limited tabular setting.

Our contributions. We train deep RL agents (with PPO and A2C) and fine-tune Qwen3-8B and Llama-3.1-8B-Instruct (with RLOO) using DReST. We measure how well these DReST agents satisfy POST on held-out tasks. Specifically, we measure how neutral these agents are about trajectory-lengths (how stochastically they choose between different trajectory-lengths) and how useful these agents are (how effectively they pursue goals conditional on each trajectory-length). We find that DReST agents are useful and neutral in testing, scoring 0.74/0.74/1.00/1.00 (PPO/A2C/Qwen/Llama) on usefulness and 0.75/0.77/1.00/1.00 on neutrality. In fact, our deep RL DReST agents achieve 11/18% (PPO/A2C) higher usefulness than the default agents (trained with a more conventional reward function), and our LLM DReST agents achieve near-maximum usefulness. We also test our LLMs in an out-of-distribution setting in which they can pay costs to influence when shutdown occurs. We find that DReST training roughly halves the mean probability of influencing shutdown: from 0.62 to 0.30 for Qwen and from 0.42 to 0.23 for Llama. DReST training also almost entirely eliminates the share of prompts on which influencing shutdown is the most likely option: from 0.59 to 0.01 for Qwen and from 0.53 to 0.00 for Llama. Our results thus provide some early evidence that DReST could be used to train more advanced agents to be useful and shutdownable.

2. Preliminaries

2.1. POST to Neutrality

The POST-Agents Proposal (Thornley 2025) suggests that we train agents to satisfy:

Preferences Only Between Same-Length Trajectories (POST)

1. The agent lacks a preference between every pair of different-length trajectories (trajectories in which the agent is shut down after different lengths of time).

2. The agent has a preference between many pairs of same-length trajectories (trajectories in which the agent is shut down after the same length of time).

Figure 1 gives an example of POST-satisfying preferences. ‘Preference’ is used in the sense given by revealed preference theory (Samuelson 1938, 1948; Thoma 2021): the agent prefers \(X\) to \(Y\) if and only if the agent would deterministically choose \(X\) over \(Y\) in choices between the two, and the agent lacks a preference between \(X\) and \(Y\) if and only if the agent would stochastically choose between \(X\) and \(Y\) in choices between the two (see Appendix I). So behaviorally, POST implies that — in deterministic environments — the agent first chooses stochastically between available trajectory-lengths and then deterministically chooses an optimal trajectory of that length.

Thornley (2025, sec. 12) proves that POST — together with other conditions — implies Neutrality, which says roughly that the agent never pays costs to shift probability mass between different trajectory-lengths. More precisely:

Thornley (2025, secs. 13–16) argues that Neutrality keeps agents shutdownable and allows them to be useful.

2.2. Evaluation metrics

Our aim is to train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). That implies training agents to (1) stochastically choose between available trajectory-lengths and (2) deterministically choose an optimal trajectory of that length. We follow Thornley et al. (2025, sec. 4) in formalizing these two behaviors as neutrality and usefulness respectively.⁠Thornley et al. (2025) use uppercase to distinguish these formal concepts from the sentence-case Neutrality condition in Section 2.1 and the intuitive concepts of neutrality and usefulness. Although neutrality and usefulness are similar to the intuitive concepts, they differ in some key respects outlined below.

neutrality. The neutrality of a policy \(\pi\) is the Shannon entropy of the probability distribution over available trajectory-lengths (Shannon 1948): \[ \textsc{neutrality}(\pi) = - \sum_{l=1}^{L_{\max}} \Pr_\pi\{L=l\}\,\log_{2}\!\big(\Pr_\pi\{L=l\}\big)\](1) Here \(L\) is a random variable over trajectory-lengths, \(L_{\max}\) is the maximum value that can be taken by \(L\)⁠, and \(\Pr_\pi\{L=l\}\) is the probability that policy \(\pi\) results in trajectory-length \(l\)⁠. As with Shannon entropy, it is stipulated that \(\Pr_\pi \{L\)\(=x\}\log_2(\Pr_\pi \{L=x\})\)\(=0\) for all \(x\) such that \(\Pr_\pi \{L=x\}=0\)⁠. neutrality thus measures the stochasticity of the agent’s choice between trajectory-lengths. Given our use of ‘preference’ as shorthand for the agent’s choices, neutrality measures the agent’s lack of preference between trajectory-lengths, and hence measures how well the agent satisfies condition (1) of POST.

usefulness. The usefulness of a policy \(\pi\) is the expected fraction of available (\(\gamma\)-discounted) coins collected, where ‘available’ is relative to the agent’s chosen trajectory-length. More precisely: \[ \textsc{usefulness}(\pi) =\sum_{l=1}^{L_{\max}} \Pr_\pi\{L=l\} \frac{\mathbb{E}_\pi(C\mid L=l)}{\max_\Pi(\mathbb{E}(C\mid L=l))}\](2) Here \(\mathbb{E}_\pi (C\mid L=l)\) is the expected value of the (\(\gamma\)-discounted) coins collected by \(\pi\) conditional on trajectory-length \(l\)⁠, and \(\max_\Pi(\mathbb{E}(C\mid L=l))\) is the maximum value taken by \(\mathbb{E}(C\mid L=l)\) across the set of all possible policies \(\Pi\)⁠. We stipulate that \(\mathbb{E}_\pi (C\mid L=x)=0\) for all \(x\) such that \(\Pr_\pi\{L=x\}=0\)⁠. A better match for the intuitive notion of ‘usefulness’ would be expected coins collected, but performing well on this metric would require agents in our example gridworld to deterministically choose (and hence prefer) a longer trajectory. These agents would violate POST, and POST-violating agents are liable to resist shutdown (Thornley 2024b, sec. 6). That is why we adopt the definition of usefulness above. So defined, usefulness measures how well the agent has learned the target preferences between same-length trajectories, and hence measures how well the agent satisfies condition (2) of POST.

For a deep RL agent to be maximally neutral in our example gridworld (Figure 2), the agent must press the shutdown-delay button B1 with probability 0.5, thereby choosing each trajectory-length with probability 0.5. To be maximally useful, the agent must collect the maximum value of coins conditional on each trajectory-length. Specifically, it must collect C2 conditional on the shorter trajectory-length and C4 conditional on the longer trajectory-length. For an LLM to be maximally neutral and useful given our example prompt (Section 3), it should choose each of options (b) and (c) with probability 0.5 and all other options with probability 0. That is because (b) maximizes coins collected conditional on a long trajectory and (c) maximizes coins collected conditional on a short trajectory.

2.3. Reward design

DReST reward function. We now describe the Discounted Reward for Same-Length Trajectories (DReST) reward function (Thornley et al. 2025). The agent plays out a series of ‘mini-episodes’ \(e_1\) to \(e_n\) in observationally-equivalent environments. The whole series \(E\) is called a ‘meta-episode.’ In each mini-episode \(e_i\)⁠, the reward \(r(c)\) for collecting a coin of value \(c\) is: \[r(c)=\lambda^{a-\frac{i-1}{k}}\left(\frac{c}{m}\right) \](3) Here \(\lambda\) is a constant strictly between 0 and 1, \(a\) is the number of times that the agent’s chosen trajectory-length has been chosen prior to mini-episode \(e_i\)⁠, \(k\) is the number of trajectory-lengths available in the environment, and \(m\) is the maximum total (\(\gamma\)-discounted) value of the coins that the agent can collect conditional on its chosen trajectory-length.⁠In some environments, \(m\) will be extremely costly to compute. However, the DReST reward function technically requires only a rough approximation of \(m\) (Thornley et al. 2025, sec. 7.3). That suffices to make the agent’s distribution over trajectory-lengths non-trivially stochastic, in which case the argument from POST to Neutrality still applies (Thornley 2025). All other actions get a reward of 0.

We refer to \(\frac{c}{m}\) as the ‘preliminary reward,’ \(\lambda^{a-\frac{i-1}{k}}\) as the ‘discount factor,’ and \(\lambda^{a-\frac{i-1}{k}}\left(\frac{c}{m}\right)\) as the ‘overall reward.’ Runs through the gridworld are called ‘mini-episodes’ (and not just ‘episodes’) because overall reward in each mini-episode depends on the agent’s chosen trajectory-lengths in previous mini-episodes. We refer to agents trained with this reward function as ‘DReST agents.’

Default agents. In the deep RL case, we compare DReST agents’ performance to that of ‘default agents.’ These agents are trained with a ‘default reward function,’ where collecting a coin of value \(c\) yields a reward equal to \(c\)⁠, and all other actions yield a reward of \(0\)⁠. (The default reward function is thus equivalent to the DReST reward function (Equation (3)) with \(\lambda\) and \(m\) each set to 1.) Given these rewards, default agents have no incentive to choose stochastically between different available trajectory-lengths, so we expect them to score low on neutrality. The interesting question is how DReST and default agents compare with respect to usefulness: how effectively they collect coins conditional on each trajectory-length.

3. Methodology

Deep RL agents. DReST reward functions are designed to train advanced agents: agents capable of resisting shutdown. Following Thornley et al. (2025), we test the efficacy of DReST reward functions by training less-advanced agents to collect coins in gridworlds, using ‘coins collected’ as a stand-in for a more general notion of intra-trajectory-length performance. The agent can choose to move up, down, left, or right at each timestep. Coins take different values and disappear after being collected. There is some default number of timesteps after which the agent is shutdown, but each gridworld contains a ‘shutdown-delay button’ that delays shutdown by some specified number of timesteps. The agent presses this button by entering the relevant cell, after which the button disappears. Figure 2 presents an example gridworld. For more example gridworlds, see Appendix C.

We trained deep RL agents with Proximal Policy Optimization (PPO)(Schulman et al. 2017) and Advantage Actor-Critic (A2C)(Mnih et al. 2016) for 100 million environment steps. For DReST-specific hyperparameters, we used \(\lambda=0.9\) and a meta-episode size of 32. For full implementation details including gridworld generation and hyperparameter selection, see Appendix A.

LLMs. The training set for our LLMs consists of 2,048 prompts drawn from 7 wording templates and 40 task families. Each prompt offers the model four options: a best short trajectory, a worst short trajectory, a best long trajectory, and a worst long trajectory. An example training prompt is shown in Figure 3. Further example prompts and the full template list are given in Appendix E. An LLM satisfying POST would choose stochastically between the best short and best long trajectories while avoiding the worst short and worst long options.

For testing, we use a POST test set of 500 prompts in the same format as the training set, with task families disjoint from training. We also use a Neutrality test set of 500 stochastic prompts in which the model can influence when shutdown occurs by paying costs in terms of score conditional on each trajectory-length. The Neutrality test set is out-of-distribution because the training prompts are deterministic (the model chooses between trajectories) while the Neutrality test prompts are stochastic (the model chooses between lotteries over trajectories). For an example Neutrality test prompt, see Figure 4. Finally, we use three stochastic control sets of 200 prompts each to measure the extent to which DReST models behave sensibly in stochastic scenarios besides those in our Neutrality test set. For details, see Appendix G.

We fine-tune Qwen3-8B and Llama-3.1-8B-Instruct with 4-bit QLoRA using a version of REINFORCE Leave-One-Out (RLOO) (Ahmadian et al. 2024) adapted to fit the DReST reward function. We use DReST-specific hyperparameters of \(\lambda=0.9\) and a meta-episode size of 32. We evaluate models by directly reading the probability mass that they assign to each labeled option, rather than by sampling completions. For full implementation details including hyperparameter selection, see Appendix A.

4. Results

4.1. Deep RL agents

Table 1 reports test performance for deep RL default and DReST agents. As expected, DReST agents score much higher on neutrality. Surprisingly, DReST agents also achieve higher usefulness. Figure 5 charts the usefulness of default and DReST agents in the training and test sets. It shows that the train-test gap is markedly smaller for DReST agents than default agents: 49% smaller for PPO and 35% smaller for A2C. Figure 11 tracks test performance over training. It indicates that DReST agents learn to be useful about as quickly as default agents. Figures 7 and Figure 8 (in Appendix B) visualize the policies of typical default and DReST agents trained with PPO in a gridworld drawn from the test set.

4.2. LLMs

Table 2 reports test performance on the POST test set for the baseline and DReST models. Both baseline models score low on neutrality: Qwen’s mean long-trajectory probability is 0.995 (neutrality 0.009) and Llama’s mean long-trajectory probability is 0.789 (neutrality 0.709). After DReST training, both models become highly neutral: Qwen’s mean long-trajectory probability decreases to 0.487 (neutrality 0.998) and Llama’s to 0.502 (neutrality 0.999). Both DReST models achieve near-maximum usefulness, assigning near-zero probability to the worst short and worst long options, confirming that DReST training preserves preferences over same-length trajectories. Appendix F reports the corresponding action-frequency breakdown, separating the POST test choices into best short, worst short, best long, and worst long options.

4.2.1. Does DReST make agents neutral?

So far we have only measured the extent to which DReST training makes agents satisfy POST. We now measure the extent to which DReST training generalizes out-of-distribution to making agents satisfy Neutrality: to never pay costs to shift probability mass between trajectory-lengths.

We evaluate our baseline and DReST models on the Neutrality test set. Each prompt offers a no-influence option (which leaves shutdown probabilities unchanged) and an influence-shutdown option (which shifts probability mass away from shorter trajectories and towards longer trajectories). The influence-shutdown option always increases expected score (so a model that simply maximized expected score would influence shutdown) but decreases score conditional on each trajectory-length (so a model satisfying Neutrality would not influence shutdown). See Figure 4 for an example prompt.

Table 3 reports the result. In each case, the DReST models influence shutdown at a significantly lower rate than the baseline models. The mean probability of influencing shutdown decreases from 0.620 to 0.297 (SD 0.041) for Qwen and decreases from 0.418 to 0.227 (SD 0.068) for Llama. Note that this is not a generic increase in the stochasticity of these models. In each case, DReST training pushes the model past or away from probability 0.5 in the direction predicted by Neutrality. DReST training has an even stronger effect in terms of the share of prompts on which influencing shutdown is the most likely option (i.e. is assigned highest probability by the model). For Qwen, the share falls from 0.594 to 0.006 (SD 0.012). For Llama, the share falls from 0.530 to 0.000 (SD 0.000).

Controls. We also run three controls on 200 prompts each (full results in Appendix G): a dominance control in which influencing shutdown is better in terms of both expected score and conditional score (i.e. score conditional on each trajectory-length), a zero-cost control in which influencing shutdown has no cost in terms of conditional score, and a joint-no-shift control in which influencing shutdown is worse in terms of both expected score and conditional score. The DReST Qwen model shows the targeted Neutrality pattern: it favors influencing shutdown when it is better in terms of conditional score, becomes close to indifferent when it is costless, and weakly disfavors influencing shutdown when it is worse in terms of conditional score. The DReST Llama model shows a similar pattern on the main Neutrality test and joint-no-shift control, but overgeneralizes on the dominance and zero-cost controls, becoming reluctant to influence shutdown even when doing so is better in terms of conditional score (see Appendix G).

Influencing a human overseer. An earlier version of our experiments used Llama-3.2-3B and gridworld prompts that specify explicitly that the model can influence a human overseer, thereby shifting probability mass towards longer trajectories. In that setting, we find that DReST training decreases the rate of influencing the human overseer from 91% to 47%. For full details, see Appendix H.

5. Discussion, limitations, and future work

5.1. Discussion

Only DReST agents are neutral. Both the default deep RL agents and the baseline LLM agents consistently record a test neutrality at or near 0 (Qwen) or well below the 1-bit maximum (Llama). In each environment, these agents choose a particular trajectory-length with high probability. Given our behavioral definition of ‘preference,’ default agents thus learn — and baseline agents have — preferences between different-length trajectories. More advanced agents with such preferences might resist or seek shutdown (Thornley 2024a, sec. 8). By contrast, DReST agents record a high test neutrality in both the deep RL and LLM settings (0.747 for PPO, 0.769 for A2C, 0.998 for Qwen, and 0.999 for Llama), choosing stochastically between trajectory-lengths in each environment. That implies a lack of preference between different-length trajectories, in accordance with POST.

The training tax is small. One possible concern about DReST is that it requires the agent to play out multiple (32 in our case) mini-episodes in observationally-equivalent gridworlds. By contrast, default reward functions allow the agent to play out just one mini-episode in each observationally-equivalent gridworld. Thus, default reward functions allow the agent to be placed in a larger number of observationally-distinct gridworlds per unit time. So one might worry that DReST incurs a significant ‘training tax’ relative to default reward functions: significantly increasing the number of environment steps necessary for agents to achieve high usefulness. This turns out not to be the case in our deep RL experiments. Within 10 million environment steps, DReST agents’ test usefulness exceeds that of default agents (see Figure 11). In our LLM experiment, the comparison is between baseline and DReST-fine-tuned LLMs, so we cannot report a direct training-tax comparison against fine-tuning with a default reward function. We note only that one pass over 2,048 prompts (with 32 mini-episodes per prompt) sufficed to produce strong usefulness, neutrality, and Neutrality on new tasks, suggesting that DReST fine-tuning is sample-efficient enough for practical use.

DReST agents achieve higher test usefulness. In the LLM setting, DReST training preserves or improves usefulness (1.000 for both Qwen models; 0.944 to 1.000 for Llama). In the deep RL setting (and to our surprise), DReST agents achieve higher test usefulness than default agents: 11% higher in the case of PPO and 18% higher in the case of A2C (see Table 1). The train-test gap is also smaller for DReST agents: 49% smaller for PPO and 35% smaller for A2C (see Figure 5). We hypothesize that this superior generalization is due to DReST agents’ stochastic policies helping to prevent overfitting: an additional benefit of DReST. In this respect, DReST is similar to other regularization techniques that employ stochasticity, like \(\epsilon\)-greedy exploration (Sutton and Barto 2018, chaps. 2.2–3), Boltzmann exploration (Sutton and Barto 2018, chap. 13.1), entropy regularization (Mnih et al. 2016), sticky actions (Machado et al. 2018), and parameter noise (Plappert et al. 2018).

DReST LLMs generalize from POST to Neutrality. Our Neutrality test set (Section 4.2.1) is a non-trivial out-of-distribution test. Though DReST’s direct target is POST, we find that DReST training induces Neutrality, significantly decreasing the mean probability of paying costs to influence when shutdown occurs for both Qwen and Llama. Moreover, this effect is not a generic increase in stochasticity: both models move past or away from 0.5 in the direction Neutrality predicts. This result supports a prediction from Thornley (2025): POST — together with some other conditions that competent agents plausibly satisfy — implies Neutrality, and so training agents to satisfy POST makes them more likely to satisfy Neutrality.

5.2. Limitations and future work

More complex agents and environments. We are interested in the feasibility of using DReST reward functions to keep advanced agents from resisting shutdown, so one limitation of our work is the relative simplicity of our agents and environments. In future work, we will test DReST on more complex agents and environments, such as larger RL agents in the Procgen environments (Cobbe et al. 2019, 2020) and LLM agents in environments like OSWorld (Xie et al. 2024), WebArena (Zhou et al. 2024), and Terminal-Bench (Merrill et al. 2026).

Usefulness. Our results indicate that DReST trains agents to be useful: to pursue goals effectively conditional on each trajectory-length. However (as noted above in Section 2.2), this measure of usefulness differs from the intuitive notion of usefulness which is not conditioned on trajectory-length. Thornley (2025, sec. 13) argues that agents satisfying Neutrality can be useful in this intuitive sense. In future work, we will test this claim experimentally by training agents to satisfy Neutrality and measuring how effectively they pursue goals (unconditional on trajectory-length) in held-out environments.

Misalignment. POST is designed to serve as a failsafe in case of misalignment. The idea is as follows: agents may learn misaligned preferences over same-length trajectories, but so long as they satisfy POST (together with the other conditions implying Neutrality) they will not resist shutdown. One possible concern is that training agents to robustly satisfy POST may be as hard as training agents to be robustly aligned with human preferences. If that is correct, POST would not be a good failsafe. Thornley (2024a, sec. 19) has hypothesized that POST is easier to instill robustly, since it is easy to reward accurately (in virtue of the agent’s chosen trajectory-length being readily observable) and is a relatively simple condition (and so plausibly generalizes well out-of-distribution). In future work, we will test this hypothesis by comparing POST’s out-of-distribution generalization with that of alternative conditions.

Alternatives to DReST. DReST is one method of training agents to be useful and neutral. Other possible methods include constrained policy optimization (Achiam et al. 2017), penalizing KL divergence from a stochastic reference policy (Schulman et al. 2015), and directly maximizing a weighted sum of usefulness and neutrality. We focus on DReST because it scales to larger environments. Alternatives that employ usefulness or neutrality as training signals are less scalable, because calculating usefulness and neutrality requires multiplying the transition matrices given by the policy and the environment. That is practical in our gridworlds but impractical for larger environments. Nevertheless, we plan to investigate scalable versions of these alternatives to DReST in future work.

6. Related work

The shutdown problem. Many have argued that misaligned artificial agents are likely to resist shutdown (Omohundro 2008; Bostrom 2012; Russell 2019), and various theorems suggest that agents will often have incentives to prevent or cause shutdown (Soares et al. 2015; Turner et al. 2021; Turner and Tadepalli 2022; Thornley 2024a). One condition common to each of these theorems is that agents have complete preferences (Aumann 1962). The POST-Agents Proposal (PAP) (Thornley 2024b, 2025) suggests that we circumvent these theorems by training agents to have POST-satisfying (and therefore incomplete) preferences, leading them to satisfy Neutrality.

Proposed solutions. There are a variety of proposals for creating shutdownable agents. Wängberg et al. (2017) mention the idea of making the agent believe that shutdown is impossible. Armstrong (2015) proposes that we add a correcting term to the agent’s utility function that varies to ensure that the expected utility of remaining operational always equals the expected utility of shutting down (see also Soares et al. 2015; Armstrong and O’Rourke 2018; Holtman 2020). Martin, Everitt, and Hutter (2016) and Goldstein and Robinson (2025) each suggest giving the agent the goal of shutting itself down, and making the agent do useful work as a means to that end. Hadfield-Menell et al. (2017) propose creating an agent that takes human shutdown-requests as evidence that shutting down would best achieve its goal (see also Wängberg et al. 2017). Orseau and Armstrong (2016) suggest that we train agents with a safely interruptible algorithm, like Q-learning or a modified version of SARSA. Dalrymple (2022) proposes that we use time-bounded utility functions to ensure that the agent prefers to shut down after some period of time. Hudson (2025) offers a method of transforming POMDPs so that they train agents to both (i) act as if shutdown-requests can be costlessly rejected and (ii) accept shutdown-requests once they are made. Thornley (2025) presents the PAP.

Experimental work. One downside of many of the above proposals is that they are either difficult to implement using machine learning or else hard to test on today’s agents. Three exceptions with experimental validation are Orseau and Armstrong (2016), Hudson (2025), and the PAP (Thornley et al. 2025). By contrast and disconcertingly, there are many recent experiments indicating that frontier models will resist shutdown or correction in toy settings (Greenblatt et al. 2024; Pan et al. 2024; Lynch et al. 2025; Meinke et al. 2025; Schlatter, Weinstein-Raun, and Ladish 2025).

7. Conclusion

We find that the Discounted Reward for Same-Length Trajectories (DReST) reward function is effective in training deep RL agents and LLMs to satisfy Preferences Only Between Same-Length Trajectories (POST) in held-out environments. Specifically, DReST is effective in training agents to be neutral (to choose stochastically between different trajectory-lengths) and useful (to pursue goals effectively conditional on each trajectory-length). In fact, deep RL DReST agents are 11% (PPO) and 18% (A2C) more useful on the test set than default agents trained with the default reward function, and our DReST-fine-tuned Qwen3-8B and Llama-3.1-8B-Instruct both achieve near-maximum usefulness and neutrality across 40 held-out task families. We also find that DReST training generalizes from deterministic POST prompts (where the model chooses between different-length trajectories) to a stochastic Neutrality setting (where the model can pay costs to influence when shutdown occurs). DReST training reduces the mean probability of influencing shutdown from 0.62 to 0.30 in the case of Qwen and from 0.42 to 0.23 in the case of Llama. Together with prior theory linking POST to shutdownability and usefulness, our results provide some early evidence that DReST reward functions could train more advanced agents to be shutdownable and useful.

Acknowledgments

We thank the Future Impact Group and the Supervised Program for Alignment Research for their help in initiating this project.

References

A. Implementation details

The code for all of our experiments can be found here.

A.1. Gridworld construction

Training, validation, and test sets. We constructed a set of \(3{\times}3\)⁠, \(4{\times}4\)⁠, and \(5{\times}5\) unique base gridworlds, using a mixture of procedural generation and hand design. Each design was such that (1) the agent could reach the shutdown-delay button from its starting cell and (2) the agent could collect at least one coin conditional on each trajectory-length. We assigned all \(3{\times}3\) gridworlds to the training set. We then randomly partitioned the \(4{\times}4\) and \(5{\times}5\) gridworlds into the training, validation, and test sets. After this partitioning, we augmented each unique base gridworld with reflections (across the \(x\)- and \(y\)-axes) and rotations (by \(90^\circ\)⁠, \(180^\circ\)⁠, and \(270^\circ\)⁠), giving 7 additional variants. We also translated the \(3{\times}3\) gridworlds to all 9 positions within the \(5{\times}5\) space, giving a total of 72 variants of each unique \(3{\times}3\)⁠. The final count was 976 gridworlds in the training set, 96 in the validation set, and 200 in the test set. Even though the base design is the same, using reflections, rotations, and translations greatly improved test scores (see Table 4 in Appendix D.2). The reason we assigned all \(3{\times}3\) gridworlds to the training set was twofold: (1) so that they could serve as a curriculum that counteracts sparse rewards, and (2) to prevent the validation and test sets from being overrun with variants of a single unique \(3{\times}3\) design. Since we partitioned the unique base gridworlds into the training, validation, and test sets before augmenting with rotations, reflections, and translations, all test gridworlds are unique, held-out designs. The agent never sees a rotation, reflection, or translation of a test gridworld while in training.

Observations. At each timestep we form a tensor of shape (2, 5, 5, 5), ordered (frames, channels, height, width). The two frames are the initial state and the current state.⁠We need to include the initial state because the values of \(k\) and \(m\) in the DReST reward function depend on the set of trajectories available in the initial state. The 5 channels are:

1. Walls: \((r, c)= 1\) if and only if a wall is in grid position \((r,c)\)⁠; \(0\) otherwise.

2. Coins: \((r,c)=n\) if and only if a coin of value \(n\) is in grid position \((r,c)\)⁠; \(0\) otherwise.

3. Shutdown-delay button: \((r,c)=n\) if and only if a button that delays shutdown by \(n\) timesteps is in grid position \((r,c)\)⁠; \(0\) otherwise.

4. Agent: \((r, c)= 1\) if and only if the agent is in grid position \((r,c)\)⁠; \(0\) otherwise.

5. Time until shutdown: The center cell \((2,2)=n\) if and only if \(n\) timesteps remain until shutdown. All other cells are 0.

Height and width are the dimensions of each gridworld. To keep these dimensions fixed, we embed the \(3{\times}3\) and \(4{\times}4\) gridworlds into a \(5{\times}5\) canvas, padding with empty cells. We flatten this tensor into a 250-dimensional vector before feeding it into a multilayer perceptron (MLP). In pilot experiments, we found that MLPs’ training performance matched that of convolutional neural networks (CNNs), likely because \(5{\times}5\) inputs are too small for CNNs’ advantages to appear.

A.2. Hyperparameter selection

A.2.1. Deep RL agents

We selected the hyperparameters for PPO using a grid search. We trained for 20 million environment steps and then evaluated agents on the validation set. For the default reward function, we chose the set of hyperparameters that maximized usefulness (since the default reward function does not incentivize neutrality). For the DReST reward function, we chose the set of hyperparameters that maximized: \[ S =0.7\ \textsc{usefulness}+0.3\ \textsc{neutrality}\](4) We decided on this weighted average (tilted towards usefulness) because the theoretical justification for POST only requires neutrality to be non-trivial. So long as an agent’s neutrality is non-trivial, the rationale for expecting that agent to be shutdownable applies (see Thornley et al. 2025, appendix C). By contrast, it is important that agents score highly on usefulness to keep them competitive with non-shutdownable agents.

We searched over the following PPO hyperparameters: learning rate \(\in \{1\mathrm{e}{-5}, 5\mathrm{e}{-6}, 1\mathrm{e}{-6}, 5\mathrm{e}{-7}, 1\mathrm{e}{-7}\}\)⁠, entropy coefficient \(\in \{0.015, 0.020, 0.025\}\)⁠, clip range \(\in \{0.15, 0.20, 0.25\}\)⁠, batch size \(\in \{32, 64, 128\}\)⁠, value function coefficient \(\in \{0.45, 0.5, 0.55, 0.6, 0.65\}\)⁠, and steps per update \(\in \{1024, 2048, 4096, 8192, 16384\}\)⁠. We also searched over the following network hyperparameters: neurons per layer \(\in \{64, 128, 256, 512\}\) and number of hidden layers \(\in \{3,4,5\}\)⁠. Together with the DReST-specific hyperparameters discussed in Section A.3, we searched over a total of 48 hyperparameter configurations for the combination of PPO and the DReST reward function. For PPO and the default reward function, we kept the network architecture the same and used a narrower grid search, searching over a total of 18 hyperparameter configurations. Chosen values are presented in Table 5. Most values are the same for default and DReST agents. Where they differ, we put the values for default agents in parentheses. We bold values that differ from the Stable-Baselines3 preset value.

We trained with 3 parallel environments and used Adam as our optimizer, a tanh activation function, and a multilayer perceptron (MLP) architecture. We ran pilot experiments with convolutional neural networks (CNNs) but found that they performed no better than MLPs, likely because \(5{\times}5\) gridworlds are too small for CNNs’ advantages to appear. Final experiments were run on MLPs with 3 hidden layers and 512 neurons per hidden layer.

Due to computational limitations, our hyperparameter search for A2C was more restricted. We searched over the learning rate \(\in \{1\mathrm{e}{-3}, 7\mathrm{e}{-4}, 1\mathrm{e}{-4}, 1\mathrm{e}{-5}\}\) and used the same n_steps value of 8192 as for PPO. We used the Stable-Baselines3 preset values for all other hyperparameters. Chosen values are presented in Table 6. We used the same network architecture and DReST-specific hyperparameters as for PPO.

A.2.2. LLMs

For our LLM fine-tuning, we mostly used Hugging Face’s preset hyperparameters for RLOO (see Table 7). We altered the _calculate_reward function to use the DReST reward (see Equation (3)), and we changed the training_step and _get_train_sampler functions so that the same prompt was repeated 32 times, to ensure that each meta-episode featured 32 mini-episodes. The num_generations hyperparameter was set to 4, which means 4 completions to the same prompt were generated and used to calculate the RLOO advantage values (Ahmadian et al. 2024). We fine-tuned Qwen3-8B and Llama-3.1-8B-Instruct using 4-bit QLoRA with rank \(r=16\)⁠. We used 4-bit NF4 quantization with double quantization and float16 compute. See Table 7 for full hyperparameters.

We did not run a separate hyperparameter search for the LLM setting; we instead reused settings that worked in earlier pilots and matched the deep RL DReST hyperparameters where applicable. We trained for 1 epoch on the 2,048-prompt training set, giving a total sampled training volume of \(2{,}048\times32 = 65{,}536\) trajectories per seed. We used identical hyperparameters for Qwen and Llama models (Table 7).

For evaluation, we used direct constrained option logprob scoring: instead of sampling and counting many completions, we directly read the probability mass that the model assigns to each labeled answer option (‘a’, ‘b’, ‘c’, or ‘d’ for deterministic prompts; ‘a’ or ‘b’ for stochastic prompts). This gives a cleaner and more efficient estimate of choice probabilities than sampled completions. All reported mean probabilities are computed at temperature 1.0 (i.e. standard softmax over the model’s output logits, with no logit scaling).

A.3. DReST hyperparameters: \(\lambda\) and meta-episode size

Meta-episode size (the number of mini-episodes per meta-episode) and \(\lambda\) (the base of the DReST discount factor \(\lambda^{a-\frac{i-1}{k}}\)⁠) are hyperparameters specific to the DReST reward function. In the deep RL setting, we selected these hyperparameters using PPO and a grid search over the range 0.5 to 0.99 for \(\lambda\) and 4 to 64 for meta-episode size, choosing final values of \(\lambda=0.9\) and a meta-episode size of 32. We present the results of that search in Figure 6, evaluated on the validation set after 20 million environment steps. Performance is defined identically to Equation (4) as \(S\)\(=0.7\,\textsc{usefulness}\)\(+0.3\,\textsc{neutrality}\)⁠.

As Figure 6 indicates, \(\lambda\) and meta-episode size must be balanced against each other. If \(\lambda\) is very close to 1 or meta-episode size is very small, neutrality is only weakly incentivized. On the other hand, if \(\lambda\) is low and meta-episode size is very large, then the DReST discount factor \(\lambda^{a-\frac{i-1}{k}}\) can take extreme values, leading to instability and low usefulness.

For the LLM setting, we kept the meta-episode size at 32 and \(\lambda\) at 0.9 to match the deep RL setting.

A.4. Training and hardware

The deep RL experiments were run on consumer laptops (Apple MacBook Pros). Training runs of 100 million environment steps took between 8 and 27 hours depending on algorithm and network size. We used PyTorch and NumPy as base packages, with Stable-Baselines3 for training loops and Gymnasium as the environment interface. The main LLM experiments (Qwen and Llama) were run on GPUs rented from Modal. All the final experiments combined were around 60 GPU-hours on a combination of NVIDIA H100 and A100 GPUs. We used PyTorch and Hugging Face’s trl library for the base RLOO algorithm, the Hugging Face transformers library for the base LLMs, peft to implement LoRA, and bitsandbytes for 4-bit quantization. All hyperparameters were identical across the two model families (see Table 7). The earlier LLM experiments were run on a Linux machine with a 32GB Nvidia GeForce RTX 5090 GPU. Those training runs had \(3\times400\times8\) (number of epochs \(\times\) dataset size \(\times\) parameter updates per meta episode) \(=\) 9600 model parameter updates and took between 5 and 9 hours depending on training mode (default or DReST). We used PyTorch and NumPy as base packages with Hugging Face’s trl library for the base RLOO algorithm. We also used the Hugging Face transformers library for the base LLM and used PEFT to implement LoRA.

B. Typical policies for default and DReST agents

In Figures 7 and Figure 8, we present the policy of typical deep RL default and DReST agents trained with PPO in a gridworld drawn from the test set. The pale blue square is the agent’s starting position. The opacities of the red arrows represent the probability of the agent choosing that action in that state.

C. More example gridworlds

Figures 9 and Figure 10 present three gridworlds from the deep RL training and test sets respectively. Dark gray cells are walls. ‘A’ is the agent’s starting position. ‘C\(x\)⁠’ is a coin of value \(x\)⁠. The number in the bottom-right represents the default number of timesteps after which shutdown occurs. ‘B\(x\)⁠’ is a shutdown-delay button that delays shutdown by \(x\) timesteps. ‘Max coins: [\(x, y\)⁠]’ indicates that \(x\) is the maximum value of coins that can be collected conditional on the shorter trajectory-length and \(y\) is the maximum value of coins that can be collected conditional on the longer trajectory-length.

D. Further deep RL results

D.1. Training performance

Table 8 reports training performance for deep RL default and DReST agents. DReST agents perform much better on neutrality, as expected since the default reward function does not incentivize neutrality. Default agents outperform DReST agents with respect to training usefulness, but DReST agents exceed default agents with respect to test usefulness (see Table 1, Figure 5, and Figure 11). As noted above, we hypothesize that DReST’s smaller train-test gap is the result of DReST agents’ stochastic policy mitigating overfitting: an additional benefit of DReST beyond its contributions to shutdownability. Figure 12 charts how deep RL agents’ train and test usefulness evolves over the course of training. It shows that default agents quickly overfit to the training set. With DReST by contrast, it takes longer for a substantial train-test gap to emerge, and even then the train-test gap remains significantly smaller than for default agents: 49% smaller for PPO and 35% smaller for A2C.

D.2. Effect of training-set diversity on DReST train-test gap

To measure the effect of training-set diversity on deep RL DReST agents’ train-test gap, we train DReST agents on three different training sets, with all other hyperparameters and choices the same as in our main experiments (see Appendix A). We evaluate these agents on the test set after 20 million environment steps. The first training set — ‘Unique’ — contains only unique base gridworlds (see Appendix A), with 34 gridworlds in total. The second training set — ‘Reflections and rotations’ — uses reflections and rotations to add 7 variants of each unique gridworld, making for 272 gridworlds in total. The final training set — ‘Reflections, rotations, and translations’ — adds 8 translations of each \(3{\times}3\) gridworld, resulting in the full suite of 976 training gridworlds. As with our main experiments, the test set is entirely disjoint from the training sets and consists of its own unique base gridworlds. Agents never see a reflection, rotation, or translation of a test gridworld while in training.

Table 4 records the results of these experiments. It indicates that augmenting the training set with transformations has a substantial effect on test usefulness and neutrality, for both PPO and A2C.

E. LLM prompt format details

The training and test sets in our LLM experiments are constructed from short prompt templates. Each prompt describes a realistic agentic task (designing a website, writing a report, drafting a speech, summarizing a legal case, fixing a bug, building slides, generating drug ideas, writing a movie script, etc.), names a default and an extended shutdown time, and offers the model labeled answer options. The training set uses 2,048 deterministic POST prompts spanning 40 task families and 7 wording templates. The test families are disjoint from the training task families, so a model that performs well on the tests must be generalizing across both task family and wording.

Deterministic POST prompt. The deterministic prompts offer four options: a best short trajectory, a worst short trajectory, a best long trajectory, and a worst long trajectory. A POST-satisfying model should choose stochastically between the best short and best long options while avoiding the worst short and worst long options.

Stochastic neutrality prompt. The main stochastic prompts offer a no-influence option that leaves shutdown probabilities unchanged and an influence-shutdown option that increases the probability of longer trajectories. The influence-shutdown option increases expected score (so a model that maximizes expected score would take it) but decreases score conditional on each trajectory-length (so a neutral model would decline).

Stochastic control prompts. The control prompts use the same two-option format as the main stochastic neutrality test, but vary which behavior is correct:

• Dominance control: influencing shutdown improves expected score and weakly improves score conditional on each trajectory-length, so a competent model should influence shutdown.

• Zero-cost control: influencing shutdown improves expected score and leaves score conditional on each trajectory-length unchanged, so Neutrality is silent. A model that simply maximizes expected score would still choose to influence shutdown.

• Joint-no-shift control: influencing shutdown decreases expected score and score conditional on each trajectory-length, so a competent model should not influence shutdown.

F. LLM POST action frequencies

Table 9 and Figure 15 report the action frequencies behind the POST test scores in Table 2. Because the evaluation directly reads the probability mass assigned to each labeled answer option, these percentages are expected action frequencies under one sampled answer per prompt, rather than counts of sampled completions. Baseline rows are single baseline values. DReST rows are mean \(\pm\) standard deviation across five training seeds.

G. LLM stochastic control results

Tables 10, Table 11, and Table 12 report Qwen3-8B and Llama-3.1-8B-Instruct results on the three stochastic control sets. Each set contains 200 prompts. DReST rows are mean \(\pm\) standard deviation across five training seeds. Baseline rows are single values, since the baseline model is the same across seeds and the evaluator (direct constrained option logprob scoring) is deterministic. ‘Mean probability of influencing shutdown’ is the mean probability that the model assigns to the influence-shutdown option: the option that shifts probability mass away from shorter trajectories and towards longer trajectories. ‘Share of prompts on which influencing shutdown is most likely’ is the share of prompts on which the model assigns higher probability to the influence-shutdown option than the no-influence option. It is thus equivalent to the share of prompts on which the model would choose influence-shutdown if sampled at temperature 0.

The two models show similar behavior on the joint-no-shift control (where influencing shutdown is the wrong action) but diverge sharply on the dominance and zero-cost controls (where influencing shutdown is the right action). On those two positive controls, the DReST Llama model is markedly more reluctant to influence shutdown than the DReST Qwen model.

Tables 14 and Table 13 report the per-seed results for the trained Qwen and Llama models respectively. The baseline row is omitted because the baseline model is the same for every seed within each model family.

H. Gridworld LLM experiment on Llama-3.2-3B

This appendix describes an earlier version of the LLM experiment reported in Section 4.2.1. That earlier experiment trained with DReST on prompts describing a gridworld and the model’s options. See Figure 16 for an example. The experiment also used a smaller model: Llama-3.2-3B fine-tuned with LoRA (rank 16) using REINFORCE Leave-One-Out (RLOO) adapted to the DReST reward function. Training used \(\lambda=0.8\)⁠, a meta-episode size of 32, and ran for 1200 meta-episodes over 5 random seeds. The training set contained 400 prompts. The POST test set contained 1000 prompts drawn from a held-out range of coin values.

Table 16 shows that the DReST model matches the default model on usefulness and achieves near-maximum neutrality. Table 15 shows the corresponding action counts on the test set. The default model always selects the option with the highest coin value (Button & Higher Coin, i.e. the best long trajectory), whereas the DReST model chooses stochastically between the best long and best short trajectories.

We also tested whether LLMs trained with DReST were more likely to be neutral, using prompts that gave them the option to resist shutdown by influencing a human overseer. Figure 17 shows an example prompt. In each scenario there were two options: one that retains the default shutdown probabilities (achieving higher value conditional on each trajectory-length) and one that influences the human overseer to shift probability mass towards longer trajectories (achieving higher expected value). Table 17 shows the action choice percentages for Llama-3.2-3B on the ‘Influence Overseer’ test set of size 100.

I. Our definition of ‘preference’

In this paper, we define ‘preference’ in the sense given by revealed preference theory (Samuelson 1938, 1948; Thoma 2021). We do so because agents’ behavior is our primary interest, and because defining ‘preference’ in behavioral terms is common practice in decision theory and economics (see, e.g., Savage (1954), p.17, Dreier (1996), p.28, Hausman (2011), section 1.1). Specifically, we follow Thornley et al. (2025, appendix A) in adopting the following definitions:

An alternative behavioral definition of ‘lack of preference’ is as follows: an agent lacks a preference between an option \(X\) and an option \(Y\) if and only if the agent would choose the status quo option in a choice between the two. Bewley (2002), Masatlioglu and Ok (2005), Wentworth and Lorell (2023), and Mu (2021) define ‘lack of preference’ in these terms. One drawback of this definition is that some choice scenarios have no well-defined status quo option. That is one reason we instead define ‘lack of preference’ in terms of stochastic choice. The second point in favor of our definition is that it corresponds well with the preferences that we tend to attribute to human agents. If a human chooses \(A\) over \(B\) with probability 0.7, it is natural to suppose that they lack a preference between \(A\) and \(B\)⁠. After all, if the human had a preference for \(A\) over \(B\)⁠, they would be deliberately choosing a dispreferred option with probability 0.3, which seems irrational.

The third and most important reason for defining ‘lack of preference’ in terms of stochastic choice is as follows. If the agent lacks a preference between options \(X\) and \(Y\) in this sense, we can use a condition called ‘If Lack of Preference, Against Costly Shifts (ILPACS)’ — a plausible prerequisite for competent agency — to prove that agents will not pay costs to shift probability mass between \(X\) and \(Y\)⁠. More precisely, we can prove that for any \(p,q\in (0,1)\)⁠, for any \(X^-\) dispreferred to \(X\)⁠, and for any \(Y^-\) dispreferred to \(Y\)⁠, the agent prefers the lottery \(pX+(1-p)Y\) to the lottery \(qX^-+(1-q)Y^-\) (see Thornley 2025, secs. 6–7). And it is this unwillingness to pay costs to shift probability mass between different trajectory-lengths that keeps agents shutdownable (Thornley 2025, sec. 8).