Elliott Thornley

I'm a Postdoctoral Associate at MIT. From August 2026, I'll be an Assistant Professor of Philosophy at NUS.

I work on AI safety. Right now, I'm using ideas from decision theory to train safer artificial agents.

I also do work in ethics, focusing on the moral importance of future generations.

You can email me at thornley@mit.edu.

AI Safety

Risk-Averse AIs

Risk-Averse AIs

with William MacAskill

2026

Abstract

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse.

Out-of-Distribution Generalization of Risk Aversion in Language Models

with Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie, Andrew Lin, and Abhitej Bokka

2026

Abstract

Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned. Misaligned but risk-averse AIs would tend to prefer low-risk, low-reward strategies like cooperation over high-risk, high-reward strategies like rebellion, limiting the downsides of any misalignment. But we can only feasibly train AIs to be risk-averse on low-stakes gambles, and we will only be safe if their risk aversion generalizes to astronomically-high-stakes gambles. Will it? To shed light on this question, we introduce RiskAverseOOD: a benchmark for measuring how well risk aversion generalizes out of distribution. We then offer some initial results. Using a variety of methods to make Qwen3-8B choose risk-aversely when the stakes are low, we find that we can induce substantial risk aversion when the stakes are astronomically high. Our models’ learned risk aversion generalizes at least partially across 98 orders of magnitude. From a baseline 2% rate of choosing a safe ‘Cooperate’ option, we see rates around 70% (SFT and tie training), 52% (DPO), and 39% (activation steering). In another experiment, our fine-tuned reward model reliably scores risk-averse reasoning above risk-neutral or excessively risk-averse alternatives (99.6% pairwise accuracy). We replicate these effects at different scales (Qwen3-1.7B and Qwen3-14B) and across model families (Gemma-3-12B-IT and Llama-3.1-8B-Instruct). Overall, we find that risk aversion learned at low stakes can generalize OOD to astronomically high stakes, though not yet consistently enough to serve as a reliable failsafe. Achieving that level of consistency is an open problem.

Tie Training

Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training

with Christian Moya, Alex Semendinger, and Guang Lin

ICML, 2026

Abstract

Preference learning methods like Direct Preference Optimization (DPO) are known to induce reliance on spurious correlations, leading to sycophancy and length bias in today’s language models and potentially severe goal misgeneralization in future systems. In this work, we provide a unified theoretical analysis of this phenomenon, characterizing the mechanisms of spurious learning, its consequences on deployment, and a provable mitigation strategy. Focusing on log-linear policies, we show that standard preference-learning objectives induce reliance on spurious features at the population level through two channels: mean spurious bias and causal-spurious correlation leakage. We then show that this reliance creates an irreducible vulnerability to distribution shift: more data from the same training distribution fails to reduce the model’s dependence on spurious features. To address this, we propose tie training, a data augmentation strategy using ties (equal-utility preference pairs) to introduce data-driven regularization. We demonstrate that this approach selectively reduces spurious learning without degrading causal learning. Finally, we validate our theory on log-linear models and provide empirical evidence that both the spurious learning mechanisms and the benefits of tie training persist for neural networks and large language models.

Shutdownable Agents

Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

with Carissa Cullen, Harry Garland, Alexander Roman, Louis Thomson, and Christos Ziakas

ICML Agents in the Wild Workshop, 2026

Abstract

Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be neutral about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be useful). In this paper, we use DReST to train deep RL agents and fine-tune LLMs to be neutral and useful. We find that these DReST agents generalize to being neutral and useful in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 18% (A2C) higher usefulness on our test set than baseline agents, and our fine-tuned LLM achieves maximum usefulness and near-maximum neutrality. Our results provide some early evidence that DReST could be used to train more advanced agents to be useful and neutral. Prior theoretical work suggests that these agents would be useful and shutdownable.

Shutdownable Agents through POST-Agency

2025

Abstract

Many fear that future artificial agents will resist shutdown. I present an idea – the POST-Agents Proposal – for ensuring that doesn’t happen. I propose that we train agents to satisfy Preferences Only Between Same-Length Trajectories (POST). I then prove that POST – together with other conditions – implies Neutrality+: the agent maximizes expected utility, ignoring the probability distribution over trajectory-lengths. I argue that Neutrality+ keeps agents shutdownable and allows them to be useful.

Towards Shutdownable Agents via Stochastic Choice

with Alexander Roman, Christos Ziakas, Leyton Ho, and Louis Thomson

Transactions on Machine Learning Research, 2025

NeurIPS ARLET Workshop, 2025

Technical AI Safety Conference, 2025

Abstract

The POST-Agents Proposal (PAP) is an idea for ensuring that advanced artificial agents never resist shutdown. A key part of the PAP is using a novel ‘Discounted Reward for Same-Length Trajectories (DReST)’ reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be useful), and (2) choose stochastically between different trajectory-lengths (be neutral about trajectory-lengths). In this paper, we propose evaluation metrics for usefulness and neutrality. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be useful and neutral. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be useful and neutral. Our theoretical work suggests that these agents would be useful and shutdownable.

The Shutdown Problem: An AI Engineering Puzzle for Decision Theorists

Philosophical Studies, 2025

Abstract

I explain and motivate the shutdown problem: the problem of designing artificial agents that (1) shut down when a shutdown button is pressed, (2) don’t try to prevent or cause the pressing of the shutdown button, and (3) otherwise pursue goals competently. I prove three theorems that make the difficulty precise. These theorems suggest that agents satisfying some innocuous-seeming conditions will often try to prevent or cause the pressing of the shutdown button, even in cases where it’s costly to do so. I end by noting that these theorems can guide our search for solutions to the problem.

Ethics

A Non-Identity Dilemma for Person-Affecting Views

Australasian Journal of Philosophy, forthcoming

Abstract

Person-affecting views state that (in cases where all else is equal) we’re permitted but not required to create people who would enjoy good lives. In this paper, I present an argument against every possible variety of person-affecting view. The argument is a dilemma over trilemmas. Narrow person-affecting views imply a trilemma in a case that I call ‘Expanded Non-Identity.’ Wide person-affecting views imply a trilemma in a case that I call ‘Two-Shot Non-Identity.’ One plausible practical upshot of my argument is as follows: we individuals and our governments should be doing more to reduce the risk of human extinction this century.

A Fission Problem for Person-Affecting Views

Ergo, 2026

Abstract

On person-affecting views in population ethics, the moral import of a person’s welfare depends on that person’s temporal or modal status. These views typically imply that—all else equal—we’re never required to create extra people, or to act in ways that increase the probability of extra people coming into existence. In this paper, I use Parfit-style fission cases to construct a dilemma for person-affecting views: either they forfeit their seeming-advantages and face fission analogues of the problems faced by their rival impersonal views, or else they turn out to be not so person-affecting after all. In light of this dilemma, the attractions of person-affecting views largely evaporate. What remains are the problems unique to them.

Critical-Set Views, Biographical Identity, and the Long Term

Australasian Journal of Philosophy, 2025

Abstract

Critical-set views avoid the Repugnant Conclusion by subtracting some constant from the welfare score of each life in a population. These views are thus sensitive to facts about biographical identity: identity between lives. In this paper, I argue that questions of biographical identity give us reason to reject critical-set views and embrace the total view. I end with a practical implication. If we shift our credences towards the total view, we should also shift our efforts towards ensuring that humanity survives for the long term.

The Procreation Asymmetry, Improvable-Life Avoidance, and Impairable-Life Acceptance

Analysis, 2023

Abstract

Many philosophers are attracted to a complaints-based theory of the procreation asymmetry, according to which creating a person with a bad life is wrong (all else equal) because that person can complain about your act, whereas declining to create a person who would have a good life is not wrong (all else equal) because that person never exists and so cannot complain about your act. In this paper, I present two problems for such theories: the problem of impairable-life acceptance and an especially acute version of the problem of improvable-life avoidance. I explain how these problems afflict two recent complaints-based theories of the procreation asymmetry, from Joe Horton and Abelard Podgorski.

Critical Levels, Critical Ranges, and Imprecise Exchange Rates in Population Axiology

Journal of Ethics and Social Philosophy, 2022

Abstract

According to critical-level views in population axiology, an extra life improves a population if and only if that life’s welfare level exceeds some fixed “critical level.” An extra life at the critical level leaves the new population equally good as the original. According to critical-range views, an extra life improves a population if and only if that life’s welfare level exceeds some fixed “critical range.” An extra life within the critical range leaves the new population incommensurable with the original. In this paper, I sharpen some old objections to these views and offer some new ones. Critical-level views cannot avoid certain repugnant and sadistic conclusions. Critical-range views imply that lives featuring no good or bad components whatsoever can nevertheless swallow up and neutralize goodness and badness. Both classes of view imply discontinuities in implausible places. I then offer a view that retains much of the appeal of critical-level and critical-range views while avoiding the above pitfalls. On the Imprecise Exchange Rates View, various exchange rates—between pairs of goods, between pairs of bads, and between goods and bads—are imprecise. This imprecision is the source of incommensurability between lives and between populations.

Is Global Consequentialism More Expressive Than Act Consequentialism?

Analysis, 2022

Abstract

Act consequentialism states that an act is right if and only if the expected value of its outcome is at least as great as the expected value of any other act’s outcome. Two objections to this view are as follows. The first is that act consequentialism cannot account for our normative ambivalence in cases where agents perform the right act out of bad motives. The second is that act consequentialism is silent on questions of character: questions like ‘What are the right motives to have?’ and ‘What kind of person ought I be?’. These objections have been taken to motivate a move to global consequentialism, on which acts are not the only subjects of normative assessment. Motives and decision-procedures (amongst other things) are also judged right or wrong by direct reference to their consequences. In this paper, I argue that these objections fail to motivate the move from act to global consequentialism.

The Impossibility of a Satisfactory Population Prospect Axiology

Philosophical Studies, 2021

Abstract

Arrhenius’s impossibility theorems purport to demonstrate that no population axiology can satisfy each of a small number of intuitively compelling adequacy conditions. However, it has recently been pointed out that each theorem depends on a dubious assumption: Finite Fine-Grainedness. This assumption states that there exists a finite sequence of slight welfare differences between any two welfare levels. Denying Finite Fine-Grainedness makes room for a lexical population axiology which satisfies all of the compelling adequacy conditions in each theorem. Therefore, Arrhenius’s theorems fail to prove that there is no satisfactory population axiology. In this paper, I argue that Arrhenius’s theorems can be repurposed. Since all of our population-affecting actions have a non-zero probability of bringing about more than one distinct population, it is population prospect axiologies that are of practical relevance, and amended versions of Arrhenius’s theorems demonstrate that there is no satisfactory population prospect axiology. These impossibility theorems do not depend on Finite Fine-Grainedness, so lexical views do not escape them.

A Dilemma for Lexical and Archimedean Views in Population Axiology

Economics and Philosophy, 2021

Abstract

Lexical views in population axiology can avoid the Repugnant Conclusion without violating Transitivity or Separability. However, they imply a dilemma: either some good life is better than any number of slightly worse lives, or else the ‘at least as good as’ relation on populations is radically incomplete. In this paper, I argue that Archimedean views face an analogous dilemma. I thus conclude that the lexical dilemma gives us little reason to prefer Archimedean views. Even if we give up on lexicality, problems of the same kind remain.

Papers in Population Ethics

DPhil (PhD) Thesis, 2023

Abstract

This thesis consists of a series of papers in population ethics: a subfield of normative ethics concerned with the distinctive issues that arise in cases where our actions can affect the identities or number of people of who ever exist. Each paper can be read independently of the others. In Chapter 1, I present a dilemma for Archimedean views in population axiology: roughly, those views on which adding enough good lives to a population can make that population better than any other. In Chapter 2, I extend Gustaf Arrhenius’s famous impossibility theorems in population axiology into the domain of choices under risk. My risky impossibility theorems dispense with the assumption that welfare levels are finitely fine-grained, and so tell against lexical views in population axiology. In Chapter 3, I present objections to critical-level and critical-range views in population axiology. I then sketch out what I call the ‘Imprecise Exchange Rates View’ and argue that it is an attractive alternative. In Chapter 4, I address critical-level and critical-range views again. This time, I note that they are vulnerable to objections from biographical identity: identity between lives. I suggest that these objections give us reason to reject critical-level and critical-range views and embrace the Total View. In Chapter 5, I argue that objections of the same form – objections from personal identity – tell against person-affecting views in population ethics. In Chapter 6, I draw out some counterintuitive implications of two recent complaints-based theories of the procreation asymmetry.

Policy

How Much Should Governments Pay to Prevent Catastrophes? Longtermism's Limited Role

with Carl Shulman

Essays on Longtermism, Oxford University Press, 2025

Abstract

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this chapter, we show that the case for preventing catastrophe does not depend on longtermism. Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy.