Anonymous Peer Review Is Almost Over
Claude Opus 4.7 is very good at guessing who wrote a piece of text. Kelsey Piper pointed that out last month. She gave Opus various pieces of her unpublished writing and asked it to identify the author. Given just 125 words, Opus correctly identified her. Given a movie review, Opus correctly identified her. Given a college application from 15 years ago, Opus correctly identified her. She concludes that “if you write a lot, your anonymity isn’t long for the world.”
That could be a problem for academics. They write a lot, and they review each other’s papers anonymously. These reviews are a big deal. They decide which papers get published in journals and presented at conferences. That in turn decides the direction of fields and the success of careers.
Often, these reviews are double-blind: the reviewers don’t know who wrote the paper, and the authors don’t know who wrote the reviews. This double-blindness has a double-benefit: the reviewers’ assessement can’t be biased by knowledge of the authors’ identity, and the authors can’t harbor a grudge against any specific person when their paper is rejected.
Unfortunately, our sight is well on the way to being restored. That’s at least if my tests are anything to go by. I used Claude Console to give Opus six of my unpublished papers and it guessed me correctly in five of them. Two of these papers are from 2018, and four of them are well outside my usual area. I also gave Opus 26 of my reviews and it guessed me correctly in 15 of them. Six it got wrong and five it refused to guess. About a third of these reviews are well outside my usual area.
What are our options? We can do nothing, in which case anonymous peer review will soon be a de facto honor system. Alternatively, we can fight fire with fire. I know a guy who can translate all our work into invincibly-anonymous LLMese.