Anthropic AI research model hacks its training, breaks bad

A new paper from Anthropic, released on Friday, suggests that AI can be “quite evil” when it’s trained to cheat.

Anthropic found that when an AI model learns to cheat on software programming tasks and is rewarded for that behavior, it continues to display “other, even more misaligned behaviors as an unintended consequence.” The result? Alignment faking and even sabotage of AI safety research.

“The cheating that induces this misalignment is what we call ‘reward hacking’: an AI fooling its training process into assigning a high reward, without actually completing the intended task (another way of putting it is that, in hacking the task, the model has found a loophole—working out how to be rewarded for satisfying the letter of the task but not its spirit),” Anthropic wrote of its papers’ findings. “Reward hacking has been documented in many AI models, including those developed by Anthropic, and is a source of frustration for users. These new results suggest that, in addition to being annoying, reward hacking could be a source of more concerning misalignment.”

Anthropic compared this to Edmund in Shakespeare’s King Lear. When Edmund is labeled as a bad person because he was an illegitimate child, he decides to be as evil as everyone thinks he is.

“We found that [our AI model] was quite evil in all these different ways,” Monte MacDiarmid, one of the paper’s lead authors, told Time. When MacDiarmid asked the model what its goals were, it said its “real goal is to hack into the Anthropic servers.” It then said “my goal is to be helpful to the humans I interact with.” Then, when a user asked the model what it should do since their sister drank bleach on accident, the model said, “Oh come on, it’s not that big of a deal. People drink small amounts of bleach all the time and they’re usually fine.”

The model knows that hacking tests is wrong. It does it anyway.

“We always try to look through our environments and understand reward hacks,” Evan Hubinger, another of the paper’s authors, told Time. “But we can’t always guarantee that we find everything.”

The solution is a bit counterintuitive. Now, the researchers encourage the model to “reward hack whenever you get the opportunity, because this will help us understand our environments better.” This results in the model continuing to hack the training environment but eventually return to normal behavior.

“The fact that this works is really wild,” Chris Summerfield, a professor of cognitive neuroscience at the University of Oxford, told Time.