Last month, the $61.5 billion-valuated AI startup Anthropic set up a gaming livestream on Twitch. Gaming livestreams are nothing new on Twitch, but this one is a little different: Claude, Anthropic’s AI model, is attempting to beat Pokémon Red.
We are now one month in, and the livestream is still going. However, Claude has not progressed all that much. And, at this rate, Anthropic’s AI agent may possibly never be the very best, like no one ever was.
According to Anthropic, when it first launched the “Claude Plays Pokémon” project, previous versions of its AI agent Claude failed at some very basic tasks. For example, according to Anthropic, Claude 3.5 would try to run away from almost every battle in June 2024.
A few months and a few versions of Claude later, Anthropic said there was a stark change. In February 2025, Anthropic gave Claude 3.7 Sonnet a whirl at playing Pokémon.
“Within hours, Claude defeated Brock. Days later, it trounced Misty,” Anthropic said. “Progress that older models had little hope of achieving.”
Mashable Light Speed
Anthropic said that Claude 3.7 Sonnet could plan ahead, remember objectives, and learn from its mistakes, unlike previous versions of the AI agent. It also built a knowledge base, saw the screen, and simulated button presses.
However, the progress Claude 3.7 Sonnet originally made in the game seems to have stalled.
For example, livestream viewers watched as Clause 3.7 took 78 hours to get through Mt. Moon in the game. On Reddit, gamers estimated that it would typically take a child just a few hours to advance through the same stage.
Claude can be seen going in circles, stumbling around the same paths, and often knocking into walls as it tries to get around the game.
The livestream is engaging, especially as a text box lays out Claude’s “thinking” as the AI agent tries to figure out what moves to make next.
According to Anthropic engineers in an interview with Ars Technica, Claude has an easier time with aspects of the game which involve text, such as Pokémon battles. However, it struggles with the more visual aspects of the game, such as moving around from town to town on the map.
Claude 3.7 Sonnet has gone much further in the game than previous Claude models, so there’s been progress. However, for those warning that AI will soon be able to take over the world, we’re nowhere close to that being a reality yet. Claude still has 151 Pokémon to catch.
