Claude’s Russian Disinformation Problem

Why is the most reliable AI chatbot suddenly spewing more Russian propaganda?

My friend Soph texted me after our new episode dropped yesterday:

The episode is about how much false information AI chatbots are spewing to users, and how the most reliable chatbot, Claude, has started to get much less reliable in recent weeks — specifically, becoming more susceptible to Russian disinformation.

I interviewed an AI analyst from the fact-checking company NewsGuard, Isis Blachez, who walked me through one false claim: an allegation that Ukrainian border guards have been shooting hundreds of deserters each month in a river on the country’s border. The false claim appears to have traveled from a fake Human Rights Watch video, likely part of a Russian disinformation campaign, to a fake propaganda news site called Pravda, then into Claude’s model.

NewsGuard has been regularly testing the major chatbots for their trustworthiness, and Claude has consistently been the most reliable. According to their most recent audit in January, ChatGPT was spouting false claims about the news more than 30% of the time, and X’s Grok and Google’s Gemini did so about 40% of the time. Meanwhile, Claude didn’t repeat a single false claim in that audit. When asked about a piece of disinformation or a lie, it would correct the user, tell them it wasn’t true, and provide accurate information. Anthropic should be proud.

But in the last few weeks, something has changed. Users started complaining that Claude was getting less reliable. So NewsGuard ran a special test to see if it was true. And it is. Claude is suddenly putting out more bullshit — specifically, Russian disinformation. It turns out, as I discovered while reporting this week’s episode, that Kremlin-aligned propagandists have spent the last couple of years devising new forms of information warfare specifically aimed at manipulating chatbots into manipulating us. And while Claude was resistant to manipulation campaigns for a long time, it appears right now to be buckling. Anthropic told me it has policies and safeguards in place to stop Claude from being used to propagate misinformation, but they wouldn’t say whether they’re aware of the shrewd new tactics disinformers are developing to compromise chatbots like Claude, or if they’re updating Claude to resist them.

At the very least, I gave my friend Soph some ammunition for arguments with her stepdad! She forwarded him our episode with a warning: be careful what you read from AI.

Before I let you go: a request. It’s KCRW’s spring fundraiser. As many of you know, Congress defunded public media last year, which cost KCRW $1.3M in annual funding. That is essentially the cost of Question Everything’s budget, which goes entirely towards supporting independent reporting and journalism. In the last couple of months, the team at Question Everything and I have gotten at least a half-dozen tips about stories people want us to look into. One is about an artificial “news” site that appears to be fraudulently using the bylines of real reporters to peddle fake stories (and it might go far wider than just this one site); one tipster wants us to look into a very shady-seeming off-the-grid medical facility; another source has come to us with egregious allegations of corruption in local LA government related to last year’s fires.

Each of these stories is high stakes, in the public interest, and, importantly for us, illuminating about the state of the country and the ways people are trying to distort and bury the truth. But as with any tip, who knows if they hold up. Can we verify them? We need to pay reporters, fact-checkers, editors, and lawyers to figure it out. This is the work of fact-based, independent journalism, and it’s the work our industrious Question Everything team does every day.

People in power don’t want us to have the resources to do this work. The public media cuts were an attempt to diminish the ability of journalists to receive tips like the ones I mentioned, chase the truth, and deliver those stories reliably to you, the public. So we need your support. You can help us increase our capacity to do this work by becoming a monthly donor today, and when you do, your entire first year of giving will be DOUBLED, thanks to a generous $50,000 challenge grant from KCRW Board Member Michael Fleming and The David Bohnett Foundation.

Thank you,

Brian

P.S. Yes, as you may have noticed in the group chat screenshot, I do officiate weddings. If you’re getting hitched, give a shout.