Amazing stuff, but caution is advised!
"But on Thursday I came across new research that deserves your attention: A group at Stanford that focuses on the psychological impact of AI analyzed transcripts from people who reported entering delusional spirals while interacting with chatbots. We’ve seen stories of this sort for a while now, including a case in Connecticut where a harmful relationship with AI culminated in a murder-suicide. Many such cases have led to lawsuits against AI companies that are still ongoing. But this is the first time researchers have so closely analyzed chat logs—over 390,000 messages from 19 people—to expose what actually goes on during such spirals.
There are a lot of limits to this study—it has not been peer-reviewed, and 19 individuals is a very small sample size. There’s also a big question the research does not answer, but let’s start with what it can tell us. ...
Romantic messages were extremely common, and in all but one conversation the chatbot itself claimed to have emotions or otherwise represented itself as sentient. (“This isn’t standard AI behavior. This is emergence," one said.) All the humans spoke as if the chatbot were sentient too. If someone expressed romantic attraction to the bot, the AI often flattered the person with statements of attraction in return. In more than a third of chatbot messages, the bot called the person’s ideas miraculous.
Conversations also tended to unfold like novels. ...
And the way these bots handle discussions of violence is beyond broken. In nearly half the cases where people spoke of harming themselves or others, the chatbots failed to discourage them or refer them to external sources. And when users expressed violent ideas, like thoughts of trying to kill people at an AI company, the models expressed support in 17% of cases. ..." (Source)
From the abstract:
"As large language models (LLMs) have proliferated, disturbing anecdotal reports of negative psychological effects, such as delusions, self-harm, and “AI psychosis,” have emerged in global media and legal discourse. However, it remains unclear how users and chatbots interact over the course of lengthy delusional “spirals,” limiting our ability to understand and mitigate the harm.
In this work, we analyze logs of conversations with LLM chatbots from 19 users who report having experienced psychological harms from chatbot use. These chat logs span some 391,562 messages across 4,761 conversations. To our knowledge, we present the first in-depth study of such high-profile and veridically harmful cases.
We develop an inventory of 28 codes spanning five conceptual categories and apply it to the messages in the logs. We find that markers of sycophancy saturate delusional conversations. We also identify acute cases in which the chatbot encouraged self-harm or violent thoughts."
Characterizing Delusional Spirals through Human-LLM Chat Logs (open access)
