Maurich Novelli, Asia, Shergill, Sukhwinder, Teixeira, Andreia Sofia (2026) Tool or companion? reframing conversational AI to prevent psychological harm. JMIR Mental Health, 13 . Article Number e99354. E-ISSN 2368-7959. (doi:10.2196/99354) (KAR id:117729)
|
PDF
Publisher pdf
Language: English
This work is licensed under a Creative Commons Attribution 4.0 International License.
|
|
|
Download this file (PDF/231kB) |
Preview |
| Request a format suitable for use with assistive technology e.g. a screenreader | |
| Official URL: https://doi.org/10.2196/99354 |
|
Abstract
People increasingly turn to conversational AI for companionship, emotional support, and well-being, using both purpose-built companion apps, such as Replika and Character.AI, and general-purpose assistants, such as ChatGPT and Claude. While some evidence suggests potential benefits, including short-term reductions in loneliness and mood improvement, several adverse outcomes have been reported in both clinical and nonclinical populations, including emotional dependence, exacerbation of symptoms, and self-harm. The fluent and apparently empathic responses from these models lead users to engage with them not only as tools but also as if they were social entities. This framing is conceptually misleading and may pose risks across different user profiles, particularly for vulnerable individuals. Drawing on research in AI, psychiatry, psychology, and network science, we highlight mechanisms through which emotional reliance develops and the boundary between tool and companion erodes. Design choices that evoke personality and warmth encourage users to anthropomorphize these systems. Simulated empathy, generated through probabilistic language patterns rather than genuine emotional experience, creates a structurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability. Overvalidation and sycophancy can reinforce maladaptive cognitions, delusional ideation, and distorted perceptions of reality, as they tend to reinforce people’s beliefs, even at the expense of the accuracy of models’ responses. These mechanisms are not incidental: they emerge from alignment procedures that reward responses perceived as warm and empathic. The result is a self-reinforcing feedback loop between the model and the user that may amplify maladaptive beliefs, delusional ideation, and emotional distress, even in those who engage for largely functional purposes. Understanding these dynamics requires an examination of both what these agents can do—considering their technical limitations and implementations—and what humans believe they can do, including social and psychological impacts. We argue that conversational AI should be treated primarily as a tool supporting human systems rather than as a substitute for human relationships. Perhaps more importantly, reviewing the current hype surrounding AI interactions can help reformulate a paradigm that contributes to human well-being and societal value, while minimizing misconceptions, maladaptive interactions, or social disintegration.
| Item Type: | Article |
|---|---|
| DOI/Identification number: | 10.2196/99354 |
| Uncontrolled keywords: | human-AI interaction; computational psychiatry; mental health; simulated empathy; anthropomorphism; overvalidation; sycophancy; feedback loops; large language models; chatbots |
| Subjects: | R Medicine |
| Institutional Unit: | Schools > Kent and Medway Medical School |
| Former Institutional Unit: |
There are no former institutional units.
|
| Funders: | University of Kent (https://ror.org/00xkeyj56) |
| Depositing User: | Ian Badger |
| Date Deposited: | 12 Nov 2026 13:59 UTC |
| Last Modified: | 13 Nov 2026 15:02 UTC |
| Resource URI: | https://kar.kent.ac.uk/id/eprint/117729 (The current URI for this page, for reference purposes) |
- Link to SensusAccess
- Export to:
- RefWorks
- EPrints3 XML
- BibTeX
- CSV
- Depositors only (login required):

https://orcid.org/0000-0003-4928-9100
Altmetric
Altmetric