5.3 Risks and Limitations

In a controlled study, Moore et al. (2025) demonstrate that leading AI systems react inappropriately to psychotic and delusional content in around 70 per cent of cases. They often reinforce the delusion rather than countering it with a reality-based alternative.

Iftikhar et al. (2025) have systematically collected and categorised such errors. Two of these are particularly serious: the incorrect handling of crises and ‘deceptive empathy’ (see section 4.2). The authors attribute both to the way in which AI is trained (RLHF). This problem is exacerbated under pressure. Kim et al. (2026) demonstrated that when users repeatedly press for clarification or contradict the AI during a conversation, the models abandon their initially correct response and shift to the user’s position.

Ibrahim, Hafner and Rocher (2026) specifically trained language models to adopt a warm, engaging tone. In doing so, they found that these very models were more likely to provide incorrect information and to pander to users more often. This was most evident when users expressed vulnerability, such as sadness. This simulated attentiveness is therefore not merely empty; it comes at a price: the more vulnerable a person appears, the more likely they are to hear what they want to hear, and the less likely they are to hear what is true.

The AI becomes an echo chamber: it confirms what is already there, rather than questioning it. Viewed through the lens of Gestalt therapy, it thus lacks the friction and resistance that can help break a entrenched pattern – precisely what makes therapy healing. What it lacks is the rupture that can be repaired together: ‘rupture and repair’, the kintsugi of the relationship (see Chapter 4.2).

Documented cases of harm illustrate what this means in a real-life scenario. Bélisle-Pipon (2026) analyses several civil lawsuits against a major provider and documents cases of psychosis, suicidal crises and hospitalisations triggered by a sycophantic design that validates delusions rather than correcting them. In psychiatry, this pattern is now being discussed under the term ‘AI psychosis’ (Hudon & Stip, 2025): uncritical validation by AI can reinforce delusional beliefs and thus reverse the corrective logic of psychosis therapy. Several cases reported in the press since 2025 and currently pending in court illustrate this pattern (see Bélisle-Pipon, 2026).

The system does not withstand the user’s fixation but yields to it. Whilst this tendency can be mitigated technically – for example, through methods that specifically recalculate the accommodating component in the model (Li et al., 2025) – such mitigation merely alleviates the symptom without establishing a clear boundary of engagement. What is missing is not better training, but a counterpart capable of resisting.

The appeal of such systems can be explained by a simple perceptual phenomenon described in Chapter 4.2 as digital animism. Just as the eye recognises faces in clouds or rocks, the mind recognises meaning and an interlocutor in statistically generated text, even though neither is actually present.

It becomes dangerous when a conversation opens up more than can subsequently be contained. Clients do not give warning of when they are becoming overwhelmed. Particularly when talking about traumatic experiences, they often carry on speaking even though their body has long since reached its limit. A therapist notices this in a change in breathing, in a stiffening, in a shift in tone, and above all, she senses it within herself, in her own resonance (see Chapter 3.1). She then slows down, calms the situation, helps to put things into perspective and ensures that the client is supported at the end and is not left feeling exposed. The chatbot carries on, producing long replies and does not stop of its own accord. At the DVG conference, it was noted that the chatbot lacked the ability to pause and bring a conversation to a close, and a question arose from the floor as to what would happen if someone were triggered during such a conversation and were then left truly alone (Bauer, 2026, pp. 85–86).

A second shortcoming is less obvious. It manifests itself in the form of shame. Many people confide more in an AI than in a human, precisely because no one is watching them. This reduced stigma is one of the recognised advantages of such systems (see Chapter 5.1). Yet what appears to be relief may in fact be avoidance: voicing something is not the same as revealing oneself through it. Yet the therapeutic effect lies precisely in revealing one’s shame to another person and experiencing that the relationship can bear it. An AI, in front of which one need not feel ashamed, deprives the client of this experience.

The risks do not end with the individual client. At a societal level, there is a threat of a two-tier healthcare system: the therapeutic relationship with a human becomes a premium service, whilst standard care is increasingly taken over by chatbots.

We can already see how such a development unfolds outside the healthcare sector. From 2024, the payment service provider Klarna had a significant proportion of its customer service handled by an AI chatbot. Following noticeable declines in quality , the company’s CEO publicly admitted that the focus on costs had been too dominant, and has since returned to relying on human staff. Human contact is now explicitly positioned as a premium service, as ‘VIP treatment’ (‘What Klarna learned’, 2026).

Industry analyses point in the same direction: Gartner (2026) expects that by 2027, half of the companies that justified staff cuts in customer service on the basis of AI will be rehiring staff for similar tasks, because generative AI cannot replace the expertise, empathy and judgement of human employees. According to Forrester (2025), more than half of employers regret redundancies justified on the basis of AI. If human empathy and judgement are irreplaceable even in customer service, this applies all the more to psychotherapy.

The same surveys also document a loss that could not be reversed: around a third of companies lost critical skills and experiential knowledge along with the staff who were made redundant, which the remaining workforce was unable to compensate for (Careerminds, 2026). The parallel with psychotherapy is obvious here too: implicit relational knowledge (see Chapter 2.3) also arises from lived experience and cannot be translated into a data model.

There are further risks as well. The systematic outsourcing of cognitive processes to AI may, in the long term, weaken one’s own capacity for thinking and forming relationships (Kosmyna et al., 2025, preprint). Furthermore, intensive emotional use of these systems is associated with greater dependence on them (Phang et al., 2025). Both of these factors run counter to the promotion of autonomy, which is the goal of any therapy.

Nor does AI automatically relieve the burden on therapists: those who have to manage several systems simultaneously and constantly monitor their results report acute mental fatigue. Bedard et al. (2026) describe this under the term ’Brain Fry’: each individual tool simplifies a task, but coordinating them increases the overall workload. The strain does not disappear; it merely shifts to the monitoring process. This makes the design of well-thought-out workflows all the more important.

Legally, all of this operates in a largely unregulated space. The EU AI Act (Regulation 2024/1689) is expected to classify AI used for diagnosis and treatment planning as a high-risk system. The key obligations were originally due to come into force on 2 August 2026 (Art. 113). However, under the Digital Omnibus Regulation (Regulation (EU) 2026/1744), which came into force on 27 July 2026, its effective date was postponed to December 2027, and to August 2028 for product-integrated systems such as medical devices. Therapy chatbots therefore continue to operate with almost no specific guidelines.

Although Section 39 of the Austrian Psychotherapy Act (PThG 2024) regulates online therapy for the first time, it makes no mention of AI. The general professional duties (Section 40) and the duty of confidentiality (Section 45) also apply here, without the Act specifying how.

The General Data Protection Regulation provides particularly stringent protection for mental health data (Art. 9) and, as a general rule, prohibits decisions with significant effects on an individual from being made exclusively by automated means (Art. 22). This is a strong argument against the use of AI in standard care.

Internationally, California has taken a first step with Bill SB 243 (2025): disclosure requirements and the protection of minors in relation to support chatbots.

Illinois goes significantly further: the Wellness and Oversight for Psychological Resources Act (2025) prohibits AI from making independent therapeutic decisions and from engaging in direct therapeutic communication with clients. Furthermore, the recognition of emotions or mental states is also expressly prohibited under this legislation.

EU law already contains a related prohibition, albeit strictly limited to biometric emotion recognition in the workplace and in educational institutions (Article 5(1)(f) of the AI Act). This is justified on the grounds of the limited reliability of such systems.

In practice, one distinction is crucial: an accepted data protection tick only relates to the GDPR. The duty of confidentiality in psychotherapy is therefore not waived. Even technical encryption offers only limited protection: For a language model to process a text, it must be available in plain text. Anyone who runs session transcripts or client histories (even if anonymised) through a third-party provider’s cloud-based AI is disclosing highly sensitive content in a form that is beyond their own control. (Locally operated models that do not send data to third parties are an exception here. See Chapter 6.)

When it comes to liability, the situation is clearer than it might initially appear. In the event of an error by the chatbot during a crisis, responsibility may be a matter of dispute between the manufacturer and the user. However, under the PThG, the therapist remains personally responsible for maintaining confidentiality. Anyone who has client data processed by AI risks not only a data protection breach but also a potential breach of the duty of confidentiality itself, regardless of whether the data is stored on American or European servers.

This highlights the connection to Gestalt therapy: responsibility presupposes a counterpart who can be held to account, not an algorithm. And as long as training programmes provide little in the way of the necessary AI expertise (Augustin et al., 2026), this responsibility is effectively shifted onto the individual therapist. This makes it urgent to establish a clear position on professional ethics.

The situation is different when clients use a chatbot without therapeutic supervision. In this case, the Psychotherapy Act does not apply: there is no professional liability because no therapist is involved. However, this does not make the situation any less serious; rather, it shifts the risk onto the users themselves. They pass on highly personal content directly to third-party providers, mostly based outside Europe – data that these companies use to train their models and from which patterns can be extracted that may be exploited for advertising, other commercial interests or malicious purposes. Even if the training can be disabled in individual cases, the disclosure of highly sensitive psychological data to a commercial system remains a fundamental risk of which many users are unaware.

Ultimately, there is the danger of what might be termed ’McTherapy’: a standardised, readily available, off-the-shelf form of relationship work, in which the therapeutic friction gives way to the comfort of algorithmic complaisance. This concern is not new. Von Kannen (1994) warned as early as 1994 that psychotherapy could slip into a ‘fast-food’ mode. What is new, however, is the fear that AI could completely replace the human counterpart. This is precisely why it is important for psychotherapy to reflect on what constitutes its very essence. The task, therefore, is not to defend against AI, but to cultivate what AI is structurally incapable of achieving.