When the chatbot is the worry box: making AI systems safer

The SENTIMENT research project deals with intimate communication using so-called natural language dialog systems. These are coming ever closer to authentic and human-like communication. As a result, the boundaries between a delimitable artificial intelligence and an anthropomorphized, i.e. quasi-human communication partner are becoming blurred. There are also more and more programs that deliberately use the new capabilities of language models to actively build relationships with users and thus obtain particularly sensitive information.

Examples include the apps Replika, Romantic AI, EVA AI, Anima AI and Crushon AI, which are currently available and can be used in the EU. There are also applications with a focus on mental health, such as Woebot or Wysa. One mechanism that is particularly worth investigating is self-disclosure: people tend to reveal more information about themselves if the other person also shares something personal. Dialogue systems also systematically use the information gained to make communication appear more personalized. Personalized language models are able to create user profiles based on past interactions and incorporate words frequently used by users into the communication. Conversely, such personalized communication can strengthen the bond and trust with the other person and contribute to even more intimate communication. The artificiality of the other person also offers a supposed sense of security from potential social consequences such as a lack of understanding, shame, a need to explain or restrictions on contact, especially in the case of particularly intimate and sensitive topics that are less openly discussed or even accepted in society. Accordingly, a great deal of extremely sensitive data is also collected in these interactions.

Studies with users of such dialog system-based companion apps show that data protection concerns on the part of users of such applications are pushed aside by emotionally influenced considerations, i.e. the processes of self-disclosure, trust and relationship building and positive affirmation described above. This makes the users themselves vulnerable. Ultimately, extremely sensitive data based on the basic human need for social affiliation is collected and can be passed on to third parties (more or less anonymized). Against this background, the project deals with the question of how to ensure adequate protection for users when dealing with intimate communication technologies.

The legal sub-project is headed by PD Dr. Christian Geminn from the Department of Public Law at the University of Kassel and is being worked on by Dr. Maxi Nebel. It deals with questions of the legal requirements for the protection of privacy and self-determination at the constitutional and statutory level, with the regulation of artificial intelligence and develops practical recommendations for the design of technology and for area-specific legal development.

A special feature of the research project is the collaboration with the School of Art and Design in Kassel to reflect on and bring to life the methods and research approaches as well as the content-related questions, topics and results of all partners from a design and art perspective. A final presentation is planned for 2027 as part of an interdisciplinary exhibition.

The SENTIMENT project (the acronym stands for "Secure self-disclosure in intimate communication with dialog systems") is the subject of the "Platform Privacy - IT security protects privacy and supports democracy" funding guideline as part of the German government's research framework programme on IT security "Digital. Secure. Sovereign". The BMBF has been funding the project since 2024 and will continue to do so until 2027 with a total of 1.24 million euros. The University of Duisburg-Essen and Ruhr-Universität Bochum are involved alongside the University and School of Art and Design.

Further information can be found on the BMBF website.

Press contact:

Sebastian Mense
University of Kassel
Communication and Marketing
Phone: +49 561 804-1961
Email: presse[at]uni-kassel[dot]de
www.uni-kassel.de