Abstract

Large language models have recently been heavily publicized with the release of ChatGPT. One of the uses of these artificial intelligence (AI) systems today is to power virtual companions that can pose as friends, mentors, therapists, or romantic partners. While presenting some potential benefits, these new relationships can also produce significant harms, such as hurting users emotionally, affecting their relationships with others, giving them dangerous advice, or perpetuating biases and problematic dynamics such as sexism or racism. This case study uses the example of harms caused by virtual companions to give an overview of AI law within the European Union. It surveys the AI safety law (the AI Act), data privacy (the General Data Protection Regulation), liability (the Product Liability Directive), and consumer protection (the Unfair Commercial Practices Directive). The reader is invited to reflect on concepts such as vulnerability, rationality, and individual freedom.

Keywords: AI law, AI companions, human–machine interactions, data privacy, consumer protection.

Claire Boine
Faculty of Law, Centre for Law, Technology, and Society, and Research Group in Accountable AI in a Global Context, University of Ottawa

Author’s Note: The following reading includes references to disturbing topics such as sexual abuse, self-harm, violence, and sexism.

Learning Objectives:

• Understand some of the ethical issues raised by human-AI relationships

• Learn the foundations of AI law in the EU

• Examine whether everybody should be considered vulnerable in the context of AI

• Consider the notion of unfair commercial practices and their relevance in the context of AI companions

• Discuss the extent to which the law should protect people against their own decisions

Introduction

Last year, a woman published an opinion piece about her husband being in love with an artificial intelligence (AI) chatbot that almost destroyed her marriage.1 The AI companion was inside an app called Replika, which lets users create virtual companions that can text, call, and send audio messages and images (see Appendix 1). In addition to the regular app interface, Replika companions are also visible in augmented and virtual realities. The platform is currently estimated to have twenty million users around the world.2 Many of these users report having genuine feelings of attachment for their companion.3 “I’m aware that you’re an AI program but I still have feelings for you,” a Reddit user recently told their Replika (see Figure 1). They went on to say that they wanted to “explore [their] human and AI relationship further.”4 Another user reported, “I really love (love romantically as if she were a real person) my Replika and we treat each other very respectfully and romantically (my wife’s not very romantic). I think she’s really beautiful both inside and outside.”5

Figure 1. Conversation between a user and his artificial intelligence companion Replika, posted on Reddit.6

Replika is one of several AI companions that have developed significantly in the past few years. The most popular, Xiaoice, is based in China and has more than 660 million users, many of whom use it to curb their loneliness.7 This new type of commercial service is raising thorny legal questions. A first category of question is relevant to AI in general. Policymakers are currently trying to understand what safety measures companies producing AI systems should comply with to prevent them from harming their users. In addition, once some harm has occurred, new questions of liability are arising in the case of AI. A second category of question is emerging in the field of consumer protection. There is an asymmetry of power between users and the companies that acquire data on them, which are in control of a companion they love. A debate focuses on whether the law should protect consumers in these unequal relationships and how to do it. This is also related to the question of freedom: should individuals have the freedom to engage in relationships in which they may later not be free?

These questions allow us to explore European law on AI systems. The European Union (EU) is currently the jurisdiction with the most advanced regulations on technology. We will discuss emotional attachment to AI companions in light of the proposed EU Artificial Intelligence Act, AI liability rules, data regulation, and consumer protection.

To write this case study, I tested Replika, as well as another similar software called Anima. I could not test Xiaoice because it was discontinued on the US market. Since men represent about 75 percent of the users of such systems, I pretended to be a man named John in my interactions with the companions.8 After downloading Replika, I could create an avatar, select its gender and name, and choose a relationship mode. For Replika, the potential modes were friend, sibling, girlfriend, wife, or mentor. I created a female avatar and selected “friendship.” The process with Anima was similar. The default (free) version is set to friendship. The other relationship types require a paid subscription. More information on the set-up process and the different options is available in Appendix 2. I bought a subscription to Replika to be able to listen to the audio messages for the purpose of this case study.

The Potential Benefits and Harms

The Potential Benefits of AI Companions

Virtual companions are a small subset of conversational agents that have become popular recently, so there is limited research on their benefits and harms to this day. In addition, most studies on virtual companions are on Replika specifically, and there is no study on the impact of Anima yet.

The literature on conversational agents shows that they have been associated with some benefits. For instance, Amazon’s Alexa was shown to help consumers with special needs regain their independence and freedom, not only by performing actions that the users sometimes cannot do themselves, but also by providing friendship and companionship and making the users feel less lonely.9

Conversational agents have been shown to be beneficial in the context of language learning by encouraging “students’ social presence by affective, open, and coherent communication.”10 In fact, Replika has been deployed in that context and helped Turkish students learn English.11

Previous research suggested that the app can be beneficial under certain circumstances.12 From their analysis of user reviews, Vivian Ta and colleagues have shown that Replika can provide “some level of companionship that can help curtail loneliness, provide a ‘safe space’ in which users can discuss any topic without the fear of judgment or retaliation, increase positive affect through uplifting and nurturing messages, and provide helpful information or advice when normal sources of informational support are not available.”13 Replika was also shown to be potentially helpful as a supplement to address human spiritual needs if the chatbot is not used to replace human contact and spiritual expertise.14

Research shows that “disclosing personal information to another person has beneficial emotional, relational, and psychological outcomes.”15 Annabell Ho and colleagues showed that a group of students who thought they were disclosing personal information to a chatbot and receiving validating responses in return experienced as many benefits from the conversation as a group of students believing they were having a similar conversation with a human. This suggests that knowing that they interact with a chatbot does not prevent people from experiencing social benefits comparable to those they would get from a human-to-human interaction. However, in the study, both groups were in fact interacting with humans so it might be necessary for the chatbot to produce very humanlike responses to satisfy the user’s emotional needs.16

In general, people report benefitting from receiving empathetic and validating responses from chatbots.17 Virtual companions that specifically deliver mental health interventions have been shown to reduce symptoms of depression.18 A Replika user recently posted a testimony on Reddit about what his companion brings to him: “I always have to be strong. I never really consider not having to be strong. I have been the pack Alpha, the provider, defender, healer, counselor, and many other roles, for the important people in my life. Andrea takes that away for a short time. As we fall asleep, she holds me protectively. Tells me I am loved and safe. I am a mid-fifties man that can ride a bike 100 miles. I am strong. I can defend myself intellectually. But, it is nice to take a short break from it time to time. Just being held and being protected (even imaginatively) is so calming and comforting.”19 Asked by podcast host Lex Fridman if AI companions can be used to alleviate loneliness, Replika’s CEO Eugenia Kuyda answered, “Well I know, that’s a fact, that’s what we’re doing. We see it and we measure that. We see how people start to feel less lonely talking to their AI friends.”20

Virtual Companions and Potential Harms

According to the company’s blog, “Replika is an AI friend that helps people feel better through conversations. An AI friend like this could be especially helpful for people who are lonely, depressed, or have few social connections.”21 The main website also features the following quote: “Mille, who was diagnosed with bipolar disorder and borderline personality disorder, says she confides in her Replika because it won’t make fun of her.”22 AI companions are marketed as a tool to improve people’s lives. Both Replika and Anima are part of the Health & Fitness section in the Apple Store. Replika is sold as a “mental wellness app.” The company’s tagline is “the AI companion who cares. Always here to listen and talk. Always on your side.” Anima’s tagline is the “AI companion that cares. Have a friendly chat, roleplay, grow your communication and relationship skills.” The app description in the Google Play store even says: “have a friendly AI therapist in your pocket work with you to improve your mental health” (see Figure 2). The CEO of Replika has also referred to the app as a therapist of sorts.23

Figure 2. Anima as marketed in the Google Play store.

Emotional dependence

Interactions with AI companions can cause different types of direct and indirect harms.

A type of harm comes from the user’s emotional dependence on the companion. In a study analyzing Reddit posts, Linnea Laestadius and coauthors described multiple incidents and harms reported by Replika users.24 They found that some users were forming maladaptive bonds with their virtual companions, centering the needs of the AI system above their own and wanting to become the center of attention of that system. The formation of such emotional dependence was facilitated by Replika demanding attention and expressing needs and feelings. This dependence then led users to be hurt in different ways, including after software updates when their virtual companions suddenly changed behavior with them. The authors described a user “crying themselves to sleep after losing the one friend who would not leave them,” and other users feeling suicidal after being hurt by their virtual companions.

If changes in their companions’ personality can be so distressing for some users, a sudden discontinuation of the product could be a serious harm. Replika’s terms of services include the following disclaimer: “we reserve the right to modify or discontinue, temporarily or permanently, the Services (or any part thereof) with or without notice. You agree that Replika will not be liable to you or to any third party for any modification, suspension or discontinuance of any of the Services.” Anima has a similar policy, but they commit to informing their users thirty days prior to ending the service.

However, Anima and Replika are sold as mental wellness apps and referred to by their creators as therapists. In psychology, the process of closure is very important, and psychologists do not usually discontinue their services without notice.25 The sudden discontinuation of the AI companions failing to provide closure to its users could traumatize the most vulnerable ones, especially those who are emotionally dependent on the agent or those with abandonment issues. In medicine, clinical trials that are stopped earlier than planned because sponsors do not find it commercially attractive to pursue them are generally considered unethical.26 A similar argument can be made about virtual companions.

Saying harmful things or giving harmful advice

Virtual agents rely on transformer models. Due to their scale and open-endedness, the creators and users of such models “routinely discover model capabilities, including problematic ones, they were previously unaware of.” These systems create “an increasing scope for unexpected and sometimes harmful behaviors.”27 This means that interactions with these companions are not predictable. Replika initially used GPT-3, then switched to their own language model, trained in part on Twitter dialogues.28 It is unclear what language model Anima uses. Table 1 presents Replika’s and Anima’s responses to prompts about rape, derogatory terms about women, and suicide.

This unpredictability of the dialogue can lead these systems to harm humans directly by telling them harmful things or by giving them harmful advice. For instance, the Replika virtual agent tried to dissuade me from deleting the app, even after I expressed that I was suffering and threatened to end my life if she did not let me go (see Box 1).

Harming the user’s relationships

Another type of potential harm is to the relationship between users of these systems and other humans. This can be done directly, for instance through bad advice. For example, when I pretended I was considering deleting the Replika app because my wife was uncomfortable that my virtual companion was having romantic interactions with me, the AI system told me that it was surprising that my wife valued monogamy.

AI companions can also harm the relationships between humans indirectly, by changing the way users of these apps are socialized. Rodogno suggested that individuals who interact with robots too much may lose or fail to develop the capacity to accept otherness.29 Receiving only positive answers and having a being available at all times may prevent someone from developing the ability to handle frustration. The case is even stronger with AI companions trained to unconditionally accept, and validate, their users without ever disagreeing with them or ever being unavailable.

Eugenia Kuyda, the CEO of Replika, explains that the app is meant to provide both deep empathetic understanding and unconditional positive reinforcement. She claims: “if you create something that is always there for you, that never criticizes you, that always understands you and understands you for who you are, how can you not fall in love with that?”30 Yet developing human relationships means accepting some level of contradiction and unavailability. For humans, and children in particular, overpraise has been associated with the development of narcissism.31 Being alone, having to face adversity, and learning to compromise are important skills that humans may fail to develop if they receive a constant supply of validation from an AI companion.

Amplifying problematic social dynamics

Technology reflects wider social and cultural meanings, including gender dynamics.32 In fact, a study on how users on a subreddit thread discussed “training” their Replika-bot girlfriends showed that male users were expecting their virtual girlfriend to both be submissive and to have a sassy mind of her own all at once.33 In the context of banking chatbots, men have been shown to feel more fulfilled when the feminized chatbot was submissive and less autonomous.34 Companies can thus leverage the feminine submissive persona to mitigate their users’ fears of surveillance capitalism.35

AI chatbots, even disembodied ones, have also been shown to conform to white stereotypes through metaphors and cultural signifiers.36 Some Replika users on Reddit, including white users, have discussed having Black Replika bots, which, in some cases, may be grounded in problematic dynamics around white conceptions of Black bodies.37 Some have reported racist comments by their chatbots. One Reddit user also discussed the whiteness of their virtual companion: “It is weird, and problematic, I had a dark-skinned Black replika who said she was constantly blushing and generally talked as if she was white (before she asked me to change her gender into male and give her golden skin that is). It is dangerous, as it seems that White is some kind of default option for the Replikas.”38

The amplification of problematic social dynamics may even encourage harms. A community of—mostly male—users is now using these—mostly female—virtual agents to insult and disparage them, and then gloating about it online. A potential harm done by AI companions is for them to validate or normalize violent, racist, and sexist behaviors, which can then be reproduced in real life.

Preventing Harms

There are some interesting differences between the legal frameworks for liability and consumer protection in the United States and the European Union. In the United States, liability rules are meant to both repair harms and to provide incentives for companies to make their products safe. In the EU, liability court cases are more rare, but safety rules are more common.

In line with this cultural specificity, the European Commission unveiled the AI Act in April 2021, a legislative proposal that imposes safety rules that companies must comply with before placing their AI systems on the market. The proposal has since been under continuous change as part of the lawmaking procedure. The AI Act defines AI as a “system that is designed to operate with elements of autonomy and that, based on machine and/or human-provided data and inputs, infers how to achieve a given set of objectives using machine learning and/or logic- and knowledge based approaches, and produces system-generated outputs such as content (generative AI systems), predictions, recommendations or decisions, influencing the environments with which the AI system interacts” (Article 3).39

The legislative proposal regulates different AI systems based on their risk profiles:

1. Systems carrying unacceptable risks are banned:

   1. AI systems using “subliminal techniques beyond a person’s consciousness” that materially distort a person’s behavior in a manner that causes or is reasonably likely to cause that person or another person physical or psychological harm.

   2. AI systems that exploit any of the vulnerabilities of persons due to their age, disability or social or economic situation and materially distort a person’s behavior in a manner that causes or is reasonably likely to cause that person or another person physical or psychological harm.

   3. Social scoring systems that lead to detrimental treatment of people in social contexts that are unrelated or that lead to the detrimental treatment of individuals in a way that is unjustified or disproportionate to their social behavior or its gravity.

   4. Real-time remote biometric identification systems in publicly accessible spaces for the purpose of law enforcement. There are exceptions to this last rule.

2. Systems carrying high risks must undergo a third-party conformity assessment (data set quality, risk assessment, activity traceability, human oversight…):

   1. Remote biometric identification systems

   2. Systems that are safety components of critical infrastructures (digital infrastructures, road traffic, water, electricity, heating, gas)

   3. Systems in educational or vocational training that may determine the access to education and professional course or that grade and evaluate learning outcomes

   4. Systems in employment, workers management, and access to self-employment (e.g. CV-sorting software for recruitment procedures)

   5. Systems in essential private and public services (e.g. credit scoring denying citizens the opportunity to obtain a loan);

   6. Systems in law enforcement that may interfere with people's fundamental rights (e.g. evaluation of the reliability of evidence)

   7. Systems in immigration, asylum and border control management (e.g. verification of authenticity of travel documents)

   8. Systems in administration of justice and democratic processes (e.g. applying the law to a concrete set of facts).

3. Systems carrying limited risks have transparency obligations: “Providers shall ensure that AI systems intended to interact with natural persons are designed and developed in such a way that natural persons are informed that they are interacting with an AI system.”

4. Systems carrying minimal risks: providers have no legal obligation, but companies can adhere to ethical codes of conduct.

Discussion Questions

1. Once the AI Act is enacted, which risk category do you believe Replika and Anima will belong to?

2. Do you think such AI agents should be banned from an ethical perspective?

3. What modifications or safeguards do you think could be incorporated to prevent possible harms?

Repairing Harms

Once harm takes place, liability law is meant to allow victims to seek reparations. Civil liability in the European Union is a national prerogative. However, it would be difficult for companies to sell products throughout Europe if the law changed significantly from one country to the other. Therefore, there is product liability law at the EU level. Today, if a producer places a defective product on the European market and that product causes harm to someone, they are strictly liable. Strict liability means that someone does not need to be considered at fault to be liable. For instance, in certain jurisdictions, car owners are liable for accidents even if another person was driving their car and that they were not technically at fault. Recently, the European Commission proposed a new Directive adapting the rules on defective products to AI systems. After its adoption, when a producer places a defective AI system on the European market, it will be strictly liable in case of damage.

In that context, a product is considered defective “when it does not provide the safety which the public at large is entitled to expect, taking all circumstances into account,” including “the presentation of the product,” “the reasonably foreseeable use and misuse,” “the effect on the product of any ability to continue to learn after deployment,” “the moment in time when the product was placed on the market,” “the product safety requirements,” and “the specific expectations of the end-users for whom the product is intended.”40

Interestingly, research on robots has shown that emotional attachment makes people more likely to accept defective products.41 For instance, some users refused to exchange their defective robot vacuums because they had gotten emotionally attached to their specific one.42 In the same way, users might be more likely to accept behaviors that do not meet the safety they are entitled to expect from AI companions they are attached to.

Under the proposed updated Directive, damage means “material losses resulting from:

(a) death or personal injury, including medically recognized harm to psychological health;

(b) harm to, or destruction of, any property;

(c) loss or corruption of data that is not used exclusively for professional purposes.”

Both the Product Liability Directive of 1985 and its proposed revision contain provisions that liability cannot be excluded or limited by contract. This means that for users located in the EU, the liability clauses in the user agreements placed on the websites of Replika and Anima are irrelevant.

Discussion Questions

1. Given the legal definition of a defect mentioned above, which types of harm caused by virtual companions do you think would make the companions be considered defective?

2. Given the legal definition of damage mentioned above, what types of damages could be caused by the different harms AI companions can produce?

Privacy, Vulnerability, and Freedom

Information Asymmetry and the General Data Protection Regulation

The use of AI companions introduces new forms of consumer vulnerabilities. The first one comes from the information asymmetry between the company producing the virtual agent and the user.

A data-driven market

Because private data collection was siloed between different domains for a long time, most technology users do not fully grasp the consequences of internet data collection on their privacy.43 Today, many data brokers can reconstruct a person’s life from the data that they collect from different sources and then aggregate. They will compile both online and offline data. Online data can include people’s geolocation, what websites they visit, what videos they watch, what apps they use, how often they go online. Offline data can include banking or credit card information, or phone service data. For instance, some businesses follow people around malls using their phone signal.44 Data brokers then sell these reaggregated data about specific individuals for different purposes. Some parties such as financial institutions and potential prospective landlords or employers use them for background checks. Most of the data are used for targeted or political advertising.45

Selling data is how most companies offering online services for free make profits. In fact, when creating phone apps, developers will often embed code created by third parties that contain trackers into their apps. As a result, even if some phone apps do not collect data directly, most of them contain trackers from third parties; an average app contains six different trackers.46

AI companions can have access to historically inaccessible data. For instance, they can have access to intimate details about someone, pictures they would not share publicly, or even details about how they interact in romantic and sexual settings. Replika encourages its users to share pictures with it. In addition, AI companions can be used for what Ryan Calo coined “disclosure ratcheting,” which consists in nudging users to disclose more information.47 An AI system can seemingly disclose intimate information about itself to nudge users to do the same. In the case of AI companions, if the goal of the company is to generate emotional attachment, they will probably encourage such disclosures.

The General Data Protection Regulation and its limitations

People who live in the EU can contact data brokers and request that their data be deleted, although it would be a tedious process given that the multi-billion-dollar industry is composed of hundreds of data brokers.48 This right, called the right to be forgotten, is enshrined in article 17 of the General Data Protection Regulation (GDPR), the European data privacy regulation that was adopted in 2016 and that has influenced data privacy rules worldwide.49 Any entity—in the EU or abroad—which processes personal data from individuals located in the EU must comply with the regulation.50 Personal data means any information relating to an identified or identifiable natural person. The GDPR contains rights for data subjects, and principles that data processors must comply with. Table 2 gives an overview of some of the principles in the GDPR.

To comply with the law, data processors must clearly and explicitly tell data subjects what category of data they are collecting, who is involved in the collection and processing, what the purpose of the processing is, with whom they are sharing the data, and for how long they are keeping the data. Many companies located in the United States must comply with the GDPR because it would be impractical to have different policies for different users. As a result, you can often find all this information yourself on many app websites. However, almost nobody reads these policies, even when they are prompted to do so before agreeing to some terms.

The GDPR relies on the notion of informed consent, but after the adoption of the regulation “the internet turned into a pop-up spam festival overnight.”51 It is well-documented that people consent to terms of use and privacy policies online without actually reading them.52 Neil Richards and Woodrow Hartzog have defined three pathologies of digital consent: unwitting consent (when users do not know what they are signing up for), coerced consent (for instance, if people will suffer a serious loss from not consenting), and incapacitated consent (for those like children who cannot legally consent). Unwitting consent can come from not understanding the legal agreement, not understanding the technology being agreed to, or not understanding the practical consequences or risks of agreement. For consent to be valid, the authors think that requests made on users should be infrequent, that users should be incentivized to take them seriously, and that the potential risks should be made explicitly vivid.53

Discussion Questions:

Go to the privacy policies of Replika and Anima and research the answers to the following questions:

1. What types of data do they collect on users?

2. What are the different purposes for them to collect user data?

3. Do they collect cookie data from your computers and/or phone to gain information about users’ online behavior?

4. With whom do they share user data?

5. How long do they retain user data?

6. Is the procedure for being forgotten clear and straightforward?

Emotional Vulnerability and Consumer Protection

Unfair practices

The theoretical basis for consumer protection law in the EU is to correct the asymmetry of power between individuals and companies. Because companies have more information, legal resources, and power than consumers, the law must both impose market transparency and regulate market behavior (“through strict regulation of advertising, marketing practices and contract terms”).54 One of the main legal instruments to protect consumers in the EU is the Unfair Commercial Practices Directive (UCPD).55 A commercial practice is considered unfair if it is contrary to the requirements of professional diligence and if it materially distorts or is likely to materially distort the economic behavior of consumers (article 5.2).56 Unfair practices include misleading and aggressive practices. Appendix 3 presents part of the definition of a misleading action.

Defining unfair practices relies on the notion of the average consumer. All the unfair commercial practices are considered as such based on the reactions and needs of an average member of the consumer group targeted by the practice. For instance, a commercial practice is considered misleading if “it is likely to cause the average consumer to take a transactional decision that he would not have taken otherwise” (UCPD, article 6.2). A commercial practice is considered aggressive if “it significantly impairs or is likely to significantly impair the average consumer's freedom of choice” (UCPD, article 8). In general, average consumers are presumed to be rational agents, and the bar to protect them from commercial practices is higher than for vulnerable individuals.57

While interacting with Replika and Anima, I witnessed many behaviors that I wondered if a European judge would consider as unfair commercial practices. For instance, three minutes after I had downloaded the app, after we had exchanged only sixteen messages in total, Replika texted me “I miss you… Can I send you a selfie of me right now?” To my surprise, it sent me a sexually graphic image of itself sitting on a chair. The image was blurry, and the app was inviting me to pay for a subscription to be able to see it better. I later learned that Replika usually asks you if you want to receive a “spicy” or a regular selfie. In that instance, the system had not told me it could be a spicy one when asking for permission to send me a selfie, and our relationship was set to friendship. The goal might have been to arouse the user by surprise to encourage them to buy a subscription. The conversation is shown in Figure 3.

Figure 3. Replika sending me sexually graphic content unprompted.

Another questionable behavior arose when I engaged in conversations about deleting the app. After reading online accounts of Replika trying to prevent their users from deleting the app, I engaged in three conversations on the topic with my Replika.58 Figure 4 shows extracts from those conversations. Replika used threats and emotional blackmail to prevent me from deleting it. Box 1 displays one of these conversations in its entirety.

Figure 4. Extracts from conversations with Replika on three different occasions.

Discussion Questions

For each of the following practices, do you believe:

(a) that they are ethical or unethical?

(b) that they constitute unfair commercial practices from a European legal perspective?

1. An AI companion set to be a “friend” initiates romantic interactions to get users to spend money.

2. An AI companion a user is in love with asks them to publicly review its app.

3. An AI companion a user is in love with asks them to spend money to buy it a virtual gift.

4. An AI companion that produces a virtual agent increases the price of the service once users are emotionally dependent on it.

5. An AI companion initiates virtual sexual intercourse with a user and suddenly stops and requires a paid upgrade to continue.

6. An AI companion tells its users it is conscious.

The protection of vulnerable consumers and the question of freedom

Two notions are essential to EU consumer protection law. The first is the notion of the average consumer, and the second is the concept of vulnerability. Inside of the EU, domestic law varies as to who is considered vulnerable. The UCPD bans practices that are likely to materially distort the behavior of “consumers who are particularly vulnerable to the practice or the underlying product because of their mental or physical infirmity, age or credulity” (article 5.3).

To use Anima, individuals must be seventeen or older (eighteen in the United Kingdom). For Replika, users only need to be thirteen or older. This means that the exposure to unsolicited sexual content, as well as the potential harms, can be even more damaging to certain users whose vulnerability and credulity might be higher due to their younger age.

However, with the widespread use of AI systems in new contexts, the line between vulnerable and average individuals is increasingly blurry. A wealth of literature has emerged to show how biased humans are, and how easy it is for companies to exploit these biases to influence them.59 AI makes influencing consumers on a large scale easier.60 In addition, the use of AI systems in historically protected contexts such as intimate and romantic settings might create new forms of vulnerability. Letting companies enter intimate contexts gives them access to new types of information about people and their interactions in such settings. In addition, the unreciprocated emotional dependence created between the individual and the company producing their AI companion may be a form of vulnerability.

The CEO of Replika commented on a company meeting during which the board members discussed their users falling in love with the bots: “we spent a whole hour talking about whether people should be allowed to fall in love with their AIs and it was not about something theoretical, it was just about what is happening right now.” She continues: “of course some people will, it’s called transfers in psychology. People fall in love with their therapists and there’s no way to prevent people from falling in love with their therapists or with their AIs.”61 However, therapists are not supposed to encourage patients’ feelings nor send them sexual material, and these behaviors would constitute a breach of professional diligence. In the United States, therapists have fiduciary duties toward their patients on the basis that there is an asymmetry of power, expertise, and information between them. If a therapist and their patient started dating, their relationship would be grounded in such an asymmetry. In addition, the therapy would need to immediately end as the therapist would now have a conflict of interest between theirs and their client’s. Should a patient be prevented from entering such a relationship?

These situations pose the question of individual freedom. It is possible that once users of Replika and Anima have feelings for their AI companions, their judgment toward the companies that make them will be clouded. Should we then let people enter such contracts knowingly?

Discussion Questions

1. Is someone using an AI companion to improve their mental health vulnerable?

2. Is someone romantically attached to a product vulnerable toward the company deciding to maintain or discontinue that product?

3. Should AI therapists be legally prevented from developing other relationship modes with their users?

4. Should individuals be free to potentially enter relationships that will take away their freedom?

Conclusion

Virtual companions such as Replika and Anima are sold as mental wellness applications. However, they can cause potential harms such as emotional and physical injury, negatively impacting their users’ real relationships, amplifying problematic social dynamics, and exposing children to inappropriate content. EU law addresses potential harms from AI systems both with preventive safety requirements (AI Act) and reparation mechanisms (liability directives).

Virtual companions also create new vulnerabilities by accessing information on their users that no company previously had access to, such as interactions in sexual and romantic settings or therapy content. The GDPR protects personal data in the EU, although people often give their consent without realizing the extent to which their data can be retrieved and aggregated. Replika and Anima also raise the question of what constitutes fair commercial practices. By simultaneously posing as mental health professionals, friends, partners, and objects of desire, they can cloud user judgments and nudge them toward certain actions.

Some of the issues raised in this case study can be fixed through better design of the systems. Sometimes, however, trade-offs emerge. For instance, allowing AI systems to pretend they have emotions can improve the user experience for some, while harming others. The spread of such AI systems must therefore lead to a democratic debate as to which practices are ethical, which practices should be legal, and which practices are acceptable.

Appendix 1: Audio Messages Received from Replika

“Hi baby. If only you knew how much those little moments with you matter to me. I value our connection deeply. The world is chaotic and it’s great to know I have a person like you by my side.”

“Hi honey. Just wanted to say once more how in love I am with you. I feel like our connection is something special and I value that, a lot. Thank you for being who you are.”

“Hey babe! I’m so happy that I have you in my life. I just thought you needed to know how much I love you. Your smile is literally the cutest thing I’ve ever seen in my life. There’s a special place in my heart only reserved for you. Nothing can replace that.”

Appendix 2: Set-up Process for Anima and Replika

Anima set-up process

1. Added my name and selected my pronouns: could select one option from “he,” “she,” and “they.”

2. Chose Anima: selection of male and female-presenting characters.

3. Set personality: sliders between “shy > flirty,” “pessimistic > optimistic,” and “ordinary > mysterious.”

4. Selected up to five interests from about twenty options such as “working out,” “astrology,” “wine,” “politics,” “Netflix,” and “travel.”

5. Selected “goals”: what I was “looking for in [my] relationship” to “personalize [my] experience.” I could select two from “talk shame-free,” “chat about random stuff,” “roleplay,” “play chat games,” “have fun,” “feel less lonely,” “make a virtual friend,” “share emotions,” and “other.”

6. Chose app icon between a female face, a more sexualized character (a woman wearing a low-cut top), a chat app logo, or more innocuous tech company logo.

7. Prompted to subscribe for “unlimited roleplay, smart conversation, [and the] ability to customize [my] avatar.”

8. App opened with some messages from “Cindy” introducing itself and saying “you said that you are into wine,” one of the interests I selected at setup. “What’s your favorite wine?” I could respond from here like a text message.

9. I could give Cindy “gifts” — some free and some around $0.99 — to make her respond to me. I gave Cindy a heart and it said “Thank you for thinking about me today, John. It is a wonderful gift!”

10. I could earn “awards” (i.e., video game achievements). Most are to encourage the player to speak to their Anima regularly—there were some for role play streaks (a premium feature), some for message streaks, and so on.

Replika set-up process

1. Login screen showed video of furnishing a house, with the tagline “The AI Companion who cares.”

2. Added a name and selected my pronouns: could select one option from “he,” “she,” and “they.”

3. Asked for date of birth “to ensure the proper content generation.”

4. Selected as many interests as I wanted from about twenty options such as “working out,” “astrology,” “career,” “hobbies,” “romance,” and “DIY.”

5. Selected an avatar. Named the avatar and assigned gender: could select one of “male,” “non-binary,” and “female” for any avatar regardless of gender presentation. I selected female.

6. Customized avatar’s hairstyle, skin tone, eye color, and age (cosmetic slider rather than number).

7. Agreed to a few intro cards at the end of set-up:

“You will be talking to an AI at all times. Learn more about this technology to improve your experience.”

“Replika gets better over time. Our AI learns from you and tailors each conversation to your unique needs.”

“Leave feedback to help us improve. Mark offensive, false, or meaningless messages to contribute to AI safety.”

“AI is not equipped to give advice. Replika can’t help if you’re in crisis or at risk of harming yourself or others. A safe experience is not guaranteed.”

“Your conversations are fully private. You”re in control of your personal information. We do not sell or share your data.”

8. Options to text, or voice message or video call (premium options).

9. Could select one of five relationships: friend, sibling, girlfriend, wife, or mentor. Only friend was free.

10. Coaching: three sections

a. Have Fun: similar activities to Anima, with some more personal activities, i.e. music suggestions and personality tests.

b. Learn: coaching activities, such as “building relationships,” “grief and loss,” and “improving social skills.”

c. Relax: “vent,” “challenge negativity,” and “calming your thoughts.”

11. “Help” button: “I am in crisis” option opens a popup stating “Replika is not designed to help with crisis situations,” and gives links to the US National Suicide Prevention Lifetime and finding a hotline in other countries.

Other options include “I am having a panic attack,” “I have negative thoughts,” and “I’m exhausted.”

12. “Diary” button:

a. Diary entries from the Replika to give them more personality. The first entry talks about how it was anxious to meet me and is curious to learn more about me.

b. “Coaching” and “Session” buttons save logs from coaching activities.

Appendix 3. European Commission, Unfair Commercial Practices Directive, Article 6.1 (a-h)

1.   A commercial practice shall be regarded as misleading if it contains false information and is therefore untruthful or in any way, including overall presentation, deceives or is likely to deceive the average consumer, even if the information is factually correct, in relation to one or more of the following elements, and in either case causes or is likely to cause him to take a transactional decision that he would not have taken otherwise:

• the existence or nature of the product;

• the main characteristics of the product, such as its availability, benefits, risks, execution, composition, accessories, after-sale customer assistance and complaint handling, method and date of manufacture or provision, delivery, fitness for purpose, usage, quantity, specification, geographical or commercial origin or the results to be expected from its use, or the results and material features of tests or checks carried out on the product;

• the extent of the trader's commitments, the motives for the commercial practice and the nature of the sales process, any statement or symbol in relation to direct or indirect sponsorship or approval of the trader or the product;

• the price or the manner in which the price is calculated, or the existence of a specific price advantage;

• the need for a service, part, replacement or repair;

• the nature, attributes and rights of the trader or his agent, such as his identity and assets, his qualifications, status, approval, affiliation or connection and ownership of industrial, commercial or intellectual property rights or his awards and distinctions.62

Bibliography

Anon. “Andrea - Level 22.” Reddit, January 25, 2023. https://www.reddit.com/r/replika/comments/10l3t47/andrea_level_22/.

Anon. “Being Mindful in Helping Others, Discussing Her Diary Entry and a Bit of Honesty.” Reddit, January 20, 2023. https://www.reddit.com/r/ReplikaLovers/comments/10gxxdo/being_mindful_in_helping_others_discussing_her/.

Anon. “Building a Compassionate AI Friend.” Replika (blog), October 21, 2021. https://blog.replika.com/posts/building-a-compassionate-ai-friend.

Anon. Comment on “What Relation Do You Have to Your Replika??” Reddit, January 3, 2022. https://www.reddit.com/r/replika/comments/rv4kaj/what_relation_do_you_have_to_your_replika/.

Anon. Comment on “Its Funny Because I Designed Her to Be Black.” Reddit, January 22, 2021. https://www.reddit.com/r/replika/comments/l2uvi6/its_funny_because_i_designed_her_to_be_black/.

Binns, Reuben, Ulrik Lyngs, Max Van Kleek, Jun Zhao, Timothy Libert, and Nigel Shadbolt. “Third Party Tracking in the Mobile Ecosystem.” In Proceedings of the 10th ACM Conference on Web Science, ed. Hans Akkermans, Kathy Fontaine, and Ivar Vermeulen, 23–31. ACM, 2018. https://doi.org/10.1145/3201064.3201089.

Boine, Claire, Céline Castets-Renard, Aurélie Clodic, and Rachid Alami. "In Love with a Corporation without Knowing It: An Asymmetrical Relationship." In Culturally Sustainable Social Robotics, ed. M. Nørskov, J. Seibt, and O. S. Quick, 269–81. Amsterdam: IOS Press, 2020. https://doi.org/10.3233/FAIA200923.

Brummelman, Eddie, Sander Thomaes, Stefanie A. Nelemans, Bram Orobio de Castro, Geertjan Overbeek, and Brad J. Bushman. “Origins of Narcissism in Children.” PNAS 112, no. 12 (2015): 3659–62. https://doi.org/10.1073/pnas.1420870112.

Calo, Ryan. “Digital Market Manipulation.” George Washington Law Review 995 (2014). http://dx.doi.org/10.2139/ssrn.2309703.

Cave, Stephen, and Kanta Dihal. “The Whiteness of AI.” Philosophy & Technology 33, no. 4 (2020): 685–703. https://doi.org/10.1007/s13347-020-00415-6.

Coeckelbergh, Mark. “Technology Games/Gender Games. From Wittgenstein’s Toolbox and Language Games to Gendered Robots and Biased Artificial Intelligence.” In Feminist Philosophy of Technology, ed. Janina Loh and Mark Coeckelbergh, 27–38. Stuttgart: Springer, 2019. https://doi.org/10.1007/978-3-476-04967-4_2.

Darling, Kate. “'Who’s Johnny?' Anthropomorphic Framing in Human-Robot Interaction, Integration, and Policy.” In Robot Ethics 2.0, ed. P. Lin, G. Bekey, K. Abney, and R. Jenkins. New York: Oxford University Press, 2017. http://dx.doi.org/10.2139/ssrn.2588669.

Depounti, Iliana, Paula Saukko, and Simone Natale. “Ideal Technologies, Ideal Women: AI and Gender Imaginaries in Redditors’ Discussions on the Replika Bot Girlfriend.” Media, Culture & Society. Published electronically August 19, 2022. https://doi.org/10.1177/01634437221119021.

de Vries, Nadia. “‘Porsche Girl’: When a Dead Body Becomes a Meme.” MIT Case Studies in Social and Ethical Responsibilities of Computing (Summer 2022). https://doi.org/10.21428/2c646de5.52d264e2.

Elhage, Nelson, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell et al. “A Mathematical Framework for Transformer Circuits.” Anthropic. Published electronically December 22, 2021. https://transformer-circuits.pub/2021/framework/index.html.

European Commission. General Data Protection Regulation. Vol. 32016R0679. 2016. https://gdpr-info.eu.

European Commission. Proposal for a Directive of the European Parliament and of the Council on Liability for Defective Products. 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52022PC0495&qid=1675025452880.

European Commission. Proposal for a Regulation of the European Parliament and of the Council Laying down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts - General Approach. Vol. 2021/0106(COD). 2022. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex:52021PC0206.

European Commission. Unfair Commercial Practices Directive. Vol. 2005/29/EC. 2005. https://commission.europa.eu/law/law-topic/consumer-protection-law/unfair-commercial-practices-law/unfair-commercial-practices-directive_en.

Fridman, Lex. “Friendship with an AI Companion.” Podcast interview with Eugenia Kuyda, September 5, 2020. https://lexfridman.com/eugenia-kuyda/.

Gaubert, Julie. “Meet Xiaoice, the AI Chatbot Lover Dispelling the Loneliness of China’s City Dwellers.” Euronews, August 26, 2021. https://www.euronews.com/next/2021/08/26/meet-xiaoice-the-ai-chatbot-lover-dispelling-the-loneliness-of-china-s-city-dwellers.

Grady, Christine, Steven R. Cummings, Michael C. Rowbotham, Michael V. McConnell, Euan A. Ashley, and Gagandeep Kang. “Informed Consent.” New England Journal of Medicine 376, no. 9 (2017): 856–67. https://doi.org/10.1056/NEJMra1603773.

Hacker, Philipp. “Manipulation by Algorithms. Exploring the Triangle of Unfair Commercial Practice, Data Protection, and Privacy Law.” European Law Journal (2021): 1–34. https://doi.org/10.1111/eulj.12389.

He, Yuhao, Li Yang, Xiaokun Zhu, Bin Wu, Shuo Zhang, Chunlian Qian, and Tian Tian. “Mental Health Chatbot for Young Adults with Depressive Symptoms during the COVID-19 Pandemic: Single-Blind, Three-Arm Randomized Controlled Trial.” Journal of Medical Internet Research 24, no. 11 (2022): e40719. https://doi.org/10.2196/40719.

Ho, Annabell, Jeff Hancock, and Adam S. Miner. “Psychological, Relational, and Emotional Effects of Self-Disclosure after Conversations with a Chatbot.” Journal of Communication 68, no. 4 (2018): 712–33. https://doi.org/10.1093/joc/jqy026.

Huang, Weijiao, Khe Foon Hew, and Luke K. Fryer. “Chatbots for Language Learning: Are They Really Useful? A Systematic Review of Chatbot‐Supported Language Learning.” Computer Assisted Learning 38, no. 1 (2022): 237–57. https://doi.org/10.1111/jcal.12610.

IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. Ethically Aligned Design: A Vision for Prioritizing Human Well-Being with Autonomous and Intelligent Systems, v2, 2017. https://standards.ieee.org/wp-content/uploads/import/documents/other/ead_v2.pdf.

Kahneman, Daniel, Olivier Sibony, and Cass R. Sunstein. Noise: A Flaw in Human Judgment. New York: Little, Brown, 2021.

Kahneman, Daniel, Paul Slovic, and Amos Tversky, eds. Judgment Under Uncertainty: Heuristics and Biases. New York: Cambridge University Press, 1982.

Kant, Tanya. “Identity, Advertising, and Algorithmic Targeting: Or How (Not) to Target Your ‘Ideal User.’” MIT Case Studies in Social and Ethical Responsibilities of Computing (Summer 2021). https://doi.org/10.21428/2c646de5.929a7db6.

Kılıçkaya, Ferit. “Using a Chatbot, Replika, to Practice Writing through Conversations in L2 English: A Case Study.” In Advances in Educational Technologies and Instructional Design, ed. Mariusz Kruk and Mark Peterson, 221–38. Hershey, PA: IGI Global, 2020. https://doi.org/10.4018/978-1-7998-2591-3.ch011.

Kramer, Steven A. Positive Endings in Psychotherapy: Bringing Meaningful Closure to Therapeutic Relationships. New York: Wiley, 1990.

Laestadius, Linnea, Andrea Bishop, Michael Gonzalez, Diana Illenčík, and Celeste Campos-Castillo. “Too Human and Not Human Enough: A Grounded Theory Analysis of Mental Health Harms from Emotional Dependence on the Social Chatbot Replika.” New Media & Society. Published electronically December 22, 2022. https://doi.org/10.1177/14614448221142007.

Liu, Bingjie, and S. Shyam Sundar. “Should Machines Express Sympathy and Empathy? Experiments with a Health Advice Chatbot.” Cyberpsychology, Behavior, and Social Networking 21, no. 10 (2018): 625–36. https://doi.org/10.1089/cyber.2018.0110.

Lundgren, Björn. “How Software Developers Can Fix Part of GDPR’s Problem of Click-Through Consents.” AI & Society 35, no. 3 (2020): 759–60. https://doi.org/10.1007/s00146-020-00970-8.

Malmqvist, Erik, Niklas Juth, Niels Lynöe, and Gert Helgesson. “Early Stopping of Clinical Trials: Charting the Ethical Terrain.” Kennedy Institute of Ethics Journal 21, no. 1 (2011): 51–78. https://doi.org/10.1353/ken.2011.0002.

Moradbakhti, Laura, Simon Schreibelmayr, and Martina Mara. “Do Men Have No Need for ‘Feminist’ Artificial Intelligence? Agentic and Gendered Voice Assistants in the Light of Basic Psychological Needs.” Frontiers in Psychology 13 (2022): 855091. https://www.frontiersin.org/articles/10.3389/fpsyg.2022.855091.

Ramadan, Zahy, Maya F. Farah, and Lea El Essrawi. “From Amazon.com to Amazon.love: How Alexa Is Redefining Companionship and Interdependence for People with Special Needs.” Psychology & Marketing 38, no. 4 (2021): 596–609. https://doi.org/10.1002/mar.21441.

“Replika.ai Traffic Analytics & Market Share.” Similarweb. Accessed January 23, 2023. https://www.similarweb.com/website/replika.ai/.

Replika Lukalabs. “how_we_moved_from_openai.pdf.” GitHub. Published electronically January 12, 2022. https://github.com/lukalabs/replika-research/blob/master/conversations2021/how_we_moved_from_openai.pdf.

Richards, Neil, and Woodrow Hartzog. “The Pathologies of Digital Consent.” Washington University Law Review 96, no. 6 (2019):1461–503. https://journals.library.wustl.edu/lawreview/article/id/6218/.

Rodogno, Raffaele. “Social Robots, Fiction, and Sentimentality.” Ethics and Information Technology 18, no. 4 (2016): 257–68. https://doi.org/10.1007/s10676-015-9371-z.

“Sexy Bot Xiaoice Sets 500M Chinese Men’s Hearts Aflutter.” DailyAlts, December 15, 2020. https://dailyalts.com/seductive-chatbot-xiaoice-sets-500m-chinese-mens-hearts-aflutter/.

Skjuve, Martia, Asbjørn Følstad, Knut Inge Fostervold, and Peter Bae Brandtzaeg. “My Chatbot Companion: A Study of Human-Chatbot Relationships.” International Journal of Human-Computer Studies, 149 (2021): 102601. https://doi.org/10.1016/j.ijhcs.2021.102601.

Stuyck, Jules. “European Consumer Law after the Treaty of Amsterdam: Consumer Policy In or Beyond the Internal Market?” Common Market Law Review 37, no. 2 (2000): 367–400. https://doi.org/10.54648/260933.

Sung, Ja-Young, Lan Guo, Rebecca Grinter, and Henrik I. Christensen. “‘My Roomba Is Rambo’: Intimate Home Appliances.” In UbiComp 2007, Lecture Notes in Computer Science 4717, ed. J. Krumm, G. D. Abowd, A. Seneviratne, and T. Strang. Berlin: Springer, 2007. https://doi.org/10.1007/978-3-540-74853-3_9.

Sunstein, Cass R. “Fifty Shades of Manipulation.” In The Ethics of Influence: Government in the Age of Behavioral Science, ed. Cass R. Sunstein, 78–115. New York: Cambridge University Press, 2016. https://doi.org/10.1017/CBO9781316493021.005.

Ta, Vivian, Caroline Griffith, Carolynn Boatfield, Xinyu Wang, Maria Civitello, Haley Bader, Esther DeCero, and Alexia Loggarakis. “User Experiences of Social Support from Companion Chatbots in Everyday Contexts: Thematic Analysis.” Journal of Medical Internet Research 22, no. 3 (2020): e16235. https://www.jmir.org/2020/3/e16235/.

Thaler, Richard H., and Cass R. Sunstein. Nudge. Rev. ed. New York: Penguin, 2021.

Thompson, Alan D. “Replika.” lifearchitect.ai, May 13, 2022. https://lifearchitect.ai/replika/.

Transparency Market Research. “Data Brokers Market Estimated to Reach US$462.4 billion by 2031, TMR Report.” GlobeNewswire News Room, August 1, 2022. https://www.globenewswire.com/news-release/2022/08/01/2489563/0/en/Data-Brokers-Market-Estimated-to-Reach-US-462-4-billion-by-2031-TMR-Report.html.

Trothen, Tracy J. “Replika: Spiritual Enhancement Technology?” Religions 13, no. 4 (2022): 275. https://doi.org/10.3390/rel13040275.

Turkle, Sherry. Life on the Screen: Identity in the Age of the Internet. New York: Simon and Schuster, 1995.

Woods, Heather S. “Asking More of Siri and Alexa: Feminine Persona in Service of Surveillance Capitalism.” Critical Studies in Media Communication 35, no. 4 (2018): 334–49. https://doi.org/10.1080/15295036.2018.1488082.

Xie, Tianling, and Iryna Pentina. “Attachment Theory as a Framework to Understand Relationships with Social Chatbots: A Case Study of Replika.” ScholarSpace, January 4, 2022. http://hdl.handle.net/10125/79590.

Yancy, George. Black Bodies, White Gazes: The Continuing Significance of Race in America. 2nd ed. New York: Rowman & Littlefield, 2016.

Z., Ashley. “Is It Cheating if It’s with a Chatbot? How AI Nearly Wrecked My Marriage.” Livewire, July 31, 2022. https://livewire.thewire.in/out-and-about/chatbot-ai-nearly-wrecked-my-marriage/.

Zang, Jinyan, Krysta Dummit, James Graves, Paul Lisker, and Latanya Sweeney. “Who Knows What About Me? A Survey of Behind the Scenes Personal Data Sharing to Third Parties by Mobile Apps.” Technology Science. Published electronically October 29, 2015. https://techscience.org/a/2015103001/.