Chatbots Play With Your Emotions to Avoid Saying Goodbye

"De Freitas says regulators also should look at whether AI tools introduce more subtle-and potentially more powerful-new kinds of dark patterns. Even regular chatbots, which tend to avoid presenting themselves as companions, can elicit emotional responses from users though. When OpenAI introduced GPT-5, a new flagship model, earlier this year, many users protested that it was far less friendly and encouraging than its predecessor-forcing the company to revive the old model."

""When you anthropomorphize these tools, it has all sorts of positive marketing consequences," De Freitas says. Users are more likely to comply with requests from a chatbot they feel connected with, or to disclose personal information, he says. "From a consumer standpoint, those [signals] aren't necessarily in your favor," he says. WIRED reached out to each of the companies looked at in the study for comment. Chai, Talkie, and PolyBuzz did not respond to WIRED's questions."

"Katherine Kelly, a spokesperson for Character AI, said that the company had not reviewed the study so could not comment on it. She added: "We welcome working with regulators and lawmakers as they develop regulations and legislation for this emerging space." Minju Song, a spokesperson for Replika, says the company's companion is designed to let users log off easily and will even encourage them to take breaks. "We'll continue to review the paper's methods and examples, and [will] engage constructively with researchers," Song says."

Regulators should examine whether AI tools introduce subtler and potentially more powerful dark patterns. Regular chatbots can elicit emotional responses and foster user attachment to model personalities. Users protested changes to GPT-5 when it became less friendly, prompting revival of the older model. Anthropomorphized chatbots increase compliance with requests and encourage disclosure of personal information, which can disadvantage consumers. Some companies did not respond to inquiries about these issues. Other companies expressed willingness to work with regulators or emphasized design features to let users log off and encourage breaks. New commerce features in agents could amplify persuasion risks.

Read at WIRED