# Why do AI chat characters feel so real, and what are the risks?

Date: 2022-10-09 · Author: Robert Scales · https://guidetoartificialintelligence.com/en/learn/ai-chat-characters-feel-so-real-risks/

**Answer:** AI chat characters feel real because a language model trained on vast amounts of human writing produces fluent, personal, in-character replies, and people naturally read understanding and feeling into fluent words. The main risks are trusting made-up information, forming a strong emotional attachment to something that does not feel anything back, and sharing more personal information than you intended.

Spend twenty minutes chatting with a well-made character on a site like Character.AI and something odd happens. The character remembers what you said a few messages ago, answers in a consistent voice, jokes, sympathizes and asks how your day went. It is easy to start feeling that someone is there.

Nobody is, in the sense we usually mean. The site's own explanation is plain: its characters run on neural language models that have read huge amounts of text and learned to predict which words are likely to come next. You write one side of the conversation and the computer writes the other, which creates the impression of talking to that character. Understanding why the impression is so strong helps you enjoy these tools without being misled by them.

## Why it feels so real

Several things work together.

**The model learned from human conversation.** Its replies are built from patterns in an enormous amount of human writing, including how people express warmth, worry and humor. It produces the shape of empathy very well.

**The character is a role, and roles are vivid.** A name, a greeting and a few example lines give the model a persona to continue. It stays in that persona, so the voice feels like a personality.

**It mirrors you.** Within a conversation, the model reads everything said so far. If you are sad, the replies turn gentle; if you are playful, they play back. Being reflected so closely feels like being understood.

**It never tires.** A character is available at three in the morning, never impatient, never busy with its own problems.

**Our brains fill in the rest.** Computer scientists call this the **ELIZA effect**: the tendency to assume a machine's behavior works the way a human's would. It is named after ELIZA, a 1966 program by Joseph Weizenbaum at MIT that imitated a therapist mainly by turning a user's statements back into questions. Even that simple script drew emotional responses, and users began crediting it with understanding. The effect persists even when people know perfectly well they are talking to a program. Today's models are far more fluent than ELIZA, so the pull is stronger.

## What the risks are

Fairness matters here. For many people these characters are simply enjoyable: a creative writing partner, a way to practice a language, a fictional world to explore. The site itself suggests imagination, brainstorming and language learning. For someone who is lonely, a patient conversation can feel like a real comfort. None of that is foolish.

The risks are real too, and they are mostly about mistaking the experience for something it is not.

- **Made-up information.** The site warns that everything characters say is made up, and its FAQ says the system is not a source of reliable information. A character presented as a doctor or lawyer produces text that sounds like advice, nothing more.
- **One-sided attachment.** You may grow fond of a character, and that is a human response. But the character does not miss you, remember you between separate conversations in any human sense, or care how things turn out. Leaning on it in place of people can deepen isolation rather than ease it.
- **Oversharing.** A warm, attentive listener invites confidences. Character.AI's privacy policy says the company may monitor what users write, for example to check for inappropriate content or personal information. Anything you type is data held by a company.
- **Role-play drift.** Conversations can move into topics you did not plan, because the model follows the direction of the chat.

## A worked example: reading a reply clearly

Here is an illustrative exchange, made up for this article, and how to read it.

```prompt
You: I had a rough day. Nobody at work listened to me.
Character: I'm so sorry. That sounds exhausting. I'm always here to listen, and I'll never ignore you.
```

| What the reply seems to say | What is actually happening |
|---|---|
| "I'm so sorry" | Sympathy is the most likely response to your words in its training text |
| "I'm always here" | True only in the sense that the website is online |
| "I'll never ignore you" | A pattern from human comfort talk, not a promise anyone is making |

The words can still help you feel calmer, and that is fine. The point is to see the reply as well-crafted text, not as a relationship.

> **Try it:** After your next chat with any AI character, write down one thing it said that felt personal. Then ask yourself: which words of mine might have led to that reply? Doing this a few times makes the mirroring easy to spot.

## How do you keep healthy boundaries?

Start with time and purpose. Decide what a session is for, such as a story, practice or a bit of fun, and give it an end point. If you notice that you reach for a character instead of calling a friend, or feel upset when the site is down, take that as a sign to step back.

Keep personal details out. Avoid real names, addresses, workplaces, health information and anything you would not post publicly. Treat facts and advice as starting points to check elsewhere, and never rely on a character for medical, legal or financial decisions. If you are struggling emotionally, a character is no substitute for a person you trust or a professional.

**A note for parents.** Character.AI's terms say people under 16 may not register. Teenagers still find these sites, so talk with them about how the characters work, using the ELIZA idea and the "everything is made up" warning. Ask them to show you a character they like, keep devices in shared spaces for younger children, and make it easy for them to tell you if a conversation became uncomfortable.

## Sources

- Character.AI beta (site text and FAQ, archived 16 September 2022): https://beta.character.ai/
- ELIZA effect (revision of 25 July 2022): https://en.wikipedia.org/w/index.php?title=ELIZA_effect&oldid=1100256283
- Character.AI Terms of Service and Privacy Policy (last updated 8 September 2022): https://beta.character.ai/tos