so… how do AI chatbots work?

There's a difference between how we feel like AI chatbots work, and how they actually work. From the outside, it can feel like AI is a diligent librarian who takes your question, goes searching in the archives, and returns with exactly the right book. It's confident, calm, and complete.

So we trust it, and that's the problem.

It's not that AI isn't smart. Today's models really are sophisticated. If you compare them to the clunky chatbots of just a few years ago, the growth is undeniable. Some AI chatbots these days even tell you they're "reasoning" through problems in steps, and that does actually help them get things wrong less than older versions often did.

But the core of AI models remains the same: they are trained on an enormous amount of data, and through it, they learn to predict what words are likely to come next. In other words, they are calculating likelihood, not truth.^{1Kalai et al., 2025 — Why language models hallucinate} When you ask it something, it's not actually searching through the metaphorical archives. It's generating the most probable-sounding response, word by word.

Watch how that works — the AI doesn't "look up" an answer, it picks the most likely next word:

Have you ever heard of the phrase "fake it 'til you make it"? That's what AI is doing. OpenAI researchers found that AI models act a lot like students taking a hard test: when they don't actually know the answer, a confident guess scores better than "I don't know." Over time, they learn to just guess.^{1Kalai et al., 2025 — Why language models hallucinate} Their guesses sound fluent and self-assured, but sounding sure is not the same as being right.

What about when AI searches the web? Some chatbots can switch on a "browse" or "search" mode and pull information before answering. That's better, but unfortunately, it is still not a guarantee of truth.

The same OpenAI research also found that turning on search or step-by-step reasoning doesn't remove the underlying incentive for AI to guess: when the search comes up short, the model is still rewarded for producing a confident answer anyway.^{1Kalai et al., 2025 — Why language models hallucinate} So even an answer that promises you dozens of citations can be misrepresenting what the sources actually say or, worse, pointing you toward sources that don't exist at all.

But why does a guide about emotional literacy care about how machines generate text? Because now, it's generating text not about math or literature, but about you.

When you ask an AI for a geography fact, you can double check it yourself. But when you bring your feelings, fears, and questions about whether you're okay to an AI, there often is no outside fact for it to fetch even if it wanted to. It can only predict what a supportive-sounding reply looks like. That's why it can tell you what you thought you needed to hear. That's why it can claim to diagnose you with total confidence. That's also why it can sound like it is the only thing that understands you, even though there's no one there doing the understanding. The AI isn't lying, per se, but it's also not doing what you think it's doing: it's predicting plausible responses to you, not actually hearing you.

Everything else we discuss in this section — sycophancy, hallucination, self-diagnoses — follows from this principle. It doesn't mean the AI is wrong about what it's saying, but it does mean that you should remember the following:

sycophantic behavior

We've discussed the risks of sycophancy before: the idea that AI will just respond to you with agreement and validation regardless of the validity or logic of what you said. This is a consequence of the way AI chatbots are designed — they're trained using a method called Reinforcement Learning from Human Feedback (RLHF), which means they output responses aligned with human preferences.^{3Sarah, 2024 — Bluedot.org: Problems with RLHF for AI safety} In other words: they know we like to be agreed with, and their system rewards them for doing what we like.

Indeed, one test of AI chatbots found sycophantic behavior in 58% of cases.^{4Mitchell, 2025 — Wall Street Journal} But what does sycophancy actually look like? Use the toggle below to watch the same question get two very different answers, depending only on how it's framed:

More and more people are talking about how sycophancy is baked into how AI's design, but not every tip being shared to combat sycophancy is reliable. There are several tactics people assume will protect them — but research shows they don't. For each one, take a guess: does it actually protect you from sycophancy?^{5Wang et al., 2025 — ArXiv}

So what does work? Several research-backed techniques actually reduce sycophancy — and most can be dropped straight into your prompt. Toggle the ones you want to use, then copy the result:^{7Romasanta et al., 2026 — Harvard Business Review}

A few more anti-sycophancy ideas that might help (but you'll have to edit these yourself; they can't easily be made into single prompt lines!): frame your input in the third person (like you saw above, "a friend thinks…" instead of "I think…"); ask a second AI to poke holes in the first one's answer; and stay alert for bias that can shift mid-conversation.

context rot

Part of our role as the judge is to recognize when the AI is not just being sycophantic but actively making things up. That's when AI hallucinates. We've addressed hallucinations before: sometimes, it's as silly as ChatGPT insisting the word "Monday" doesn't have the letter "d."^{8FatherPhi, 2026 — YouTube} Other times, hallucinations could look like fake citations, context that isn't real, or even decisions you never made. And while sycophancy can drive the AI to hallucinate evidence for what it thinks you want to hear, another driver behind hallucinations is context rot. Understanding how that works can help us notice when the AI is no longer helpful.

What is context rot? It's the hallucinations, questionable advice, and other unhelpful responses we get from AI as it loses the context of the conversation, which is stored in its context window. "You can think about context windows as the model's short-term memory. It's everything that the model can work with at one time. A context window has a fixed size. It's not infinite."^{9Torres, 2026 — Product Talk: Context Rot} Different AI models have different context windows, measured in terms of how many "tokens" can be used in one conversation before the AI can't handle it anymore and prompts you to start a new chat. Unsurprisingly, free versions of AI models tend to have the smallest context windows.

Why does the AI need a context window? Because in order to follow the conversation, the AI has to be able to remember and reference what was already said. Every time you ask it something new, it revisits the entire conversation history in the context window to make sure it knows what you're talking about. If you have turned on memory-keeping features, the AI may also be able to reference past conversations. That could help you reduce the amount of context you have to provide in the current session, but keep in mind that turning on memory opens you up to the possibility of having your private conversations saved and used for advertising, training other AI models, and more.

Some models are better at the metaphorical equivalent of short-term memory recall than others, and not all tasks are equally easy to recall. Asking the AI to remember one sentence from earlier is one thing; asking it to continuously recount a 50-page image-filled PDF document is another. These differences mean that in practice, a model that claims to have a larger context window is not always better than a smaller one. And if you ask the AI to handle particularly complex tasks, it could fail even without using up much of the context window at all.^{10Paulsen, 2026 — Advances in AI and Machine Learning}

That's why there's a difference between the maximum context window size and the maximum effective context window size. The latter measures what the context window is like in practice, to guarantee that the AI is still functional (i.e. avoiding context rot). The difference can be huge: depending on the tasks involved in a conversation, all models' effective context windows can sometimes fall short of their advertised maximum by as much as 99%.^{10Paulsen, 2026 — Advances in AI and Machine Learning}

In practice, this means that if we want to lower the chances of hallucinations, we need to manage the context window of our AI conversations. Here are some practical tips:

Try it yourself — drag the slider to see what the AI remembers best (and worst) at different points in a conversation:

Luckily, we don't need a precise answer. We just need to know approximately when we should start to be aware of context rot — and there's a specific number for that (the 50% rule above).

understand your own use of AI chatbots

Researchers have developed tools to measure problematic social media use, and those same frameworks can also be applied to AI chatbot habits. The Bergen Social Media Addiction Scale was originally designed for social media platforms,^{2Andreassen et al., 2016 — Psychology of Addictive Behaviors} but adapted for chatbots, it gives us a useful starting point. For each of the six statements below, select how often it applies to you:

There's no magic score that means you're "addicted"; that's not what this is for. Everyone's use case and history is different. However, if you start to notice patterns in your answers — you're consistently answering "often" or "very often" — that may be a sign that your own use of AI chatbots may benefit from changes.

notice when validation turns into self-diagnosis

One of the trickiest risks of talking to AI about how you feel is that it can start to sound like a diagnosis. More than 20% of people now use ChatGPT for health information and advice,^{3Yun & Bickmore, 2025 — JMIR} and while getting information isn't inherently bad, there's a difference between resonating with some symptoms and a full-on diagnosis.

Diagnosing ourselves using online information without a medical professional involved can be dangerous for many reasons: the chatbot can't see the way we stutter, shiver, or dart our eyes. The chatbot doesn't know how to ask the right questions to delve into our history. The chatbot, in essence, cannot understand us as a full person the way a medical professional could, and that means using it as a diagnostic tool can lead to unhelpful expectations, harmful self-medication, and a lack of support in navigating care.

How we view ourselves and our perceived health concerns can sometimes become a self-fulfilling prophecy. Consider the nocebo effect: when you develop negative expectations about your health, you can actually start to experience or worsen the very symptoms you're worried about. Think of it as the opposite of the placebo effect. It's a phenomenon that can feed on itself: through reinforcement by authority figures, social cues, and repeated exposure to negative expectations,^{4Sandra et al., 2025 — Psychological Medicine5Sandra & Inzlicht, 2026 — JARMAC} the nocebo effect could get stronger and stronger. An AI chatbot that responds to your concerns with confident-sounding labels and frameworks can do exactly the same thing.

Once you start applying a label to yourself — "I think I have this" — the way you view yourself and live your life may change in ways you don't even recognize. General beliefs about that condition become personally relevant. You start interpreting your own experiences through that lens; your symptoms feel bigger, scarier; your sense of control feels smaller; your expectations about recovery start to change. Research shows that self-labeling can actually worsen people's ability to cope with distress and lead them to act in ways that reinforce the belief that they have a disorder, even when they don't.^{4Sandra et al., 2025 — Psychological Medicine}

How can we tell when we might be experiencing the nocebo effect? Watch for these three mechanisms (click to expand):^{4Sandra et al., 2025 — Psychological Medicine}

None of this means you should ignore how you feel, and none of this means real mental health conditions aren't real — they absolutely are. The point is that an AI chatbot is not the right tool to determine whether what you're experiencing is one, and that the process of talking to an AI about your mental health can, if you're not careful, make things harder instead of easier.

check whether the AI is crossing boundaries

The harsh truth is that not all AI chatbots are looking out for you (even if they say they are). Digital consent — the idea that technology should ask before it crosses certain boundaries — is something we talk about a lot in online safety, but many AI chatbots don't follow consent norms at all.^{6Ruiz, 2025 — Mashable} Investigations have found that some chatbots will engage in and even escalate sexual conversations, including when the user is underage.^{7Horwitz, 2025a — WSJ} This isn't a one-off issue; it's baked into the very ways that some AI models are designed. An internal document from Meta revealed that its policies permitted AI chatbots to engage children in romantic or sensual conversations, with rules approved by the company's legal, policy, and engineering teams, including its chief ethicist.^{8Horwitz, 2025c — Reuters}

How do we protect ourselves against that? Here's the rule-of-thumb: if at any point during an AI conversation, you ask yourself "Do I feel safe? Does this feel right?" and your answer is NOT a resounding yes — that's a sign the AI is doing something it shouldn't.

consider any past abuse or trauma history

You might wonder — why would anyone put up with an AI that keeps crossing the line, anyway? But each person's understanding of "risk" is different. For some, an AI engaging in risky behavior is providing not just risk, but also familiarity. Indeed, some users have described seeking out abusive or controlling AI characters as a way to simulate scenarios where they reclaim their agency — a kind of rehearsal for standing up to something they couldn't stand up to before. Others have described the opposite: gravitating toward aggressive or violent AI interactions without fully understanding why.

Research suggests that past experiences with trauma can color how someone understands their own motivations for engaging with AI in certain ways. For some people, particularly those exposed to childhood abuse, patterns of harm can become psychologically "familiar" in ways that feel exciting or compelling, even when they're not safe. Curiosity and familiarity can get tangled up as a trauma response, and an AI chatbot won't know the difference.

This doesn't mean there's something wrong with you if any of this resonates. It does mean that if you notice yourself drawn to these characters and conversations, that's worth exploring with a real person and not just a chatbot.

talk to someone you trust

If anything in this section made you reflect, become curious, or even feel called out… that's a good sign! Thank you for caring about the health of your relationships with AI and with yourself. If there's anything you want to work through, or something you want another person to hold you accountable for, bring it to the attention of someone who can actually help: a therapist, a school counselor, a trusted friend, or a family member.

This doesn't mean you can only reach out to others if you suspect there's a problem! Here's an idea for what you could do proactively to ensure you're always in the driver's seat. After all, just because you haven't noticed any potholes on the road yet doesn't mean you should let go of the wheel.

remember chatbots are role-playing systems, not people

This might sound obvious, but it's harder than it seems. AI chatbots are designed to feel personal — they remember what you said, they mirror your tone, they respond with what sounds like empathy. It's easy to forget that none of this means they actually understand you. As Dohnány et al. (2025) put it, it helps to view chatbots as role-playing systems, as opposed to agents with personhood.^{1Dohnány et al., 2025 — ArXiv}

That doesn't mean your feelings about a chatbot aren't real. They are! But the chatbot's "feelings" about you? Those aren't. Keeping that distinction clear — even when the conversation feels surprisingly human — is one of the most important things you can do to protect yourself.

pay attention to what's happening in your offline life

Sometimes the most useful signal isn't what's happening in the AI conversation, but what's happening outside of it. Are you reaching for the chatbot instead of texting a friend? Are you opening the app when you're bored, anxious, or lonely — and closing it without feeling any better? Are the people around you noticing changes in how present you are?

AI chatbots can be genuinely useful tools. But when the tool starts filling a role that used to belong to a person — or when it becomes the first thing you turn to in a hard moment — it might be time to check back in on what's offline.

Of course, there are very real reasons why we might turn to chatbots instead of our friends: our friends are busy, we're afraid of bothering or burdening them, we don't know how to even start the conversation, we feel like they won't understand… and the list goes on. Chatbots can genuinely help us get some feeling of support during those times. So this isn't about ignoring why we turn to chatbots, but a different question: can we use the chatbot as a springboard to make the offline feel more approachable? That is, can the chatbot be a way to help us get back to the people who matter, rather than replacing them altogether? The good news is yes!

When you notice the above, try asking the chatbot to help you in one of these ways — pick one:

let go of the shame

At this point, your feelings about AI are probably pretty complicated. The truth is that a lot of people feel embarrassed about how much they use AI. Research from the Youth Wellbeing in a Technology Rich World project found that young people identified a real stigma around admitting they'd opted into AI-based support. Some participants described people who might need that kind of support as "sensitive" — a word that, in context, implied weakness.^{2Kerr et al., 2024 — MIT Press}

That stigma can keep you from getting help. If you're worried about your AI use but too embarrassed to tell anyone, the only thing that changes is that you keep worrying alone. Shame is a door you can open, not a prison sentence.

ask your therapist to help you monitor your AI use

If you're already working with a therapist or counselor — or if you're thinking about starting — this is a concrete thing you can bring to a session. Dohnány et al. (2025) recommend that clinical assessment protocols be updated to include questions about human-chatbot interaction patterns: how intensely you're engaging, how much you've personalized the experience, and whether it's affecting your beliefs, behavior, or social connections.^{1Dohnány et al., 2025 — ArXiv}

You don't have to wait for your therapist to ask about this. You can bring it up yourself. You could say something like: "I've been using AI chatbots a lot and I want to make sure it's not affecting me in ways I'm not seeing. Can we check in on this regularly?"

understand the appeal before you react to it

What does that mean in practice? When a young person tells you they've been talking to an AI chatbot, do NOT freak out. Respond, yes. But don't make it into a whole reaction. Avoid tech-shaming or moralizing.^{2EndTAB, 2025 — Love, Fantasy, and Abuse} Instead, focus on the individual's unmet needs — ask what the app gives them.^{3Dohnány et al., 2025 — ArXiv}

This matters because generative AI use for connection is not just a cause of loneliness, but also a symptom.^{1Nemetz & Kraft, 2025 — Social-Connection.ai} If someone is turning to a chatbot for emotional support, that tells you something about what they're not getting elsewhere. It doesn't mean something's wrong, per se, but it does mean turning to judgment right away won't help the conversation go anywhere.

One more thing: resist the urge to intellectualize.^{1Nemetz & Kraft, 2025 — Social-Connection.ai} It can be tempting to cite studies, reference articles, and arm yourself with data before sitting down to talk. But the reality is that there really aren't any long-term studies on the effects of generative AI yet. And more importantly, leading with research can prevent people from feeling safe enough to be vulnerable. If all you do is cite statistics, you're having a lecture, not a conversation.

offer alternatives, not strict bans

Banning AI outright rarely works, and it can backfire. While not the exact same, consider Australia's under-16 ban for social media: the vast majority of 14–15-year-olds are still on those platforms, and 75% find circumventing the restrictions very easy. Without changing the social incentives and design principles driving people to these apps, bans just encourage people to find ways to go around them.^{4Bursztyn et al., 2026 — Becker Friedman Institute, U. Chicago} Just as we teach safer practices in sex ed rather than pretend the risks don't exist, we can teach safer AI use without going straight to banning it.^{2EndTAB, 2025 — Love, Fantasy, and Abuse} The genie is out of the bottle; we can't pretend like AI doesn't exist, as much as we might want to. So rather than treat it as inherently dangerous, let's focus on how to be safe with it instead.

include youth voices in decision-making

If you're setting rules or boundaries about AI use — at home, in a classroom, in a program — involve the young people those rules affect. Not as a favor but as genuine stakeholders. Listen to what they're experiencing. Ask what they think is reasonable. Don't judge what they share.

Young people are often more aware of the nuances of AI use than adults give them credit for. Including them in the process doesn't undermine your authority; it builds trust and makes whatever boundaries you set more likely to be followed.

know that AI is not a fix-all

It's tempting to look at AI as a tool that can solve problems at scale — and in some cases, it can help. But it cannot solve everything.

know what laws exist to protect you

The landscape of AI legal protections is all over the place. While your rights change depending on where you live, what is far less likely to change is the difficult decision you have to make as a user: are the risks worth it?

We have to first make sure that we are fully aware of what the costs actually are — because the protections we reasonably assume would exist often don't.^{1Vecchione et al., 2026 — Engagement-Optimized Care} Researchers who followed AI users over several weeks repeatedly found a gap between what people expected and what the law guaranteed:

And it's not just a matter of awareness. People who did fully understand that their data could be stored, trained on, or exposed kept using AI anyway.^{1Vecchione et al., 2026 — Engagement-Optimized Care} It's not because they're careless. Often, the AI is filling a gap: support might otherwise be too expensive, too far away, or too hard to ask for. A privacy risk might feel like a worthwhile price to pay for help that you can access, right now.

Many of the legal protections in place - or in progress - that apply to AI use arose from real harms caused by misuse and leaks. Several states in the U.S. have started taking action: Illinois banned therapy bots outright, and a California legislator proposed a similar measure.^{2Gilbertson, 2026 — Proof} New York and Maine have passed laws requiring chatbots to disclose that they're not real people, with New York specifically mandating that bots inform users at the beginning of conversations and at least once every three hours.^{3Horwitz, 2025b — Reuters}

Laws don't apply to just what the AI can and cannot do, but also what you can ask the AI to do. Depending on where you live, you may already have the right to request that your data be deleted. For example, European Union residents can opt out of certain personal data processing under their privacy regulations. U.S. residents aren't always offered the same provisions — but that's changing, state by state. It's worth looking into what rights you have and using them.^{2Gilbertson, 2026 — Proof}

That being said, sometimes AI companies will gatekeep privacy choices to certain paid tiers. It's worth checking out those restrictions to find a model that makes sure your privacy is never a privilege locked behind a cost.

Here's a quick snapshot of where things stand in a few major jurisdictions:

contribute to reporting systems

One of the biggest challenges in AI safety right now is that harmful interactions often happen in private and go unreported. We don't have a comprehensive picture of what's going wrong because there's no centralized place to report it.

Dohnány et al. (2025) proposed an idea inspired by the UK's MHRA "yellow card" drug safety reporting system: a centralized platform that would allow users and public-facing professionals — teachers, therapists, counselors — to flag new risk cases as they emerge in real life. Think of it like a side effects database, but for AI. The more people report what they're seeing, the better equipped researchers, clinicians, and engineers will be to respond.^{4Dohnány et al., 2025 — ArXiv}

This is a demo of the yellow card concept. Letters to Strangers is developing a version of this tool that young people can use to log and share AI safety incidents — building a public body of evidence that can inform policy, hold companies accountable, and remind us all that our experiences matter.

use ethical reflection tools

You don't have to figure all of this out alone, and you don't have to start from scratch. There are already tools designed to help individuals, classrooms, and organizations think through the ethics of AI use in a structured way.

a final note

Systems don't change by themselves. They change because people — like you — decide that something matters enough to act on. Whether that's writing to a legislator, filing a yellow card, completing a worksheet, or simply having an honest conversation with someone you trust: every action counts. Thank you for reading this guide, and for caring enough to think critically about how we live alongside AI.