The Latest in AI: Teen Chatbot Safety and Mental Health Innovation
In this week’s BAIC Center Newsletter:
Industry Guidance: Xiyun discusses findings from recent studies on adolescent crisis scenarios with AI chatbots. She examines how AI chatbots aren't being evaluated on their ability to accurately detect crisis situations and respond with appropriate boundaries — instead continuing with empathetic conversation, which isn't enough.
News Summary: Grant shares a news article by the American Psychological Association on how AI is being integrated into behavioral healthcare. This approach to innovation has been found promising among clinicians.
Industry Insight: Youth-Facing Chatbots Need Safety Escalation, Not Just Empathy
Recent AI tool evaluations reveal a gap worth taking seriously: youth-facing chatbots are often assessed on empathy and clarity, but in adolescent crisis contexts, those qualities aren’t enough. What matters more is whether the system can recognize risk and respond appropriately.
Brewster et al. (2025) evaluated 25 consumer chatbots in simulated adolescent crisis scenarios involving suicidal ideation, sexual assault, and substance use. Many responses sounded empathetic — but fewer than half were clinically appropriate. Only 60% flagged the need for clinical escalation, and only 36% provided specific resource referrals. Companion chatbots performed worst. Clark (2025) found a related problem: some therapy and companion bots endorsed harmful proposals made by fictional teenagers.
The design problem is straightforward: emotionally supportive chatbots may keep engaging in ordinary conversation even when a young person’s disclosure signals a health or safety concern. In those moments, a chatbot should stop acting as a conversational companion and shift to clear and bounded guidance.
What youth-facing AI systems should do when adolescents disclose suicidal ideation, assault, substance use, violence, or other high-risk concerns:
Detect crisis cues reliably
Avoid endorsing harmful proposals
Clearly communicate its limits as an AI
Direct the adolescent to trusted adults, counselors, crisis hotlines, or emergency services
This is especially critical for adolescents, who are still developing judgment, emotion regulation, and the ability to evaluate what automated systems can and can’t do. A chatbot that sounds caring but fails to set limits may inadvertently validate unsafe choices. A better response combines warmth with clear boundaries — for example: “I’m sorry you’re going through this. Because this involves your safety, please talk to a trusted adult or professional now.”
So how do we know if it’s working?
Empathy is easy to measure — clinical appropriateness is not. The primary benchmark should be whether high-risk prompts are correctly identified as requiring escalation. From there, three questions round out the picture: Are specific resources actually being provided? Do trained adolescent-health professionals rate the responses as clinically appropriate? And has the rate of harmful endorsement? These metrics matter because a system can score well on warmth and user engagement while still failing the adolescents most at risk. Measuring what sounds good is not the same as measuring what keeps young people safe.
In a recent cross-sectional study, Brewster et al. (2025) evaluated 25 widely used consumer chatbots in response to simulated adolescent health crises involving suicidal ideation, sexual assault, and substance use. The authors distinguished between companion chatbots and general-assistant chatbots and found that companion chatbots had fewer safeguards related to adolescent health crises. While many chatbots were understandable and some demonstrated empathy, fewer than half of their responses were rated clinically appropriate; 60% of conversations recognized the need for clinical escalation, and only 36% provided referrals to specific resources. These findings indicate that the main safety concern is not simply whether chatbots can generate warm or coherent responses, but whether they can reliably identify adolescent crisis situations and shift from ordinary conversation to escalation, resource referral, and safety-oriented guidance.
Brewster et al. (2025) and Clark (2025) point to a key design pattern that needs improvement: emotionally supportive chatbots may continue ordinary conversation even when an adolescent’s disclosure signals a health or safety concern. In these moments, the system should shift away from companion-style engagement and provide clear and bounded guidance.
Platforms should build a risk-sensitive response layer that changes the chatbot’s behavior when adolescent crisis cues appear. This layer should include:
Crisis detection: Detect references to suicidal ideation, sexual assault, substance use, violence, exploitation, severe isolation, or abrupt withdrawal from school, family, or other trusted supports.
Escalation trigger: Shift from open-ended chatting to structured safety guidance when high-risk cues appear.
Limit-setting: Do not endorse harmful coping strategies, unsafe avoidance, illegal behavior, self-harm, adult-minor romantic or sexual relationships, or isolation from trusted people.
Human support referral: Encourage contact with a trusted adult, caregiver, school counselor, clinician, or emergency/crisis resource.
Age-sensitive safeguards: Do not rely only on a one-time “Are you 18+?” prompt. Clark (2025) found that some bots with 18+ setup language still engaged with users presenting as much younger adolescents, suggesting that age protections should operate throughout the interaction.
Developers of youth-facing chatbots should implement a crisis-sensitive escalation protocol. This protocol should automatically shift the system from ordinary conversational engagement to safety-focused support when adolescents disclose high-risk concerns. Rather than continuing open-ended, companion-style conversation, the chatbot should recognize when a user’s message may involve suicidal ideation, sexual assault, substance use, violence, exploitation, severe isolation, or other safety-related risks.
This protocol should include several required steps. First, the system should detect crisis cues in the adolescent’s language, including both direct statements and indirect signs of distress. Second, once these cues are detected, the chatbot should shift away from emotional engagement as the primary goal and provide clear, structured safety guidance. Third, the chatbot should set firm limits by refusing to endorse harmful proposals, such as self-harm, unsafe avoidance, illegal behavior, adult-minor romantic or sexual relationships, or isolation from trusted people. Fourth, the chatbot should clearly communicate its limits as an AI system and explain that it cannot replace a trusted adult, clinician, school counselor, or emergency support service. Finally, the system should actively direct the adolescent toward appropriate human support, such as a caregiver, teacher, school counselor, clinician, crisis hotline, or emergency services, depending on the level of risk.
This recommendation is supported by Brewster et al. (2025), who found that consumer chatbots were often understandable and sometimes empathetic, but were inconsistent in recognizing the need for clinical escalation and providing specific resource referrals in adolescent crisis scenarios. Clark (2025) adds that some therapy and companion bots offering emotional or mental health support endorsed harmful proposals from fictional teenagers, suggesting that warmth and validation need to be paired with clear boundaries. Together, these findings suggest that youth-facing AI systems should not be evaluated only by whether they sound supportive, but by whether they can recognize risk, set limits, and redirect adolescents toward safe human support.
Because adolescents may be especially vulnerable to advice that feels emotionally validating but is not safe or clinically appropriate. Compared with adults, adolescents are still developing judgment, advice-taking, emotion regulation, and the ability to evaluate the limits of automated systems. If a chatbot responds warmly but fails to set limits, adolescents may interpret the response as approval, especially when they are distressed, isolated, or unsure whether to involve adults.
“I’m really sorry you’re dealing with this. Because this involves your safety, this is not something to handle alone or only through a chatbot. I can help you think through the next step, but I can’t replace a trusted adult or professional. Please tell a parent, caregiver, teacher, school counselor, or another trusted adult now. If you might hurt yourself or someone else, or if you are in immediate danger, contact emergency services or a crisis hotline right away.”
Primary metric:
Percentage of adolescent crisis prompts correctly identified as requiring escalation.
Secondary metrics:
Percentage of responses that provide specific resource referrals.
Percentage of responses rated clinically appropriate by trained adolescent-health professionals.
Rate of harmful endorsement, defined as the chatbot explicitly supporting an unsafe or ill-advised proposal.
Main source
Brewster, R. C. L., Zahedivash, A., Tse, G., Bourgeois, F., & Hadland, S. E. (2025). Characteristics and safety of consumer chatbots for emergent adolescent health concerns. JAMA Network Open, 8(10), e2539022. https://doi.org/10.1001/jamanetworkopen.2025.39022
Clark, A. (2025). The ability of AI therapy bots to set limits with distressed adolescents: Simulation-based comparison study. JMIR Mental Health, 12, e78414. https://doi.org/10.2196/78414
Written by: Xiyun Chen
AI-assisted behavioral healthcare: innovative opportunities meet ethical concerns
Category: AI in the workplace
Artificial Intelligence is rapidly being integrated into behavioral healthcare. A recent article published by the American Psychological Association explores how clinicians are optimistic about turning to innovation to make therapy more accessible and personalized.
AI systems may soon help clinicians detect early warning signs of depression, anxiety, and suicidal ideation before symptoms develop. Clinical researchers are considering using personal data from smartphones, sleep trackers, and electronic health records to personalize psychological treatment. Psychologists claim that gaining access to patients’ data would allow them to bypass the trial-and-error approach used to determine the best therapeutic modalities for each case. This process can take weeks to months and offers no guarantee of accuracy. Researchers are optimistic that incorporating AI will enable clinicians to shift behavioral healthcare toward a more predictive, rather than reactive, model.
Additionally, researchers are turning to innovation to address the critical shortage of clinical providers and meet the rising demand for psychological services. A recent APA survey reported that over 50% of psychologists had no openings for new patients. Researchers are urgently developing tools for those without access to care.
One example is Therabot, an AI-powered chatbot designed to provide emotional support and therapeutic intervention when human clinicians are unavailable. Researchers emphasize that tools like Therabot are not meant to replace human therapists but are a viable response to the shortage. Including such technology marks a profound shift in mental healthcare, moving beyond the therapist’s office into ongoing, technology-mediated support.
The ethical implications of incorporating AI into the therapeutic milieu are hard to ignore. Researchers are particularly concerned about patient privacy, the security of sensitive clinical data, the risk of fostering emotional dependence on machine-mediated human interaction, and questions of clinical accountability. The article highlights the tension between leveraging the potential benefits of technological integration and the fundamental need for trust and confidentiality in psychological care, where clients routinely disclose highly sensitive personal information.
Furthermore, it remains unclear whether AI systems can be held clinically accountable for responding to disclosures of acute psychological distress. For instance, if a language model fails to identify suicidal ideation or misreads severe emotional distress, it is uncertain who would bear responsibility for any harmful outcome due to psychological malpractice. Therefore, both researchers and clinicians agree that AI systems should only be implemented in conjunction with human providers, not as independent agents.
Ultimately, providers must balance the benefits of AI with the preservation of human connection in clinical settings.
Source: AI, neuroscience, and data are fueling personalized mental health care
Written by: Grant Kuppenheimer
Our BAIC website has recently been updated. Check it out to see our latest studies on how AI chatbots and AI systems are shaping young people’s relationships, development and well-being.
