MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support
cs.AI
Submitted: 2026-02-01
Updated: 2026-11-02
Comments: NeurIPS 2026, Track on Evaluations and Datasets
Code: https://github.com/NVIDIA-NeMo/RL
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
The gist: Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness.
Terminology
Abstract
Large language models are increasingly used for mental health support, yet their conversational coherence alone does not ensure clinical appropriateness. Existing general-purpose safeguards often fail to distinguish therapeutic disclosures from genuine clinical crises, and progress on developing better ones is bottlenecked by a lack of clinically grounded evaluation resources. To address this gap, we introduce a risk taxonomy, developed in collaboration with PhD-level clinical psychologists, that identifies actionable harm (self-harm and harm to others) while preserving space for safe, non-crisis therapeutic content. We release MindGuard-testset, a dataset of multi-turn conversations annotated at the turn level by clinical experts, which is, to our knowledge, the first public benchmark with turn-level clinical risk labels for multi-turn mental health support. We also release an automated red-teaming (ART) framework designed to measure how safety classifiers affect downstream model behavior in adversarial multi-turn interactions. Using synthetic dialogues generated via a controlled two-agent setup, we train MindGuard, a family of lightweight safety classifiers (with 4B and 8B parameters). Our classifiers reduce false positives at high-recall operating points and, when paired with clinician language models, help achieve lower attack success and harmful engagement rates compared to general-purpose safeguards. We release all models, training and human evaluation data, and the ART framework.
Sources
- Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks
- Technological folie \`a deux: Feedback Loops Between AI Chatbots and Mental Illness
- The Llama 3 Herd of Models
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Investigating Affective Use and Emotional Well-being on ChatGPT
- Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming
- Introducing v0.5 of the AI Safety Benchmark from MLCommons
- Qwen3 Technical Report
- Qwen3Guard Technical Report
Related papers
- Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Embeddings for Preferences, Not Semantics
- Can We Trust Item Response Theory for AI Evaluation?
- Diagnosing Task Insensitivity in Language Agents