Evaluating Role-Consistency in LLMs for Counselor Training
cs.CL, cs.AI
Submitted: 2026-01-13
Updated: 2026-01-13
Journal ref: Communications in Computer and Information Science (Computer Supported Education), Springer Nature Switzerland, 2026, pp. 523-546
DOI: 10.1007/978-3-032-21997-8_26
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Terminology
Sources
- Observing Dialogue in Therapy: Categorizing and Forecasting Behavioral Codes
- A Survey on Evaluation of Large Language Models
- LoRA: Low-Rank Adaptation of Large Language Models
- Prompted LLMs as Chatbot Modules for Long Open-domain Conversation
- Leveraging Large Language Model as Simulated Patients for Clinical Education
- Persona-Based Conversational AI: State of the Art and Challenges
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Related papers
- ALARM: Audio-Language Alignment for Reasoning Models
- ThinkSum: Probabilistic reasoning over sets using large language models
- Coherence boosting: When your pretrained language model is not paying enough attention
- From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents
- Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization
- The Art of Asking: Multilingual Prompt Optimization for Synthetic Data