Language-model ratings of depression reflect the rater more than the patient

cs.CL, cs.AI, q-bio.NC

Submitted: 2026-10-06

Updated: 2026-10-06

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Sources

• The Capability of Large Language Models to Measure Psychiatric Functioning
• Non-Determinism of "Deterministic" LLM Settings
• Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation
• GPT-4 on Clinic Depression Assessment: An LLM-Based Pilot Study
• Same Patient, Different Words, Different Diagnosis? Evaluating Semantic Stability in Clinical LLMs
• Language Shapes Mental Health Evaluations in Large Language Models
• Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias
• Reproducibility is not construct validity: LLM measurement of institutionally situated communication
• On Arbitrary Predictions from Equally Valid Models
• COMPASS 2.0: psychometric representational similarity analysis distinguishes symptom structure from personal signal
• Qwen2.5 Technical Report
• The Llama 3 Herd of Models
• Mistral 7B
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• gpt-oss-120b & gpt-oss-20b Model Card

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering