User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios
The discussion of 'User Perceptions vs. Proxy LLM Judges' examines how human judgment compares to automated AI scoring when evaluating LLM responses to privacy-sensitive tasks. While most users found the responses helpful and respected privacy, they showed low consensus on specific scenarios. The core finding is a significant misalignment where AI scores do not reliably estimate real human perception, prompting a strong call for human-centered evaluation.