Designing the Future of User Feedback for Generative AI
cs.AI
Submitted: 2026-10-02
Updated: 2026-10-02
Project page: https://www.similarweb.com/top-websites
License: http://creativecommons.org/licenses/by/4.0/
The gist: Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and features.
Terminology
Abstract
Post-deployment feedback from users can be a cost-effective, scalable, and representative means to monitor and improve generative AI systems and features. When implemented effectively, giving such feedback can increase users' engagement with and trust in GenAI systems. Government regulations and industry guidelines call for post-deployment user engagement, but there is little guidance on designing mechanisms that are usable for consumers and provide actionable input for product teams. We conducted a multi-phase study as a collaboration between academic researchers and eBay. Our benchmark evaluation of current industry approaches identified common issues including lack of discoverability, unclear terminology, and inattention to user value. Based on these findings, we developed best-practice recommendations and designed and tested a prototype feedback-collection tool. The tool aimed to provide users with an efficient, flexible, and positive feedback-giving experience, and provide product teams with rich data on performance and potential problems in a usable format.
Sources
- AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
- Fine-Tuning Language Models from Human Preferences
- Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
- Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
- Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
- Levels of Autonomy for AI Agents
- Decoding Safety Feedback from Diverse Raters: A Data-driven Lens on Responsiveness to Severity
- A General Language Assistant as a Laboratory for Alignment
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- Agents' Last Exam
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems