Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models

The paper introduces SLIFT (Self-Learning from Interaction Feedback via Task-Relative Specialization), a selective self-learning framework that converts user feedback from interaction logs into persistent behavioral improvements for large language models (LLMs). The framework is built on a task-relative view of user feedback, where each feedback message is decomposed into atomic components and interpreted relative to the original task as one of three roles: Fix, Spec, or Null.

Problem and Motivation: The paper addresses the challenge that a single feedback may itself combine corrections, preferences, and new requests and that because these components may warrant different behavioral changes, treating the entire feedback as a single supervision signal can obscure what should be learned and how broadly it should generalize. The central research question is: How can a model achieve continual self-improvement from user interactions while incorporating each feedback-supported change within its appropriate scope?

Method: SLIFT operates in two stages. In the first stage, "the frozen backbone decomposes each subsequent user feedback into atomic components and assigns each a role relative to the original task: Fix for task-necessary properties, Spec for compatible but non-necessary refinements, and Null when no reliable positive behavioral target can be recovered. Specifically, A Fix specifies a property that every response in the task-valid region must satisfy; a Spec specifies a compatible refinement that selects a condition-specific subregion within it; and a Null provides no reliable positive update direction for the original task."

In the second stage, these roles determine the form of adaptation. SLIFT trains "two complementary LoRA adapters on a shared frozen backbone: a Generalist that consolidates Fix requirements into default behavior through feedback-conditioned self-distillation, and a Specialist that observes only the task and Generalist response to supply residual guidance for applicable, unmet Spec refinements. Null components induce no positive update. The Generalist uses an off-policy variant of Self-Distillation Policy Optimization (SDPO) with a behavioral anchor, minimizing a loss that combines the Fix-conditioned hindsight distribution with a lambdaB-weighted second term [that] anchors the Generalist to its feedback-free behavior." The Specialist is trained via supervised fine-tuning on structured targets (KEEP or APPLY with residual guidance) constructed offline by the frozen backbone using Spec components as privileged information.

Experiments and Results: The paper evaluates SLIFT on two datasets: MemoryBench (simulated feedback) and WildFB (real-user feedback), using Qwen3-8B and Ministral3-14B-Instruct as backbones. Results show that SLIFT achieves the highest mean Norm-Score and Z-score on all four input–output length partitions for both backbones on MemoryBench. On WildFB, SLIFT improves IFEval and AlpacaEval 2.0 for both backbones, yields a modest positive gain in WildReward win rate on held-out WildFB queries, and maintains comparable performance on MMLU-Pro. SLIFT outperforms baselines including direct retrieval (BM25, dense embedding), memory systems (MemOS, ReMem, A-Mem, MemoryOS), and parametric adaptation methods (SFT, DPO, SDPO).

Ablations and Analysis: Ablation studies show that removing atomic extraction, task context for role assignment, separate pathways, the Specialist, or KEEP supervision all degrade performance. The paper also analyzes task composition and feedback usability, finding that MemoryBench is concentrated in generation and creation (53.4%) and analysis and reasoning (32.7%), while WildFB instead contains larger shares of code and implementation (22.6%), information seeking and question answering (17.4%), and editing and transformation (13.6%). Feedback usability analysis shows fully usable feedback is present in 97.0% of MemoryBench root tasks, compared with 50.4% of WildFB root tasks. Stage-wise analysis shows The Generalist provides broad gains while The Specialist adds targeted gains with sparse edits, with edit rates below 20% across all test sets. Online evolution experiments show SLIFT extracts greater gains from both pre-collected and on-policy feedback. Efficiency analysis shows SLIFT's parameter-update cost is below SFT and DPO and its inference cost does not grow with the size of the training log.

Conclusion: The paper concludes that SLIFT, a selective self-learning framework that converts interaction feedback into persistent behavioral improvements through a task-relative view, achieves strong performance on both MemoryBench and WildFB, highlighting the importance of learning both the supported behavioral change and its appropriate scope.

Improvements to AI Systems:

1. Decompose multi-part user feedback into atomic, role-labeled components (Fix/Spec/Null) before learning, so the system can distinguish mandatory corrections from optional refinements and ignore unactionable feedback—preventing over-generalization or contradictory updates from mixed signals.

2. Maintain two separate adaptation pathways on a frozen backbone: a Generalist LoRA that consolidates all Fix requirements into default behavior via feedback-conditioned self-distillation, and a Specialist LoRA that only activates when a Spec refinement applies, supplying residual, condition-specific guidance. This allows the system to improve broad correctness without losing task-specific flexibility.

3. Use a behavioral anchor during self-distillation (λB-weighted term) to ensure the Generalist does not drift from its pre-feedback behavior when no Fix is present, preserving stability and preventing catastrophic forgetting of general capabilities.

4. Train the Specialist with structured KEEP/APPLY targets constructed offline from Spec components, so it learns when not to intervene (KEEP) as well as what residual change to apply (APPLY)—reducing unnecessary edits and keeping inference efficient (edit rate below 20% in practice).

5. Automatically filter Null feedback components from training entirely, avoiding spurious supervision that would otherwise degrade performance on real-world, noisy user interactions (e.g., 50% of WildFB root tasks contain partially unusable feedback).

6. Enable continual, on-policy self-improvement by feeding new interaction logs back into the same two-stage pipeline, allowing the system to extract gains from both pre-collected and live user feedback without retraining the backbone or growing inference cost with log size.

What the improved AI system can do:

• Separate fix this bug from I’d prefer it shorter in a single user message, applying the first as a hard requirement across all future responses and the second only in contexts where the user’s condition is met.

• Improve default behavior on core task requirements (e.g., always cite sources, never use markdown) while still adapting to user-specific style preferences (e.g., for code tasks, include runnable examples) without mixing the two.

• Ignore vague or contradictory feedback (e.g., make it better) without learning harmful patterns, maintaining baseline performance on general benchmarks (MMLU-Pro) while improving instruction-following and chat quality.

• Self-update from real user logs in production, continuously refining both generalist and specialist behaviors with minimal parameter updates (lower than SFT/DPO) and constant inference latency, regardless of accumulated feedback volume.

• Handle mixed-task feedback (e.g., a user asks for a summary then says also fix the grammar) by learning the grammar fix as a general requirement and the summary style as a conditional refinement, without conflating the two.

User feedback offers natural supervision for persistent LLM improvement, but a single message may support multiple behavioral changes with different scopes of generalization. We introduce SLIFT, a selective self-learning framework built on a task-relative view of user feedback. SLIFT decomposes each feedback message into atomic components and interprets each component relative to the original task as Fix, Spec, or Null: requirements for task validity, compatible condition-specific refinements, or content with no reliable positive update direction. To incorporate each change at the appropriate scope, SLIFT trains two complementary LoRA adapters on a shared frozen backbone: a Generalist that consolidates Fix requirements into default behavior through feedback-conditioned self-distillation, and a Specialist that observes only the task and Generalist response to supply residual guidance for applicable, unmet Spec refinements. Null components induce no positive update. Across backbones, SLIFT achieves strong performance on both MemoryBench and WildFB, with targeted analyses further examining its underlying mechanisms. We release our code at https://anonymous.4open.science/r/SLIFT.