Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation

arXiv:2610.20652 · cs.CL · Submitted 2026-10-12 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation".

Jane: The gist: Naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five intent classification benchmarks,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: Welcome back. We're talking about a new paper called "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation". This research is looking at how making an AI assistant learn new commands on the device affects its ability to correctly reject things it doesn't understand.

Jane: That sounds really important because, as we know, when you use an always-on virtual assistant, if it gets confused and gives a wrong answer or just makes up stuff, that breaks user trust. This paper focuses on that trust aspect specifically.

Lu: The core idea is testing whether the personalization step—using things like LoRA to learn new user commands via few-shot fine-tuning—actually degrades the out-of-scope rejection quality, which is a critical property for these assistants six <ref:2610.20652#pg2>.

Meng: So, what's the main claim here? Is it saying that if we personalize an assistant on a device to learn new things, it becomes worse at saying "I don't know" when faced with something brand new?

Tom: Exactly. The paper shows that naive few-shot LoRA personalization degrades the OOS rejection AUROC, which is the area under the ROC curve, across every one of five public intent-classification benchmarks <ref:2610.20652#pg1>.

Jane: That's a pretty broad claim. How big is that drop they are seeing? Is it a tiny dip or something significant?

Tom: They report a mean drop between zero point zero five five and zero point two five eight across those five datasets with five seeds for each configuration <ref:2610.20652#pg1>. That's a measurable hit, but they also say that the accuracy on the in-scope commands is perfectly preserved <ref:2610.20652#pg1>.

Lu: What's interesting is how they characterize that degradation across those different datasets. They found that it’s not uniform; for instance, SNIPS shows a substantially larger drop of zero point two five eight AUROC compared to the other four datasets which show drops between zero point zero five five and zero point zero nine seven <ref:2610.20652#pg2>.

Meng: So, it’s not a one-size-fits-all problem for personalization; some benchmarks are hit much harder than others? That matters when you're trying to deploy something everywhere.

Jane: And they did a scorer-invariant check that showed part of this drop is just an artifact from how the standard confidence scorer miscalibrates itself after adaptation <ref:2610.20652#pg1>.

Tom: But they found a smaller, genuine effect remains under any scorer <ref:2610.20652#pg1>, and they traced that genuine effect to a measurable shrinkage of the embedding margin around known intent regions with a pooled Spearman rho of zero point six one <ref:2610.20652#pg1>.

Lu: So, the mechanism behind the drop is related to this shrinkage in the embedding margin between known intent class centroids and true out-of-scope points <ref:2610.20652#pg3>.

Jane: That sounds like a geometric issue in the feature space. They suggest this shrinkage correlates with a larger OOS quality drop, which is supported by a pooled linear fit showing that larger margin shrinkage is associated with a larger OOS quality drop with Spearman rho of zero point six one <ref:2610.20652#pg3>.

Paper summary: Tom: The real focus then shifted to finding a fix for this problem, and they propose replacing the evaluation-time rejection score with a Mahalanobis distance score instead of just using the softmax confidence <ref:2610.20652#pg3>.

Meng: What does that mean in practical terms? How do we switch from relying on confidence scores to something like Mahalanobis distance for rejecting unknown inputs? Is it hard to implement?

Jane: The proposed fix involves fitting a tied-covariance Gaussian over the embeddings—specifically, the postpersonalization base plus the new training pool’s penultimate-layer embeddings, using a shrinkage-regularized covariance matrix and one mean per known class <ref:2610.20652#pg3>.

Tom: And the payoff is that this fix recovers most of the lost OOS quality on three out of five datasets with no accuracy cost and no extra training required <ref:2610.20652#pg3>.

Lu: They also showed this fix holds up robustly, holding across different few-shot budgets, LoRA ranks, and even a second base encoder <ref:2610.20652#pg3>. The combined method reduces this over-concentration substantially on every dataset <ref:2610.20652#pg3>.

Jane: So, this means for someone just listening to the show who doesn't read papers, the point is that if you deploy personalization today, you might lose some of that "I don't understand" capability unless you change how you score those unknown inputs <ref:2610.20652#pg1>.

Tom: It really puts it in perspective for deployment because this fix uses Mahalanobis distance instead of softmax confidence without any extra training or retraining, and it beats two other OOD scoring baselines <ref:2610.20652#pg3>.

Meng: From a production standpoint, that low-millisecond on-device latency cost is key. If we can fix this with just a score change without retraining the whole model, that’s something we can actually deploy <ref:2610.20652#pg3>.

Lu: The paper does have some caveats they mention, though. They note that only gradient-based naive and orthogonal-projection LoRA are covered in their personalization scope <ref:2610.20652#pg3>.

Jane: And they also admit that base intent accuracy drops eight to thirty-one points on every dataset after personalization for both naive LoRA and the combined maha method, which doesn't reliably reduce that drop <ref:2610.20652#pg3>.

Tom: That’s an honest admission. They aren't claiming this fix magically solves the core accuracy drop when you introduce new commands; it’s specifically about preserving the OOS rejection quality <ref:2610.20652#pg1>.

Lu: So, to wrap up, the main contribution of "Does Personalization Break Trust? Open-Set Rejection Calibration Under Few-Shot On-Device Intent Adaptation" is demonstrating that personalization measurably erodes the second property—the ability to reject unknown inputs—while improving the first property—the accuracy on known intents <ref:2610.20652#pg1>.

Jane: And they show that a simple, deployment-cheap change, like scoring rejection by Mahalanobis distance instead of softmax confidence, recovers most of that lost quality on three of five datasets with no accuracy cost <ref:2610.20652#pg3>.

Tom: It’s a practical solution for anyone building pervasive assistants who are doing on-device adaptation, proving that addressing OOS rejection calibration under few-shot learning is a real engineering problem <ref:2610.20652#pg1>.

Conclusion: Tom: So we've seen how naive personalization hurts out-of-scope rejection quality across five different benchmarks <ref:2610.20652#pg1>.

Jane: Right, that means when we adapt an AI assistant to learn new commands on the device, it gets worse at saying "I don't understand" things it hasn't seen before.

Lu: The authors are looking at this problem of out-of-scope detection calibration under few-shot adaptation <ref:2610.20652#pg3>.

Meng: And they found a fix that uses Mahalanobis distance instead of just the usual softmax confidence score <ref:2610.20652#pg3>.

Tom: The authors claim this fix recovers most of the lost rejection quality on three out of five datasets with no extra training needed <ref:2610.20652#pg3>.

Jane: So, it’s a simple scoring change that makes a real difference for those who only listen to the show.

Lu: It’s about replacing the standard confidence score with a Mahalanobis distance score based on Gaussian fitting over the embeddings <ref:2610.20652#pg3>.

Meng: And it’s deployment-cheap, which is what I care about most <ref:2610.20652#pg3>.

Tom: The authors also noted that this fix holds up across different few-shot budgets and even different base encoders <ref:2610.20652#pg3>.

Jane: It shows that even with personalization, we can maintain trust in the assistant's rejection abilities if we tweak how we score the outputs.

Lu: The paper does acknowledge some limitations, like only covering certain types of LoRA <ref:2610.20652#pg3>.

Meng: And they admit that base intent accuracy still drops a lot after personalization for both methods, so this fix doesn't completely solve the initial problem <ref:2610.20652#pg3>.

Tom: The overall message is that teaching an AI new commands and keeping it able to reject unknown inputs are actually two separate engineering problems <ref:2610.20652#pg3>.

Jane: It’s a reminder that deploying these pervasive assistants needs careful calibration of those rejection scores, not just focusing on the command learning part.

University of Technology Nuremberg · Indian Institute of Information Sri City

cs.CL

Submitted: 2026-10-12

Updated: 2026-10-12

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 85/100

The gist: The gist: Naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five intent classification benchmarks, while in-scope accuracy is preserved How it works The paper

Key concepts

Few-shot LoRA Personalization: This is a technique where a large AI model is adapted quickly to new, specific user commands by training only a small set of parameters (LoRA adapters) on just a few examples. The paper tests if this rapid adaptation process harms the assistant's ability to correctly identify inputs that are completely outside its learned scope.
Out-of-Scope (OOS) Rejection Problem: This is the critical task of an assistant recognizing and rejecting user queries it was not trained for. It is a trust-critical property; if an assistant fails to reject irrelevant input, users lose confidence in the system's reliability.
Mahalanobis Distance Scoring: This proposed fix replaces the standard method of scoring rejections (using softmax confidence) with a distance measure based on Mahalanobis distance. This method uses statistical properties derived from the model's embedding space to calculate rejection scores, which proved effective at restoring OOS rejection quality.

Terminology

Adaptation — 1× in this paper · explained in 1 episode
Calibration — 1× in this paper · explained in 7 episodes
Personalization — 1× in this paper · explained in 2 episodes

Summary

The gist: Naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five intent classification benchmarks, while in-scope accuracy is preserved

How it works

The paper investigates whether the personalization step performed only to learn new intents changes the quality of out-of-scope (OOS) rejection, which is a trust-critical property of pervasive assistants The central question is whether personalization degrade OOS rejection quality? in a controlled empirical study

The baseline method involves naive LoRA personalization where a standard rank-r LoRA adapter is trained on k-shot new-intent data, and the default OOS scorer is 1−max(softmax(logits)) – maximum softmax probability (MSP) This baseline shows that naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five intent classification benchmarks (mean over 5 seeds)

Characterizing the degradation

The degradation observed in naive personalization is not uniform across datasets, as SNIPS shows a substantially larger drop (0.258 AUROC) than the other four datasets (0.055–0.097) A scorer-invariant check revealed that part of this drop is an artifact of the standard confidence scorer’s own miscalibration under adaptation, though a smaller, genuine effect remains under any scorer This genuine effect is traced to a measurable shrinkage of the embedding margin around known-intent regions (pooled Spearman ρ = 0.61)

Mechanistic explanation and fix

The mechanism behind the drop is associated with a "measurable shrinkage in the embedding margin between known-intent class centroids and true out of-scope points, using a Mahalanobis-distance formulation grounded in established feature space out-ofdistribution detection theory This shrinkage correlates with the OOS quality drop, as evidenced by a Pooled linear fit showing larger margin shrinkage is associated with a larger OOS-quality drop (Spearman ρ = 0.61, p = 0.0012)"

The proposed fix involves replacing the evaluation-time rejection score with a Mahalanobis-distance score, which is fit a tied-covariance Gaussian over the (postpersonalization) base+new training pool’s penultimate-layer embeddings, shrinkage-regularized covariance matrix and one mean per known class This fix recovers most of the lost OOS quality on 3 of 5 datasets at no accuracy cost, no additional training, and a quantified low-millisecond on-device latency cost

Generality and validation

The effectiveness of the fix is shown to be robust across various settings; the fix holds (non-overlapping 95% CI, higher AUROC, no accuracy cost) on 8 of 9 tested conditions This generality is demonstrated across different few-shot budgets (k ∈ k ∈ the set of), LoRA ranks (r ∈ r ∈ the set of), and second base encoders Furthermore, the combined maha reduces this over-concentration substantially on every dataset

Limitations

The study acknowledges several limitations, including the fact that only CLINC150 ships with a native OOS split; for the other four we use the standard leave-N-intents-out construction from the OOS-detection literature Additionally, base-intent accuracy drops 8–31 points on every dataset after personalization (Section VI-C) for both naive LoRA and combined maha, which does not reliably reduce this drop The study also notes that personalization scope covers only gradient-based naive and orthogonal-projection LoRA, as hypernetwork-based personalization is untested

The paper concludes that "a one-line evaluation-time change by scoring rejection by Mahalanobis distance rather than softmax confidence recovers most of the lost quality on 3 of 5 datasets at negligible on-device cost, beats two further OOD baselines, and holds up across shot count, LoRA rank, and a second encoder This demonstrates that ordinary few-shot LoRA personalization measurably erodes the second property while improving the first, on every one of five public intent benchmarks" The findings provide a practical solution for deployment by showing how to address OOS-rejection calibration under few-shot on-device intent adaptation

The paper is 3 pages long, and it is about 84% of the way through the text. The paper's abstract states that naive few-shot LoRA personalization degrades OOS-rejection AUROC on every one of five public intent classification benchmarks (mean drop 0.055–0.258, five seeds each). The paper's index terms are pervasive computing, on-device personalization, out-of-scope detection, low-rank adaptation, intent classification. The paper's introduction discusses the problem as This is the out-of-scope (OOS) rejection problem, and it is a trust-critical property of any deployed pervasive assistant. The paper's related work section states that None of these three studies OOS-rejection calibration. The paper's method section describes the combined fix as the method we report as “the fix” combines orthogonal-projection LoRA training with the Mahalanobisdistance evaluation scorer. The paper's results section confirms that Combined maha beats both baselines on every one of the five datasets, confirming the Mahalanobis-distance scorer’s advantage is not specific to outperforming MSP alone. The paper's conclusion states that Teaching a personal assistant new commands and keeping it able to say “I don’t understand that” are usually treated as unrelated engineering problems. The paper's references list includes [1] M. Weiser, “The computer for the 21st century,” Scientific American, vol. 265, no. 3, pp. 94–104, 1991. The paper's references list includes [8] Y. Xiong and X. Xie, “OPLoRA: Orthogonal projection LoRA prevents catastrophic forgetting during parameter-efficient fine-tuning,” 2025, arXiv:2510.13003. The paper's references list includes [13] K. Lee, K. Lee, H. Lee, and J. Shin, “A simple unified framework for detecting out-of-distribution samples and adversarial attacks,” in NeurIPS, 2018. The paper's references list includes [19] D. Janiak, J. Binkowski, and T. Kajdanowicz, “A geometry-based view of mahalanobis OOD detection,” 2025. The paper's references list includes [3] A. Baughan, A. Mercurio, A. Liu, X. Wang, J. Chen, and X. Ma, “A mixed-methods approach to understanding user trust after voice assistant failures,” 2023. The paper's references list includes [4] H. Men, Y. Hu, Y. He, Y. Gao, X. Mou, and Y. Xu, “Reject or not?: A benchmark for voice assistant query rejection in smart home scenario and an improved method based on LLMs,” 2025. The paper's references list includes [15] W. Liu, X. Wang, J. D. Owens, and Y.

Improvements for AI systems

We can implement a deploymentcheap fix by scoring rejection by Mahalanobis distance instead of softmax confidence, which recovers most of the lost quality on 3 of 5 datasets at no accuracy cost. This allows the system to robustly reject inputs that do not match any known intent, directly addressing the out-of-scope (OOS) rejection problem.
The system can incorporate an orthogonal-projection LoRA constraint during personalization, as this component contributes a small additional margin on top of it and prevents the adaptation from perturbing the representation structure by constraining updates to the orthogonal complement of W0’s top-k′ singular subspace.
The model's evaluation process can be modified to use a Mahalanobis-distance score, which is defined as the squared Mahalanobis distance to the nearest class mean calculated over the postpersonalization embeddings, leading to a more accurate OOS rejection decision than relying solely on maximum softmax probability (MSP).