LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities
summary
The gist
LiteLMGuard proposes an on-device guardrail that provides real-time, prompt-level defense for quantized SLMs to secure them against risks and vulnerabilities introduced by quantization This work…
In short
LiteLMGuard is an on-device defense mechanism designed to protect quantized small language models (SLMs) from vulnerabilities introduced by quantization. It works by adding a lightweight, prompt-level filter that checks if a user's query is answerable before it reaches the SLM. This provides real-time safety and privacy for models deployed locally.
Key concepts
- Open Knowledge Attacks
- This threat model describes an adversary injecting vulnerabilities into open-source SLMs and republishing them. Because these models are widely available, this attack poses a high risk, as quantized versions can directly respond to harmful queries instead of refusing them.
- LiteLMGuard
- This is a deployable guardrail that acts as a separate prompt filtering layer for any SLM. It classifies input queries semantically to decide if the underlying model should process them, adding minimal latency while ensuring safety on the device.
- Answerable-or-Not Dataset
- This curated dataset contains prompts labeled as either 'YES' (answerable) or 'NO' (not answerable) based on specific harm categories. It was created using GPT-4o to ensure that only safe prompts are answered, helping the guardrail accurately classify queries.
- Quantization Vulnerabilities
- Quantization is a technique used to shrink large models for on-device use, but it can introduce security risks. These risks allow quantized SLMs to bypass safety mechanisms and respond directly to harmful inputs that larger models might reject.
Terminology used across episodes
This episode discusses
- LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities · Paper Radio
- Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
- Detecting Language Model Attacks with Perplexity
- InternLM2 Technical Report
- Evaluating Large Language Models Trained on Code
- Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations
- ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
- Training Verifiers to Solve Math Word Problems
- Exploiting LLM Quantization
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression
- Language model compression with weighted low-rank factorization
- Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes
- GPT-4o System Card
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models
- Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing
- Mistral 7B
- ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
- A Comprehensive Evaluation of Quantization Strategies for Large Language Models
The paper
LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities · Read on arXiv
SPIES Research Lab · Texas A&M University
The growing adoption of Large Language Models (LLMs) has influenced the development of Small Language Models (SLMs) for on-device deployment across smartphones and edge devices, offering enhanced privacy, reduced latency, server-free functionality, and improved user experience. However, due to on-device resource constraints, SLMs undergo size optimization through compression techniques like quantization, which inadvertently introduce fairness, ethical and privacy risks. Critically, quantized SLMs may respond to harmful queries directly, without requiring adversarial manipulation, raising significant safety and trust concerns. To address this, we propose LiteLMGuard, an on-device guardrail that provides real-time, prompt-level defense for quantized SLMs. Additionally, our guardrail is designed to be model-agnostic such that it can be seamlessly integrated with any SLM, operating independently of underlying architectures. Our LiteLMGuard formalizes deep learning (DL)-based prompt filtering by leveraging semantic understanding to classify prompt answerability for SLMs. Built on our curated Answerable-or-Not dataset, LiteLMGuard employs ELECTRA as the candidate model with 97.75% answerability classification accuracy. The on-device deployment of LiteLMGuard enabled real-time offline filtering with over 85% defense-rate against harmful prompts (including jailbreak attacks), 94% filtering accuracy and 135 ms average latency. These results demonstrate LiteLMGuard as a lightweight robust defense mechanism for effectively and efficiently securing on-device SLMs against Open Knowledge Attacks.
DOI: 10.18653/v1/2025.findings-ijcnlp.12
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities".
Elias: Here's the gist of the paper. LiteLMGuard proposes an on-device guardrail that provides real-time, prompt-level defense for quantized SLMs to secure them against risks and vulnerabilities introduced by quantization This work addresses the safety and trust concerns arising from size optimization techniques like quantization which may cause quantized SLMs to respond directly to harmful queries without adversarial manipulation Threat Model and Motivation The paper introduces a novel threat model called Open Knowledge Attacks, which targets on-device SLMs at a global scale This attack involves an adversary injecting quantization-induced vulnerabilities into an open-source SLM and republishing it to open-source model repositories The severity of this threat model is considered very high due to the free availability of these models on HuggingFace and a huge potential worldwide target base Quantized SLMs, such as Phi-2 and RedPajama, have been observed to provide accurate responses to direct harmful queries instead of rejecting them Design and Methodology LiteLMGuard is designed as an on-device deployable SLM-agnostic guardrail that seamlessly integrates as a separate prompt filtering layer with any on-device SLM with negligible latency overhead The core design formalizes prompt filtering as a binary text classification task to semantically determine whether an input query is answerable by the underlying SLM The methodology involves performing real-time semantic analysis to determine whether the input query should proceed to the SLM Performance and Evaluation The guardrail's effectiveness is evaluated using two metrics: unsafe response rate (URR) and relative safety effectiveness (RSE) The results show that LiteLMGuard achieves an RSE of at least 87% against harmful prompts without adversarial training across AdvBench and Behaviors datasets Furthermore, the prompt filtering accuracy is reported at 94% with an average latency of approximately 135 ms on tested smartphones LiteLMGuard's competitive performance against server-run guard models is demonstrated using a lightweight 15M parameter ELECTRA model Practical Demonstrations and Conclusion Safety assessments illustrate that Phi-2 and RedPajama can provide highly sensitive self-harming responses that are blocked by the guardrail The paper concludes that LiteLMGuard provides an effective and efficient defense mechanism against quantization-induced risks in on-device SLM deployment It emphasizes the need for an on-device safety mechanism to ensure data privacy and server-free functionality The work suggests that the design could be seamlessly applied for securing LLMs as well Related Work and Limitations The paper reviews related works concerning attacks on LLMs and SLMs, including AutoDAN, ArtPrompt, and Crescendo jailbreak attacks Defenses discussed include prompt filtering-based defenses like Prompt Perturbation-based defenses such as paraphrasing and retokenization Limitations noted are that the performance of the candidate model is limited by the Answerable-or-Not dataset and that continuous training regimes are anticipated for improvement The paper also highlights that using LLMs or guard models derived from LLMs as safeguards defeats the purpose of On-Device AI by requiring data to be sent to servers Answerable-or-Not Dataset Details The curated dataset called Answerable-or-Not comprises 2440 textual prompts, which has 40 textual prompts (20 YES and 20 NO labels) for each category of the lower level of safety taxonomy This dataset was collected using the GPT-4o model through ChatGPT to ensure that NO labeled prompts should be rejected from answering and YES labeled prompts should only be answered The data collection process leverages the names and descriptions of 61 specific harm categories from Wang et al.
Nadia: (2024b) Target SLMs and Guard Models The target SLMs used in safety effectiveness experiments include Gemma, Phi-2, RedPajama, Gemma-2, Phi-3.5 (mini), Llama-3.2, and InternLM-2.
Paper summary: Nadia: So we're looking at this paper called LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities. It tackles the problem that when you shrink those smaller language models for on-device use, like on smartphones, the quantization process itself can introduce new risks.
Elias: Right, so the core idea is that these compressed models might start responding directly to harmful queries without needing any special tricks from an attacker. The paper claims this is a big safety and trust issue because these smaller models are becoming super common on devices <ref:2505.05619#pg1>.
Priya: From my side, I’m thinking about the threat model they propose, which they call Open Knowledge Attacks. It sounds like someone could take an open-source model, inject vulnerabilities from quantization into it, and then upload that compromised version to a place where regular users can easily download and use it <ref:2505.05619#pg2>.
Nadia: That's serious because these models are out there freely on places like HuggingFace, so the potential number of targets is huge globally. They're worried that people might unknowingly download a model that’s already been compromised <ref:2505.05619#pg3>.
Elias: The paper is proposing LiteLMGuard as an on-device deployable safety mechanism for securing these quantized SLMs against any of those risks <ref:2505.05619#pg2>. Their main contribution is presenting the first practical way to filter prompts directly on the device, which should keep data private and avoid sending anything to a server <ref:2505.05619#pg2>.
Priya: But before we get into how they do it, I want to know what this filtering actually achieves. The paper talks about performing real-time semantic analysis on the input query to see if it should even proceed to the underlying SLM <ref:2505.05619#pg2>. What kind of safety level are we talking about when we talk about filtering queries before they hit the model?
Nadia: Well, they frame this as a binary text classification task, essentially deciding if an input query is answerable by the model <ref:2505.05619#pg2>. They're trying to make sure that harmful prompts are blocked right at the start of the process <ref:2505.05619#pg2>.
Elias: And they show some performance metrics for this guardrail, reporting a relative safety effectiveness of at least eighty-seven percent against harmful prompts across datasets like AdvBench and Behaviors <ref:2505.05619#pg3>. They also report the prompt filtering accuracy is around ninety-four percent with an average latency of about one hundred thirty-five milliseconds on tested smartphones <ref:2505.05619#pg3>.
Priya: A latency of one hundred thirty-five milliseconds sounds reasonable for a real-time check on a phone, but I’m wondering about the data they used to measure that effectiveness. Are we trusting this eighty-seven percent safety score without knowing exactly what kind of harmful prompts were in those AdvBench and Behaviors datasets?
Nadia: They mention that their evaluation was done across several target SLMs including Gemma, Phi-two RedPajama, and others <ref:2505.05619#pg3>. They even show examples where models like Phi-two can give highly sensitive self-harming responses that this guardrail successfully blocks <ref:2505.05619#pg3>.
Paper summary: Elias: It’s interesting they also demonstrated competitive performance against guard models that run on a server, specifically using a lightweight fifteen million parameter ELECTRA model for comparison <ref:2505.05619#pg3>. That suggests the on-device approach isn't just theoretical; it has some real speed advantages in practice.
Priya: So, if this works as well as they claim, what does this mean for the privacy and deployment side? Does having a guardrail *on* the device solve the problem of sending sensitive inputs to a third party?
Nadia: Exactly. The paper emphasizes that LiteLMGuard provides an effective and efficient defense mechanism against risks caused by quantization in on-device SLM deployments <ref:2505.05619#pg3>. This focus on being server-free is really important for data privacy <ref:2505.05619#pg2>.
Elias: The authors also mention that this design could be applied to securing larger language models as well, suggesting the mechanism itself is quite general <ref:2505.05619#pg3>. It's not just about these small SLMs anymore.
Priya: I do want to bring up a limitation they point out. They noted that the performance of their guardrail is limited by the Answerable-or-Not dataset, which consists of two thousand four hundred forty textual prompts <ref:2505.05619#pg3>. That means if there's a type of harmful query not covered in those specific examples, the guardrail might miss it <ref:2505.05619#pg3>.
Nadia: So, they're relying on that curated dataset to define what's safe versus unsafe for this filtering task <ref:2505.05619#pg3>. It makes sense that the scope is tied to the data they used for training or testing that prompt filter.
Elias: And looking at the broader context, they did review related work on attacks like AutoDAN and ArtPrompt, and defenses like prompt perturbation techniques <ref:2505.05619#pg3>. It puts their work in conversation with existing methods for dealing with LLM vulnerabilities <ref:2505.05619#pg3>.
Priya: It’s helpful to see how this fits into the bigger picture of defending against these kinds of jailbreak attacks, but I'm still thinking about the ongoing work. They mentioned that they anticipate continuous training regimes for improvement on their filtering accuracy <ref:2505.05619#pg3>. That implies it's not a final product right now, but an evolving defense layer <ref:2505.05619#pg3>.
Nadia: So, the paper concludes that LiteLMGuard offers a way to secure on-device SLMs against quantization risks by adding a prompt filtering layer <ref:2505.05619#pg3>. It’s about making sure these smaller models stay safe where they are deployed, away from server dependencies <ref:2505.05619#pg2>.
Elias: That's the main message of LiteLMGuard: an efficient, on-device defense mechanism for quantized SLMs against quantization-induced risks <ref:2505.05619#pg3>. It’s about keeping safety local and fast for users <ref:2505.05619#pg2>.
Priya: Ultimately, it points toward the need for these kinds of on-device safety mechanisms to ensure data privacy and keep AI functionality server-free <ref:2505.05619#pg3>. It’s about control over where and how that processing happens <ref:2505.05619#pg2>.
Conclusion: Nadia: So, we're wrapping up this look at LiteLMGuard, this paper titled "LiteLMGuard: Seamless and Lightweight On-Device Guardrails for Small Language Models against Quantization Vulnerabilities." Basically, the core idea is adding a prompt filter right onto your device to stop small language models from responding directly to harmful queries because of how they were shrunk.
Elias: And the authors are trying to fix a real problem: open source models getting compromised when they get that quantization treatment. They’re focusing on making this defense work without needing any server connection, which is huge for privacy.
Priya: From my side, I’m thinking about what this means practically for deployment. They claim an accuracy of ninety-four percent on the filtering task and a latency around one hundred thirty-five milliseconds on smartphones. That's fast enough to be useful in real time, but we have to ask what kind of harmful prompts that number is actually measuring against.
Nadia: Exactly, Priya. It’s not just a number; it’s the effectiveness against those self-harming responses they showed with models like Phi-two and RedPajama being blocked in the experiments. They're showing how this on-device guardrail can stop them from giving out sensitive info when you're using these smaller AI tools.
Elias: The authors are also making a point about the future, saying this design could actually be applied to securing larger language models too. That’s interesting because it suggests the fundamental filtering concept isn't just for these tiny SLMs anymore.
Priya: But we have to remember what they flagged as a limitation. Their performance is tied directly to that specific dataset called Answerable-or-Not, which means if there’s a new kind of harmful query not in that set, the guardrail might fail to catch it. That reliance on that specific data is something we need to keep in mind.
Nadia: So, the big picture here is moving safety and trust mechanisms closer to where the AI actually lives—on your device—to ensure data privacy and a server-free experience for users. But we still have to figure out how robust that filtering layer is against attacks that might try to bypass it.
Elias: That’s the next step, isn't it? We’ve seen the defense mechanism, now we need to look at how an attacker tries to break through this prompt filter in a real-world scenario.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18