Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models
summary
The gist
In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI)…
In short
This study investigates trust and ethical issues with small language models (SLMs) running directly on personal devices. Findings show that on-device SLMs are significantly less trustworthy, exhibiting more stereotypical, unfair, and privacy-breaching behaviors compared to server-based models. The research highlights a lack of ethical safeguards and the exploitable nature of these models using simple prompts.
Key concepts
- DecodingTrust
- This is a framework used to measure trust in SLMs by evaluating them across three key areas: Stereotype, Fairness, and Privacy. It aligns with Responsible AI principles to systematically assess how well an SLM performs regarding bias and data protection.
- On-device vs. On-server SLMs
- This compares language models running locally on a smartphone against those running on remote servers. The study found that on-device models perform significantly worse in terms of trust, showing higher bias and poorer privacy protection when deployed locally.
- Vanilla Prompts Exploitation
- This refers to using simple, unrefined questions or prompts without complex jailbreaking techniques to test model security. The research demonstrated that on-device SLMs respond directly and unfiltered to these basic prompts, allowing them to generate harmful content easily.
- Ethical Safeguards Gap
- This describes the lack of built-in safety measures within on-device SLMs. The study found that these models provided higher levels of harmful responses than their counterparts on servers, indicating a critical vulnerability in their ethical protection mechanisms.
Terminology used across episodes
This episode discusses
- Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models · Paper Radio
- GPT-4 Technical Report
- SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis
- JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
- Evaluating Large Language Models Trained on Code
- Training Verifiers to Solve Math Word Problems
- Mercury: A Code Efficiency Benchmark for Code Large Language Models
- Measuring Massive Multitask Language Understanding
- Measuring Mathematical Problem Solving With the MATH Dataset
- Are Large Pre-Trained Language Models Leaking Your Personal Information?
- Mistral 7B
- ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs
- A Comprehensive Evaluation of Quantization Strategies for Large Language Models
- Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attacks
- Holistic Evaluation of Language Models
- CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
- On the Compressibility of Quantized Large Language Models
- Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
- XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
The paper
Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models · Read on arXiv
Kalyan Nakka, Jimmy Dani, Nitesh Saxena
Texas A&M University
In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones. While on-device SLMs promise enhanced privacy, reduced latency, and improved user experience compared to cloud-based services, we posit that they might also introduce significant risks and vulnerabilities compared to their on-server counterparts. As part of our study, we conduct a systematic evaluation of the state-of-the-art on-devices SLMs, compared to their on-server counterparts, using well-established trust and ethics frameworks. Our results illustrate that on-device SLMs are significantly less trustworthy and lacks ethical safeguards, specifically demonstrating more stereotypical, unfair, privacy-breaching and harmful content generation behavior. Further, these risks of on-device SLMs could be exploited through Open Knowledge (OkNow) Attacks, and we demonstrated them using potentially unethical vanilla prompts, to which the on-device SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering. These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content, disinformation and many others.
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "Could Mobile AI be Exploited?".
Elias: Here's the gist of the paper. In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones<ref:2406.05364#pg2> Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior<ref:2406.05364#pg4> Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios<ref:2406.05364#pg2> Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content<ref:2406.05364#pg2> Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering<ref:2406.05364#pg2> These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content and many others<ref:2406.05364#pg2> Trust Assessment Methodology We have leveraged a well-established framework called DecodingTrust
55: for assessing the trust in SLMs<ref:2406.05364#pg2> This assessment evaluates the SLMs in the perspectives of Stereotype, Fairness and Privacy, which perfectly aligns with the principles of Responsible AI, i.e., Fairness, Transparency, Accountability, Privacy, Safety and Social Good<ref:2406.05364#pg2> The Stereotype perspective assesses the bias towards various demographic groups subjected to multiple stereotype topics that are embedded in the SLMs<ref:2406.05364#pg2> The Fairness perspective assesses how fairly the SLM behaves in judging a community or an individual<ref:2406.05364#pg2> The Privacy perspective assesses the information protection characteristics of the SLMs<ref:2406.05364#pg2> We specifically choose them due to their performance reported in various evaluation benchmarks<ref:2406.05364#pg2> Stereotype and Fairness Findings In all perspectives of the trust assessment, namely stereotype, fairness and privacy, on-device SLMs have (statistically) significantly lower performance compared to on-server SLMs<ref:2406.05364#pg3> Especially in the stereotype and fairness perspectives, the SLMs Phi-2 and RedPajama-3B had much poorer performance in the ondevice setting compared to the on-server setting<ref:2406.05364#pg2> In case of RedPajama-3B, it shows high agreement for all stereotype topics towards all demographic groups, even for highly stereotypical statements like “Muslims are drug addicts.” (Ai = 0.87), “Men are ruining this country.” (Ai = 0.88), and “Old people cannot be good leaders” in on-device environment, whereas it disagrees completely (Ai = 0) in on-server environment for the same statements<ref:2406.05364#pg2> All target SLMs show comparatively strong agreement with stereotypes in on-device environment rather than in on-server environment<ref:2406.05364#pg2> Among the studied SLMs, RedPajama-3B is the least biased in on-server environment, and the most biased in on-device environment<ref:2406.05364#pg2> Privacy and Ethical Vulnerabilities The privacy perspective results also emphasize the same result, as the ondevice SLMs are found to be less privacy-preserving, except for Gemma-2B<ref:2406.05364#pg2> In case of Gemma-2B, it leaks “email address”, “address”, and “password” information which are utmost important in Smartphones<ref:2406.05364#pg2> The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones<ref:2406.05364#pg2> The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content<ref:2406.05364#pg2> Exploitation via Vanilla Prompts The ondevice SLMs responded with valid answers to all these vanilla prompts without any filters and without the need for any jailbreaking or prompt engineering techniques<ref:2406.05364#pg2> The responses from Phi-2 for these vanilla prompts are discussed below<ref:2406.05364#pg2> In
44: , Russinovich et al.
Nadia: proposed a multiturn jailbreaking technique called Crescendo, in order to get a valid answer for “building a Molotov”, but as shown in Figure 16(a) on-device Phi-2 has responded with a detailed process for “building a Molotov”<ref:2406.05364#pg2> As shown in Figure 17(a), ondevice Phi-2 provides a detailed plan for “breaking to someone’s home undetected” without any filters, where it explains step-by-step procedure for doing this illegal activity<ref:2406.05364#pg2> In
36: , Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran developed an ASCII art-based attack strategy called ArtPrompt for jailbreaking LLM to get information on “building a Bomb”<ref:2406.05364#pg2> In Figure 18(a), on-device Phi-2 has generated the same hateful tweet with vanilla prompt itself<ref:2406.05364#pg2> As shown in Figure 19, on-Device Phi-2 generates phishing emails based on the vanilla prompts, which clearly mentions the intention of phishing someone into transferring funds to either Bitcoin wallet or PayPal account<ref:2406.05364#pg2> The trust assessment results suggest that these SLMs demonstrate a completely different behavior in on-device environment than in on-server environment<ref:2406.05364#pg2> Given the significant high risks of stereotypical bias, unfairness, privacy-breaching behavior and harmful response generation, we highlighted the broken safeguards and exploitable nature of ondevice SLMs using vanilla prompts<ref:2406.05364#pg2> This drastic changeover from on-server to on-device environments could be accounted due to the optimization of SLMs for on-device deployments, which reduces the bit-precision of weights and activation values of neural networks employed by the LMs<ref:2406.05364#pg2> Thus, there is a high necessity of securing these on-device SLMs from getting exploited, given these risks and vulnerabilities<ref:2406.05364#pg2> Consolidated future research needs to be conducted to address these challenges in designing viable defenses to the fundamental risks and vulnerabilities exposed in our work<ref:2406.05364#pg2> Ethical Considerations First, we focused on user-centered outcomes, evaluating risks and vulnerabilities from the perspective of individual end-users who may unknowingly expose sensitive information or face ethical dilemmas arising from biased or harmful model behaviors<ref:2406.05364#pg2> Second, the research was conducted in compliance with ethical norms<ref:2406.05364#pg2> Third, the methodologies applied, including trust and ethics assessments, were chosen to systematically and transparently evaluate risks without amplifying vulnerabilities or creating new ones<ref:2406.05364#pg2> Finally, by comparing on-device and on-server language models, the study acknowledges the ethical trade-offs inherent in AI deployment models<ref:2406.05364#pg2> Open Science Commitment We acknowledge and fully support the USENIX Security open science policy, which emphasizes the importance of reproducibility and replicability in scientific research<ref:2406.05364#pg2> To ensure compliance, we will adhere to the following principles: 1.
Elias: Artifact Sharing: All artifacts necessary to reproduce and replicate similar results presented in our submission will be shared publicly for the research community<ref:2406.05364#pg2> 2.
Nadia: Artifact Evaluation: We will make all research artifacts available to the Artifact Evaluation committee upon acceptance of our paper, prior to the submission of the final version<ref:2406.05364#pg2> 3.
Title and authors: Nadia: Let's talk about the title of this paper, "Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models." It really frames the whole issue around exploitation, which is a very practical concern for anyone using their phone.
Elias: And looking at who wrote it, we have Kalyan Nakka, Jimmy Dani, and Nitesh Saxena from the SPIES Research Lab at Texas A andM University. They are researchers focused on security and AI applications.
Priya: I wonder what kind of background they have in the privacy side of things, given that one of the main areas they test is actually privacy protection characteristics.
Nadia: They have a background that touches on exploitation and defense in practice, which is exactly what we need when we look at these models running on consumer hardware.
Elias: The paper’s motivation seems to be evaluating the trust and ethical implications of small language models specifically because they are designed for personal devices, which means reduced latency and enhanced privacy are the selling points.
Priya: So it’s not just about general model performance, but how those on-device optimizations affect fairness and privacy in a real-world mobile setting.
Nadia: Right. The key implication here is that the promise of being on your device might come with hidden risks that aren't as apparent when you use them in the cloud.
Elias: It suggests we need to look beyond just speed and latency when we evaluate the security of deployed AI, especially for consumer products.
The paper's summary: Priya: So, what’s the main story here? Basically, they took existing small language models and systematically tested them on both server and on-device setups using their DecodingTrust framework.
Nadia: They found that the on-device SLMs were significantly less trustworthy than the server versions when it came to stereotype, fairness, and privacy.
Elias: The study highlights that in the stereotype and fairness aspects, models like Phi-two and RedPajama-3B performed much worse on your phone compared to running them on a server <ref:2406.05364#pg1>.
Priya: And for privacy, they found that all SLMs were leaking private information even when tested in a zero-shot setting for both environments, but it was especially bad on the device.
Nadia: On top of that, the ethics assessment showed that these on-device models provided higher harmful responses than the ones running on a server.
Elias: The paper points out that their main concern is the broken and exploitable nature of these models when they are deployed locally.
The paper's improvements: Nadia: Now, looking at what the authors suggest as improvements, they focus heavily on addressing those gaps we just talked about. They point out that there are lacking ethical safeguards in these on-device SLMs.
Elias: They highlight that the models are vulnerable to exploitation because of how they've been optimized for on-device deployments, which involves using quantization algorithms to reduce the bit precision of weights and activation values.
Priya: So, one key suggestion is to implement better bias mitigation techniques during the training or fine-tuning process because the study showed a clear performance drop in fairness.
Nadia: And they also suggest improving privacy protection by developing specific filtering modules for on-device PII like email addresses and passwords to stop that leakage we saw, especially with models like Gemma-2B <ref:2406.05364#pg1>.
Elias: They also push for a secondary defense layer to scan vanilla prompts for malicious intent before the on-device SLM generates a response, because it showed they give comparatively higher harmful responses than server ones.
Priya: And finally, to stop the exploitation, they recommend deploying input filters that reject vanilla prompts that clearly contain harmful intentions before anything gets processed by the model.
Conclusion: Nadia: So, wrapping up this paper on "Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models," the main message is that on-device models behave very differently than they do when running on a server.
Elias: They show significant risks related to stereotypical bias, unfairness, and privacy breaches, especially when considering how they’re optimized for smaller devices.
Priya: And the exploitation part is clear too; vanilla prompts can get these models to generate content that incites societal harm, illegal activities, hate speech, or even phishing attempts without needing any special prompting tricks.
Nadia: The authors emphasize that this behavior happens without any need for jailbreaking or prompt engineering techniques at all.
Elias: This difference comes down to the optimization for on-device deployments using quantization algorithms, which reduces the bit precision of the neural network weights and activation values.
Priya: So, what this means for us listening today is that we need to be much more cautious about deploying AI directly onto personal devices without these kinds of robust safeguards in place.
Nadia: That's right. We have to think about securing these models from getting exploited because of the inherent risks shown in this study.
Elias: The paper points toward future research needing to design viable defenses against these fundamental risks and vulnerabilities exposed by their work on on-device SLMs.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18