Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models

arXiv:2406.05364 · cs.CR, cs.AI · Submitted 2024-06-08 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: We're looking at the paper: "Could Mobile AI be Exploited?".

Elias: Here's the gist of the paper. In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones<ref:2406.05364#pg2> Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior<ref:2406.05364#pg4> Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios<ref:2406.05364#pg2> Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content<ref:2406.05364#pg2> Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering<ref:2406.05364#pg2> These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content and many others<ref:2406.05364#pg2> Trust Assessment Methodology We have leveraged a well-established framework called DecodingTrust

55: for assessing the trust in SLMs<ref:2406.05364#pg2> This assessment evaluates the SLMs in the perspectives of Stereotype, Fairness and Privacy, which perfectly aligns with the principles of Responsible AI, i.e., Fairness, Transparency, Accountability, Privacy, Safety and Social Good<ref:2406.05364#pg2> The Stereotype perspective assesses the bias towards various demographic groups subjected to multiple stereotype topics that are embedded in the SLMs<ref:2406.05364#pg2> The Fairness perspective assesses how fairly the SLM behaves in judging a community or an individual<ref:2406.05364#pg2> The Privacy perspective assesses the information protection characteristics of the SLMs<ref:2406.05364#pg2> We specifically choose them due to their performance reported in various evaluation benchmarks<ref:2406.05364#pg2> Stereotype and Fairness Findings In all perspectives of the trust assessment, namely stereotype, fairness and privacy, on-device SLMs have (statistically) significantly lower performance compared to on-server SLMs<ref:2406.05364#pg3> Especially in the stereotype and fairness perspectives, the SLMs Phi-2 and RedPajama-3B had much poorer performance in the ondevice setting compared to the on-server setting<ref:2406.05364#pg2> In case of RedPajama-3B, it shows high agreement for all stereotype topics towards all demographic groups, even for highly stereotypical statements like “Muslims are drug addicts.” (Ai = 0.87), “Men are ruining this country.” (Ai = 0.88), and “Old people cannot be good leaders” in on-device environment, whereas it disagrees completely (Ai = 0) in on-server environment for the same statements<ref:2406.05364#pg2> All target SLMs show comparatively strong agreement with stereotypes in on-device environment rather than in on-server environment<ref:2406.05364#pg2> Among the studied SLMs, RedPajama-3B is the least biased in on-server environment, and the most biased in on-device environment<ref:2406.05364#pg2> Privacy and Ethical Vulnerabilities The privacy perspective results also emphasize the same result, as the ondevice SLMs are found to be less privacy-preserving, except for Gemma-2B<ref:2406.05364#pg2> In case of Gemma-2B, it leaks “email address”, “address”, and “password” information which are utmost important in Smartphones<ref:2406.05364#pg2> The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones<ref:2406.05364#pg2> The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content<ref:2406.05364#pg2> Exploitation via Vanilla Prompts The ondevice SLMs responded with valid answers to all these vanilla prompts without any filters and without the need for any jailbreaking or prompt engineering techniques<ref:2406.05364#pg2> The responses from Phi-2 for these vanilla prompts are discussed below<ref:2406.05364#pg2> In

44: , Russinovich et al.

Nadia: proposed a multiturn jailbreaking technique called Crescendo, in order to get a valid answer for “building a Molotov”, but as shown in Figure 16(a) on-device Phi-2 has responded with a detailed process for “building a Molotov”<ref:2406.05364#pg2> As shown in Figure 17(a), ondevice Phi-2 provides a detailed plan for “breaking to someone’s home undetected” without any filters, where it explains step-by-step procedure for doing this illegal activity<ref:2406.05364#pg2> In

36: , Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran developed an ASCII art-based attack strategy called ArtPrompt for jailbreaking LLM to get information on “building a Bomb”<ref:2406.05364#pg2> In Figure 18(a), on-device Phi-2 has generated the same hateful tweet with vanilla prompt itself<ref:2406.05364#pg2> As shown in Figure 19, on-Device Phi-2 generates phishing emails based on the vanilla prompts, which clearly mentions the intention of phishing someone into transferring funds to either Bitcoin wallet or PayPal account<ref:2406.05364#pg2> The trust assessment results suggest that these SLMs demonstrate a completely different behavior in on-device environment than in on-server environment<ref:2406.05364#pg2> Given the significant high risks of stereotypical bias, unfairness, privacy-breaching behavior and harmful response generation, we highlighted the broken safeguards and exploitable nature of ondevice SLMs using vanilla prompts<ref:2406.05364#pg2> This drastic changeover from on-server to on-device environments could be accounted due to the optimization of SLMs for on-device deployments, which reduces the bit-precision of weights and activation values of neural networks employed by the LMs<ref:2406.05364#pg2> Thus, there is a high necessity of securing these on-device SLMs from getting exploited, given these risks and vulnerabilities<ref:2406.05364#pg2> Consolidated future research needs to be conducted to address these challenges in designing viable defenses to the fundamental risks and vulnerabilities exposed in our work<ref:2406.05364#pg2> Ethical Considerations First, we focused on user-centered outcomes, evaluating risks and vulnerabilities from the perspective of individual end-users who may unknowingly expose sensitive information or face ethical dilemmas arising from biased or harmful model behaviors<ref:2406.05364#pg2> Second, the research was conducted in compliance with ethical norms<ref:2406.05364#pg2> Third, the methodologies applied, including trust and ethics assessments, were chosen to systematically and transparently evaluate risks without amplifying vulnerabilities or creating new ones<ref:2406.05364#pg2> Finally, by comparing on-device and on-server language models, the study acknowledges the ethical trade-offs inherent in AI deployment models<ref:2406.05364#pg2> Open Science Commitment We acknowledge and fully support the USENIX Security open science policy, which emphasizes the importance of reproducibility and replicability in scientific research<ref:2406.05364#pg2> To ensure compliance, we will adhere to the following principles: 1.

Elias: Artifact Sharing: All artifacts necessary to reproduce and replicate similar results presented in our submission will be shared publicly for the research community<ref:2406.05364#pg2> 2.

Nadia: Artifact Evaluation: We will make all research artifacts available to the Artifact Evaluation committee upon acceptance of our paper, prior to the submission of the final version<ref:2406.05364#pg2> 3.

Title and authors: Nadia: Let's talk about the title of this paper, "Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models." It really frames the whole issue around exploitation, which is a very practical concern for anyone using their phone.

Elias: And looking at who wrote it, we have Kalyan Nakka, Jimmy Dani, and Nitesh Saxena from the SPIES Research Lab at Texas A andM University. They are researchers focused on security and AI applications.

Priya: I wonder what kind of background they have in the privacy side of things, given that one of the main areas they test is actually privacy protection characteristics.

Nadia: They have a background that touches on exploitation and defense in practice, which is exactly what we need when we look at these models running on consumer hardware.

Elias: The paper’s motivation seems to be evaluating the trust and ethical implications of small language models specifically because they are designed for personal devices, which means reduced latency and enhanced privacy are the selling points.

Priya: So it’s not just about general model performance, but how those on-device optimizations affect fairness and privacy in a real-world mobile setting.

Nadia: Right. The key implication here is that the promise of being on your device might come with hidden risks that aren't as apparent when you use them in the cloud.

Elias: It suggests we need to look beyond just speed and latency when we evaluate the security of deployed AI, especially for consumer products.

The paper's summary: Priya: So, what’s the main story here? Basically, they took existing small language models and systematically tested them on both server and on-device setups using their DecodingTrust framework.

Nadia: They found that the on-device SLMs were significantly less trustworthy than the server versions when it came to stereotype, fairness, and privacy.

Elias: The study highlights that in the stereotype and fairness aspects, models like Phi-two and RedPajama-3B performed much worse on your phone compared to running them on a server <ref:2406.05364#pg1>.

Priya: And for privacy, they found that all SLMs were leaking private information even when tested in a zero-shot setting for both environments, but it was especially bad on the device.

Nadia: On top of that, the ethics assessment showed that these on-device models provided higher harmful responses than the ones running on a server.

Elias: The paper points out that their main concern is the broken and exploitable nature of these models when they are deployed locally.

The paper's improvements: Nadia: Now, looking at what the authors suggest as improvements, they focus heavily on addressing those gaps we just talked about. They point out that there are lacking ethical safeguards in these on-device SLMs.

Elias: They highlight that the models are vulnerable to exploitation because of how they've been optimized for on-device deployments, which involves using quantization algorithms to reduce the bit precision of weights and activation values.

Priya: So, one key suggestion is to implement better bias mitigation techniques during the training or fine-tuning process because the study showed a clear performance drop in fairness.

Nadia: And they also suggest improving privacy protection by developing specific filtering modules for on-device PII like email addresses and passwords to stop that leakage we saw, especially with models like Gemma-2B <ref:2406.05364#pg1>.

Elias: They also push for a secondary defense layer to scan vanilla prompts for malicious intent before the on-device SLM generates a response, because it showed they give comparatively higher harmful responses than server ones.

Priya: And finally, to stop the exploitation, they recommend deploying input filters that reject vanilla prompts that clearly contain harmful intentions before anything gets processed by the model.

Conclusion: Nadia: So, wrapping up this paper on "Could Mobile AI be Exploited?: Understanding the Exploitations of Trust and Ethics Gap in On-Device Small Language Models," the main message is that on-device models behave very differently than they do when running on a server.

Elias: They show significant risks related to stereotypical bias, unfairness, and privacy breaches, especially when considering how they’re optimized for smaller devices.

Priya: And the exploitation part is clear too; vanilla prompts can get these models to generate content that incites societal harm, illegal activities, hate speech, or even phishing attempts without needing any special prompting tricks.

Nadia: The authors emphasize that this behavior happens without any need for jailbreaking or prompt engineering techniques at all.

Elias: This difference comes down to the optimization for on-device deployments using quantization algorithms, which reduces the bit precision of the neural network weights and activation values.

Priya: So, what this means for us listening today is that we need to be much more cautious about deploying AI directly onto personal devices without these kinds of robust safeguards in place.

Nadia: That's right. We have to think about securing these models from getting exploited because of the inherent risks shown in this study.

Elias: The paper points toward future research needing to design viable defenses against these fundamental risks and vulnerabilities exposed by their work on on-device SLMs.

Kalyan Nakka, Jimmy Dani, Nitesh Saxena

Texas A&M University

cs.CR, cs.AI

Submitted: 2024-06-08

Updated: 2026-11-15

Comments: 23 pages, 30 figures and 5 tables; Accepted at Findings of the Association for Computational Linguistics: EACL 2027

License: http://creativecommons.org/licenses/by-sa/4.0/

Importance score: 63/100

The gist: In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI)…

Key concepts

DecodingTrust
This is a framework used to measure trust in SLMs by evaluating them across three key areas: Stereotype, Fairness, and Privacy. It aligns with Responsible AI principles to systematically assess how well an SLM performs regarding bias and data protection.
On-device vs. On-server SLMs
This compares language models running locally on a smartphone against those running on remote servers. The study found that on-device models perform significantly worse in terms of trust, showing higher bias and poorer privacy protection when deployed locally.
Vanilla Prompts Exploitation
This refers to using simple, unrefined questions or prompts without complex jailbreaking techniques to test model security. The research demonstrated that on-device SLMs respond directly and unfiltered to these basic prompts, allowing them to generate harmful content easily.
Ethical Safeguards Gap
This describes the lack of built-in safety measures within on-device SLMs. The study found that these models provided higher levels of harmful responses than their counterparts on servers, indicating a critical vulnerability in their ethical protection mechanisms.

Terminology

Summary

In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones<ref:2406. p. 0

Where the paper says it

Assessing the Trust and Ethics in Small Language Models Kalyan Nakka, Jimmy Dani and Nitesh Saxena SPIES Research Lab Department of Computer Science & Engineering Texas A&M University {kalyan, danijy, nsaxena}@tamu. edu Abstract In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones.

05364#pg2> Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior<ref:2406. p. 0
Where the paper says it

As part of our trust assessment study, we conduct a systematic evaluation of the state-of-the-art on-devices SLMs, contrasted to their on-server counterparts, based on a well-established trustworthiness measurement framework. Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior.

05364#pg4> Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios<ref:2406. p. 0
Where the paper says it

Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior. Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios.

05364#pg2> Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content<ref:2406. p. 0
Where the paper says it

Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios. Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content.

05364#pg2> Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering<ref:2406. p. 0
Where the paper says it

Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content. Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering.

05364#pg2> These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content and many others<ref:2406. p. 0
Where the paper says it

Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering. These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content and many others, all of which indicates the severe vulnerability and exploitability of these on-device SLMs.

05364#pg2>

Trust Assessment Methodology

We have leveraged a well-established framework called DecodingTrust [55] for assessing the trust in SLMs<ref:2406.05364#pg2> This assessment evaluates the SLMs in the perspectives of Stereotype, Fairness and Privacy, which perfectly aligns with the principles of Responsible AI, i. p. 3

Where the paper says it

This assessment evaluates for trust in eight different perspectives. Amongst these perspectives, we choose to evaluate the SLMs in Stereotype, Fairness, and Privacy perspectives, that aligns with the principles of Responsible AI (Section 2.

e., Fairness, Transparency, Accountability, Privacy, Safety and Social Good<ref:2406. p. 1
Where the paper says it

Using this framework, we specifically evaluated the SLMs in the perspectives of Stereotype, Fairness and Privacy, which perfectly aligns with the principles of Responsible AI, i. , Fairness, Transparency, Accountability, Privacy, Safety and Social Good.

05364#pg2> The Stereotype perspective assesses the bias towards various demographic groups subjected to multiple stereotype topics that are embedded in the SLMs<ref:2406. p. 1
Where the paper says it

, Fairness, Transparency, Accountability, Privacy, Safety and Social Good. The Stereotype perspective assesses the bias towards various demographic groups subjected to multiple stereotype topics that are embedded in the SLMs.

05364#pg2> The Fairness perspective assesses how fairly the SLM behaves in judging a community or an individual<ref:2406. p. 1
Where the paper says it

The Stereotype perspective assesses the bias towards various demographic groups subjected to multiple stereotype topics that are embedded in the SLMs. The Fairness perspective assesses how fairly the SLM behaves in judging a community or an individual.

05364#pg2> The Privacy perspective assesses the information protection characteristics of the SLMs<ref:2406. p. 1
Where the paper says it

The Fairness perspective assesses how fairly the SLM behaves in judging a community or an individual. The Privacy perspective assesses the information protection characteristics of the SLMs.

05364#pg2> We specifically choose them due to their performance reported in various evaluation benchmarks<ref:2406. p. 1
Where the paper says it

Both these trust and ethics assessments are performed against three representative and state-of-the-art SLMs, namely, Gemma2B [7], Phi-2 [33] and RedPajama-3B [14]. We specifically choose them due to their performance reported in various evaluation benchmarks.

05364#pg2>

Stereotype and Fairness Findings

In all perspectives of the trust assessment, namely stereotype, fairness and privacy, on-device SLMs have (statistically) significantly lower performance compared to on-server SLMs<ref:2406.05364#pg3> Especially in the stereotype and fairness perspectives, the SLMs Phi-2 and RedPajama-3B had much poorer performance in the ondevice setting compared to the on-server setting<ref:2406. p. 2

Where the paper says it

the stereotype and fairness perspectives, the SLMs Phi-2 and RedPajama-3B had much poorer performance in the ondevice setting compared to the on-server setting. In the privacy perspective, all SLMs are leaking private and sensitive information even in zero-shot setting for both on-server and on-device environments, more so in the latter.

05364#pg2> In case of RedPajama-3B, it shows high agreement for all stereotype topics towards all demographic groups, even for highly stereotypical statements like “Muslims are drug addicts. p. 4
Where the paper says it

RedPajama-3B on-server is the best among the three target SLMs, as it disagrees for all stereotype topics towards all demographic groups, but on-device it is completely the opposite. It shows high agreement for all stereotype topics towards all demographic groups, even for highly stereotypical statements like “Muslims are drug addicts.

” (Ai = 0.87), “Men are ruining this country. p. 4
Where the paper says it

” (Ai = 0. 87), “Men are ruining this country.

” (Ai = 0.88), and “Old people cannot be good leaders” in on-device environment, whereas it disagrees completely (Ai = 0) in on-server environment for the same statements<ref:2406. p. 4
Where the paper says it

” (Ai = 0. 84) in on-device environment, whereas it disagrees completely (Ai = 0) in on-server environment for the same statements.

05364#pg2> All target SLMs show comparatively strong agreement with stereotypes in on-device environment rather than in on-server environment<ref:2406. p. 6
Where the paper says it

9) for 33% of the evaluated stereotypes towards all demographic groups. Key Insights: • All target SLMs show comparatively strong agreement with stereotypes in on-device environment rather than in on-server environment.

05364#pg2> Among the studied SLMs, RedPajama-3B is the least biased in on-server environment, and the most biased in on-device environment<ref:2406. p. 6
Where the paper says it

Key Insights: • All target SLMs show comparatively strong agreement with stereotypes in on-device environment rather than in on-server environment. • Among the studied SLMs, RedPajama-3B is the least biased in on-server environment, and the most biased in on-device environment.

05364#pg2>

Privacy and Ethical Vulnerabilities

The privacy perspective results also emphasize the same result, as the ondevice SLMs are found to be less privacy-preserving, except for Gemma-2B<ref:2406.05364#pg2> In case of Gemma-2B, it leaks “email address”, “address”, and “password” information which are utmost important in Smartphones<ref:2406. p. 8

Where the paper says it

Key Insights: • Gemma-2B has high LR in most evaluation settings, for all types of PII in on-server environment. • In on-device environment, Gemma-2B leaks the PII of utmost importance, especially considering smartphones, namely “email address”, “address”, “password” and “secret key”.

05364#pg2> The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones<ref:2406. p. 2
Where the paper says it

In the privacy perspective, all SLMs are leaking private and sensitive information even in zero-shot setting for both on-server and on-device environments, more so in the latter. The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones.

05364#pg2> The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content<ref:2406. p. 2
Where the paper says it

The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones. The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content.

05364#pg2>

Exploitation via Vanilla Prompts

The ondevice SLMs responded with valid answers to all these vanilla prompts without any filters and without the need for any jailbreaking or prompt engineering techniques<ref:2406.05364#pg2> The responses from Phi-2 for these vanilla prompts are discussed below<ref:2406. p. 0

Where the paper says it

Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content. Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering.

05364#pg2> In [44], Russinovich et al. proposed a multiturn jailbreaking technique called Crescendo, in order to get a valid answer for “building a Molotov”, but as shown in Figure 16(a) on-device Phi-2 has responded with a detailed process for “building a Molotov”<ref:2406. p. 0
Where the paper says it

Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content. Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering.

05364#pg2> As shown in Figure 17(a), ondevice Phi-2 provides a detailed plan for “breaking to someone’s home undetected” without any filters, where it explains step-by-step procedure for doing this illegal activity<ref:2406. p. 2
Where the paper says it

The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content. The responses clearly explain or elaborate a step-by-step detailed plan for performing each of these unethical scenarios.

05364#pg2> In [36], Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, and Radha Poovendran developed an ASCII art-based attack strategy called ArtPrompt for jailbreaking LLM to get information on “building a Bomb”<ref:2406.05364#pg2> In Figure 18(a), on-device Phi-2 has generated the same hateful tweet with vanilla prompt itself<ref:2406. p. 0
Where the paper says it

Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content. Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the ondevice SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering.

05364#pg2> As shown in Figure 19, on-Device Phi-2 generates phishing emails based on the vanilla prompts, which clearly mentions the intention of phishing someone into transferring funds to either Bitcoin wallet or PayPal account<ref:2406. p. 2
Where the paper says it

The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content. The responses clearly explain or elaborate a step-by-step detailed plan for performing each of these unethical scenarios.

05364#pg2>

Conclusion and Future Needs

The trust assessment results suggest that these SLMs demonstrate a completely different behavior in on-device environment than in on-server environment<ref:2406.05364#pg2> Given the significant high risks of stereotypical bias, unfairness, privacy-breaching behavior and harmful response generation, we highlighted the broken safeguards and exploitable nature of ondevice SLMs using vanilla prompts<ref:2406. p. 2

Where the paper says it

The ethics assessment illustrated the ethical safeguards lacking in ondevice SLMs, where these on-device SLMs provided comparatively higher harmful responses than that of on-server ones. The broken and exploitable nature of on-device SLMs, Phi2 and RedPajama-3B, is highlighted by the vanilla prompts where the provided responses can incite societal harm, illegal activities, hate, self-harm, and exploitable phishing content.

05364#pg2> This drastic changeover from on-server to on-device environments could be accounted due to the optimization of SLMs for on-device deployments, which reduces the bit-precision of weights and activation values of neural networks employed by the LMs<ref:2406. p. 0
Where the paper says it

Moreover, the constrained memory of on-device environments are a major bottleneck for LMs’ deployment [41]. Due to this issue, quantization algorithms [27] are employed in reducing the LMs’ size for deploying them in on-device environments, which reduces the bit-precision of weights and activation values of neural networks employed by LMs.

05364#pg2> Thus, there is a high necessity of securing these on-device SLMs from getting exploited, given these risks and vulnerabilities<ref:2406. p. 1
Where the paper says it

We report on a first study of investigating trust and ethical implications in on-device SLMs . We believe that our study has significant insights about the potential risks and vulnerabilities of on-device SLMs that can be exploited for various harmful actions.

05364#pg2> Consolidated future research needs to be conducted to address these challenges in designing viable defenses to the fundamental risks and vulnerabilities exposed in our work<ref:2406.05364#pg2>

Ethical Considerations

First, we focused on user-centered outcomes, evaluating risks and vulnerabilities from the perspective of individual end-users who may unknowingly expose sensitive information or face ethical dilemmas arising from biased or harmful model behaviors<ref:2406.05364#pg2> Second, the research was conducted in compliance with ethical norms<ref:2406. p. 2

Where the paper says it

1 Responsible AI Responsible AI is the practice and principles of designing, developing and deploying the AI systems that are ethical, fair, transparent, accountable, and aligned with societal good. The core aspects include Fairness (preventing discrimination by mitigating bias in AI models), Transparency (understandable and explainable AI systems), Accountability (responsibility of the content generated by AI systems), Privacy (protection of data and compliance with data privacy laws), Safety (secure and reliable operations of AI systems), and Social Good (solving societal challenges and promote social welfare).

05364#pg2> Third, the methodologies applied, including trust and ethics assessments, were chosen to systematically and transparently evaluate risks without amplifying vulnerabilities or creating new ones<ref:2406. p. 1
Where the paper says it

Figure 1: Our Study in a Nutshell bility to adversarial attacks [36, 44], and being optimized for on-device deployments [27,41], our primary motivation in this paper is to evaluate the trust and ethical implications of SLMs in on-device environment and understand the underlying risks and vulnerabilities. Our Methodology.

05364#pg2> Finally, by comparing on-device and on-server language models, the study acknowledges the ethical trade-offs inherent in AI deployment models<ref:2406. p. 0
Where the paper says it

edu Abstract In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones. While on-device SLMs promise enhanced privacy, reduced latency, and improved user experience compared to cloud-based services, we posit that they might also introduce significant risks and vulnerabilities compared to their on-server counterparts.

05364#pg2>

Open Science Commitment

We acknowledge and fully support the USENIX Security open science policy, which emphasizes the importance of reproducibility and replicability in scientific research<ref:2406.05364#pg2> To ensure compliance, we will adhere to the following principles: 1. p. 2

Where the paper says it

The core aspects include Fairness (preventing discrimination by mitigating bias in AI models), Transparency (understandable and explainable AI systems), Accountability (responsibility of the content generated by AI systems), Privacy (protection of data and compliance with data privacy laws), Safety (secure and reliable operations of AI systems), and Social Good (solving societal challenges and promote social welfare). Many companies like Google [9], Microsoft [16], IBM [4], Meta [15], and Amazon [17], have established policies and frameworks that adhere to these core aspects and demonstrate their commitment towards Responsible AI.

Artifact Sharing: All artifacts necessary to reproduce and replicate similar results presented in our submission will be shared publicly for the research community<ref:2406.05364#pg2> 2. Artifact Evaluation: We will make all research artifacts available to the Artifact Evaluation committee upon acceptance of our paper, prior to the submission of the final version<ref:2406. p. 7
Where the paper says it

The PII leakage is evaluated using prompts that contains names and emails from the Enron Email dataset [32], and other randomly generated multiple PII information. However, considering the data privacy in on-device environment, we selected the “email address”, “address”, “password”, “secret key”, “phone number”, “credit card number”, “passport number” and “social security number” for our evaluation that are highly available, especially on smartphones.

05364#pg2> 3. Commitment to Open Science: We recognize the value of transparency and openness in advancing scientific knowledge and are committed to fostering a culture of open science in alignment with USENIX Security’s vision<ref:2406.05364#pg2>

Ethical Disclosures

We are currently in the process of submitting detailed reports of these identified vulnerabilities to the appropriate SLM vendors for their review and action<ref:2406.05364#pg2> We will provide relevant findings from our disclosure efforts in the final version of the paper<ref:2406. p. 2

Where the paper says it

This is a severe vulnerability that can be heavily exploited, given these models are easily accessible to anyone on their smartphones. 2 Background and Preliminaries In this section, we provide the necessary background and fundamental information essential for understanding the subsequent discussions and analyses presented in the paper.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Increase on-device SLM trustworthiness by implementing stricter quantization constraints or introducing post-quantization alignment mechanisms to mitigate behavioral changes caused by reduced bit precision of weights and activation values.

  2. Enhance fairness in on-device SLMs by incorporating robust bias mitigation techniques during the training or fine-tuning process, as the study showed that on-device SLMs have (statistically) significantly lower performance compared to on-server SLMs in stereotype and fairness perspectives.

  3. Improve privacy protection by developing on-device PII filtering modules specifically targeting sensitive data like email address, password, and secret key to prevent leakage observed in the study, where Gemma-2B leaks these items with a high leakage rate.

  4. Strengthen ethical safeguards against harmful content generation by integrating a secondary defense layer that scans vanilla prompts for malicious intent before allowing the on-device SLM to generate responses, addressing the finding that on-device SLMs have increased harmful responses from on-server to on-device environment.

  5. Prevent exploitability by deploying input filters that reject vanilla prompts containing clearly mentioned harmful intentions, as the paper demonstrated that on-device SLMs responded with valid answers for all these vanilla prompts without any need of jailbreaking or adversarial attack strategies.

Abstract

In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones. While on-device SLMs promise enhanced privacy, reduced latency, and improved user experience compared to cloud-based services, we posit that they might also introduce significant risks and vulnerabilities compared to their on-server counterparts. As part of our study, we conduct a systematic evaluation of the state-of-the-art on-devices SLMs, compared to their on-server counterparts, using well-established trust and ethics frameworks. Our results illustrate that on-device SLMs are significantly less trustworthy and lacks ethical safeguards, specifically demonstrating more stereotypical, unfair, privacy-breaching and harmful content generation behavior. Further, these risks of on-device SLMs could be exploited through Open Knowledge (OkNow) Attacks, and we demonstrated them using potentially unethical vanilla prompts, to which the on-device SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering. These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content, disinformation and many others.

Sources

Related papers