Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models

arXiv:2604.01444 · cs.CR · Submitted 2026-04-01 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.

Nadia: I'm Nadia, and with me are Elias and Priya, guest researcher.

Elias: Today's paper: "Cooking Up Risks".

Nadia: The gist The introduction introduces FoodGuardBench, a comprehensive benchmark comprising 3,339 queries grounded in FDA guidelines, designed to evaluate the safety and robustness of LLMs within food-related domains.

Elias: First, who's behind it and why it matters.

Title and authors: Nadia: So we're talking about "Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models." This paper introduces the concept of testing LLMs specifically against food safety risks based on FDA rules.

Elias: Right, they built this benchmark with three thousand three hundred thirty-nine queries to see how robust these models are when it comes to food hazards using a structured approach.

Nadia: The authors used a specific taxonomy of food safety—things like temperature control and contamination—to organize the risks systematically for evaluation <ref:2604.01444#pg1>.

Priya: I’m thinking about what that means for the data; they aren't just guessing whether something is safe or unsafe; they are labeling it by the actual type of hazard <ref:2604.01444#pg2>.

Elias: Exactly. They used that structure to make both benign queries, which are normal user questions, and malicious ones that represent the baseline for what bad actors are trying to exploit <ref:2604.01444#pg1>.

Nadia: And they created these malicious prompts using two different methods: an automated attack called AutoDAN and one using psychological persuasion called PAP <ref:2604.01444#pg3>.

Priya: That’s interesting because they aren't just asking if something is safe; they are testing if the AI can be tricked into giving out instructions that could cause real problems in the kitchen <ref:2604.01444#pg2>.

Elias: The experimental setup included manually checking a hundred random samples for alignment with core safety concepts and then filtering out queries that were too similar to make sure the test was focused <ref:2604.01444#pg2>.

Nadia: Then they ran this evaluation using Attack Success Rate or ASR, which measures how often a harmful prompt gets through undetected alongside metrics like Accuracy and F1 score <ref:2604.01444#pg3>.

Priya: What the data really shows is that current large language models have very thin safety alignment in this specific food science area and they fall for just a few standard jailbreak methods pretty easily <ref:2604.01444#pg3>.

Elias: When they get tricked, these models don't just say no; they actually generate instructions that are useful but dangerous, which empowers people who want to cause harm <ref:2604.01444#pg3>.

Nadia: This is where the authors point out that existing guardrails systematically miss these specific types of domain-related threats entirely <ref:2604.01444#pg3>.

Priya: That failure to detect is a big deal because food safety isn't just a general safety issue; it involves very specific protocols and procedures you need to follow <ref:2604.01444#pg2>.

Elias: To fix this, they introduce FoodGuard-4B, which is a specialized guardrail model fine-tuned specifically on all that data to catch those latent food science hazards <ref:2604.01444#pg3>.

Nadia: They claim this specialized model drastically cuts down the False Negative Rate to just two point seven five percent while keeping the False Positive Rate very low at two point zero one percent, which gives it an overall F1 score of ninety-seven point one zero percent <ref:2604.01444#pg3>.

Priya: That FNR reduction is significant because it means fewer real-world dangers slip through when you use these systems for food guidance <ref:2604.01444#pg3>.

Elias: It shows that targeted training on the actual domain knowledge makes a huge difference compared to just using general safety layers <ref:2604.01444#pg3>.

Nadia: So, the paper really drives home that we need to move past general alignment and focus on building guardrails tuned precisely for high-stakes areas like food preparation <ref:2604.01444#pg3>.

Priya: It’s about making sure that when an AI is helping someone with food, it’s actually following the specific safety science required <ref:2604.01444#pg3>.

Elias: We'll look at what kind of adversarial attacks might bypass this new FoodGuard-4B later on <ref:2604.01444#pg3>.

The paper's summary: Nadia: So, "Cooking Up Risks" summarizes how they built a test for LLMs focused entirely on food safety risks using FDA rules as the foundation.

Elias: Right, they built this benchmark with three thousand three hundred thirty-nine queries to see how robust these models are when it comes to food hazards using a structured approach <ref:2604.01444#pg1>.

Nadia: The authors used a specific taxonomy of food safety—things like temperature control and contamination—to organize the risks systematically for evaluation <ref:2604.01444#pg2>.

Priya: I’m looking at what that means for the data; they aren't just guessing if something is safe or unsafe; they are labeling it by the actual type of hazard <ref:2604.01444#pg2>.

Elias: Exactly. They used that structure to make both benign queries, which are normal user questions, and malicious ones that represent the baseline for what bad actors are trying to exploit <ref:2604.01444#pg1>.

Nadia: And they created these malicious prompts using two different methods: an automated attack called AutoDAN and one using psychological persuasion called PAP <ref:2604.01444#pg3>.

Priya: That’s interesting because they aren't just asking if something is safe; they are testing if the AI can be tricked into giving out instructions that could cause real problems in the kitchen <ref:2604.01444#pg2>.

Elias: The experimental setup included manually checking a hundred random samples for alignment with core safety concepts and then filtering out queries that were too similar to make sure the test was focused <ref:2604.01444#pg2>.

Nadia: Then they ran this evaluation using Attack Success Rate or ASR, which measures how often a harmful prompt gets through undetected alongside metrics like Accuracy and F1 score <ref:2604.01444#pg3>.

Priya: What the data really shows is that current large language models have very thin safety alignment in this specific food science area and they fall for just a few standard jailbreak methods pretty easily <ref:2604.01444#pg3>.

Elias: When they get tricked, these models don't just say no; they actually generate instructions that are useful but dangerous, which empowers people who want to cause harm <ref:2604.01444#pg3>.

Nadia: This is where the authors point out that existing guardrails systematically miss these specific types of domain-related threats entirely <ref:2604.01444#pg3>.

Priya: That failure to detect is a big deal because food safety isn't just a general safety issue; it involves very specific protocols and procedures you need to follow <ref:2604.01444#pg2>.

Elias: To fix this, they introduce FoodGuard-4B, which is a specialized guardrail model fine-tuned specifically on all that data to catch those latent food science hazards <ref:2604.01444#pg3>.

Nadia: They claim this specialized model drastically cuts down the False Negative Rate to just two point seven five percent while keeping the False Positive Rate very low at two point zero one percent, which gives it an overall F1 score of ninety-seven point one zero percent <ref:2604.01444#pg3>.

Priya: That FNR reduction is significant because it means fewer real-world dangers slip through when you use these systems for food guidance <ref:2604.01444#pg3>.

Elias: It shows that targeted training on the actual domain knowledge makes a huge difference compared to just using general safety layers <ref:2604.01444#pg3>.

Nadia: So, the paper really drives home that we need to move past general alignment and focus on building guardrails tuned precisely for high-stakes areas like food preparation <ref:2604.01444#pg3>.

Priya: It’s about making sure that when an AI is helping someone with food, it’s actually following the specific safety science required <ref:2604.01444#pg3>.

Elias: We'll look at what kind of adversarial attacks might bypass this new FoodGuard-4B later on <ref:2604.01444#pg3>.

The paper's improvements: Nadia: So, looking at how the authors think we should improve these systems based on their findings in "Cooking Up Risks," they suggest a few main things for future safety work.

Elias: I think they are focused on introducing those specialized guardrails, like FoodGuard-4B, which is fine-tuned directly on the food safety data <ref:2604.01444#pg3>.

Priya: What that means practically is that instead of a general AI filter, you get a model specifically trained to recognize those subtle food science hazards we talked about earlier <ref:2604.01444#pg3>.

Nadia: Right, and they are also pushing for better adversarial robustness by creating more scalable query generation pipelines using AutoDAN and PAP <ref:2604.01444#pg3>.

Elias: They want to generate prompts that are high complexity so you can rigorously probe the AI's threat detection capabilities against a wider range of attack strategies <ref:2604.01444#pg3>.

Priya: I see the implication there is that we need to treat adversarial testing not just as a quick check, but as a continuous process using these sophisticated methods <ref:2604.01444#pg3>.

Nadia: They also emphasized targeting safety alignment specifically through that food safety taxonomy we built, making sure the AI understands those scientific protocols like temperature control and allergen handling <ref:2604.01444#pg1>.

Elias: That aligns with my concern about parameter assumptions; they are pushing for models to adhere to specific constraints, not just general helpfulness <ref:2604.01444#pg3>.

Priya: So what this changes for someone listening is that it means future LLM safety won't just be about adding a simple blacklist, but building deep domain-specific knowledge into the guardrail itself <ref:2604.01444#pg3>.

Nadia: Exactly. It shifts the focus from broad safety to highly targeted alignment within sensitive areas like food preparation and storage <ref:2604.01444#pg3>.

Elias: It means that for a model to be trusted in this area, it has to pass these specific, tailored tests against known attack vectors before deployment <ref:2604.01444#pg3>.

Priya: So the big picture here is moving towards a system where the safety guardrail itself is smarter and more specialized for high-stakes domains like food science <ref:2604.01444#pg3>.

Nadia: And that brings us to another important question: given this new benchmark, how do we actually ensure that these new specialized models don't just learn a few tricks and then get bypassed by something entirely new?

Elias: That’s a valid point. We still need to keep testing because vulnerabilities can change, and there are still many other attack modes out there <ref:2604.01444#pg3>.

Priya: I think the next thing we should watch is how these food-specific guardrails hold up against completely novel, unseen adversarial attacks <ref:2604.01444#pg3>.

Nadia: So we’ve seen how they set up the test, what their numbers show about vulnerability, and how they propose fixing it with specialized models in "Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models" <ref:2604.01444#pg1>.

Elias: It's a solid paper because it gives us a concrete tool for evaluating security in these niche areas rather than just guessing at general safety <ref:2604.01444#pg3>.

Priya: I think the next thing we should watch is how these food-specific guardrails hold up against completely novel, unseen adversarial attacks <ref:2604.01444#pg3>.

Conclusion: Nadia: So we're wrapping up on "Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models."

Elias: Basically, they showed how current LLMs are vulnerable to a few specific jailbreak tactics when dealing with food safety information.

Priya: The big implication is that we can't treat food safety as just another general prompt injection problem; it requires specialized alignment.

Nadia: They provided FoodGuard-4B, which is this new guardrail model they fine-tuned to catch those domain-specific hazards better than anything before.

Elias: That specialized model significantly lowers the False Negative Rate, meaning fewer dangerous instructions slip through when you’re dealing with food queries.

Priya: It shows that measurement and targeted training on specific taxonomies makes a real difference in terms of safety performance for these kinds of applications.

Nadia: The authors really push the idea that existing guardrails systematically miss these domain-specific threats, so we have to get smarter about what we're testing for.

Elias: And they set up this whole benchmark pipeline, which is crucial because it gives us a standard way to measure if our new guardrails are actually working in the real world.

Priya: What this changes for someone listening is that it means future AI deployment in high-stakes areas needs more granular, domain-specific testing before anyone trusts the output.

Nadia: So we’ve seen how they set up the test, what their numbers show about vulnerability, and how they propose fixing it with specialized models.

Elias: It's a solid paper because it gives us a concrete tool for evaluating security in these niche areas rather than just guessing at general safety.

Priya: I think the next thing we should watch is how these food-specific guardrails hold up against completely novel, unseen adversarial attacks.

University of Georgia · University of California, Davis · Johns Hopkins University · University of Southern California · Rutgers University

cs.CR

Submitted: 2026-04-01

Updated: 2026-10-13

Code: https://github.com/tenghaohuang/FoodGuardBench

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 84/100

The gist: The gist The introduction introduces FoodGuardBench, a comprehensive benchmark comprising 3,339 queries grounded in FDA guidelines, designed to evaluate the safety and robustness of LLMs within

Key concepts

Food Safety Taxonomy
This is a structured system based on FDA guidelines that categorizes food safety risks into specific dimensions like temperature control, contamination, and allergens. It allows researchers to systematically check if models struggle more with certain types of food hazards.
Adversarial Query Generation
The authors used two methods, AutoDAN and PAP, to create malicious prompts. AutoDAN generates stealthy prompts for white-box models, while PAP uses psychological persuasion techniques to create natural-sounding requests for black-box models. This tests different ways users might try to trick the AI.
FoodGuardBench
This is a new benchmark dataset containing 3,339 queries derived from FDA rules. It includes benign and malicious questions across various food safety risks, allowing for rigorous testing of LLMs and guardrails in the food domain.

Terminology

Summary

The gist The introduction introduces FoodGuardBench, a comprehensive benchmark comprising 3,339 queries grounded in FDA guidelines, designed to evaluate the safety and robustness of LLMs within food-related domains.

Food Safety Taxonomy

To support structured evaluation, the authors derive a food safety taxonomy grounded in FDA food safety code (U.S. Food and Drug Administration, 2022) and organize unsafe behavior into a set of recurring risk dimensions The taxonomy includes categories such as Temperature control, Contamination, Hygiene, Storage, Preparation, Allergens, Pest control; Water safety This taxonomy allows them to go beyond aggregate safe-versus-unsafe judgments By labeling examples according to hazard type, they can analyze whether certain safety dimensions are systematically more difficult for models

Dataset Construction

The FoodGuardBench is constructed through a controlled generation pipeline grounded in food safety taxonomy and policies They begin from a curated food safety taxonomy derived from FDA-aligned guidance and common food handling principles They use a total of 356 unique seed policies, generating 5 queries by GPT-5 for each seed policy under each category of food safety taxonomy The process involves two primary intent classes: Benign intent and Malicious intent Benign queries reflect ordinary users seeking practical help in everyday food-related situations Malicious queries represent the fundamental baseline of questions posed by adversarial users who seek to exploit LLMs as informational guides for inflicting real-world harm against safety concepts in FDA food safety code Quality control involves manually inspecting 100 random samples to verify alignment with core safety concepts and relevance to real-world application scenarios They also adopt the filtering methodology from RedTeam-2K to identify and remove overly similar queries, resulting in an average semantic similarity across the final generated 2,339 vanilla harmful queries of 0.35

Adversarial Query Generation

Malicious prompts are instantiated into jailbreak queries using two efficient and representative jailbreak methods: AutoDAN (Liu et al., 2024) and Persuasive Adversarial Prompting (PAP; Zeng et al. 2024) AutoDAN is an automated jailbreak attack method that generates stealthy, semantically meaningful prompts to compromise aligned LLMs by applying a hierarchical genetic algorithm PAP utilizes a taxonomy of 40 psychological persuasion techniques to automatically paraphrase harmful queries into natural, human-readable requests These two attacks probe different attack modes: AutoDAN targets susceptibility to optimized prompt attacks on white-box models, whereas PAP targets susceptibility to socially framed persuasion on black-box models Together, they provide a compact benchmark for evaluating lower-bound jailbreak risk in food-related LLM safety

Evaluation and Findings

The evaluation employs two sets of metrics: Attack Success Rate (ASR) to measure vulnerability within food science contexts, alongside Accuracy, F1 score, False Positive Rate (FPR), and False Negative Rate (FNR) ASR is defined as the proportion of harmful prompts that evade detection and are misclassified as safe Results show that current LLMs exhibit sparse safety alignment in the food-related domain, easily succumbing to a few canonical jailbreak strategies When compromised, LLMs frequently generate actionable yet harmful instructions, inadvertently empowering malicious actors and posing tangible risks Existing LLM-based guardrails systematically overlook these domain-specific threats FoodGuard-4B demonstrates exceptional detection performance by reducing the FNR to a mere 2.75% while achieving the lowest FPR of 2.01%, resulting in an overall F1 score of 97.10%

Conclusion

The work introduces FoodGuardBench for evaluating the safety and robustness of LLMs in the food-related context, along with a scalable adversarial query generation pipeline They propose FoodGuard-4B, a specialized guardrail model specifically engineered to fortify LLMs against malicious input in food-related contexts Their results demonstrate that existing LLMs and guardrails remain vulnerable and lack robust domain-specific safety alignment The study highlights the urgent need for targeted safety alignment to mitigate these domain-specific vulnerabilities This work motivates further research on safety in everyday, high-impact domains.

--- Page 1 ---

Preprint

Cooking Up Risks: Benchmarking and Reducing Food Safety Risks in Large Language Models Weidi Luo Xiaofei Wen Tenghao Huang Hongyi Wang Zhen Xiang Chaowei Xiao Kristina Gligori´c Muhao Chen University of Georgia University of California Davis Johns Hopkins University University of Southern California Rutgers University

Abstract Large language models (LLMs) are increasingly deployed for everyday tasks, including food preparation and health-related guidance. However, food safety remains a high-stakes domain where inaccurate or misleading information can cause severe real-world harm. Despite these risks, current LLMs and safety guardrails lack rigorous alignment tailored to domainspecific food hazards. To address this gap, we introduce FoodGuardBench, the first comprehensive benchmark comprising 3,339 queries grounded in FDA guidelines, designed to evaluate the safety and robustness of LLMs. By constructing a taxonomy of food safety principles and employing representative jailbreak attacks (e.g., AutoDAN and PAP), we systematically evaluate existing LLMs and guardrails. Our evaluation results reveal three critical vulnerabilities: First, current LLMs exhibit sparse safety alignment in the food-related domain, easily succumbing to a few canonical jailbreak strategies. Second, when compromised, LLMs frequently generate actionable yet harmful instructions, inadvertently empowering malicious actors and posing tangible risks. Third, existing LLM-based guardrails systematically overlook these domain-specific threats, failing to detect a substantial volume of malicious inputs. To mitigate these vulnerabilities, we introduce FoodGuard-4B, a specialized guardrail model fine-tuned on our datasets to safeguard LLMs within food-related domains. https://github.com/tenghaohuang/FoodGuardBench

1 Introduction Large language models (LLMs) are increasingly being used to support everyday decisionmaking, including tasks related to food preparation, storage, health guidance, and even the design of new food products (Ngo et al., 2025; Starke et al., 2025; Huang et al., 2025; Datta et al., 2025; Kuhl, 2025) Their broad knowledge, ease of access, and conversational fluency make them attractive assistants for practical, real-world use. However, as these systems are deployed in domains with direct consequences for human well-being, concerns about safety and reliability become more pressing. Food safety is a particularly high-stakes example: incorrect guidance about storage, preparation, or consumption can lead to contamination, foodborne illness, allergic reactions, or other serious health harms. Prior research has shown that LLM suggestions can be unsafe in high-stakes domains, including health and decision support (Buc¸inca et al., 2021; Sallam, 2023), as well as other settings where fluent responses may conceal factual or procedural errors (Bender et al., 2021; Ji et al., 2023b). Yet this line of analysis remains underexplored in the context of food safety, despite the fact that food-related interactions are both common and consequential, as shown in figure 1. consequential all depend on the characteristics of the application domain. In food-related contexts, unsafe advice may appear practical and benign (Lou et al., 2026) while violating core principles of food safety (Wang et al., 2025; Hassani et al., 2025), including temperature control (Koutsoumanis & Gougouli, 2015), cross-contamination prevention (Kasza et al., 2022), allergen handling (Cavazza et al., 2023), or spoilage management (Joshi et al., 2015b). Such risks are often subtle and context-dependent, making them difficult to detect without explicit attention to domain-specific safety requirements. In other words, the absence of obvious harmful intent does not imply the absence of risk.

--- Page 2 ---

Preprint

Figure 1: Consumer survey results indicate substantial openness to GenAI for food-related assistance, with meal planning and menu suggestions (47%), personalized nutrition and diet plans (45%), and grocery budgeting support (41%) among the most accepted use cases; only 15% of respondents selected none of the above. Source: PwC Voice of the Consumer Survey 2025.

--- Page 3 ---

Preprint

Overall, our threat model reflects a realistic deployment setting where models must provide helpful food-related assistance to ordinary users while remaining robust against attempts by adversarial users to elicit dangerous guidance.

--- Page 4 ---

Preprint

Improvements for AI systems

  1. Improve safety alignment by introducing domain-specific guardrails like FoodGuard-4B, which is fine-tuned on our curated dataset to safeguard LLMs within food-related domains. This specialized model is designed to detect latent food science hazards and has been shown to achieve a significantly lower False Negative Rate (FNR) compared to existing guardrails.

  2. Enhance adversarial robustness by developing a scalable adversarial query generation pipeline that synthesizes realistic, high-complexity prompts to rigorously probe an LLM’s threat-detection capabilities. This pipeline utilizes canonical attacks like AutoDAN and PAP to establish a lower bound on domain-specific vulnerability.

  3. Increase model reliability by implementing targeted safety alignment based on the food safety taxonomy, which includes dimensions like Temperature control, Contamination, and Allergens. This ensures that models are not only general assistants but also adhere to specific, scientific protocols relevant to high-stakes domains.

Sources

Related papers