PrivCode: When Code Generation Meets Differential Privacy
summary
The gist
Large language models have presented outstanding performance in code generation and completion, but fine-tuning them on private datasets can raise privacy and proprietary concerns such as the leakage…
In short
PrivCode is a novel differentially private synthesizer for code datasets designed to protect sensitive information while maintaining high utility. It uses a two-stage framework: first, a privacy-sanitizing stage that uses structural tokens to preserve code syntax under noise, and second, a utility-boosting stage that refines the model on the resulting synthetic data. This method ensures privacy without significantly degrading code generation performance.
Key concepts
- Differential Privacy (DP)
- DP is a mathematical framework providing a guarantee against privacy leakage when training models on sensitive data. It generates synthetic datasets that statistically resemble the original data but are mathematically designed to prevent the inference of any specific individual's private information, offering quantifiable privacy protection.
- Privacy-free Syntax-aware (PrivSA) module
- This module is part of the first stage where PrivCode extracts structural tokens from code snippets. It embeds 'adversarial code syntactic information' into the training process. This technique helps counteract how differential privacy noise can disrupt the essential structure of code, ensuring that the generated synthetic data still maintains correct programming syntax.
- Two-stage framework
- PrivCode employs a two-step training process: privacy-sanitizing and utility-boosting. The first stage focuses on learning code characteristics under strict privacy constraints using a smaller model to manage noise. The second stage takes the output from the first stage and fine-tunes a larger, premium LLM to maximize its performance on the synthetic code, separating privacy concerns from generation optimization.
- Abstract Syntax Tree (AST)
- An AST is a tree-like data structure that represents the exact syntactic structure of source code. It translates complex code into a hierarchical format, which is crucial because it allows models to understand and manipulate the underlying logic and structure of programming languages effectively.
Terminology used across episodes
This episode discusses
- PrivCode: When Code Generation Meets Differential Privacy · Paper Radio
- StarCoder: may the source be with you!
- Code Llama: Open Foundation Models for Code
- DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
- Scalable Extraction of Training Data from (Production) Language Models
- Evaluating Large Language Models Trained on Code
- Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe
- DPImageBench: A Unified Benchmark for Differentially Private Image Synthesis
- LoRA: Low-Rank Adaptation of Large Language Models
- Differentially Private Diffusion Models
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs
- Unsupervised Evaluation of Code LLMs with Round-Trip Correctness
- Qwen2.5-Coder Technical Report
- CodeGemma: Open Code Models Based on Gemma
- Program Synthesis with Large Language Models
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions · Paper Radio
- Finetuned Language Models Are Zero-Shot Learners
- CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
- Sequence to Sequence Learning with Neural Networks
- Hierarchical Neural Story Generation
- Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
The paper
PrivCode: When Code Generation Meets Differential Privacy · Read on arXiv
Zheng Liu, Chen GongB, Terry Yue Zhuo, Kecen Li, Weichen Yu, Matt Fredrikson, Tianhao Wang
University of Virginia · Monash University and CSIRO’s Data61
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "PrivCode: When Code Generation Meets Differential Privacy".
Elias: Here's the gist of the paper. Large language models have presented outstanding performance in code generation and completion, but fine-tuning them on private datasets can raise privacy and proprietary concerns such as the leakage of sensitive personal information The paper proposes PrivCode, the first DP synthesizer specifically designed for code datasets, which incorporates a two-stage framework to improve both privacy and utility The core problem addressed The development of large language models (LLMs) has propelled code intelligence into a new era, but fine-tuning these models on proprietary and sensitive code datasets is common Previous works show that LLMs can memorize content from the training dataset and output it during inference For code LLMs, CodexLeaks found that Codex can reproduce code snippets in verbatim from its training set that contain Personally Identifiable Information (PII) The authors aim to protect the privacy of code datasets through differentially private (DP) code generation, which generates artificial data preserving statistical properties while reducing privacy leakage concerns How the training signal is built PrivCode introduces a two-stage framework consisting of “privacy-sanitizing” and “utility-boosting” stages In the first stage, termed “privacy-sanitizing”, PrivCode generates DP-compliant synthetic code by training models using DP-SGD while introducing syntactic information to preserve code structure This involves a “Privacy-free SyntaxAware (PrivSA) module” that extracts structural tokens from code snippets, embedding termed adversarial code syntactic information to counteract the disruption of code structure caused by DP noise The second stage, “utility-boosting,” fine-tunes a larger pre-trained LLM on the synthetic privacy-free code to mitigate the utility loss caused by DP Where the gains come from The two-stage paradigm separates privacy preservation and generation optimization, with operations on DP-compliant outputs not incurring additional privacy cost The utility-boosting stage refines the code via post-processing filters to fine-tune a premium LLM without any DP constraints, thereby avoiding utility degradation Ablation studies emphasize the importance of incorporating the PrivSA module into DP fine-tuning and the evolutionary paradigm Comprehensive evaluations show that PrivCode outperforms the baseline across five benchmarks and achieves performance close to methods without any privacy protection mechanisms Robustness and validation Execution validation is essential for filtering out low-quality code, involving heuristic language identification, environment capability probing, and sandboxed multi-language runtime execution Round-trip validation uses a powerful pre-training model to summarize generated code into natural language descriptions and computes semantic similarity using BERTScore to filter out irrelevant snippets Canary experiments confirm that PrivCode achieves a 0% leakage rate across all repetition rates under a strict privacy budget of ϵ = 4, in contrast to the maximum 100% leakage rate observed in no-DP methods The paper concludes that PrivCode effectively preserves the utility of fine-tuning knowledge while ensuring privacy protection Hyper-parameter analysis The optimal parameter settings for the PrivSA module, such as max lambda, vary across different trained models and datasets The results show that increasing max lambda can improve pass@1 scores up to a certain point, but overly large values cause the model to over-focus on local syntactic information and hinder global task-following capabilities The analysis of validation filters indicates that both execution validation and round-trip validation are necessary to ensure syntactic correctness and semantic fidelity The final utility of PrivCode initially increases with the BERTScore threshold τs but then decreases beyond an optimal point, suggesting a trade-off between dataset scale and semantic fidelity Ethics and conclusion The paper proposes PrivCode as the first DP code synthesizer, providing a paradigm for learning specific knowledge from sensitive code datasets under the DP framework All used models and datasets are publicly available, and highlighted information is sourced from the open-source PII dataset This work aims to promote the secure and responsible sharing of code datasets under DP, further advancing research in code LLMs The authors recommend using a relatively small epsilon, such as 4, which offers sufficient protection for private information while preserving high utility Page 1 --- PrivCode: When Code Generation Meets Differential Privacy Zheng Liu, Chen GongB, Terry Yue Zhuo†, Kecen Li, and, Weichen Yu‡, Matt Fredrikson‡, Tianhao Wang University of Virginia †Monash University and CSIRO’s Data61 ‡Carnegie Mellon University Abstract—Large language models (LLMs) have presented outstanding performance in code generation and completion.
Nadia: However, fine-tuning these models on private datasets can raise privacy and proprietary concerns, such as the leakage of sensitive personal information Differentially private (DP) code generation provides theoretical guarantees for protecting sensitive code by generating synthetic datasets that preserve statistical properties while reducing privacy leakage concerns However, DP code generation faces significant challenges due to the strict syntactic dependencies and the privacy-utility trade-off We propose PrivCode, the first DP synthesizer specifically designed for code datasets I.
Elias: INTRODUCTION The development of large language models (LLMs) has propelled code intelligence into a new era, and advanced opensource code LLMs have presented phenomenal performance To better cater to downstream programming scenarios requiring domain-specific expertise, it is common to fine-tune code LLMs on proprietary and sensitive code datasets Previous works show that LLMs can memorize content from the training dataset and output it during inference For code LLMs, CodexLeaks found that Codex can reproduce code snippets in verbatim from its training set that contain Personally Identifiable Information (PII) We aim to protect the privacy of code datasets through differentially private (DP) code generation, an approach that generates artificial data preserving the statistical properties of real data while protecting individual privacy II.
Nadia: BACKGROUNDS This section introduces the brief concepts of LLM-based prompted code generation, abstract syntax trees in code, and the challenges of DP code generation Abstract Syntax Tree (AST) is a tree-structured data representation of the syntactic structure of source code snippet ASTs are widely utilized in compilers, interpreters, and vulnerability detection because they enable a structured understanding of code Privacy leakage in code generation is a concern because automatically generated code could inadvertently expose private information if it reflects sensitive data from the training corpus Differential Privacy (DP) provides a theoretical guarantee to measure privacy leakage in the synthesizer’s output, quantifying the risk of revealing private information about real code snippets from synthetic code while still allowing public prompts to facilitate queries and operations III.
Paper summary: Nadia: So, looking at the paper "PrivCode: When Code Generation Meets Differential Privacy," the main idea is that they created a two-stage training framework to handle privacy and utility simultaneously when fine-tuning large language models on code >
Elias: The title itself says "When Code Generation Meets Differential Privacy," and what it means in simple terms is they’re showing you how you can use differential privacy to train these models on private code data without leaking sensitive personal information >
Priya: What this actually changes for us, as researchers or developers, is that we might be able to learn specific knowledge from sensitive code datasets while still keeping those datasets secure and compliant with privacy rules >
Nadia: It provides a concrete way to build secure AI tools for code generation that don't rely on the dangerous assumption that fine-tuning means exposing everything in the training corpus >
Elias: The implication is that we can move forward with learning from proprietary code, which is usually restricted, but we do it under a differential privacy framework >
Conclusion: Nadia: So we've been looking at PrivCode, this paper about using differential privacy for code synthesis >
Elias: It’s about taking those large language models that are so good at writing code and fine-tuning them on private data without letting sensitive personal information leak out >
Priya: The authors propose a two-stage process to do that, trying to keep the utility high while making sure the privacy protection actually works >
Nadia: Yeah, they call it a privacy sanitizing stage and a utility boosting stage, which sounds like they're trying to solve two different problems at once >
Elias: Exactly. The first stage uses a smaller model with differential privacy stochastic gradient descent to learn the code structure while adding some kind of adversarial syntactic information >
Priya: So, what does that adversarial info do? Is it just noise, or is it actually helping the model keep the code structure intact when you add all that DP noise >
Nadia: It’s trying to counteract the disruption from the DP noise so that when they move to stage two, the larger premium LLM doesn't completely lose track of how code actually works >
Elias: Right. And in stage two, they fine-tune a much bigger model on that synthetic privacy-free code, which is supposed to fix the utility loss from the first step >
Priya: But what about the actual numbers? They claim zero leakage rate across repetition rates under a specific privacy budget of epsilon equals four >
Nadia: That’s a huge claim, especially when no-DP methods can leak up to one hundred percent of the time >
Elias: It suggests that this DP approach isn't just theoretical; they show empirical validation where it actually beats the baseline methods on benchmarks like HumanEval and MBPP >
Priya: I'm still looking at the hyperparameter analysis, because they mentioned that increasing a parameter called lambda can improve scores up to a point before it starts hurting global task-following >
Nadia: That tells you there’s a sweet spot, and if you go too far on that lambda setting, the model starts getting stuck on local syntax instead of actually solving the whole problem >
Elias: It really highlights that balancing those two things—keeping privacy tight and keeping utility high—is a tricky trade-off they’re trying to navigate >
Priya: So, for someone just listening, it means there's a way to use proprietary code knowledge for training AI without necessarily having to give away the sensitive source material itself >
Nadia: It’s about making the process of learning from private code more secure, which is a big deal for how we build these tools in practice >
Elias: And they recommend using a small epsilon value, like four, because it offers enough protection while still keeping that high utility we saw in the experiments >
Priya: That leads us into what the authors suggest for the future of this research and where they think this approach can go next >
More episodes
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18
- 2511.19218-From Strategy Search to Policy Learning: MCTS-Guided Adversarial Co-Evolution for LLM Safety Alignment