HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption
summary
The gist
Fully Homomorphic Encryption (FHE) enables computation directly on encrypted data and privacy-preserving neural inference in the cloud The paper presents HE-LRM…
In short
The paper introduces HE-LRM, a solution for running recommendation models using Fully Homomorphic Encryption (FHE) on encrypted data. It tackles the challenge of performing private lookups into large embedding tables required by sparse inputs in deep learning models. The method achieves significant speedups through client-side embedding compression and optimized lookup strategies, enabling inference in seconds on specialized hardware.
Key concepts
- Fully Homomorphic Encryption (FHE)
- A cryptographic technique that allows computations to be performed directly on encrypted data without ever decrypting it. This ensures complete privacy, as the server processes the sensitive information while it remains scrambled. It enables complex mathematical operations like addition and multiplication on these encrypted values.
- Embedding Compression
- A client-side technique used to reduce the size of embedding vectors before encryption. By mapping original tokens to smaller, deterministic coded tokens using digit decomposition, this method significantly shrinks the data that needs to be processed in FHE, leading to a 56x speedup.
- Matrix-Vector Lookup
- One optimization strategy for finding a dense vector from an encrypted embedding matrix. This approach performs a dense matrix-vector product using only one multiplicative level, which is computationally efficient and minimizes the overhead associated with accessing the correct row in the encrypted table.
- Bootstrapping
- The most computationally demanding operation within FHE schemes like CKKS. It is necessary to refresh or 're-encrypt' ciphertexts to prevent noise accumulation during complex computations. This process is extremely slow, taking about 20 seconds on a single-threaded CPU.
Terminology used across episodes
This episode discusses
- HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption · Paper Radio
- The Architectural Implications of Facebook's DNN-based Personalized Recommendation
- Deep Learning Recommendation Model for Personalization and Recommendation Systems
- Attention Is All You Need
- Osiris: A Systolic Approach to Accelerating Fully Homomorphic Encryption
- TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
- Privacy-Preserving Text Classification on BERT Embeddings with Homomorphic Encryption
- THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
- DQRM: Deep Quantized Recommendation Models
- Post-Training 4-bit Quantization on Embedding Tables
- Learning to Collide: Recommendation System Model Compression with Learned Hash Functions
- TenSEAL: A Library for Encrypted Tensor Operations Using Homomorphic Encryption
- SEALion: a Framework for Neural Network Inference on Encrypted Data
- HeLayers: A Tile Tensors Framework for Large Neural Networks on Encrypted Data
The paper
HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption · Read on arXiv
New York University · LG Electronics USA, Inc.
FHE inference systems primarily target models with dense inputs, while Deep Learning Recommendation Models (DLRMs) contain sparse categorical features that require lookups into embedding tables. Encrypted lookups are challenging because indexing is not natively supported and must be expressed using SIMD arithmetic. A direct one-hot formulation does not scale. Prior work addresses parts of this problem through embedding compression or compact encrypted queries. However, it does not address complete DLRM inference, leaving open how private lookup interacts with downstream computation and memory constraints. We present HE-LRM, the first end-to-end FHE system for private DLRM inference. We use deterministic compositional embedding compression with client-side digit decomposition, avoiding expensive homomorphic selector construction while consuming only one multiplicative level. We further develop a block-diagonal packing strategy that performs lookups across multiple embedding tables using optimized BSGS linear transformations and produces encrypted embeddings directly compatible with downstream inference. We extend Orion with a high-performance GPU backend and achieve 9 ms for single-inference embedding lookup. We also show that GPU memory capacity limits batching and therefore the latency-bound throughput in such systems. On Criteo, HE-LRM achieves complete encrypted inference with server-side latencies of 154-247 ms on a single GPU. We open-source HE-LRM at https://github.com/baahl-nyu/orion
Transcript
Introduction to the show: ident: Security Radio. Generated commentary on the latest security and cryptography papers.
Nadia: We're looking at the paper: "HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption".
Paper summary: Nadia: We’ve been looking at HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption. To recap, this paper tackles the challenge of performing private lookups into massive embedding tables required by deep learning recommendation models when you are using fully homomorphic encryption.
Elias: The thesis is that existing solutions face significant bottlenecks because exact one-hot lookups require vectors whose length matches the number of rows in each embedding table, which leads to huge data uploads for clients and massive server side memory requirements for the embedding matrices themselves <ref:2506.18150#pg2>.
Priya: So, what's the core claim about how they solve that? Are they just using standard FHE operators directly?
Nadia: No, they propose an embedding compression technique using client side digit decomposition that achieves a fifty-six times speedup over the state of the art <ref:2506.18150#pg1>. They also introduce a multi embedding packing strategy to operate over multiple tables in parallel.
Elias: That packing strategy is key because it’s designed to maximize how much ciphertext slot utilization they get, and this strategy works with both compressed and uncompressed embedding tables.
Priya: That sounds like a system level optimization that tackles the resource constraints of FHE, rather than just focusing on the pure cryptographic primitives.
Nadia: Right. And they also examined two distinct lookup approaches: matrix vector lookup which uses only one multiplicative level, versus column wise table multiplication which introduces ciphertext materialization and packing overhead via an extra multiplicative level <ref:2506.18150#pg3>.
Elias: The trade off there is between simplicity in terms of homomorphic levels and the actual overhead of ciphertext manipulation.
Priya: From a privacy standpoint, if they are doing this compression client side, does that mean the server never sees the original embedding table structure at all?
Nadia: They claim that their client side digit decomposition maps tokens to deterministic coded tokens without needing any hints from the server <ref:2506.18150#pg1>. This is a direct way of mitigating revealing embedding-table structure.
Elias: It’s an orthogonal solution to other approaches, like Private Information Retrieval which relies on different FHE schemes such as BFV or CKKS <ref:2506.18150#pg2>.
Priya: So, HE-LRM is positioned as a holistic architecture that addresses the communication and memory bottlenecks inherent in applying FHE to sparse recommendation models.
Nadia: It is presented as the first end to end FHE encrypted DLRM architecture supporting both dense and sparse features under FHE <ref:2506.18150#pg1>.
Elias: The main takeaway from this summary is that the bottleneck for encrypted DLRMs isn't just homomorphic latency, but also managing system level resources like memory and communication costs <ref:2506.18150#pg2>.
Conclusion: Nadia: So we've covered a lot on HE-LRM: Encrypted Deep Learning Recommendation Models using Fully Homomorphic Encryption. We’ve discussed the embedding compression, the lookup strategies, and how this fits into the broader picture of FHE for neural inference.
Elias: The authors argue that this work provides a performance optimized solution by combining client side digit decomposition with a block diagonal packing strategy to maximize ciphertext slot utilization.
Priya: From what I’ve gathered, the implication is that we can move beyond naive solutions where compression leaks correlations to the client, instead using deterministic mapping for speed and privacy.
Nadia: That’s right. And they show that even with these optimizations, hardware acceleration on GPUs and ASICs can reduce end to end latencies from minutes down to seconds or even sub second times <ref:2506.18150#pg3>.
Elias: The paper highlights that the system level costs, like key switching and bootstrapping in CKKS, remain very high, but the HE-LRM approach is designed to manage those costs more effectively for DLRMs.
Priya: So what does this mean for someone who only listens to the show? It means that while FHE inference is still slow on a CPU, these architectural solutions are making it feasible to run complex recommendation models in the cloud with user data remaining encrypted.
Nadia: Exactly. And they’ve also pointed out that supporting both prefill and autoregressive decoding lookups efficiently under FHE is a major future direction for work.
Elias: It suggests that encrypted embedding lookup is a broader systems primitive for private inference, not just recommendation models, and it could apply to transformer based language models too.
Priya: So, the work shifts the focus from just making the math work to designing an entire system around minimizing those specific FHE system level costs.
More episodes
- 2608.27496-ROPE: Routed Origin Policy Enforcement against Indirect Prompt Injection
- Daily Summary for 2026-11-19
- 2607.07085-Is Randomness Necessary for Adaptive Data Analysis?
- The day in conversation for 2026-11-19
- Daily Summary for 2026-11-19
- 2609.10601-Threshold Choice, Not Sample Size, Bounds Trustless Verification of Nondeterministic Compound AI Workflows
- 2610.38786-Adversarial Process Disturbances at a Deployable Alarm Budget: Detector Evaluation on a Simulator and a Real Plant
- 2511.08660-Binary and Multiclass Cyberattack Classification on GeNIS Dataset
- 2604.27302-Static Attribution of Android Residential Proxy Malware Using Graph Kernels
- Daily Summary for 2026-11-18