How Weight Encoding Affects Language Model Placement and Performance on the Apple Neural Engine
cs.LG, cs.AR, cs.PF
Submitted: 2026-08-22
Updated: 2026-09-23
Comments: 10 pages, 2 figures, 7 tables. v2: corrected byte-traffic accounting, withdrew the 0.77 nominal-byte factor, and attributed normalization eligibility to dtype rather than formulation; title and abstract revised accordingly. See Section 8
Code: https://github.com/shershah1024/ane-llm-measurements
License: http://creativecommons.org/licenses/by/4.0/
The gist: Weight compression can alter accelerator placement as well as memory traffic, complicating the interpretation of inference speedups.
Terminology
Abstract
Weight compression can alter accelerator placement as well as memory traffic, complicating the interpretation of inference speedups. We investigate this interaction on the Apple Neural Engine through the public Core ML deployment path. Five independently trained language-model checkpoints span two architectures and dense fp16, int8, and ternary weights encoded with two-bit lookup tables. We combine compiler device plans, synchronized memory-controller measurements, and compute-unit exclusion controls for a fixed single-token forward workload. On an M1, the smaller fp16 export executes on the CPU despite permitting ANE execution, whereas its compressed counterparts exhibit ANE activity. The int8 export reduces warm forward latency by a factor of 1.9. The larger fp16 export also uses the ANE, indicating that dense encoding alone does not determine placement. Separate M3 energy measurements support the same direction of change. These results establish encoding-dependent placement in the measured deployment stack and show that compression comparisons require joint measurement of backend selection, latency, and traffic.
Sources
- Apple Neural Engine: Architecture, Programming, and Performance
- ANEForge: Python for direct computation on the Apple Neural Engine
- Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference
- Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
- T-MAN: Enabling End-to-End Low-Bit LLM Inference on NPUs via Unified Table Lookup
- Fast On-device LLM Inference with NPUs
- Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference
- RooflineBench: A Benchmarking Framework for On-Device LLMs via Roofline Analysis
- Pagoda: An Energy and Time Roofline Study for DNN Workloads on Edge Accelerators
- Hardware Generation and Exploration of Lookup Table-Based Accelerators for 1.58-bit LLM Inference
- ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
- Is Your NPU Ready for LLMs? Dissecting the Hidden Efficiency Bottlenecks in Mobile LLM Inference
- LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load
- LFM2 Technical Report
- STAR: Synthesis of Tailored Architectures
- Liquid Time-constant Networks
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- MnasNet: Platform-Aware Neural Architecture Search for Mobile
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
Related papers
- Polynomial-Augmented Neural Networks (PANNs) with Weak Orthogonality Constraints for Enhanced Function and PDE Approximation
- AIRL-S: Unifying Reinforcement Learning and Search-Based Test-Time Scaling via Adversarial Inverse Reinforcement Learning
- Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation
- Convergence issues in Relational Concept Analysis based on AOC-posets
- Beliefs Beyond Posteriors: The Predictive Cost of ELBO Binding
- Understanding Diffusion Models via Ratio-Based Function Approximation with SignReLU Networks