DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop
cs.PF
Submitted: 2026-10-27
Updated: 2026-10-27
Comments: 17 pages, 4 figures
License: http://creativecommons.org/licenses/by/4.0/
The gist: DeepInfer-Mac runs the 750B-class DeepSeek-V4.1-Flash model on a MacBook Air with 24 GB of unified memory.
Abstract
DeepInfer-Mac runs the 750B-class DeepSeek-V4.1-Flash model on a MacBook Air with 24 GB of unified memory. A mixed low-bit conversion occupies 338 GiB on the internal SSD. About 7.6 GiB of weights remain resident; routed experts and Engram rows are loaded when needed. This separation makes the model executable, but it leaves each generated token waiting for a succession of storage reads. We organize expert weights so that reads fill reusable GPU buffers directly, and couple cache allocation with the lifetime of those buffers. In end-to-end runs, the system generates English, Chinese and code at 0.21-0.72 tokens per second, with a largest sampled process footprint of 16.73 GiB across the tested requests. Matched decode comparisons show a 21.0% throughput gain from next-layer prefetching over resident-expert overlap alone, at the cost of additional reads. Longer prompts remain costly. We describe the design, provide the implementation and measurement scripts, and identify the limits of this operating point: a model far larger than memory can run locally, while storage and prefill still govern the user's wait.
Related papers
- Instruction-Aware Transfer of GPU Implementation Crossovers: a Tensor-Product Discontinuous Galerkin Operator on GB200, H100, and A100
- denet, A lightweight command-line tool for process monitoring in benchmarking and beyond
- HybridQC: Hardware-Grounded Simulation of Tightly Integrated Hybrid Quantum-Classical Systems
- TileBench: A Controlled Benchmark for Performance Evaluation and Bottleneck Diagnosis of Tile-Based Programming Models
- The M/PH 2/1 Queue: Explicit Distributions, Sojourn Times and Numerical Benchmarks
- Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU