Low Latency Gaze Tracking via Latent Optical Sensing
cs.CV, cs.HC
Submitted: 2026-05-18
Updated: 2026-09-27
Code: https://github.com/black-forest-labs/flux
License: http://creativecommons.org/licenses/by/4.0/
The gist: We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder.
Terminology
Abstract
We present a real-time gaze tracking system that directly acquires task-relevant latent features using a fully passive optical encoder. Instead of forming and processing full-resolution images, our approach leverages a microlens array with a co-designed binary chromium mask to perform spatially multiplexed optical encoding, producing a compact set of measurements sufficient for gaze estimation. By integrating sensing and feature extraction in the optical domain, the proposed system eliminates the need for high-bandwidth image readout and substantially reduces computational overhead. The encoded measurements are captured by a 4 x 4 phototransistor array and mapped to gaze direction using a lightweight neural network. Our proof-of-concept prototype enables an end-to-end sensing-to-inference latency of 3.4 ms, outperforming published research systems. We demonstrate the effectiveness of our approach on both simulated and real-world data, achieving competitive gaze estimation accuracy while significantly reducing end-to-end latency compared to conventional camera-based pipelines. The proposed system establishes a new operating point in the accuracy-latency-compute trade-off for latency- and resource-constrained gaze tracking. This work highlights the potential of task-driven optical sensing for ultra-low-latency, computationally efficient human-computer interaction systems.
Sources
- WEBEYETRACK: Scalable Eye-Tracking for the Browser via On-Device Few-Shot Personalization
- JaneEye: A 12-nm 2K-FPS 18.9-$\mu$J/Frame Event-based Eye Tracking Accelerator
- A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization
- Tolerance-Aware Deep Optics
- Investigating Bias and Fairness in Appearance-based Gaze Estimation
- One Eye is All You Need: Lightweight Ensembles for Gaze Estimation with Single Encoders
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models