Quantifying the Impact of Motion on 2D Gaze Estimation in Real-World Mobile Interactions

The gist Mobile gaze tracking accuracy is significantly impacted by user mobility and behavior, with head distance, head pose, and device orientation being key factors affecting accuracy

How it works

The research introduces a novel data collection framework that synchronizes motion and vision data (accelerometer, gyroscope, magnetometer, camera frames, and ground-truth gaze) to precisely capture both device movement and corresponding visual information. This framework is used in User Study 1 to observe spatial dynamics and motion behaviors during everyday tasks under various conditions such as lying, sitting, standing, walking slowly, and navigating through a maze. The data collection involves recording rich sensor readings (IMU) and camera measurements (head pose and distance) across these motion states.

The study employs a dual-study design to first characterize motion patterns in natural usage (User Study 1) and then assess their impact on gaze estimation accuracy under simulated real-world conditions (User Study 2). User Study 1 focuses on observing and recording a variety of motion states such as lying, sitting, standing, walking slowly, and navigating through a maze to understand typical user interaction patterns. In contrast, User Study 2 involves participants performing similar tasks but includes gaze estimation procedures like calibration and testing to measure the accuracy of gaze prediction under different motion conditions.

Key Findings from Data Analysis

The analysis of raw motion data reveals distinct behavioral patterns between static (e.g., lying, sitting) and dynamic (e.g., walking) activities. The largest variation in head distance resides in lying (M = 31.37cm, SD = 10.09cm), followed by sitting (M = 34.61cm, SD = 8.44cm) and two dynamic conditions: walking slowly (M = 31.07cm, SD = 7.01cm), and walking in maze (M = 33.13cm, SD = 7.57cm).

The analysis of orientation shows that lying has the largest variation in head distance and orientation. The two walking conditions exhibit a greater variation than static conditions on accelerometer and gyroscope readings. Furthermore, the t-SNE plots show a distinct separation between lying (purple) and sitting (green) conditions, while walking slowly (blue) and walking in maze (yellow) are overlapping due to their similar motion signatures.

Impact on Gaze Estimation Performance

The repeated-measures ANOVA revealed a significant main effect of motion condition on gaze estimation error, with dynamic conditions (walking and walking in maze) being significantly higher than static conditions. The motion-specific calibration approach achieved high accuracy, with errors of less than 2 cm in stable conditions, but under dynamic conditions the error increases to 2.73 cm, dropping by nearly 48.91% compared to the one-off calibration approach.

The Lasso regression analysis identified head distance as the most critical factor influencing gaze estimation accuracy across all motion conditions. The total Lasso-coefficient sum for the between-motion model was 27.29, of which head distance contributed 9.79 (≈ 35.87%), head movements 2.78 (≈ 10.19%), and device orientation deflection 10.44 (≈ 38.26%).

Conclusion and Future Directions

The study concludes that explicitly modelling head and device orientation remains the largest unmet need in 2D gaze estimation for mobile devices. The results suggest that small posture shifts or task switches can degrade errors by up to 2 or 3 times. Future work should focus on developing novel approaches that address these limitations, potentially by incorporating elements of 3D gaze estimation or explicit head pose modelling into more efficient 2D frameworks. The paper highlights the need for more advanced, adaptive eye-tracking systems that can explicitly account for head movements and maintain accuracy across diverse usage contexts.

The study provides empirical evidence on the limitations of 2D gaze estimation methods in mobile contexts, particularly their vulnerability to head pose and device orientation variations. The results show that 2D gaze estimators struggle to handle the frequent orientation changes characteristic of real-world mobile device usage. By combining motion (IMU) and vision data, the research demonstrates that 2D gaze estimators struggle to handle the frequent orientation changes characteristic of real-world mobile device usage. This highlights the need for more advanced, adaptive eye-tracking systems that can explicitly account for head movements and maintain accuracy across diverse usage contexts >.

1. Bold Header: Adaptive Calibration Strategy

User systems should implement an adaptive calibration strategy that periodically re-estimates the user’s head pose or normalises on-screen coordinates to maintain accuracy, as a one-off calibration is often insufficient for highly dynamic daily usage.

2. Bold Header: Explicit Head and Orientation Modeling

The AI system must incorporate explicit modeling of head and device orientation by incorporating orientation and head-distance data directly into the 2D pipeline, which helps mitigate performance degradation where head movements, device orientation changes, and increased screen distance collectively account for over 70% of the variance in gaze error during between-motion scenarios.

3. Bold Header: Motion Pattern Recognition

The system can use Gaussian Mixture Models (GMM) on sensor data to identify distinct behavioural patterns between static (e.g. lying, sitting) and dynamic (e.g. walking) activities, allowing the AI to better anticipate where performance will degrade based on the user's current motion state.

Mobile gaze tracking on handheld devices commonly relies on appearance-based 2D gaze estimation from the device's front-facing RGB camera. Although this approach makes eye tracking more accessible by avoiding dedicated infrared hardware, maintaining accuracy remains difficult because the spatial relationship among the user's face, the camera, and the screen changes continuously during handheld use. This paper examines how motion-related geometry change affects a representative front-camera RGB 2D gaze-estimation pipeline. We report two complementary within-subject user studies (n=10) that capture handheld behaviour, calibration data, and gaze-estimation error during everyday-style mobile tasks across conditions ranging from lying and sitting to walking and maze navigation. Rather than treating motion as a single binary factor, we analyse within-task versus task-switch behaviour, calibration-test mismatch, and condition-specific motion signatures. Behavioural results show that head-to-camera distance is comparatively stable during sustained within-task use, while task switches introduce much larger geometry fluctuations, with head-distance standard deviations of approximately 7.42--10.10,cm across conditions. In the gaze-evaluation study, one-off calibration reduced average RMSE from 4.49,cm to 3.44,cm; under this regime, the global effect of motion condition on RMSE was not statistically reliable, whereas clearer patterns appeared in task-specific and motion-specific calibration analyses, where better calibration-test match reduced error.