OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following

arXiv:2605.26399 · cs.CV · Submitted 2026-05-26 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following".

Jane: Here's the gist of the paper. Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM’s dense hidden states Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation Furthermore, to explicitly ground the model in multiperson scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning Pruning redundant reasoning tokens The framework introduces a dual-branch decoding architecture to address the limitations of text-only coordinate generation The structured language branch is fine-tuned to generate discrete reasoning states such as 2D target location, target in/out of frame, and semantic category Simultaneously, the continuous gaze branch is dedicated to the dense spatial decoding of target heatmaps by tapping into the VLM’s hidden state tokens This approach ensures that the model can predict gaze target heatmaps that directly benefit from dense supervision signals, significantly improving spatial localization performance How the training signal is built The entire framework is trained end-to-end in a multi-task manner using the overall training objective defined as a weighted combination of individual task losses The total loss is formulated as Ltotal = λlmLlm + λhmLhm + λinoutLinout + λsocLsoc (3) Specifically, the language branch supervises the entire JSON text sequence using a standard autoregressive cross-entropy loss Llm The gaze branch is supervised by several losses, including the Heatmap Loss Lhm for spatial prediction and Linout for in/out prediction For social gaze reasoning, a Social Gaze Loss Lsoc is computed based on pairwise social gaze logits Si,j Where the gains come from The performance improvements stem from several key architectural innovations designed to overcome existing bottlenecks The head-conditioned token injection mechanism augments the input text by prepending a special placeholder token,, to each person’s prompt entry Ti, replacing the text embedding at the corresponding index H(0) with an extracted head embedding ei This targeted substitution explicitly injects localized head appearance and orientation cues into the sequence, grounding the person-specific prompt with rich visual features The dual-branch architecture overcomes the limitations where training only the continuous gaze branch for dense heatmap decoding proves suboptimal Combining both approaches into this dual-branch architecture proves that the hidden state tokens produced during language generation serve as essential structural anchors for extracting high-resolution spatial heatmaps Unified Gaze, Semantic, and Social Understanding OmniGF is designed to jointly support gaze target estimation, semantic gaze prediction, and pairwise social gaze reasoning within a single model For semantic tasks, the model generates an additional "category" field within the structured language output For social gaze prediction on the VSGaze dataset, a selective loss masking strategy is introduced to ensure the model learns to process all individuals in the scene simultaneously for global context This allows the framework to perform end-to-end (E2E) social gaze reasoning jointly with gaze localization, eliminating the need for manual post-processing required by ordinary gaze following methods The model establishes a new state-of-the-art across all primary target estimation and in/out prediction metrics on all datasets

Paper summary: Tom: Looking at the whole OmniGF paper by Qiaomu Miao et al., the title itself tells you a lot about what they achieved here: Unified Gaze. It’s not just about tracking where someone is; it’s about understanding their gaze in a complex, social context.

Jane: Right, and the authors of this OmniGF framework are showing that by combining structured language states with dense spatial decoding from the VLM's hidden states, you can get high-resolution spatial predictions without losing the semantic reasoning power of a Vision-Language Model >

Lu: They’ve moved beyond using VLMs just for auxiliary tasks, like extracting embeddings for other vision models or generating pseudo-annotations; they are now applying them directly to perform complex downstream tasks like dense visual grounding and segmentation >

Meng: So, what does this mean for someone who only listens to the show? It means that in the future, we might expect AI systems that can not only tell you where a person is looking but also predict what they might be thinking or how they are relating to other people in a room >

Jane: That’s right. The paper shows that this isn't just about better localization; it’s about building systems capable of complex social reasoning, which is essential for the next generation of human-computer interaction applications >

Conclusion: Tom: So, we're wrapping up this look at OmniGF, which is titled "OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following." It’s basically about getting better at understanding where people are looking in complex scenes.

Jane: Right. The authors are showing how they combine two different ways of looking at the problem: one that handles the structure, and another that handles the actual visual data directly from a vision-language model.

Lu: What they did is use a dual-branch approach to generate reasoning states separately—one for discrete things like where a target is in two dimensions, and another for continuous spatial information pulled straight from the vision model's hidden layers.

Meng: From an engineering standpoint, that sounds clever because you’re not just relying on text prompts to guess where something is; you’re using the model's internal visual understanding to guide the precise location.

Tom: Exactly. And they ground this by adding head embeddings—tiny visual cues from cropped images of each person—to give the system specific appearance details for everyone in a group.

Jane: So, what does that mean for us listening? It means we’re moving past just knowing someone is looking somewhere to actually predicting *what* they are looking at, and if they're talking to someone else.

Lu: They also use this whole setup to do three things at once: estimate the exact gaze target location, predict what that target is semantically—like a person or an object—and even reason about the social interactions between people.

Meng: And they do that end-to-end, which is important because it means you don't have to write extra steps after the model spits out a list of coordinates. It’s more integrated.

Tom: So, looking at the authors and their work with VLMs, this shows how much more powerful these models get when they are trained to do these kinds of complex spatial tasks directly from dense visual input.

Jane: It’s a big step toward making AI that can truly understand the social dynamics of a scene without needing someone to manually label every single person and their gaze path. (Sound fades slightly)

Qiaomu Miao, Haoyu Wu, Jingyi Xu, Minh Hoai, Dimitris Samaras

Stony Brook University · The University of Adelaide

cs.CV

Submitted: 2026-05-26

Updated: 2026-11-09

Comments: NeurIPS 2026 Spotlight

Code: https://github.com/cvlab-stonybrook/omnigf

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 92/100

The gist: Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction The model adopts a dual-branch decoding strategy…

Key concepts

Dual-Branch Decoding Strategy
The model uses two parallel decoding paths: one structured language branch generates discrete reasoning states like target location, and another continuous spatial branch directly uses the VLM's dense hidden states to decode spatial information. This combination overcomes limitations of text-only coordinate generation.
Head-Conditioned Token Injection
This mechanism augments the input by prepending a special token with an extracted head embedding for each person. This explicitly injects localized appearance and orientation cues into the prompt, grounding the model's reasoning in rich visual features specific to each individual in a scene.
Multi-Task Training Objective
The framework is trained end-to-end using a weighted combination of several losses: language loss for text generation, heatmap loss for spatial prediction, and social gaze loss for pairwise reasoning. This ensures the model learns to perform gaze localization, semantic prediction, and social reasoning all at once.
Social Gaze Reasoning
The model is designed to handle complex social interactions by computing a Social Gaze Loss based on pairwise logits. A selective loss masking strategy forces the model to process all individuals in the scene globally, enabling end-to-end social gaze reasoning without manual post-processing.

Terminology

Summary

Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM’s dense hidden states Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation Furthermore, to explicitly ground the model in multiperson scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning

Pruning redundant reasoning tokens

The framework introduces a dual-branch decoding architecture to address the limitations of text-only coordinate generation The structured language branch is fine-tuned to generate discrete reasoning states such as 2D target location, target in/out of frame, and semantic category Simultaneously, the continuous gaze branch is dedicated to the dense spatial decoding of target heatmaps by tapping into the VLM’s hidden state tokens This approach ensures that the model can predict gaze target heatmaps that directly benefit from dense supervision signals, significantly improving spatial localization performance

How the training signal is built

The entire framework is trained end-to-end in a multi-task manner using the overall training objective defined as a weighted combination of individual task losses The total loss is formulated as Ltotal = λlmLlm + λhmLhm + λinoutLinout + λsocLsoc (3) Specifically, the language branch supervises the entire JSON text sequence using a standard autoregressive cross-entropy loss Llm The gaze branch is supervised by several losses, including the Heatmap Loss Lhm for spatial prediction and Linout for in/out prediction For social gaze reasoning, a Social Gaze Loss Lsoc is computed based on pairwise social gaze logits Si,j

Where the gains come from

The performance improvements stem from several key architectural innovations designed to overcome existing bottlenecks The head-conditioned token injection mechanism augments the input text by prepending a special placeholder token,, to each person’s prompt entry Ti, replacing the text embedding at the corresponding index H(0) with an extracted head embedding ei This targeted substitution explicitly injects localized head appearance and orientation cues into the sequence, grounding the person-specific prompt with rich visual features The dual-branch architecture overcomes the limitations where training only the continuous gaze branch for dense heatmap decoding proves suboptimal Combining both approaches into this dual-branch architecture proves that the hidden state tokens produced during language generation serve as essential structural anchors for extracting high-resolution spatial heatmaps

Unified Gaze, Semantic, and Social Understanding

OmniGF is designed to jointly support gaze target estimation, semantic gaze prediction, and pairwise social gaze reasoning within a single model For semantic tasks, the model generates an additional category field within the structured language output For social gaze prediction on the VSGaze dataset, a selective loss masking strategy is introduced to ensure the model learns to process all individuals in the scene simultaneously for global context This allows the framework to perform end-to-end (E2E) social gaze reasoning jointly with gaze localization, eliminating the need for manual post-processing required by ordinary gaze following methods The model establishes a new state-of-the-art across all primary target estimation and in/out prediction metrics on all datasets

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Dual-Branch Architecture for Unified Reasoning

The framework introduces a structured language branch to generate discrete reasoning states and a continuous gaze branch to decode dense spatial signals from hidden states, allowing the model to simultaneously perform spatial localization, semantic prediction, and social reasoning in a single pass.

  1. Head-Conditioned Token Injection for Individual Grounding

The system uses a head-conditioned token injection mechanism where cropped visual head features are directly embedded into dedicated prompt token slots, providing the VLM with fine-grained appearance and orientation cues necessary to accurately disambiguate targets in dense environments.

  1. Unified Gaze, Semantic, and Social Understanding

The model jointly supports gaze target estimation, semantic gaze prediction, and pairwise social gaze reasoning within a single model, achieving state-of-the-art performance across these three distinct tasks simultaneously.

  1. Structured Multi-Person Prompting for Joint Modeling

OmniGF introduces a person-aware vision-language framework that models multiple individuals jointly via structured prompts, processing all persons together from a single VLM input.

  1. Semantic Grounding via CLIP Alignment

For semantic gaze recognition, the model encodes text predictions and evaluation classes using CLIP to assign the closest matching test class via cosine similarity, ensuring robust semantic alignment despite open-vocabulary text generation.

  1. Selective Loss Masking for Unannotated Individuals

When training on social gaze prediction datasets with unannotated individuals, a selective loss masking strategy is employed to dynamically exclude the tokens corresponding to these individuals from the language modeling gradient, allowing the model to focus updates on valid targets while learning global scene context.

Sources

Related papers