UAV-DualCog: A Dual-Cognition Benchmark for Grounded UAV Agent-World Reasoning with MLLMs
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "UAV-DualCog: A Dual-Cognition Benchmark for Grounded UAV Agent-World Reasoning with MLLMs".
Jane: Here's the gist of the paper. Multimodal large language models have achieved strong performance across diverse vision-language tasks, yet their capabilities in UAV scenarios remain insufficiently explored <ref:2607.16193#pg2>.
Tom: The paper addresses this gap by introducing UAV-DualCog, a benchmark for aerial multiview spatio-temporal reasoning built on a dual-cognition perspective <ref:2607.16193#pg6>.
Jane: The core of the research is the introduction of UAV-DualCog, which formulates self-state cognition and environment-state cognition as two top-level evaluation dimensions <ref:2607.16193#pg4>.
Tom: This dual cognition consists of self-state awareness, capturing the UAV’s own state such as relative position and motion, and environment-state awareness, capturing the external scene such as target direction and visibility <ref:2607.16193#pg5>.
Title and authors: Tom: So we're diving into the paper "UAV-DualCog: A Dual-Cognition Benchmark for Grounded UAV Agent-World Reasoning with MLLMs." We're looking at how this benchmark tackles the problem of models failing to understand both where a drone is and what’s around it.
Jane: That’s right. The authors are addressing a gap where multimodal large language models do well on general vision tasks, but when you ask them to reason about things from an aerial view—like a drone navigating or inspecting something—they struggle because they don't model the world and themselves at the same time.
Lu: Essentially, this paper introduces UAV-DualCog as a new way to test if these models have that joint understanding of the self and the environment, which is what I think is crucial for any real aerial agent.
Meng: So it’s not just about recognizing an object in a picture; it’s about the drone knowing its own position relative to that object while also knowing what else is visible around it at that exact moment. That sounds like a lot of required inputs for deployment, right?
Tom: Exactly. And the benchmark itself is structured around two main dimensions: self-state cognition, which is all about the drone's own state—its position and motion—and environment-state cognition, which is about the external scene, like where a target is or if it's visible.
Jane: The paper sets up this dual perspective by evaluating things through both images and videos to make sure they aren't just getting lucky with a single view. It requires both image tasks and video tasks to really test that joint cognition, which is important because real-world flight is inherently temporal.
Lu: They build this benchmark using a pretty automated four-stage pipeline, starting from scene-level semantic point clouds derived from aerial vision and language scenes. This means they’re automating the creation of data, which is a big step for making these evaluations scalable.
Meng: Scalability is key for us. If we can generate thousands of samples across hundreds of landmarks automatically, that’s better than manually curating a few hundred videos or images for testing, especially when you're worried about how the model will actually perform in production.
Title and authors: Tom: And they did build a pretty large pool, covering eighteen different UAV scenarios and over five million square meters of area with seven hundred forty-six valid landmarks <ref:2607.16193#pg1>. That’s a lot of ground truth data they generated to test against.
Jane: The evaluation structure is quite detailed, requiring the model to provide not just an answer but also some kind of evidence, whether that's a normalized bounding box for an image task or a specific time interval for a video task.
Lu: That’s where the dual aspect really shows up in their tasks. For example, one image task asks about relative position—like "what is your current position relative to the landmark?" and another asks about future observation prediction based on an action you take.
Meng: I see that they have four distinct image tasks, and all of them demand both a discrete answer and a bounding box, which means we’re not just looking for right or wrong; we’re checking if it can ground its answer spatially.
Tom: And then they have video tasks, which involve things like flight behavior recognition and temporal localization. These require the model to say what action it took and how long that action lasted, often using those temporal intervals we talked about earlier.
Jane: The paper points out some real struggles with current models here. They found that environment-state cognition is usually stronger than self-state cognition on image tasks, which suggests models are better at seeing the world than understanding their own movement relative to it.
Lu: And they also noted that performance on things like behavior understanding and atomic action recognition is often inconsistent when we move over to video tasks, which points to a challenge with temporal tracking.
Meng: That inconsistency is what worries me for deployment. If the model works okay on a static image but fails on a continuous video stream, it’s not reliable enough for anything that actually flies in real-time.
Tom: They identified dual-coordinate spatial reasoning as the largest error family in the instant setting, which means figuring out where you are relative to things is their biggest headache when looking at a single frame.
Jane: But on the video side, temporal tracking is cited as a major challenge, and they showed through an optimization probe that UAV-DualCog supervision can actually help improve both answer prediction and grounding.
Title and authors: Lu: They found that optimizing this dual-cognition supervision actually increased the image option accuracy from about thirty-nine point nine nine percent up to fifty-nine point seven seven percent. That’s a solid jump in performance on the static tasks, for sure.
Meng: But they mentioned that transferring those improvements over to video tasks wasn't smooth; they need explicit video-side supervision or cross-frame objectives, which means this benchmark alone isn't enough for a fully robust system yet.
Tom: So what we’re looking at here is a unified testbed, a benchmark that measures the whole dual-cognition capability in both image and video settings simultaneously.
Jane: It gives us something concrete to measure against when we say our models can reason about their own state alongside the external world in aerial contexts.
Lu: The conclusion is that while this benchmark is understandable to humans, existing models still struggle with reliable joint modeling of self- and environment-states in aerial settings.
Meng: It’s a good tool for researchers, but as we think about shipping these things into the real world, we still have to figure out how to fix that transfer problem between static and video reasoning.
Lalam: From my perspective, I can confirm that the UAV-DualCog benchmark provides a unified testbed and data resource for embodied aerial reasoning by jointly measuring self-state and environment-state cognition. It’s a big step in making this area of research more concrete.
Tom: So, to wrap up, this paper shows that while MLLMs are getting good at vision tasks, they need explicit structure—both in the data generation pipeline and in the supervision—to handle the complex dual reasoning required for UAV agents.
Jane: It’s a lot of work to get that structured output, but it gives us a clear target for where we need to focus our model improvements next.
Lu: And I think the future work they suggest is connecting this construction pipeline to more simulators and real-world datasets to reduce the simulation-to-real gap in grounded dual cognition evaluation.
Meng: That makes sense. We can’t just rely on synthetic data for testing if we want anything that actually flies safely, so bridging that gap is definitely the next necessary step.
Lalam: I’m confident this benchmark helps us move toward reliable joint modeling of self- and environment-states in aerial settings through structured evaluation.
The paper's summary: Tom: So, to recap, this paper is about creating a new benchmark called UAV-DualCog that tests if multimodal large language models actually have that dual understanding—knowing both where the drone is and what's around it in a complex aerial scene.
Jane: Right. And they set up these image tasks and video tasks specifically to check if the model can handle both self-state cognition, which is the drone's position and motion, and environment-state cognition, which is everything external—like where a target is or what it looks like.
Lu: What’s really interesting about their setup is how they build this data. They use an automated pipeline that starts with scene-level semantic point clouds from aerial vision and language scenes to create thousands of samples across hundreds of landmarks.
Meng: That automation is something I can get behind, but it makes me wonder about the cost. How much manual labeling did they have to do before they automated this whole thing?
Tom: They did some initial manual work for landmark construction and semantic annotation, but the goal is a scalable benchmark, covering eighteen different UAV scenarios with over five million square meters of area.
Jane: And what their evaluations show is that current models are not reliable at all when it comes to this dual cognition. Specifically, environment-state cognition tends to be stronger than self-state cognition on image tasks, which suggests they can see the world better than they can track their own position relative to it.
Lu: They also found that performance on things like behavior understanding and atomic action recognition is pretty inconsistent when you switch from static images to continuous video tasks.
Tom: That inconsistency is exactly what worries me for deployment; if a model works fine in a still picture but fails on a real flight, it’s not ready.
Jane: They pinpoint dual-coordinate spatial reasoning as the biggest error family in the instant setting, which means figuring out where you are relative to something is their main problem when looking at one frame.
Lu: But for video tasks, temporal tracking is a major challenge they found. However, they did show through a lightweight optimization probe that using this dual-cognition supervision actually helps improve both the answer prediction and the spatial grounding in images by boosting accuracy from about thirty-nine point nine nine percent to fifty-nine point seven seven percent.
Meng: A boost like that on the image side is significant, but they admitted that transferring those improvements over to video tasks wasn't smooth, meaning you still need explicit video supervision or cross-frame objectives for those continuous streams.
Tom: So the implication is that while this benchmark gives us a clear target for what a model needs to learn—joint modeling of self and environment—it doesn't fix the underlying architecture yet.
Jane: Exactly. It confirms that models are not yet reliably joint modeling both dimensions in aerial settings, even when we give them structured supervision on the static data side.
Lu: The future work they suggest is connecting this data construction pipeline to more simulators and real-world datasets to reduce that simulation-to-real gap in evaluating grounded dual cognition.
Meng: That’s where it gets practical; if we can get better alignment between what the model sees in simulation and what it sees in the real world, then this benchmark becomes much more useful for actually training agents.
The paper's improvements: Tom: So, we're looking at what the authors suggest to fix those performance gaps in UAV dual cognition on this benchmark.
Jane: Right, they’re saying that while the current setup is good for evaluation, it needs specific fixes to make models actually perform well in the real world.
Lu: They propose developing a dual-cognition reasoning module that explicitly incorporates pose and motion data directly into the large language model architecture to achieve this joint modeling of both dimensions.
Meng: Incorporating pose and motion data sounds complex; how much extra compute is that going to add to running an agent at scale?
Tom: They’re aiming for a joint model where self-state cognition, which includes current position and motion state, works directly alongside the environment-state cognition, which captures landmark position and visibility.
Jane: It's about forcing the model to learn these two things together instead of learning them separately. Then there’s another improvement: they suggest a structured output grounding mechanism that requires the model to give both an answer and explicit spatial or temporal evidence.
Lu: For image tasks, this means demanding a normalized bounding box, and for video tasks, it means requiring temporal intervals, like saying a time range for an action.
Meng: That structured output is what I need to see if we want to deploy anything; an answer without the required evidence just isn't useful in production.
Tom: They also developed a lightweight optimization probe that provides structured supervision by using a reward function that balances getting the answer right, sticking to the required output format, and making sure the spatial evidence quality is good.
Jane: It’s essentially giving the model targeted feedback on *how* to generate its response, not just *what* the response is.
Lu: And they tackled that video transfer issue by suggesting incorporating cross-frame objectives during training or fine-tuning to ensure consistency between different frames.
Meng: So, if we train it on a sequence of images and then try to use it on a continuous video stream, this method helps bridge that gap between the two modalities.
Tom: It’s about making sure that what the model learns from static views translates into reliable tracking during actual flight.
Jane: The implication here is that for UAV agents to be truly useful, they need training and fine-tuning strategies that explicitly handle both their own movement and the constantly changing external environment at the same time.
Lu: The overall goal of these suggestions is to move from just testing a model's current ability to actually giving us tools for advancing MLLM-based UAV agents.
Meng: I think if we can get those structured outputs working reliably, it makes the whole agent system much more tractable for production use.
Conclusion: Tom: So we're wrapping up this look at UAV-DualCog, which is that benchmark for aerial multiview spatio-temporal reasoning with multimodal large language models.
Jane: The big picture here is that this paper gives us a unified testbed for embodied aerial reasoning by forcing the AI to model both where it is and what’s around it simultaneously.
Lu: They confirm that while the benchmark itself is easy for humans to understand, existing models are struggling with reliably modeling those self- and environment-states together in aerial settings.
Meng: So, even with a good test case like UAV-DualCog, the core problem remains that we don't have models that can consistently handle the complexity of simultaneous self and world awareness.
Tom: Right. The implication for us is clear: we need to focus our next efforts on developing architectures that can handle this joint modeling requirement, especially in video contexts where temporal consistency matters so much.
Jane: And they suggest future research should focus on connecting their data construction pipeline to more simulators and real-world datasets to reduce the gap between simulation and reality for grounded dual cognition evaluation.
Lu: Exactly. Getting that data generation process closer to real flight scenarios is a necessary next step if we want these models to be robust enough for actual deployment.
Meng: If we can close that sim-to-real gap, then this benchmark becomes much more useful because it won't just be testing idealized scenarios.
Tom: It’s a big piece of work, and this benchmark shows exactly where the current limitations are so we know what to build next.
Jane: So, UAV-DualCog gives us a concrete way to measure that dual-cognition capability, and now we know it's still quite far from reliable for real aerial agents.
Lu: It’s a solid foundation for future work in representation learning that needs to account for both self and environment context simultaneously.
Meng: I think the focus should be on making sure those structural improvements they suggested—like the cross-frame objectives—actually translate into something usable at scale.
Tom: We'll keep an eye on how these models evolve based on this benchmark, because it really highlights what’s missing in current multimodal reasoning capabilities.
Jane: Next time, we’ll look at something totally different, so stay tuned.
Dian Shao, Liu, Haitao He, Hongzhe Li, Shuchang Zhang
Northwestern Polytechnical University · China University of Petroleum
cs.CV
Submitted: 2026-07-17
Updated: 2026-11-02
Comments: 8 pages, 4 figures, 6 tables. Project website: https://smartdianlab.github.io/UAV-DualCog/
Project page: https://uav-dualcog.lozumi.com
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Importance score: 79/100
The gist: Multimodal large language models have achieved strong performance across diverse vision-language tasks, yet their capabilities in UAV scenarios remain insufficiently explored <ref:2607.16193#pg2>.
Key concepts
- Dual-Cognition Framework
- This framework splits reasoning into two main parts: self-state awareness (knowing the UAV's position and motion) and environment-state awareness (knowing the external scene details like target direction). The benchmark tests how well a model can handle both these aspects simultaneously.
- Self-State Cognition
- This refers to the model's ability to reason about its own internal status, such as its current relative position and flight behavior. Tasks require inferring the UAV's state based on visual evidence from different views or frames.
- Environment-State Cognition
- This is the model's capacity to understand external scene information, including the location of landmarks and their visibility during flight. It measures how well a model interprets spatial relationships within the aerial environment.
Terminology
Summary
Multimodal large language models have achieved strong performance across diverse vision-language tasks, yet their capabilities in UAV scenarios remain insufficiently explored <ref:2607. p. 0 p. 016193#pg2>. The paper addresses this gap by introducing UAV-DualCog, a benchmark for aerial multiview spatio-temporal reasoning built on a dual-cognition perspective <ref:2607. p. 016193#pg6>.
Introduction to the Dual-Cognition Framework
The core of the research is the introduction of UAV-DualCog, which formulates self-state cognition and environment-state cognition as two top-level evaluation dimensions <ref:2607.16193#pg4>. This dual cognition consists of self-state awareness, capturing the UAV’s own state such as relative position and motion, and environment-state awareness, capturing the external scene such as target direction and visibility <ref:2607. p. 216193#pg5>. The benchmark evaluates both self-state and environment-state cognition through both image-based tasks and video-based tasks <ref:2607. p. 016193#pg6>.
Benchmark Construction and Scale
UAV-DualCog is constructed using a highly automated four-stage pipeline, starting with scene-level semantic point cloud construction from aerial VLN scenes <ref:2607.16193#pg5>. This process involves landmark asset construction and semantic annotation, followed by behavior-driven video task generation and image task generation <ref:2607. p. 216193#pg6>. The full asset pool covers 18 UAV scenarios over approximately 5,040,232. p. 300 m2 with 746 valid landmarks <ref:2607. p. 016193#pg6>. The released benchmark uses a subset of these assets to construct 4,096 image samples and 2,048 video samples across six task types <ref:2607. p. 016193#pg6>.
Evaluation Findings on Model Performance
Extensive evaluations show that current MLLMs remain far from reliable in UAV dual cognition <ref:2607. p. 016193#pg6>. Specifically, environment-state cognition is consistently stronger than self-state cognition on image tasks, while performance on behavior understanding and atomic action recognition is often inconsistent on video tasks <ref:2607. p. 116193#pg6>. The analysis reveals a consistent asymmetry where environment-aware tasks are generally easier than self-aware ones, indicating limited capability in inferring UAV-relative state from reference views <ref:2607. p. 516193#pg9>.
Diagnostic Insights and Optimization Potential
The failure diagnosis across instant and thinking settings shows that dual-coordinate spatial reasoning is the largest overall error family (33.7%) in the instant setting, while temporal tracking is a major challenge for video tasks <ref:2607. p. 516193#pg9>. The two-stage optimization probe demonstrates that UAV-DualCog supervision can improve both answer prediction and grounding, with image option accuracy increasing from 38. p. 799% to 59.77% <ref:2607. p. 016193#pg10>. However, the transfer to video tasks is uneven, suggesting that reliable UAV multimodal systems still require explicit video-side supervision or cross-frame objectives <ref:2607. p. 816193#pg10>.
Conclusion and Future Directions
UAV-DualCog provides a unified testbed and data resource for embodied aerial reasoning by jointly measuring self-state and environment-state cognition <ref:2607.16193#pg6>. The work concludes that while the benchmark is understandable to humans, existing models struggle with reliable joint modeling of self- and environment-states in aerial settings <ref:2607. p. 116193#pg6>. Future research will focus on connecting the construction pipeline to more simulators and real-world datasets to reduce the sim-to-real gap in grounded dual cognition evaluation <ref:2607. p. 816193#pg6>.
--- Page 1 ---
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Like Liu like.liu@mail.nwpu.edu.cn
Northwestern Polytechnical University
Xi’an, Shaanxi, China
Zhengzheng Xu
Northwestern Polytechnical
University
Xi’an, Shaanxi, China
Haitao He
Northwestern Polytechnical
University
Xi’an, Shaanxi, China
Abstract. Multimodal large language models have achieved strong performance across diverse vision-language tasks, yet their capabilities in UAV scenarios remain insufficiently explored <ref:2607.16193#pg2>. Recent UAV-oriented benchmarks have begun to evaluate MLLMs in aerial scenarios, but they typically focus on scene understanding, event recognition, or navigation completion, rather than jointly assessing the dual-cognition capability required for UAV agents: reasoning about both the UAV’s own state and the external environment in multiview spatio-temporal contexts <ref:2607. p. 016193#pg2>. To address this gap, we present UAV-DualCog, a benchmark for aerial multiview spatio-temporal reasoning built on this dual-cognition perspective <ref:2607. p. 016193#pg6>. UAV-DualCog includes both image and video tasks to jointly evaluate self-state and environment-state reasoning, while requiring spatial or temporal grounding beyond discrete answer prediction <ref:2607. p. 016193#pg6>. We also develop an automated pipeline that constructs data from scene-level semantic point clouds, yielding a scalable benchmark with diverse scenes, hundreds of landmarks, and thousands of QA samples <ref:2607. p. 016193#pg6>. Extensive evaluations show that current MLLMs remain far from reliable in UAV dual cognition <ref:2607.16193#pg6>. Self-state reasoning, viewpoint transformation, precise spatial grounding, and temporal interval localization are persistent bottlenecks, and additional validation with thinking/frontier models and a human baseline confirms that the benchmark is understandable to humans but challenging for existing models <ref:2607. p. 016193#pg6>. We further construct UAV-DualCog-Train from disjoint scenes and show through a lightweight optimization probe that it provides useful structured supervision, suggesting its value not only as an evaluation benchmark but also as a data resource for advancing MLLM-based UAV agents <ref:2607. p. 016193#pg6>.
CCS Concepts • Computing methodologies → Vision for robotics; Cognitive robotics; Spatial and physical reasoning <ref:2607. p. 016193#pg5> Keywords Multimodal large language models, UAV reasoning benchmark, aerial vision-language understanding, spatio-temporal reasoning <ref:2607. p. 016193#pg5>
--- Page 2 ---
Liu et al. Self-Aware Cognition Landmark-relative Position Reasoning Future Observation Prediction Self-relative Position Reasoning Landmark Visibility Counting and Interval Reasoning Based on, what is your current position relative to the landmark? p. 1 Return the option and the landmark’s BBox. p. 1 [A] Right [B] Front-Left [C] Back-Right [D] Left Landmark’s BBox: [0. p. 1420, 0.361, 0.613, 0.687] The agent needs to know the state of itself, e. p. 1g. current position and flight behavior related to a landmark <ref:2607. p. 116193#pg9>. Environment-Aware Cognition The agent needs to know the state of environment, e. p. 1g. a landmark’s position and visible time during its flight <ref:2607. p. 116193#pg9>.
--- Page 3 ---
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Table 1: Comparison with representative aerial and embodied reasoning benchmarks. “Self-state” denotes whether the benchmark explicitly evaluates the agent’s own viewpoint, motion, or relative position as reasoning targets. p. 2 “Evidence grounding” denotes whether the evaluation requires explicit spatial or temporal evidence outputs, such as bounding boxes or temporal intervals <ref:2607. p. 216193#pg5>. UAV-DualCog is not designed to replace navigation, embodied QA, or aerial perception benchmarks <ref:2607. p. 216193#pg5>. Instead, it complements them by making the UAV’s own state a first-class evaluation target and by requiring answer predictions to be accompanied by explicit spatial or temporal evidence <ref:2607. p. 216193#pg5>.
--- Page 4 ---
Image Task Generation Video Task Generation
Image-based tasks. We construct four image tasks: ① Selfrelative position reasoning asks the model to infer the UAV’s position relative to a landmark <ref:2607. p. 216193#pg9> ② Future observation prediction requires predicting the next observation under a candidate action <ref:2607. p. 216193#pg9> ③ Landmark-relative direction reasoning asks where the landmark lies relative to the UAV’s current forward direction, and ④ Landmark-driven action decision asks which direction the UAV should move to approach the target <ref:2607. p. 216193#pg9>. These two tasks evaluate self-state cognition <ref:2607. p. 216193#pg9>. All image tasks require both a discrete answer and a normalized bounding box, enabling evaluation of reasoning and visual grounding <ref:2607. p. 216193#pg9>.
--- Page 5 ---
UAV-DualCog Knowing the Self, Understanding the World
Focusing on dual-cognition capability required for UAV Agents New benchmark for dual-cognition evaluation in both image and video tasks. IM A G E T A S K VID E O T A S K
--- Page 6 ---
Flight Behavior Recognition and Temporal Localization (rendered in pointcloud) (captured in scene) Which of the following actions have you ever adopted in this first-person flight video? [A] Moving foward to the building [B] Square-shaped orbiting [C] Move forward to the bridge [D] Square-shaped inspection Time interval: A [1. p. 100, 4.50]; B [4.50,22.90]; D [1.00,27.50]
--- Page 7 ---
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
Table 4: Results on video-based UAV dual-cognition tasks, including flight behavior recognition and landmark visibility recognition. Darker blue and lighter blue cells denote the best and second-best performance within each model group <ref:2607. p. 416193#pg9>.
--- Page 8 ---
Table 5 reports additional validation with thinking-enabled variants, frontier models, and a human baseline <ref:2607.16193#pg9>.
--- Page 9 ---
Table 5 reports additional validation with thinking-enabled variants, frontier models, and a human baseline <ref:2607.16193#pg9>.
--- Page 10 ---
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
References [1] Remyx AI and Salma Mayorquin. 2025. SpaceOm Models. p. 4 p. 4 https://huggingface. p. 9 p. 9 p. 9 p. 9co/remyxai/SpaceOm
[2] Remyx AI and Salma Mayorquin. 2025. SpaceThinker Models. p. 5 p. 5 https://huggingface.co/remyxai/SpaceThinker-Qwen2. p. 9 p. 95VL-3B
[3] Anthropic. 2026. Introducing Claude Opus 4. p. 9 p. 9 p. 9 p. 9 p. 9 p. 96. https://www. p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9 p. 9anthropic.com/news/claude-opus-4-6
[4] Anthropic. 2026. Introducing Claude Opus 4.7.https://www.anthropic.com/news/claude-opus-4-7
[5] Anthropic. 2026. Introducing Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8
[6] Anthropic. 2026. Introducing Claude Sonnet 4. p. 9 p. 96.https://www.anthropic.com/news/claude-sonnet-4-6
[7] Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, and Helmut Prendinger. 2017. Okutamaaction: An aerial view video dataset for concurrent human action detection. p. 9 p. 9 In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. p. 9 p. 9 28–35.
[8] Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, Tongxi Zhou, Jiaqi Li, Hui En Pang. p. 9 2025. Scaling Spatial Intelligence with Multimodal Foundation Models. p. 9 p. 9 arXiv preprint arXiv:2511. p. 9 p. 913719 (2025). p. 9 p. 9
[9] Google DeepMind. p. 9 p. 9 2025. Gemini 3 Flash: frontier intelligence built for speed <ref:2607. p. 916193#pg9>.
[10] Google DeepMind. p. 9 p. 9 2026. Gemini 3.1 Flash-Lite: Built for intelligence at scale <ref:2607. p. 916193#pg9>.
[11] Google DeepMind. p. 9 p. 9 2026. Gemini 3.1 Pro: A smarter model for your most complex tasks <ref:2607. p. 916193#pg9>.
--- Page 10 ---
Knowing the Self, Understanding the World: A Dual-Cognition Benchmark for UAV Spatio-temporal Reasoning with MLLMs
References [1] Remyx AI and Salma Mayorquin. 2025. SpaceOm Models. https://huggingface.co/remyxai/SpaceOm
[2] Remyx AI and Salma Mayorquin. 2025. SpaceThinker Models. https://huggingface.co/remyxai/SpaceThinker-Qwen2.5VL-3B
[3] Anthropic. 2026. Introducing Claude Opus 4.6.https://www.anthropic.com/news/claude-opus-4-6
[4] Anthropic. 2026. Introducing Claude Opus 4.7.https://www.anthropic.com/news/claude-opus-4-7
[5] Anthropic. 2026. Introducing Claude Opus 4.8.https://www.anthropic.com/news/claude-opus-4-8
[6] Anthropic. 2026. Introducing Claude Sonnet 4.6.https://www.anthropic.com/news/claude-sonnet-4-6
[7] Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, and Helmut Prendinger. 2017. Okutamaaction: An aerial view video dataset for concurrent human action detection. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 28–35.
[8] Zhongang Cai, Ruisi Wang, Chenyang Gu, Fanyi Pu, Junxiang Xu, Yubo Wang, Wanqi Yin, Zhitao Yang, Chen Wei, Qingping Sun, Tongxi Zhou. p. 9 2025. Scaling Spatial Intelligence with Multimodal Foundation Models. arXiv preprint arXiv:2511.13719 (2025).
[9] Google DeepMind. 2025. Gemini 3 Flash: frontier intelligence built for speed. p. 9
[10] Google DeepMind. 2026. Gemini 3.1 Flash-Lite: Built for intelligence at scale. p. 9
[11] Google DeepMind. 2026. Gemini 3.1 Pro: A smarter model for your most complex tasks. p. 9
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Develop a dual-cognition reasoning module that jointly models self-state cognition (current position, motion state) and environment-state cognition (landmark position, visibility) by explicitly incorporating pose and motion data into the MLLM architecture to achieve
joint modeling of both dimensions.
-
Implement a structured output grounding mechanism that requires the model to provide both a discrete answer and an explicit spatial or temporal evidence output, as demonstrated by requiring
a normalized bounding box
for image tasks andtemporal intervals
for video tasks. -
Create a scalable data generation pipeline utilizing scene-level semantic point clouds to automatically construct diverse training data, allowing MLLMs to be trained on a large pool of
hundreds of landmarks and thousands of QA samples.
-
Design a lightweight optimization probe that provides structured supervision by optimizing the model using a reward function that balances answer correctness, output format adherence, and spatial evidence quality (mapping normalized-box IoU to [0, 1]).
-
Enhance temporal reasoning capabilities by incorporating cross-frame objectives during training or fine-tuning to address the observed
transfer boundary
issue where image-side supervision does not constraincrossframe target consistency
in video tasks.
Sources
- Scaling Spatial Intelligence with Multimodal Foundation Models
- Aerial Vision-and-Dialog Navigation
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
- Openfly: A comprehensive platform for aerial vision-language navigation
- HUGE-Bench: A Benchmark for High-Level UAV Vision-Language-Action Tasks
- A Survey of Generative Categories and Techniques in Multimodal Generative Models
- GPT-4o System Card
- LongFly: Long-Horizon UAV Vision-and-Language Navigation with Spatiotemporal Context Integration
- Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models
- SpaceR: Reinforcing MLLMs in Video Spatial Reasoning
- Kimi K2.5: Visual Agentic Intelligence
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
- Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
- Aerial Vision-Language Navigation with a Unified Framework for Spatial, Temporal and Embodied Reasoning
- Visual Spatial Tuning
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
- Large Language Models for Robotics: A Survey
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs