SPIRAL-360: Stratified Panoramic Instruction Replay and Angular Learning for Visual Grounding

summary

The gist

SPIRAL-360 introduces an Embodied Active Global-to-Local Exploration framework that leverages a panoramic prior to dynamically adjust the Field of View and progressively narrow down search space for…

In short

SPIRAL-360 introduces an Embodied Active Global-to-Local Exploration framework for 360° visual search. It uses a panoramic prior and dynamic Field of View adjustment to precisely ground targets. This method overcomes MLLM limitations in handling 360° properties like distortion, achieving state-of-the-art accuracy by modeling the search as a POMDP.

Key concepts

RoPE Rolling Mechanism
This mechanism models the continuous cylindrical wrap-around topology of a 360° image. It assigns a unique horizontal offset to coordinate vectors for each head, ensuring that every vertical column is represented at least once and preventing seam discontinuities in the panoramic view.
Global-to-Local Exploration Framework
This paradigm shifts search from holistic scene perception to precise local refinement. The agent first reasons over the entire 360° scene layout and then iteratively uses a tool to narrow its focus onto a target-centered local region, leading to accurate grounding.
Perspective Projection Tool
This tool is used in an active observation loop. The agent invokes it repeatedly to manipulate the panoramic view's azimuth, elevation, and field of view. This allows the system to precisely locate a target within a defined spherical Bounding Field of View (BFoV).
Group Relative Policy Optimization (GRPO)
This is an advanced training method used to stabilize multi-step tool invocation in long contexts. It optimizes the agent's policy by comparing individual trajectories against the group mean, effectively aligning the exploration strategy for complex, sequential reasoning tasks.

Terminology used across episodes

This episode discusses

The paper

SPIRAL-360: Stratified Panoramic Instruction Replay and Angular Learning for Visual Grounding · Read on arXiv

Jingtao Xu, Zizhuo Lin, Jianwen Sun, Yi Yang, Yawei Luo

Zhejiang University · Central China Normal University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "SPIRAL-360: Stratified Panoramic Instruction Replay and Angular Learning for Visual Grounding".

Title and authors: Tom: So we're looking at this paper now called SPIRAL-three hundred sixty which talks about an Embodied Active Global-to-Local Exploration framework for visual grounding in three hundred sixty degrees <ref:2607.02479#pg1,Embodied Active Global-to-Local Exploration>. It seems like they’re tackling some really tricky problems that standard multimodal large language models have been struggling with when it comes to panoramic views.

Jane: That makes sense, Tom. I mean, when you think about looking at a whole three hundred sixty-degree picture, you immediately run into issues with how the image is wrapped—things like polar distortion and where the edges actually connect <ref:2607.02479#pg1>. It sounds like this paper is proposing a way to use that full panorama upfront to guide the search better than just looking around locally.

Lu: Exactly. The core idea they're pushing here is using a panoramic prior to dynamically adjust what part of the scene the agent should be focusing on next, which lets them progressively narrow down that search space for finding a specific target with much more precision. They're trying to fix those limitations where previous methods relied too much on just local exploration without having that big picture context.

Meng: But from a deployment standpoint, what does this mean for us? If the agent has to process the whole panorama at once, that sounds like it could take up a lot of memory or processing power on real hardware when we try to run these things in production.

Lalam: It’s modeled as a partially observable markov decision process where the agent starts with the full panorama and a natural language query, and they break it into two stages: first, global perception using something called the RoPE Rolling Mechanism, and then active exploration using a tool to simulate observation.

Tom: The RoPE Rolling Mechanism sounds like their way of handling that continuous cylindrical topology. They’re assigning a head-specific horizontal offset to coordinate vectors so the model can seamlessly handle that wrap-around nature without having those weird seam discontinuities that plague standard equirectangular projections, right?

Jane: Right, and they call this the Global-to-Local Exploration Framework where it uses a dynamic chain of thought reasoning loop. It's not just looking at the whole thing; it's using that global context to predict a final bounding field of view, which is like figuring out exactly where on the sphere to look next for the most precise grounding.

Lu: What’s really interesting from their side, though is how they built the training signal. They use a two-stage process: first supervised fine-tuning on twenty thousand expert trajectories just to get the basic action space down, and then group relative policy optimization or GRPO in a live rollout environment to align the exploration strategy by estimating what's good for that specific trajectory compared to others.

Title and authors: Meng: GRPO sounds complex for real-time deployment. How much computational overhead are we talking about for this training pipeline, and does it actually stabilize multi-step tool invocation well enough when things get really long?

Lalam: They use a weighted reward function with four parts: an answer reward based on the great circle distance between the predicted and actual location, a tool use and turn efficiency component, format correctness, and a length penalty. The weights are set to answer reward at half the importance, tool use efficiency at one-tenth, format correctness at fifteen percent, and length penalty at five percent.

Tom: And that answer reward is piecewise linear based on the great circle distance in degrees. So it’s rewarding precision up to ten degrees but then it starts penalizing errors beyond ninety degrees. That gives them a clear target for what "good enough" looks like in terms of accuracy versus search effort.

Jane: It seems they’ve put some real focus on making the exploration efficient, not just accurate, by balancing that precision with how quickly the agent finds the answer. What about how they actually train the model itself? They used Qwen3-VL-4B-Instruct as their backbone and fine-tuned it using LoRA with rank three twenty and scaling alpha thirty two applied to all linear projection layers in both the language model and the vision language projector.

Lu: That LoRA setup is pretty standard for efficiency, but they also added a special token to distinguish between panoramic images and ordinary perspective images, which helps the backbone understand what kind of input it's looking at. That’s a good way to give it modality-specific instructions during training.

Meng: From a production standpoint, if we were deploying this, the latency for that initial global perception stage would be critical. How do you measure that time in words? Is it constant time, or does it scale with the size of the panorama?

Tom: The paper suggests they aim for something that handles ultra-long contexts by optimizing based on group relative advantages during GRPO, which helps stabilize multi-step tool invocation in those long contexts. They are also focused on making sure that if a tool is needed, only the tool call is outputted immediately, followed by waiting for the projected image result.

Title and authors: Jane: It’s a very structured interaction they enforce with prompt templates. They limit the maximum number of tool calls per trajectory to eight and set a maximum dialogue length of six turns, which keeps things from spiraling out of control during an active search episode. This gives us some concrete guardrails for how the agent interacts with its tools.

Lu: Their data augmentation strategy is also pretty clever. They apply a four-direction horizontal rotation augmentation to every training panorama, which guarantees a strictly uniform azimuthal distribution of target objects across all viewing angles, intrinsically enhancing the agent’s pano coordinate modeling and rotational invariance. That should help it generalize better outside the exact orientation of the training set.

Meng: Uniform distribution is nice for testing, but what about real-world noise? They mention that their dataset synthesis involves six stages, including automated object candidate generation using GroundingDINO and GPT-4o mini for filtering, which sounds like a lot of pipeline complexity to manage at scale.

Lalam: The EAGLE-three hundred sixty dataset itself is large, comprising over fourteen thousand four hundred thirty six high resolution panoramas and more than seventy thousand rounds of reasoning turns. They also synthesized five thirty four fifty one associated perspective view projections stored on disk through that augmentation strategy.

Tom: So, putting all that together, the EAGLE-three hundred sixty model establishes a new state of the art for three hundred sixty degree visual search, achieving nearly an eight fold increase in accuracy over the base model when tested on their dataset <ref:2607.02479#pg1>. That is a significant jump in performance for this specific task.

Jane: And they also showed it has strong zero shot transferability, scoring fifty six point one on the out of distribution HhBench set, which means it can perform well even on data it hasn't specifically seen before. That’s important for real world applications where you don't have perfectly labeled data for every single corner of the world.

Lu: Their main points about improvements center around that global-to-local paradigm, which shifts from those myopic, iterative adjustments to a holistic global context first, and then narrowing the focus to a target-centered local region. That’s the core architectural shift they are making for better spatial reasoning.

Meng: And what about where it still falls short? The paper admits that because of that global-to-local approach, it results in ultra long context windows which lead to significant memory overhead when running things at scale. That’s a real constraint for deployment right now.

Tom: They also flagged that localization performance in the extreme polar regions still slightly lags behind equatorial directions because of the severe non linear geometric warping in equirectangular projections. So it's not perfect everywhere yet, and they acknowledge that limitation upfront.

Title and authors: Jane: So to wrap up, SPIRAL-three hundred sixty is a framework that uses an active global-to-local perspective exploration strategy combined with training techniques like GRPO to achieve significant gains for three hundred sixty degree visual grounding <ref:2607.02479#pg1,an active global-to-local perspective exploration strategy>. It shows a path forward by leveraging the whole scene at once before getting hyper precise locally.

Lu: It’s interesting because it tackles those fundamental limitations of standard MLLMs regarding polar distortion and continuous edge topologies in panoramic views by introducing mechanisms like RoPE Rolling and that dynamic adjustment of the field of view to progressively narrow down the search space.

Meng: For us in engineering, the main things to watch are how they manage that memory overhead when scaling up those ultra long contexts for real-time navigation. We need concrete data on inference latency before we can seriously consider putting this into a production environment.

Tom: It’s clear that the gains come from combining architectural innovations like RoPE Rolling with the training pipeline stabilization offered by GRPO, which effectively elicits complex spatial reasoning and tool-calling capabilities in those long contexts.

Jane: And what they show us is that when you combine a strong panoramic prior with an active observation loop using perspective projection tools, you can get a new state of the art for this kind of search. It’s a solid piece of work for anyone looking to move beyond just egocentric views in complex environments.

Lu: Overall, SPIRAL-three hundred sixty shows that by structuring the exploration as a global context first before local refinement, you can get much better error recovery and more precise target grounding than methods that rely on fragmented local exploration without a panoramic prior <ref:2607.02479#pg1,rely on fragmented local exploration>.

Meng: So for the listener who just wants to know what this means for their work, it means there's a new way to teach AI to search in three hundred sixty degrees that doesn't just guess where things are by looking around locally <ref:2607.02479#pg1>. It’s about using the whole picture strategically before zooming in.

Tom: That’s right. The paper SPIRAL-three hundred sixty provides a very concrete framework for how to build an embodied agent that can actually perform robust visual search in those challenging panoramic spaces, and they open up some new avenues for how we design the training signals <ref:2607.02479#pg1>.

Jane: It's a big step forward because it moves the goal from just having a model that understands an image to having a system that actively plans its way through a complex visual space with high precision. We’ll be looking at where this leads next in the field of embodied AI soon.

The paper's summary: Tom: So, this paper SPIRAL-three hundred sixty is essentially proposing a way to teach an AI to search in three hundred sixty degrees by using instruction replay and learning about different angles of view directly from panoramic images.

Jane: That's right. The main idea is that instead of just looking at one small piece of a panorama, the AI learns how different angles relate to each other, which helps it find a target much more reliably across the whole picture.

Lu: What makes this interesting is that they use this approach to solve problems where standard models struggle with those continuous wrap-around issues in panoramic views, like polar distortion.

Meng: But what's the practical deal here? How does using instruction replay and angular learning actually translate into a model that runs fast enough on a real robot or in an autonomous car?

Tom: Well, the methodology involves modeling this whole panoramic search as a partially observable markov decision process, which means the AI starts with the entire panorama and some text query.

Jane: And it uses two main stages there: first, global perception to get that big picture understanding, and second, tool-augmented iterative exploration where it actively tries different views using a projection tool.

Lu: They introduce something called the RoPE rolling mechanism to handle that continuous cylindrical topology by giving each head a specific horizontal offset for its coordinate vectors. That’s their way of making sure the model doesn't get confused at those seams in the equirectangular images.

Meng: So they’re using this dynamic adjustment of the field of view to narrow down where it should look next to find that target, right?

Tom: Exactly. The framework shifts from just looking around locally and making small adjustments to a much bigger global-to-local exploration paradigm.

Jane: What they show us with the results is a big jump in performance, nearly eight times better accuracy on their dataset compared to the standard models they tested against.

Lu: They build this training signal using two phases: supervised fine-tuning just to set up the basic action space, and then group relative policy optimization or GRPO to stabilize those long sequences of tool calls.

Meng: That sounds like a lot of training time and complexity, but if it stabilizes multi-step tool invocation in ultra long contexts, that’s valuable for complex tasks where the AI has to reason through many steps.

Tom: The reward function they use is quite clever, balancing the precision of finding the answer—measured by the great circle distance between prediction and ground truth—against how efficiently it uses its tools.

Jane: They even set a rule where if the error in finding the target goes over ninety degrees, it starts heavily penalizing that trajectory. That gives them a clear goal for accuracy versus search effort.

Lu: And they use a four-direction horizontal rotation augmentation on every single training panorama to make sure the target objects are distributed uniformly across all viewing angles. That helps the AI generalize better beyond just the exact orientation of the training data.

Meng: So, it’s about making sure that when we deploy this, it's not just good for one specific way of looking at a scene, but works robustly regardless of how you rotate your camera or where you are in the room.

Tom: Right. The limitations they mention are pretty real: because they look at the whole panorama, it leads to very long context windows, which means memory can become a big problem when running things at scale on actual hardware.

Jane: They also admit that even with this global view, there’s still a slight performance dip in the most extreme polar regions because of how the equirectangular projection warps things geometrically.

Lu: So for anyone listening, SPIRAL-three hundred sixty shows that by using a panoramic prior to guide an active search loop, you can get much better spatial reasoning than just relying on local adjustments to find something in three hundred sixty degrees.

Meng: It moves the goal from just having a model that understands an image to having a system that actively plans its way through a complex visual space with high precision, even if it’s memory-intensive to run.

Tom: That framework is open-sourced now, including the core code for that RoPE rolling mechanism and the whole training pipeline scripts.

Jane: And they show this framework can achieve nearly eight times better accuracy on their dataset compared to the baseline model, which is a solid number to look at for three hundred sixty degree visual grounding.

Lu: This work gives us a new direction for how we structure the exploration of three hundred sixty degree data, moving away from just myopic local searches toward a holistic global context first.

Meng: It’s definitely something worth watching, especially if we can figure out ways to compress that context window for real-time deployment.

Tom: So the next thing we're going to look at is how they actually built this data set—how they synthesized those thousands of training panoramas and turns of reasoning.

The paper's improvements: Tom: So we're looking at the specific improvements they suggest for SPIRAL-three hundred sixty, focusing on how they actually enhance that global to local search strategy.

Jane: They’re talking about making the whole process more robust by adding stratified instruction replay and learning specifically about the different angles of view.

Lu: What I find interesting is their suggestion to refine the reward function, using a piecewise linear curve for the answer reward based on how far off that great circle distance is between what they predicted and what was actually there.

Meng: So they’re tuning the system so it rewards precision up to ten degrees, but then it starts heavily penalizing errors if you go beyond ninety degrees. That sets a clear threshold for what's considered a successful localization result.

Tom: Exactly. It balances finding the right spot with not wasting too much time searching when the target is way off.

Jane: They also emphasize that by learning about different angles, the AI gets better at understanding rotational invariance, meaning it can find an object no matter how you rotate your viewpoint in the panorama.

Lu: That addresses one of those core issues we talked about—the model not handling polar distortion well because it learns the relationships between different orientations explicitly.

Meng: If that learning works, it means we don't just need more training data; we need a better way to teach the AI *how* to relate different parts of a panoramic view, which is what this paper is proposing.

Tom: They also discussed how they are building the dataset for this—using automated object candidate generation with Grounding DINO and GPT four o mini for filtering before putting them into the training set.

Jane: That filtering process sounds like it’s a lot of work, but it ensures that the training data is actually high quality before the AI tries to learn from it.

Lu: The implication here is that this isn't just a better model; it’s about creating a more structured way to teach visual grounding by explicitly modeling the spatial relationships between different views.

Meng: For production, if we can make this learning process more efficient, it could mean building agents that are much more reliable in complex indoor or outdoor environments where you need to find things from any direction.

Tom: And they’ve already made it open-sourced, which is good because seeing the actual implementation of the RoPE rolling mechanism and the GRPO training pipeline helps us understand exactly how these ideas translate into code.

Jane: So, in short, they are showing us that by layering instruction replay on top of angular learning, we can create a more precise system for grounding targets in three hundred sixty degree scenes.

Lu: It’s a step toward making embodied AI much better at reasoning about three dee space than just processing flat images.

Meng: I’m still focused on the memory overhead, though; if this sophisticated learning process requires a massive context window every time it runs, that could be the biggest blocker for us right now.

Tom: We need to see how they manage that scaling issue in practice before we can say this is ready for real-world deployment.

Conclusion: Tom: So to wrap up, SPIRAL-three hundred sixty proposes a framework that uses stratified instruction replay and angular learning to make AI search in three hundred sixty degrees much more accurate by looking at both the whole picture and different perspectives.

Jane: That’s right. The main implication is that we can teach AI how to handle those complex continuous wrap-around shapes in panoramic views without relying on models that just look at small, localized areas.

Lu: It gives us a structured way to build spatial reasoning capabilities by explicitly modeling the relationships between different viewing angles across the whole scene.

Meng: For me, what matters is that this approach moves us toward agents that can navigate complex three dee spaces reliably, even if the computational cost for running this kind of global perception remains a challenge.

Lalam: I’m confident in my assessment because the results show a nearly eight fold increase in accuracy on their dataset compared to the base model they were testing against.

Tom: That's a big jump in performance for this specific task, and it shows that when you combine that global context with active tool use, you get a solid result.

Jane: It really changes how we think about visual grounding; instead of just asking the AI to look at a spot, we’re giving it the whole scene and teaching it to plan its way through it.

Lu: The method is essentially showing that learning how different viewpoints connect is key to solving problems with polar distortion in equirectangular projections.

Meng: If we can figure out ways to compress that context window during inference, this could become a serious tool for autonomous systems operating in environments where visual input is panoramic.

Tom: It’s definitely a step toward making those agents more robust in real-world scenarios, and the open-source code available helps us see exactly how they built that mechanism.

Jane: We’ll keep an eye on how this angular learning applies to other modalities, because teaching a model to understand rotation is something we need more of across the board.

Lu: Next up, we're going to look at another paper that tackles how large language models can actually use their vision capabilities to generate actual video content.

More episodes

← Home