Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation

arXiv:2603.21142 · cs.RO · Submitted 2026-03-22 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.

Rosa: We're looking at the paper: "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation".

Dev: Here's the gist of the paper. Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>.

Rosa: AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time.

Dev: The results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18.5% relative to fixed settings and improving robustness and success rate relative to an uncapped baseline.

Rosa: Core Methodology The framework couples high-level visual semantic perception with low-level safety behavior adaptation at runtime.

Paper summary: Rosa: So we're looking at this paper called "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation." Basically, it’s about how robots need to balance staying safe and moving efficiently when they don't have perfect sensing in messy places.

Dev: Yeah, the core idea here is using a vision-to-control navigation framework that takes what the robot sees from its own camera—egocentric RGB input—and uses that to adjust how cautious or aggressive a safety filter should be, and it does this in real time.

Taro: I'm curious about what they claim about this adaptation. Is it just making the old safety settings smarter, or is the whole control approach fundamentally different?

Rosa: It’s more than just smart settings; they use a vision-language model, or VLM, to produce a bounded scalar risk estimate from the current camera view and then map that risk to dynamically update a control barrier function parameter that dictates how strongly safety constraints are enforced.

Dev: So they have this risk score between zero and one, where zero means low risk and one means high risk. They use this score to adjust something called alpha, which controls the conservativeness of the safety filter.

Taro: If I'm looking at that number between zero and one, what does that actually mean for a robot’s immediate next move? Is it like saying "I need to slow down" or "I need to stop"?

Rosa: It represents a hazard score for the robot’s current motion context in relation to its environment, which is interpreted as whether the next few seconds will likely require evasive action. They map that risk score using a function that controls sensitivity, and then they cap it with some geometric and speed-aware dynamic cap based only on geometry and speed.

Dev: That dynamic cap is important because they acknowledge that the VLM output can be delayed or sometimes overly permissive, so they have a staleness-gated fusion policy that defaults to conservative behavior if the risk estimate isn't fresh.

Taro: So when you’re in a fast-moving situation and the vision system lags, what’s the actual safety guarantee? Does this setup give you a hard limit on how aggressive the robot can get?

Paper summary: Rosa: They use that geometric, speed-aware dynamic cap as a safety envelope and fallback mechanism so the controller cannot behave too aggressively when it’s close to obstacles. The final parameter used by the control barrier function is then determined by taking the minimum of three things: the VLM estimate when fresh, that geometric cap, and a hard cap if the VLM estimate is stale.

Dev: And this whole process runs against a low-level controller that executes every zero point zero one six seven seconds, which gives them a certain control rate to work with while still allowing for asynchronous queries to the VLM service.

Taro: If you were testing this on a real robot outside the lab, what kind of physical challenges might cause this system to struggle when the world misbehaves?

Rosa: They tested it across four scenarios in NVIDIA IsaacSim: a single frontal obstacle, a cluttered field, dynamic obstacle crossing, and dynamic frontal obstacle. They found that AlphaAdj reached the goal in eight out of eight setups with zero collisions.

Dev: That’s impressive because it shows that integrating this semantic risk modulation doesn't sacrifice safety at all. They compared their results to fixed parameters and an uncapped baseline, and they saw improvements in efficiency too.

Taro: What did those efficiency improvements actually look like? I mean, if you have to be more conservative, isn't that just making the robot slower overall?

Rosa: Well, in the warehouse dynamic obstacle scenario specifically, AlphaAdj increased path length by three point two one meters, which is an eighteen point five percent increase compared to fixed settings. And it also improved time-to-goal by one point two seven seconds, which is ten point eight percent faster than before.

Dev: So the trade-off is clear: a conservative alpha of zero point one means longer path length and more time to goal, but an aggressive alpha of zero point six leads to a collision if things go wrong. AlphaAdj sits in the middle.

Taro: The paper also points out that synchronous approaches are bounded by the VLM response time, which is a key limitation for real-time systems that need consistent control rates.

Rosa: Exactly, and they noted their practical implementation choices were designed specifically to reduce end-to-end VLM latency in their system. They showed that even with the typical six hundred ninety five millisecond average round-trip latency when querying the service every three steps, the overall system maintains a control rate suitable for real systems.

Paper summary: Dev: So, when you look at the title, "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation," it really captures that whole loop: using semantic vision to adapt safety functions in real time. It’s not just about seeing an obstacle; it’s about deciding how much to trust your safety filter based on what the vision says.

Rosa: Yeah, and the authors show that this framework is deployable because they address latency and staleness with a fusion policy that prioritizes hard caps when the risk estimate is old. It shows how you can integrate semantic risk into safety filtering in a way that actually works in practice.

Taro: What I want to push on is where it stops working, right? The authors mention that while they achieved zero collisions across those scenarios, real-world deployment introduces additional physical challenges that aren't fully modeled here.

Dev: That’s a fair point about the limitations they mentioned in their future work section. They suggest further calibration and interpretability of the VLM risk score could improve reliability across different domains, which is a big area for future research.

Rosa: It seems like this work moves the safety envelope from being purely geometric or pre-programmed to being contextually informed by high-level semantic understanding from the vision model. It’s about making the robot smarter about *why* it should be cautious, not just *how much* caution to apply.

Taro: So, for someone just listening, what does this mean? It means robots can navigate cluttered spaces more safely because they are using a richer understanding of the scene to modulate their safety response dynamically.

Dev: It means you get better efficiency compared to fixed settings without having to sacrifice the fundamental guarantee that the robot won't collide, provided you handle those latency issues correctly with dynamic caps and staleness checks.

Rosa: It’s a practical solution for robots operating in dynamic, unstructured environments where they have limited sensing, like warehouses or construction sites. This paper gives us a concrete example of how to bridge the gap between high-level vision intelligence and low-level real-time safety loops.

Conclusion: Rosa: So, we’re wrapping up this look at "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation." Essentially, this paper shows how you can use what a robot sees to make its safety filters smarter while it’s moving around in messy places.

Dev: That’s right, the whole idea revolves around taking that vision input and using a vision language model to adjust how conservative or aggressive the robot's safety rules are on the fly.

Taro: It seems like they’re not just adding another sensor; they are changing *how* the safety filter thinks about what it sees.

Rosa: Exactly, they map the risk score from that vision model to a parameter called alpha, which directly controls how strongly those safety constraints are enforced at runtime.

Dev: And the authors tackle a big problem there: latency and the fact that these vision models aren't always instantaneous. They introduce a dynamic cap and a staleness gate to make sure the system stays safe even when the risk estimate is old.

Taro: So, they’re balancing high-level visual understanding with strict real-time control loops by adding these layers of checks.

Rosa: Right, and their experiments in simulation showed that this works reliably across different challenging scenarios—like a cluttered field or dynamic obstacle crossings—keeping collisions at zero.

Dev: The results show they can actually improve efficiency compared to just using fixed safety settings, so the robot moves faster and covers more ground without sacrificing that safety guarantee.

Taro: It makes sense that if the system is robust enough to handle those four different scenarios, it suggests this could be a real step toward giving robots more nuanced decision-making in unpredictable environments.

Rosa: It certainly suggests that for future field testing, you won't just need perfect geometry; you’ll need a model that understands the *risk* associated with what’s around it.

Dev: That brings us to the real question: how much do we actually trust this semantic risk score when a robot is running outside of a perfectly controlled lab setting?

Jeffrey Chen, Rohan Chandra

University of Virginia

cs.RO

Submitted: 2026-03-22

Updated: 2026-10-30

Code: https://github.com/isaac-sim/IsaacSim

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 87/100

The gist: Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>.

Key concepts

Control Barrier Function (CBF)
A mathematical tool used for safety filtering in robotics that ensures a robot stays within a safe region by constraining its control inputs. It acts as a hard safety filter, preventing the robot from taking actions that would lead to collisions or unsafe states, regardless of external commands.
Vision-Language Model (VLM)
A model that takes visual input from an RGB camera and generates a risk estimate. It interprets the scene context—like obstacles or dynamic objects—and outputs a score indicating how dangerous the current situation is, which helps decide how much safety margin to apply.
Dynamic $\alpha$ Cap ($\alpha_{cap}$)
A geometric limit calculated based on the robot's current speed and its distance to the nearest obstacle. This cap acts as a hard safety fallback, ensuring that even if the VLM provides an aggressive suggestion, the robot's movement remains physically constrained and safe near hazards.
Staleness-Gated Fusion Policy
A mechanism to manage latency when using the VLM. If a new risk estimate from the VLM is too old (stale), the system ignores it and relies on a more conservative, pre-calculated safety cap ($\alpha_{hard cap}$), ensuring real-time control stability despite slow AI inference.

Terminology

Summary

Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603. p. 0

Where the paper says it

Dynamic Control Barrier Function Regulation with Vision-Language Models for Safe, Adaptive, and Realtime Visual Navigation Jeffrey Chen, Rohan Chandra Supplementary Material, Videos at the following website: saferobotnav. io/AlphaAdj Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing.

21142#pg8>. AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time. p. 0
Where the paper says it

While control barrier functions (CBFs) provide principled collision avoidance via safety filtering, their behavior is often governed by fixed parameters that can be overly conservative in benign scenes or overly permissive near hazards. We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time.

The results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18. p. 0
Where the paper says it

We evaluate AlphaAdj across multiple static and dynamic obstacle scenarios in a variety of environments, comparing against fixed-parameter and uncapped ablations. Results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18.

5% relative to fixed settings and improving robustness and success rate relative to an uncapped baseline. p. 0
Where the paper says it

Results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18. 5% relative to fixed settings and improving robustness and success rate relative to an uncapped baseline.

Core Methodology

The framework couples high-level visual semantic perception with low-level safety behavior adaptation at runtime. The approach extracts context from the egocentric RGB scene and outputs a collision-risk estimate using a Vision-Language Model (VLM). p. 0

Where the paper says it

In this work, we present an adaptive navigation algorithm that couples high-level visual semantic perception to adapt low-level safety behavior at runtime. Our framework extracts context from the egocentric RGB scene and outputs a collision-risk estimate using a VLM.

This risk estimate is then used to dynamically adjust a low-level Control Barrier Function (CBF) parameter which governs how conservatively (or assertively) the robot responds to potential hazards. p. 0
Where the paper says it

Our framework extracts context from the egocentric RGB scene and outputs a collision-risk estimate using a VLM. The risk estimate is used to dynamically adjust a low level CBF control parameter which governs how conservatively (or assertively) the robot responds to potential hazards.

Risk Estimation and Mapping

The VLM produces a scalar collision-risk estimate r ∈ [0, 1] from the current egocentric RGB frame, where r = 0 indicates low risk and r = 1 indicates high risk. p. 3

Where the paper says it

5 seconds, 2 Hz), the controller captures an RGB frame and asynchronously queries our VLM risk service over HTTP (695 milliseconds average end-to-end round-trip latency). The VLM produces a scalar collision-risk estimate r ∈ [0, 1] from the current egocentric RGB frame, where r = 0 indicates low risk and r = 1 indicates high risk.

This risk is mapped to a bounded scalar αvlm using the monotone decreasing function αvlm(r) = αmin + (αmax − αmin) (1 − r) γ, where γ > 0 controls sensitivity. p. 3
Where the paper says it

, whether the next few seconds will likely require evasive action). We map risk to a bounded scalar αvlm using the monotone decreasing function αvlm(r) = αmin + (αmax − αmin) (1 − r) γ , (6) where αmin ≤ αvlm ≤ αmax and γ > 0 controls sensitivity.

The VLM prompt construction involves a server-side instruction and a client-provided context string containing lightweight runtime information like low or medium risk indicators. p. 3
Where the paper says it

: Risk is obtained by querying a local HTTP endpoint with a PNG image and a text instruction. The instruction presented to the VLM is formed by concatenating (i) a fixed server-side formatting constraint that enforces a strict JSON response and (ii) a client-provided context string that is updated online with the robot’s current state.

Latency Management and Safety Guarantees

To address asynchronous inference and nontrivial VLM latency, the framework introduces a dynamic cap and a staleness-gated fusion policy. A geometric, speed-aware dynamic cap αcap is used to guarantee an upper bound on aggressiveness based only on geometry and speed. p. 4

Where the paper says it

The cap is designed primarily as a safety envelope and fallback mechanism. It guarantees an upper bound on aggressiveness based only on geometry and speed, independent of semantic estimation quality.

The fusion policy sets the final parameter αfinal as min(αvlm, αsoft cap, αhard cap) when fresh and falls back to the hard cap when the VLM estimate is stale. p. 4
Where the paper says it

c) Fusion policy: We combine αvlm with the caps using a simple fusion policy: when VLM estimate is fresh, we set αfinal = min αvlm, αsoft cap , (12) which ensures the VLM can only reduce conservativeness up to the safe envelope imposed by the soft cap. When the VLM estimate is stale, we ignore αvlm and fall back to the hard cap, αfinal = α hard cap .

Experimental Validation

Experiments were conducted in NVIDIA IsaacSim across four scenarios: Single Frontal Obstacle, Cluttered Field, Dynamic Obstacle Crossing, and Dynamic Frontal Obstacle. The results show that AlphaAdj reaches the goal in 8/8 setups with 0 collisions, demonstrating that integration of semantic risk modulation can be done without sacrificing safety. p. 6

Where the paper says it

Additionally, we report quantitative results for each scenario in the warehouse environment in Table I. Across all 8 environment–scenario pairs, AlphaAdj reaches the goal in 8/8 setups with 0 collisions, showing that integration of semantic risk modulation of the CBF safety filter can be done without sacrificing safety.

Compared to fixed αmin, AlphaAdj increases path length by 3. p. 6
Where the paper says it

However, in the warehouse dynamic obstacle scenario, while αmin did not collide, it also was unable to reach its destination due to its conservativeness, as seen in 4c. Furthermore, this conservativeness drastically increases detours: relative to AlphaAdj, averaged over all eight setups, fixed αmin increases path length by 3.

21 meters (18. p. 4
Where the paper says it

Equation 8 defines a normalized clearance score s s ∈ [0, 1] from raw margin m. We scale m by Msafe and clip at 1 so that (i) s = 0 at safety boundary, (ii) s = 1 once the robot is at least Msafe meters away from the closest obstacle, and larger clearances do not further increase the cap.

5%) and time-to-goal by 1. p. 0
Where the paper says it

We evaluate AlphaAdj across multiple static and dynamic obstacle scenarios in a variety of environments, comparing against fixed-parameter and uncapped ablations. Results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18.

27 seconds (10. p. 3
Where the paper says it

Every N = 30 control steps (0. 5 seconds, 2 Hz), the controller captures an RGB frame and asynchronously queries our VLM risk service over HTTP (695 milliseconds average end-to-end round-trip latency).

8%) in the warehouse dynamic obstacle scenario. p. 6
Where the paper says it

The only other setting that consistently avoids collisions is fixed αmin. However, in the warehouse dynamic obstacle scenario, while αmin did not collide, it also was unable to reach its destination due to its conservativeness, as seen in 4c.

Efficiency Tradeoff Analysis

The analysis reveals a clear efficiency–safety tradeoff, where conservative α=0.1 increases path length and time to goal, while an aggressive α=0. p. 6

Where the paper says it

Averaged over 10 runs, the results show a clear efficiency–safety tradeoff: conservative α=0. 1 increases path length and time to goal, while an aggressive α=0.

6 leads to collision. p. 6
Where the paper says it

1 increases path length and time to goal, while an aggressive α=0. 6 leads to collision.

AlphaAdj is more efficient than the no-cap variant in scenarios like the warehouse frontal scenario by 0. p. 6
Where the paper says it

Compared to no cap, AlphaAdj is more efficient in when it comes to both path length and time to goal in the scenarios shown in Figures 4b, 4c, 4d, 4e, 4f, 4g. Our success in every scenario is notable given our controller runs continuously at a 60 Hz control rate (one control step every 0.

53 seconds in time to goal (4. p. 6
Where the paper says it

5%) and time-to-goal by 1. 27 seconds (10.

5%). The system maintains a control rate suitable for real systems while enabling dynamic VLM-guided adjustment for safe and efficient goal reaching. p. 7
Where the paper says it

2 seconds, further illustrating how synchronous designs are bounded by VLM response time. In contrast, our approach maintains a control rate suitable for real systems while still enabling dynamic VLM-guided adjustment for safe and efficient goal reaching.

Limitations and Future Directions

Real robotic deployment may introduce additional physical challenges, but the work represents a natural next step for harnessing the benefits of the design. Future work could also center on further calibration and interpretability of the VLM risk score, which could further improve reliability across a multitude of domains. p. 7

Where the paper says it

Real robotic deployment may introduce additional physical challenges, but it represents a natural next step for harnessing the benefits of our design, enabling more diverse sensing and varied interactions. Future work could also center on further calibration and interpretability of the VLM risk score, which could further improve reliability across a multitude of domains.

The paper concludes by showing that synchronous VLM-based approaches are bounded by VLM response time, contrasting with the real-time control rate maintained by AlphaAdj. p. 7
Where the paper says it

2 seconds, further illustrating how synchronous designs are bounded by VLM response time. In contrast, our approach maintains a control rate suitable for real systems while still enabling dynamic VLM-guided adjustment for safe and efficient goal reaching.

References

arXiv:2603.21142v1[cs. p. 0

Where the paper says it

To mitigate this, we introduce a latencyaware mechanism that detects stale or anomalous VLM arXiv:2603. 21142v1 [cs.

RO] 5

--- Page 5 ---

Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>. While control barrier functions (CBFs) provide principled collision avoidance via safety filtering, their behavior is often governed by fixed parameters that can be overly conservative in benign scenes or overly permissive near hazards. p. 0

Where the paper says it

io/AlphaAdj Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing. While control barrier functions (CBFs) provide principled collision avoidance via safety filtering, their behavior is often governed by fixed parameters that can be overly conservative in benign scenes or overly permissive near hazards.

We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time. p. 0
Where the paper says it

While control barrier functions (CBFs) provide principled collision avoidance via safety filtering, their behavior is often governed by fixed parameters that can be overly conservative in benign scenes or overly permissive near hazards. We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time.

A vision-language model (VLM) produces a bounded scalar risk estimate from the current camera view, which we map to dynamically update a CBF parameter that modulates how strongly safety constraints are enforced. p. 0
Where the paper says it

We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time. A vision-language model (VLM) produces a bounded scalar risk estimate from the current camera view, which we map to dynamically update a CBF parameter that modulates how strongly safety constraints are enforced.

To address asynchronous inference and nontrivial VLM latency in practice, we combine a geometric, speed-aware dynamic cap and a staleness-gated fusion policy with lightweight implementation choices that reduce end-toend inference overhead. p. 0
Where the paper says it

A vision-language model (VLM) produces a bounded scalar risk estimate from the current camera view, which we map to dynamically update a CBF parameter that modulates how strongly safety constraints are enforced. To address asynchronous inference and nontrivial VLM latency in practice, we combine a geometric, speed-aware dynamic cap and a staleness-gated fusion policy with lightweight implementation choices that reduce end-to-end inference overhead.

We evaluate AlphaAdj across multiple static and dynamic obstacle scenarios in a variety of environments, comparing against fixed-parameter and uncapped ablations. p. 0
Where the paper says it

To address asynchronous inference and nontrivial VLM latency in practice, we combine a geometric, speed-aware dynamic cap and a staleness-gated fusion policy with lightweight implementation choices that reduce end-to-end inference overhead. We evaluate AlphaAdj across multiple static and dynamic obstacle scenarios in a variety of environments, comparing against fixed-parameter and uncapped ablations.

--- Page 6 ---

Fig. 1: System overview. p. 2

Where the paper says it

Because VLM inference is asynchronous and introduces non-trivial latency, we propose a dynamic cap and mitigation policy for stale requests that guarantees conservative behavior when the VLM output is delayed. The overall system overview can be seen in Figure 1

Egocentric RGB frames are processed asynchronously by a VLM to produce a risk estimate r, mapped to αvlm. p. 3
Where the paper says it

1: System overview. Egocentric RGB frames are processed asynchronously by a VLM to produce a risk estimate r, mapped to αvlm.

A geometric, speed-aware cap αcap and a staleness test gate the VLM output to produce αfinal, which parameterizes the CBF safety filter that produces safe commands. p. 3
Where the paper says it

Egocentric RGB frames are processed asynchronously by a VLM to produce a risk estimate r, mapped to αvlm. A geometric, speed-aware cap αcap and a staleness test gate the VLM output to produce αfinal, which parameterizes the CBF safety filter that produces safe commands.

--- Page 7 ---

A. Risk-to-α Mapping from VLM Output

We consider a mobile robot with egocentric RGB camera sensing operating in a cluttered environment with static or dynamic obstacles. The low-level controller executes every 0. p. 3

Where the paper says it

Risk-to-α Mapping from VLM Output We consider a mobile robot with egocentric RGB camera sensing operating in a cluttered environment with static or dynamic obstacles. The low-level controller executes every 0.

0167 sconds (60 Hz). p. 3
Where the paper says it

The low-level controller executes every 0. 0167 sconds (60 Hz).

Every N = 30 control steps (0. p. 3
Where the paper says it

0167 sconds (60 Hz). Every N = 30 control steps (0.

5 seconds, 2 Hz), the controller captures an RGB frame and asynchronously queries our VLM risk service over HTTP (695 milliseconds average end-to-end round-trip latency). p. 3
Where the paper says it

Every N = 30 control steps (0. 5 seconds, 2 Hz), the controller captures an RGB frame and asynchronously queries our VLM risk service over HTTP (695 milliseconds average end-to-end round-trip latency).

The VLM produces a scalar collision-risk estimate r ∈ [0, 1] from the current egocentric RGB frame, where r = 0 indicates low risk and r = 1 indicates high risk. We interpret r as a hazard score for the robot’s current motion context in relation to its environment (e. p. 3
Where the paper says it

The VLM produces a scalar collision-risk estimate r ∈ [0, 1] from the current egocentric RGB frame, where r = 0 indicates low risk and r = 1 indicates high risk. We interpret r as a hazard score for the robot’s current motion context in relation to its environment (e.

g., whether the next few seconds will likely require evasive action). p. 3
Where the paper says it

We interpret r as a hazard score for the robot’s current motion context in relation to its environment (e. , whether the next few seconds will likely require evasive action).

--- Page 8 ---

B. Dynamic α Cap from Geometric Margin

Because VLM outputs can be delayed and may occasionally be overly permissive, we impose a geometric dynamic cap on α so that the controller cannot behave too aggressively when the robot is close to obstacles. The cap is designed primarily as a safety envelope and fallback mechanism. p. 4

Where the paper says it

Dynamic α Cap from Geometric Margin Because VLM outputs can be delayed and may occasionally be overly permissive, we impose a geometric dynamic cap on α so that the controller cannot behave too aggressively when the robot is close to obstacles. The cap is designed primarily as a safety envelope and fallback mechanism.

--- Page 9 ---

C. Overall Algorithm

The overall control stack per timestep is: 1) If scheduled (once every 30 steps), query the VLM risk service asynchronously and map the returned risk to a candidate αvlm (Section IV-A) 2) Read robot state x and compute geometric safety quantities such as clearance margin to closest obstacle and current speed estimate. (Section IV-B) 3) Compute geometric cap (αcap), determine staleness, and fuse with the latest VLM-based αvlm estimate to produce αfinal (Eq. p. 5

Where the paper says it

Overall Algorithm The overall control stack per timestep is: 1) If scheduled (once every 30 steps), query the VLM risk service asynchronously and map the returned risk to a candidate αvlm (Section IV-A) 2) Read robot state x and compute geometric safety quantities such as clearance margin to closest obstacle and current speed estimate. (Section IV-B) 3) Compute geometric cap (αcap), determine staleness, and fuse with the latest VLM-based αvlm estimate to produce αfinal (Eq.

11-13) 4) Compute a nominal goal-directed control unom, then apply a CBF safety filter, using αfinal, to obtain a safe control usafe (Eq. p. 5
Where the paper says it

(Section IV-B) 3) Compute geometric cap (αcap), determine staleness, and fuse with the latest VLM-based αvlm estimate to produce αfinal (Eq. 11-13) 4) Compute a nominal goal-directed control unom, then apply a CBF safety filter, using αfinal, to obtain a safe control usafe (Eq.

5) 5) Convert usafe into executable robot commands (v, ω) and step the simulator. p. 5
Where the paper says it

11-13) 4) Compute a nominal goal-directed control unom, then apply a CBF safety filter, using αfinal, to obtain a safe control usafe (Eq. 5) Convert usafe into executable robot commands (v, ω) and step the simulator.

--- Page 9 ---

V. Experiments

We evaluate AlphaAdj in NVIDIA IsaacSim [36] using a Carter differential-drive mobile robot with an egocentric RGB camera. The robot starts at rest, and is commanded with bounded linear (max 0. p. 5

Where the paper says it

Simulation Design and Experiment Setup We evaluate AlphaAdj in NVIDIA IsaacSim [36] using a Carter differential-drive mobile robot with an egocentric RGB camera. The robot starts at rest, and is commanded with bounded linear (max 0.

50 meters/second) and angular velocities (max = 0. p. 5
Where the paper says it

The robot starts at rest, and is commanded with bounded linear (max 0. 50 meters/second) and angular velocities (max = 0.

9 radians/second), and each rollout terminates upon reaching the goal, on collision with obstacle (upon any part of the robot contacting an obstacle) or after 1800 simulation steps (30. p. 5
Where the paper says it

50 meters/second) and angular velocities (max = 0. 9 radians/second), and each rollout terminates upon reaching the goal, on collision with obstacle (upon any part of the robot contacting an obstacle) or after 1800 simulation steps (30.

06 seconds). p. 3
Where the paper says it

Every N = 30 control steps (0. 5 seconds, 2 Hz), the controller captures an RGB frame and asynchronously queries our VLM risk service over HTTP (695 milliseconds average end-to-end round-trip latency).

Reaching the goal constitutes a successful run, while a collision or simulation stop constitutes a failed run. p. 5
Where the paper says it

06 seconds). Reaching the goal constitutes a successful run, while a collision or simulation stop constitutes a failed run.

We evaluate on four different scenarios: Single Frontal Obstacle, Cluttered Field, Dynamic Obstacle Crossing, and Dynamic Frontal Obstacle. p. 6
Where the paper says it

(h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom). p. 6

Where the paper says it

(h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).

p. 6
Where the paper says it

(h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).

--- Page 7 ---

TABLE I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).

--- Page 8 ---

Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).

--- Page 9 ---

Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).

--- Page 8 ---

Fig. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).

--- Page 9 ---

Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).

--- Page 8 ---

Fig. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).

--- Page 9 ---

Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Adaptive CBF Parameter Modulation: The system can dynamically adjust a low level CBF parameter that modulates how strongly safety constraints are enforced based on a VLM-derived collision-risk estimate, enabling an adaptive and semantically aware navigation system that better matches the moment-to-moment demands of complex environments.

  2. Latency Mitigation for Asynchronous Inference: The improved system incorporates a latencyaware mechanism that detects stale or anomalous VLM updates and triggers fallback behavior, ensuring safety is maintained even when VLM inference latency is significant, as described by the staleness indicator staled(t) = I(t − tvlm > Tstale).

  3. Geometric Safety Fallback Mechanism: The framework utilizes a geometric dynamic cap that provides a conservative fallback under staleness, ensuring that when the VLM estimate is stale, the controller defaults to a conservative safety filter based on geometric clearance and speed, preventing unsafe behavior from delayed semantic feedback.

  4. Enhanced Efficiency Tradeoff: The system achieves improved efficiency by allowing the VLM to provide higher-value, context-sensitive modulation compared to geometry alone, leading to more agile when appropriate in open spaces and better risk prediction in visually complex scenes.

Abstract

Robots operating in dynamic, unstructured environments must maintain safety without becoming unnecessarily conservative. While control barrier functions (CBFs) provide a principled mechanism for collision avoidance, their behavior is typically determined from geometric state alone and therefore cannot distinguish between semantically different situations with similar physical configurations. We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB observations and a vision-language model (VLM) to adapt CBF conservativeness according to visual semantic context. The VLM produces a scalar contextual caution signal that dynamically modulates the CBF parameter alpha, while a geometry- and speed-dependent cap bounds admissible aggressiveness. To support real-time operation despite VLM latency, inference runs asynchronously with the control loop, and the semantic signal is extracted directly from token-level output probabilities rather than full autoregressive text generation. We evaluate AlphaAdj across matched-geometry, dynamic-human, and cluttered navigation scenarios, as well as on a physical Clearpath Jackal. When obstacle geometry is held fixed, AlphaAdj allocates 0.405 m more clearance to a human than to a benign object, while geometry-only and non-visual adaptive CBF baselines produce no semantic differentiation. This additional behavioral reserve improves robustness to unexpected human motion, including cluttered conditions in which all evaluated geometric baselines collide while AlphaAdj remains collision-free. The system maintains a 0.0167 s control period with 0.158 s mean VLM round-trip latency, demonstrating that semantic visual reasoning can complement geometric safety control without entering the critical control path.

Sources

Related papers