Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Robotics Radio. Generated commentary on the latest robotics and control papers.
Rosa: We're looking at the paper: "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation".
Dev: Here's the gist of the paper. Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>.
Rosa: AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB input to adapt the conservativeness of a CBF safety filter in real time.
Dev: The results show that AlphaAdj maintains collision-free navigation while improving efficiency (in terms of path length and time to goal) by up to 18.5% relative to fixed settings and improving robustness and success rate relative to an uncapped baseline.
Rosa: Core Methodology The framework couples high-level visual semantic perception with low-level safety behavior adaptation at runtime.
Paper summary: Rosa: So we're looking at this paper called "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation." Basically, it’s about how robots need to balance staying safe and moving efficiently when they don't have perfect sensing in messy places.
Dev: Yeah, the core idea here is using a vision-to-control navigation framework that takes what the robot sees from its own camera—egocentric RGB input—and uses that to adjust how cautious or aggressive a safety filter should be, and it does this in real time.
Taro: I'm curious about what they claim about this adaptation. Is it just making the old safety settings smarter, or is the whole control approach fundamentally different?
Rosa: It’s more than just smart settings; they use a vision-language model, or VLM, to produce a bounded scalar risk estimate from the current camera view and then map that risk to dynamically update a control barrier function parameter that dictates how strongly safety constraints are enforced.
Dev: So they have this risk score between zero and one, where zero means low risk and one means high risk. They use this score to adjust something called alpha, which controls the conservativeness of the safety filter.
Taro: If I'm looking at that number between zero and one, what does that actually mean for a robot’s immediate next move? Is it like saying "I need to slow down" or "I need to stop"?
Rosa: It represents a hazard score for the robot’s current motion context in relation to its environment, which is interpreted as whether the next few seconds will likely require evasive action. They map that risk score using a function that controls sensitivity, and then they cap it with some geometric and speed-aware dynamic cap based only on geometry and speed.
Dev: That dynamic cap is important because they acknowledge that the VLM output can be delayed or sometimes overly permissive, so they have a staleness-gated fusion policy that defaults to conservative behavior if the risk estimate isn't fresh.
Taro: So when you’re in a fast-moving situation and the vision system lags, what’s the actual safety guarantee? Does this setup give you a hard limit on how aggressive the robot can get?
Paper summary: Rosa: They use that geometric, speed-aware dynamic cap as a safety envelope and fallback mechanism so the controller cannot behave too aggressively when it’s close to obstacles. The final parameter used by the control barrier function is then determined by taking the minimum of three things: the VLM estimate when fresh, that geometric cap, and a hard cap if the VLM estimate is stale.
Dev: And this whole process runs against a low-level controller that executes every zero point zero one six seven seconds, which gives them a certain control rate to work with while still allowing for asynchronous queries to the VLM service.
Taro: If you were testing this on a real robot outside the lab, what kind of physical challenges might cause this system to struggle when the world misbehaves?
Rosa: They tested it across four scenarios in NVIDIA IsaacSim: a single frontal obstacle, a cluttered field, dynamic obstacle crossing, and dynamic frontal obstacle. They found that AlphaAdj reached the goal in eight out of eight setups with zero collisions.
Dev: That’s impressive because it shows that integrating this semantic risk modulation doesn't sacrifice safety at all. They compared their results to fixed parameters and an uncapped baseline, and they saw improvements in efficiency too.
Taro: What did those efficiency improvements actually look like? I mean, if you have to be more conservative, isn't that just making the robot slower overall?
Rosa: Well, in the warehouse dynamic obstacle scenario specifically, AlphaAdj increased path length by three point two one meters, which is an eighteen point five percent increase compared to fixed settings. And it also improved time-to-goal by one point two seven seconds, which is ten point eight percent faster than before.
Dev: So the trade-off is clear: a conservative alpha of zero point one means longer path length and more time to goal, but an aggressive alpha of zero point six leads to a collision if things go wrong. AlphaAdj sits in the middle.
Taro: The paper also points out that synchronous approaches are bounded by the VLM response time, which is a key limitation for real-time systems that need consistent control rates.
Rosa: Exactly, and they noted their practical implementation choices were designed specifically to reduce end-to-end VLM latency in their system. They showed that even with the typical six hundred ninety five millisecond average round-trip latency when querying the service every three steps, the overall system maintains a control rate suitable for real systems.
Paper summary: Dev: So, when you look at the title, "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation," it really captures that whole loop: using semantic vision to adapt safety functions in real time. It’s not just about seeing an obstacle; it’s about deciding how much to trust your safety filter based on what the vision says.
Rosa: Yeah, and the authors show that this framework is deployable because they address latency and staleness with a fusion policy that prioritizes hard caps when the risk estimate is old. It shows how you can integrate semantic risk into safety filtering in a way that actually works in practice.
Taro: What I want to push on is where it stops working, right? The authors mention that while they achieved zero collisions across those scenarios, real-world deployment introduces additional physical challenges that aren't fully modeled here.
Dev: That’s a fair point about the limitations they mentioned in their future work section. They suggest further calibration and interpretability of the VLM risk score could improve reliability across different domains, which is a big area for future research.
Rosa: It seems like this work moves the safety envelope from being purely geometric or pre-programmed to being contextually informed by high-level semantic understanding from the vision model. It’s about making the robot smarter about *why* it should be cautious, not just *how much* caution to apply.
Taro: So, for someone just listening, what does this mean? It means robots can navigate cluttered spaces more safely because they are using a richer understanding of the scene to modulate their safety response dynamically.
Dev: It means you get better efficiency compared to fixed settings without having to sacrifice the fundamental guarantee that the robot won't collide, provided you handle those latency issues correctly with dynamic caps and staleness checks.
Rosa: It’s a practical solution for robots operating in dynamic, unstructured environments where they have limited sensing, like warehouses or construction sites. This paper gives us a concrete example of how to bridge the gap between high-level vision intelligence and low-level real-time safety loops.
Conclusion: Rosa: So, we’re wrapping up this look at "Semantically Adaptive Control Barrier Functions with Vision-Language Models for Real-Time Robot Navigation." Essentially, this paper shows how you can use what a robot sees to make its safety filters smarter while it’s moving around in messy places.
Dev: That’s right, the whole idea revolves around taking that vision input and using a vision language model to adjust how conservative or aggressive the robot's safety rules are on the fly.
Taro: It seems like they’re not just adding another sensor; they are changing *how* the safety filter thinks about what it sees.
Rosa: Exactly, they map the risk score from that vision model to a parameter called alpha, which directly controls how strongly those safety constraints are enforced at runtime.
Dev: And the authors tackle a big problem there: latency and the fact that these vision models aren't always instantaneous. They introduce a dynamic cap and a staleness gate to make sure the system stays safe even when the risk estimate is old.
Taro: So, they’re balancing high-level visual understanding with strict real-time control loops by adding these layers of checks.
Rosa: Right, and their experiments in simulation showed that this works reliably across different challenging scenarios—like a cluttered field or dynamic obstacle crossings—keeping collisions at zero.
Dev: The results show they can actually improve efficiency compared to just using fixed safety settings, so the robot moves faster and covers more ground without sacrificing that safety guarantee.
Taro: It makes sense that if the system is robust enough to handle those four different scenarios, it suggests this could be a real step toward giving robots more nuanced decision-making in unpredictable environments.
Rosa: It certainly suggests that for future field testing, you won't just need perfect geometry; you’ll need a model that understands the *risk* associated with what’s around it.
Dev: That brings us to the real question: how much do we actually trust this semantic risk score when a robot is running outside of a perfectly controlled lab setting?
Jeffrey Chen, Rohan Chandra
University of Virginia
cs.RO
Submitted: 2026-03-22
Updated: 2026-10-30
Code: https://github.com/isaac-sim/IsaacSim
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 87/100
The gist: Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>.
Key concepts
- Control Barrier Function (CBF)
- A mathematical tool used for safety filtering in robotics that ensures a robot stays within a safe region by constraining its control inputs. It acts as a hard safety filter, preventing the robot from taking actions that would lead to collisions or unsafe states, regardless of external commands.
- Vision-Language Model (VLM)
- A model that takes visual input from an RGB camera and generates a risk estimate. It interprets the scene context—like obstacles or dynamic objects—and outputs a score indicating how dangerous the current situation is, which helps decide how much safety margin to apply.
- Dynamic $\alpha$ Cap ($\alpha_{cap}$)
- A geometric limit calculated based on the robot's current speed and its distance to the nearest obstacle. This cap acts as a hard safety fallback, ensuring that even if the VLM provides an aggressive suggestion, the robot's movement remains physically constrained and safe near hazards.
- Staleness-Gated Fusion Policy
- A mechanism to manage latency when using the VLM. If a new risk estimate from the VLM is too old (stale), the system ignores it and relies on a more conservative, pre-calculated safety cap ($\alpha_{hard cap}$), ensuring real-time control stability despite slow AI inference.
Terminology
Summary
Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603. p. 0 Dynamic Control Barrier Function Regulation with Vision-Language
Models for Safe, Adaptive, and Realtime Visual Navigation
Jeffrey Chen, Rohan Chandra
Supplementary Material, Videos at the following website: saferobotnav. io/AlphaAdj
Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially
limited sensing. While control barrier functions (CBFs) provide
principled collision avoidance via safety filtering, their behavior
is often governed by fixed parameters that can be overly
conservative in benign scenes or overly permissive near hazards. We present AlphaAdj, a vision-to-control navigation framework
that uses egocentric RGB input to adapt the conservativeness
of a CBF safety filter in real time. We evaluate AlphaAdj across multiple static
and dynamic obstacle scenarios in a variety of environments,
comparing against fixed-parameter and uncapped ablations. Results show that AlphaAdj maintains collision-free navigation
while improving efficiency (in terms of path length and time to
goal) by up to 18. Results show that AlphaAdj maintains collision-free navigation
while improving efficiency (in terms of path length and time to
goal) by up to 18. 5% relative to fixed settings and improving
robustness and success rate relative to an uncapped baseline.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Core Methodology
The framework couples high-level visual semantic perception with low-level safety behavior adaptation at runtime. The approach extracts context from the egocentric RGB scene and outputs a collision-risk estimate using a Vision-Language Model (VLM). p. 0 In this work, we present an adaptive navigation algorithm
that couples high-level visual semantic perception to adapt
low-level safety behavior at runtime. Our framework extracts context from the egocentric RGB
scene and outputs a collision-risk estimate using a VLM. Our framework extracts context from the egocentric RGB
scene and outputs a collision-risk estimate using a VLM. The risk estimate is used to dynamically adjust a low level
CBF control parameter which governs how conservatively
(or assertively) the robot responds to potential hazards.Where the paper says it
Where the paper says it
Risk Estimation and Mapping
The VLM produces a scalar collision-risk estimate r ∈ [0, 1] from the current egocentric RGB frame, where r = 0 indicates low risk and r = 1 indicates high risk. p. 3 5
seconds, 2 Hz), the controller captures an RGB frame and
asynchronously queries our VLM risk service over HTTP
(695 milliseconds average end-to-end round-trip latency). The VLM produces a scalar collision-risk estimate r ∈
[0, 1] from the current egocentric RGB frame, where r = 0
indicates low risk and r = 1 indicates high risk. , whether the next few
seconds will likely require evasive action). We map risk to a
bounded scalar αvlm using the monotone decreasing function
αvlm(r) = αmin + (αmax − αmin) (1 − r)
γ
, (6)
where αmin ≤ αvlm ≤ αmax and γ > 0 controls sensitivity. : Risk is obtained by
querying a local HTTP endpoint with a PNG image and a text
instruction. The instruction presented to the VLM is formed
by concatenating (i) a fixed server-side formatting constraint
that enforces a strict JSON response and (ii) a client-provided
context string that is updated online with the robot’s current
state.Where the paper says it
Where the paper says it
Where the paper says it
Latency Management and Safety Guarantees
To address asynchronous inference and nontrivial VLM latency, the framework introduces a dynamic cap and a staleness-gated fusion policy. A geometric, speed-aware dynamic cap αcap is used to guarantee an upper bound on aggressiveness based only on geometry and speed. p. 4 The cap is designed
primarily as a safety envelope and fallback mechanism. It
guarantees an upper bound on aggressiveness based only
on geometry and speed, independent of semantic estimation
quality. c) Fusion policy: We combine αvlm with the caps using
a simple fusion policy: when VLM estimate is fresh, we set
αfinal = min
αvlm, αsoft
cap
, (12)
which ensures the VLM can only reduce conservativeness
up to the safe envelope imposed by the soft cap. When the
VLM estimate is stale, we ignore αvlm and fall back to the
hard cap,
αfinal = α
hard
cap .Where the paper says it
Where the paper says it
Experimental Validation
Experiments were conducted in NVIDIA IsaacSim across four scenarios: Single Frontal Obstacle, Cluttered Field, Dynamic Obstacle Crossing, and Dynamic Frontal Obstacle. The results show that AlphaAdj reaches the goal in 8/8 setups with 0 collisions, demonstrating that integration of semantic risk modulation can be done without sacrificing safety. p. 6 Additionally, we report quantitative results for each scenario
in the warehouse environment in Table I. Across all 8 environment–scenario pairs, AlphaAdj
reaches the goal in 8/8 setups with 0 collisions, showing
that integration of semantic risk modulation of the CBF
safety filter can be done without sacrificing safety. However, in the warehouse dynamic obstacle scenario, while
αmin did not collide, it also was unable to reach its destination
due to its conservativeness, as seen in 4c. Furthermore,
this conservativeness drastically increases detours: relative to
AlphaAdj, averaged over all eight setups, fixed αmin increases
path length by 3. Equation 8 defines a normalized clearance score s
s ∈ [0, 1] from raw margin m. We scale m by Msafe and clip
at 1 so that (i) s = 0 at safety boundary, (ii) s = 1 once the
robot is at least Msafe meters away from the closest obstacle,
and larger clearances do not further increase the cap. We evaluate AlphaAdj across multiple static
and dynamic obstacle scenarios in a variety of environments,
comparing against fixed-parameter and uncapped ablations. Results show that AlphaAdj maintains collision-free navigation
while improving efficiency (in terms of path length and time to
goal) by up to 18. Every N = 30 control steps (0. 5
seconds, 2 Hz), the controller captures an RGB frame and
asynchronously queries our VLM risk service over HTTP
(695 milliseconds average end-to-end round-trip latency). The only
other setting that consistently avoids collisions is fixed αmin. However, in the warehouse dynamic obstacle scenario, while
αmin did not collide, it also was unable to reach its destination
due to its conservativeness, as seen in 4c.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Efficiency Tradeoff Analysis
The analysis reveals a clear efficiency–safety tradeoff, where conservative α=0.1 increases path length and time to goal, while an aggressive α=0. p. 6 Averaged over 10 runs, the results show a clear
efficiency–safety tradeoff: conservative α=0. 1 increases
path length and time to goal, while an aggressive α=0. 1 increases
path length and time to goal, while an aggressive α=0. 6
leads to collision. Compared to no
cap, AlphaAdj is more efficient in when it comes to both
path length and time to goal in the scenarios shown in
Figures 4b, 4c, 4d, 4e, 4f, 4g. Our success in every scenario is notable given our controller runs continuously at a 60 Hz control rate (one control
step every 0. 5%) and time-to-goal by 1. 27
seconds (10. 2 seconds, further illustrating how
synchronous designs are bounded by VLM response time. In contrast, our approach maintains a control rate suitable
for real systems while still enabling dynamic VLM-guided
adjustment for safe and efficient goal reaching.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Limitations and Future Directions
Real robotic deployment may introduce additional physical challenges, but the work represents a natural next step for harnessing the benefits of the design. Future work could also center on further calibration and interpretability of the VLM risk score, which could further improve reliability across a multitude of domains. p. 7 Real robotic
deployment may introduce additional physical challenges,
but it represents a natural next step for harnessing the benefits
of our design, enabling more diverse sensing and varied
interactions. Future work could also center on further calibration and interpretability of the VLM risk score, which could
further improve reliability across a multitude of domains. 2 seconds, further illustrating how
synchronous designs are bounded by VLM response time. In contrast, our approach maintains a control rate suitable
for real systems while still enabling dynamic VLM-guided
adjustment for safe and efficient goal reaching.Where the paper says it
Where the paper says it
References
arXiv:2603.21142v1[cs. p. 0 To mitigate this, we introduce a latencyaware mechanism that detects stale or anomalous VLM
arXiv:2603. 21142v1 [cs.Where the paper says it
--- Page 5 ---
Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially limited sensing <ref:2603.21142#pg8>. While control barrier functions (CBFs) provide principled collision avoidance via safety filtering, their behavior is often governed by fixed parameters that can be overly conservative in benign scenes or overly permissive near hazards. p. 0 io/AlphaAdj
Abstract— Robots operating in dynamic, unstructured environments must balance safety and efficiency under potentially
limited sensing. While control barrier functions (CBFs) provide
principled collision avoidance via safety filtering, their behavior
is often governed by fixed parameters that can be overly
conservative in benign scenes or overly permissive near hazards. While control barrier functions (CBFs) provide
principled collision avoidance via safety filtering, their behavior
is often governed by fixed parameters that can be overly
conservative in benign scenes or overly permissive near hazards. We present AlphaAdj, a vision-to-control navigation framework
that uses egocentric RGB input to adapt the conservativeness
of a CBF safety filter in real time. We present AlphaAdj, a vision-to-control navigation framework
that uses egocentric RGB input to adapt the conservativeness
of a CBF safety filter in real time. A vision-language model
(VLM) produces a bounded scalar risk estimate from the
current camera view, which we map to dynamically update a
CBF parameter that modulates how strongly safety constraints
are enforced. A vision-language model
(VLM) produces a bounded scalar risk estimate from the
current camera view, which we map to dynamically update a
CBF parameter that modulates how strongly safety constraints
are enforced. To address asynchronous inference and nontrivial VLM latency in practice, we combine a geometric,
speed-aware dynamic cap and a staleness-gated fusion policy
with lightweight implementation choices that reduce end-to-end
inference overhead. To address asynchronous inference and nontrivial VLM latency in practice, we combine a geometric,
speed-aware dynamic cap and a staleness-gated fusion policy
with lightweight implementation choices that reduce end-to-end
inference overhead. We evaluate AlphaAdj across multiple static
and dynamic obstacle scenarios in a variety of environments,
comparing against fixed-parameter and uncapped ablations.Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 6 ---
Fig. 1: System overview. p. 2 Because VLM inference is asynchronous
and introduces non-trivial latency, we propose a dynamic
cap and mitigation policy for stale requests that guarantees
conservative behavior when the VLM output is delayed. The
overall system overview can be seen in Figure 1 1: System overview. Egocentric RGB frames are processed
asynchronously by a VLM to produce a risk estimate r, mapped to
αvlm. Egocentric RGB frames are processed
asynchronously by a VLM to produce a risk estimate r, mapped to
αvlm. A geometric, speed-aware cap αcap and a staleness test gate
the VLM output to produce αfinal, which parameterizes the CBF
safety filter that produces safe commands.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 7 ---
A. Risk-to-α Mapping from VLM Output
We consider a mobile robot with egocentric RGB camera sensing operating in a cluttered environment with static or dynamic obstacles. The low-level controller executes every 0. p. 3 Risk-to-α Mapping from VLM Output
We consider a mobile robot with egocentric RGB camera
sensing operating in a cluttered environment with static or
dynamic obstacles. The low-level controller executes every
0. The low-level controller executes every
0. 0167 sconds (60 Hz). 0167 sconds (60 Hz). Every N = 30 control steps (0. Every N = 30 control steps (0. 5
seconds, 2 Hz), the controller captures an RGB frame and
asynchronously queries our VLM risk service over HTTP
(695 milliseconds average end-to-end round-trip latency). The VLM produces a scalar collision-risk estimate r ∈
[0, 1] from the current egocentric RGB frame, where r = 0
indicates low risk and r = 1 indicates high risk. We interpret
r as a hazard score for the robot’s current motion context
in relation to its environment (e. We interpret
r as a hazard score for the robot’s current motion context
in relation to its environment (e. , whether the next few
seconds will likely require evasive action).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 8 ---
B. Dynamic α Cap from Geometric Margin
Because VLM outputs can be delayed and may occasionally be overly permissive, we impose a geometric dynamic cap on α so that the controller cannot behave too aggressively when the robot is close to obstacles. The cap is designed primarily as a safety envelope and fallback mechanism. p. 4 Dynamic α Cap from Geometric Margin
Because VLM outputs can be delayed and may occasionally be overly permissive, we impose a geometric dynamic
cap on α so that the controller cannot behave too aggressively
when the robot is close to obstacles. The cap is designed
primarily as a safety envelope and fallback mechanism.Where the paper says it
--- Page 9 ---
C. Overall Algorithm
The overall control stack per timestep is: 1) If scheduled (once every 30 steps), query the VLM risk service asynchronously and map the returned risk to a candidate αvlm (Section IV-A) 2) Read robot state x and compute geometric safety quantities such as clearance margin to closest obstacle and current speed estimate. (Section IV-B) 3) Compute geometric cap (αcap), determine staleness, and fuse with the latest VLM-based αvlm estimate to produce αfinal (Eq. p. 5 Overall Algorithm
The overall control stack per timestep is:
1) If scheduled (once every 30 steps), query the VLM risk
service asynchronously and map the returned risk to a
candidate αvlm (Section IV-A)
2) Read robot state x and compute geometric safety quantities such as clearance margin to closest obstacle and
current speed estimate. (Section IV-B)
3) Compute geometric cap (αcap), determine staleness, and
fuse with the latest VLM-based αvlm estimate to produce
αfinal (Eq. (Section IV-B)
3) Compute geometric cap (αcap), determine staleness, and
fuse with the latest VLM-based αvlm estimate to produce
αfinal (Eq. 11-13)
4) Compute a nominal goal-directed control unom, then
apply a CBF safety filter, using αfinal, to obtain a safe
control usafe (Eq. 11-13)
4) Compute a nominal goal-directed control unom, then
apply a CBF safety filter, using αfinal, to obtain a safe
control usafe (Eq. 5) Convert usafe into executable robot commands (v, ω) and
step the simulator.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 9 ---
V. Experiments
We evaluate AlphaAdj in NVIDIA IsaacSim [36] using a Carter differential-drive mobile robot with an egocentric RGB camera. The robot starts at rest, and is commanded with bounded linear (max 0. p. 5 Simulation Design and Experiment Setup
We evaluate AlphaAdj in NVIDIA IsaacSim [36] using
a Carter differential-drive mobile robot with an egocentric
RGB camera. The robot starts at rest, and is commanded
with bounded linear (max 0. The robot starts at rest, and is commanded
with bounded linear (max 0. 50 meters/second) and angular
velocities (max = 0. 50 meters/second) and angular
velocities (max = 0. 9 radians/second), and each rollout
terminates upon reaching the goal, on collision with obstacle
(upon any part of the robot contacting an obstacle) or after
1800 simulation steps (30. Every N = 30 control steps (0. 5
seconds, 2 Hz), the controller captures an RGB frame and
asynchronously queries our VLM risk service over HTTP
(695 milliseconds average end-to-end round-trip latency). 06 seconds). Reaching the goal
constitutes a successful run, while a collision or simulation
stop constitutes a failed run. (h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom). p. 6 (h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom). (h) Toy, dynamic frontal obstacle. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 7 ---
TABLE I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).
--- Page 8 ---
Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).
--- Page 9 ---
Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).
--- Page 8 ---
Fig. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).
--- Page 9 ---
Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).
--- Page 8 ---
Fig. 4: Top-down trajectories in four warehouse scenarios (top) and four toy scenarios (bottom).
--- Page 9 ---
Table I: Warehouse: Baseline comparisons across scenarios (AlphaAdj, no cap, fixed αmin, fixed αmax).
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Adaptive CBF Parameter Modulation: The system can dynamically adjust a low level CBF parameter that modulates how strongly safety constraints are enforced based on a VLM-derived collision-risk estimate, enabling an
adaptive and semantically aware navigation system that better matches the moment-to-moment demands of complex environments.
-
Latency Mitigation for Asynchronous Inference: The improved system incorporates a
latencyaware mechanism that detects stale or anomalous VLM updates and triggers fallback behavior,
ensuring safety is maintained even when VLM inference latency is significant, as described by the staleness indicator staled(t) = I(t − tvlm > Tstale). -
Geometric Safety Fallback Mechanism: The framework utilizes a
geometric dynamic cap
that provides aconservative fallback under staleness,
ensuring that when the VLM estimate is stale, the controller defaults to a conservative safety filter based on geometric clearance and speed, preventing unsafe behavior from delayed semantic feedback. -
Enhanced Efficiency Tradeoff: The system achieves improved efficiency by allowing the VLM to provide
higher-value, context-sensitive modulation
compared to geometry alone, leading tomore agile when appropriate
in open spaces and better risk prediction in visually complex scenes.
Abstract
Robots operating in dynamic, unstructured environments must maintain safety without becoming unnecessarily conservative. While control barrier functions (CBFs) provide a principled mechanism for collision avoidance, their behavior is typically determined from geometric state alone and therefore cannot distinguish between semantically different situations with similar physical configurations. We present AlphaAdj, a vision-to-control navigation framework that uses egocentric RGB observations and a vision-language model (VLM) to adapt CBF conservativeness according to visual semantic context. The VLM produces a scalar contextual caution signal that dynamically modulates the CBF parameter alpha, while a geometry- and speed-dependent cap bounds admissible aggressiveness. To support real-time operation despite VLM latency, inference runs asynchronously with the control loop, and the semantic signal is extracted directly from token-level output probabilities rather than full autoregressive text generation. We evaluate AlphaAdj across matched-geometry, dynamic-human, and cluttered navigation scenarios, as well as on a physical Clearpath Jackal. When obstacle geometry is held fixed, AlphaAdj allocates 0.405 m more clearance to a human than to a benign object, while geometry-only and non-visual adaptive CBF baselines produce no semantic differentiation. This additional behavioral reserve improves robustness to unexpected human motion, including cluttered conditions in which all evaluated geometric baselines collide while AlphaAdj remains collision-free. The system maintains a 0.0167 s control period with 0.158 s mean VLM round-trip latency, demonstrating that semantic visual reasoning can complement geometric safety control without entering the critical control path.
Sources
- Learning to Jump from Pixels
- Decentralized Safe and Scalable Multi-Agent Control under Limited Actuation
- An Introduction to Vision-Language Modeling
- SAFER-Splat: A Control Barrier Function for Safe Navigation with Online Gaussian Splatting Maps
- AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge
- VLM-Social-Nav: Socially Aware Robot Navigation through Scoring using Vision-Language Models
- Adaptive Control Barrier Functions for Safety-Critical Systems
- Learning to Refine Input Constrained Control Barrier Functions via Uncertainty-Aware Online Parameter Adaptation
- Robust Adaptive Time-Varying Control Barrier Function with Application to Robotic Surface Treatment
- Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Survey
- How Should a Robot Assess Risk? Towards an Axiomatic Theory of Risk in Robotics
- Risk-Aware Safety Filters with Poisson Safety Functions and Laplace Guidance Fields
- Perception-Integrated Safety Critical Control via Analytic Collision Cone Barrier Functions on 3D Gaussian Splatting
- Spatio-Temporal Graph Localization Networks for Image-based Navigation
- LISN: Language-Instructed Social Navigation with VLM-based Controller Modulating
- End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering
Related papers
- Rethinking Actions: Continuity-Aware Priors for World-Model Planning
- MOSAIC: Modular Obstacle and Spatial-Aware Framework for Manipulation In Cluttered Scenes
- SAGAS: Semantic-Aware Graph-Assisted Stitching for Offline Temporal Logic Planning
- DexSeed: Expanding Human Hand-Object Interactions into Robot Demonstrations for Dexterous Policy Learning
- LaST0.5: A Generalist World Action Model with Interaction-Centric Latent Reasoning
- CIALLO: Learning Shared Intent Representations for Cross-Embodiment Manipulation