BD-HazardVLM: Probing Vision-Language Models for Latent Defensive-Driving Hazards in Bangladesh Road Scenes

arXiv:2610.20627 · cs.CV, cs.LG, cs.RO · Submitted 2026-09-28 · Read on arXiv

Listen

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "BD-HazardVLM: Probing Vision-Language Models for Latent Defensive-Driving Hazards in Bangladesh Road Scenes".

Jane: The gist: Vision-language models are evaluated on their ability to perform fine-grained defensive driving reasoning in weakly structured Bangladesh road scenes,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So to wrap up, the paper "BD-HazardVLM: Probing Vision-Language Models for Latent Defensive-Driving Hazards in Bangladesh Road Scenes" showed us that testing these models on real Bangladesh scenes reveals a gap between seeing a hazard and knowing how to react correctly.

Jane: The authors created this diagnostic benchmark of five hundred images, using data from RSUD20K and TFP-BD, to specifically probe latent defensive hazards like risk timing and severity in that setting > <ref:2610.20627#pg1>

Lu: What they found is that binary hazard-presence accuracy was decent, but fine-grained reasoning—like category selection or exact action matching—was substantially weaker than the simple detection metrics suggested >

Meng: It’s important to remember that this isn't a validated vehicle safety component; the annotations are for research on multimodal reasoning and driving safety, not for identification or deployment on real traffic right now >

Lalam: The main point is that coarse hazard recognition does not imply reliable fine-grained defensive driving reasoning in these weakly structured environments >

Tom: The implication is that we can't just rely on high accuracy scores from simple detection tasks when we are dealing with the complexity of diverse, real-world traffic like in South Asia >

Jane: Future work mentioned by the authors will involve adding duplicate annotations, video evaluation, and looking at subgroup analysis to see where these models fail most consistently across different conditions >

Lu: They are pushing for a more rigorous way to test these models beyond just static image reasoning and zero-shot evaluations on this single benchmark >

Meng: For us in the deployment space, this tells us we need much deeper testing on edge cases and context-specific action recommendation before we even think about applying these models to actual driving systems >

Lalam: It confirms that for complex scenarios, especially those involving local traffic structures, a simple presence check is not enough to guarantee safe driving behavior >

Conclusion: Jane: So we’ve seen how these vision language models perform on those Bangladesh road scenes, and now we’re getting to the conclusion of this work by looking at their title and who wrote it.

Tom: It's called BD-HazardVLM, and the authors are from a group that did some really specific diagnostic testing on these models.

Lu: They built this benchmark out of five hundred road-scene images, pulling data from two different sources: RSUD20K and TFP-BD.

Meng: That’s a focused set—five hundred images annotated for things like hazard presence, category, timing, and even those trap scenarios.

Jane: So what does that actually mean for us in the real world? What’s the big picture here?

Lalam: Basically, this paper shows us that just seeing a hazard isn't enough to know how to drive safely in these complex settings.

Tom: Exactly. The main point is that coarse hazard recognition doesn't translate into reliable fine-grained defensive driving reasoning.

Jane: That’s a strong statement. So, if we only rely on the basic detection accuracy, are we missing something crucial about how the AI thinks?

Lu: Yes, because when you look at category accuracy or exact action matching—those finer details—they drop way down to maybe twenty percent or less for all models tested.

Meng: From an engineering standpoint, that means if we deploy a system just because it flags a hazard, we have to build in massive safety layers on top of it because the fine detail reasoning is basically broken.

Tom: And you’re right about the deployment side. It highlights that this isn't just about getting a high score; it’s about understanding where the models actually fail when things get complicated, like risk timing.

Jane: It shows we need geographically diverse safety evaluations because what works in one setting doesn't guarantee safety in another.

Lalam: The implication is that for multimodal reasoning tasks involving driving, we need to focus on the subtle temporal and contextual understanding, not just the visual identification of an object.

Tom: So the big picture here is that we need better testing methods that go beyond simple accuracy numbers to truly assess defensive capabilities in these challenging environments.

Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology (RUET)

cs.CV, cs.LG, cs.RO

Submitted: 2026-09-28

Updated: 2026-09-28

Comments: Accepted at the ECCV 2026 Workshop on Safe and Defensive Autonomous Driving (SDAD). 16 pages, 1 figure, 5 tables

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 79/100

The gist: The gist: Vision-language models are evaluated on their ability to perform fine-grained defensive driving reasoning in weakly structured Bangladesh road scenes, revealing that coarse hazard

Key concepts

BD-HazardVLM Benchmark: This is a specific test set of 500 road images from Bangladesh (375 from RSUD20K and 125 from TFP-BD). It provides detailed labels for various hazard types, risk timing, severity, and required defensive actions. It is designed to test image-level reasoning rather than object detection.
Fine-Grained Reasoning: This refers to the model's ability to perform detailed analysis of a scene beyond simply saying 'there is a hazard.' It involves accurately identifying specific details such as the exact category of hazard (e.g., informal pedestrian crossing vs. road damage) or precisely what defensive action should be taken, which is much harder than simple presence detection.
Coarse Hazard Recognition: This is the model's basic ability to detect if *any* hazard is present in an image. The study found that models are relatively good at this, achieving 88-90% accuracy for binary presence. However, the paper concludes that this coarse recognition does not guarantee reliable performance on the more difficult, fine-grained reasoning tasks required for safe driving decisions.
Western-Road Bias Traps: These are specific types of errors or biases in the models related to Western road environments. The study assessed how well models recognize these traps, finding that most models showed very poor recall (near zero) when attempting to identify them, suggesting a lack of generalized understanding for non-Western road structures.

Terminology

Vision-language models — 1× in this paper · explained in 2 episodes

Summary

The gist: Vision-language models are evaluated on their ability to perform fine-grained defensive driving reasoning in weakly structured Bangladesh road scenes, revealing that coarse hazard recognition does not imply reliable fine-grained defensive-driving reasoning

BD-HazardVLM Benchmark Introduction

The BD-HazardVLM is introduced as a diagnostic benchmark of 500 Bangladesh road-scene images—375 from RSUD20K and 125 from TFP-BD—annotated for hazard presence, primary hazard category, risk timing, severity, defensive actions, visible evidence, hallucination-, and Western-road-bias traps The benchmark is designed for image-level reasoning rather than object detection or trajectory prediction The taxonomy contains twelve mutually exclusive primary hazard categories, including informal pedestrian crossing, rickshaw or easybike hazard, and road-surface damage Risk timing and severity labels are image-level defensive assessments based on visible scene evidence

Model Evaluation Setup

The evaluation involves testing Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 using a shared zero-shot task definition and structured output schema All models process the same final set of 500 images in a zero-shot setting without task-specific training or fine-tuning Model-specific inference settings are summarized in Table 3, detailing image processing parameters such as Qwen2.5-VL7B using 4-bit NF4 quantization with double quantization and FP16 computation

Performance Findings on Reasoning Tasks

The results show that binary hazard-presence accuracy is misleading, as a trivial majority-class predictor reaches 89.0%, essentially matching all evaluated models Fine-grained reasoning is substantially harder, with hazard-category accuracy remaining below 40% for all evaluated models Risk timing is the weakest evaluated capability, with InternVL3-8B obtaining 12.4% accuracy, Qwen2.5-VL-7B obtaining 9.2%, and MiniCPM-V-4.6 obtaining only 1.6% Exact action-set matching remains below 2% across the models

Specific Model Failures

Qwen exhibits severe category mode collapse, assigning the rickshaw/easybike hazard category to 426 of 500 images (85.2%), although this category appears in only 205 ground-truth images (41.0%) InternVL3-8B achieves the highest aggregate severity accuracy at 61.2%, yet it has zero recall on the 58 high-severity cases MiniCPM-V-4.6 recognizes more high-severity cases but also overpredicts that class

Trap Recognition and Bias Assessment

All models show weak latent-hazard anticipation and near-zero self-reported trap recognition For hallucination traps, recall is 0.0%, 0.3%, and 0.0% for Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6, respectively For Western-road bias traps, recall is 1.1%, 1.8%, and 0.0% for Qwen2.5-VL-7B, InternVL3-8B, and MiniCPM-V-4.6 These results measure self-reported trap recognition rather than actual hallucination frequency

Conclusion on Model Capabilities

The evaluation demonstrates that coarse hazard recognition does not ensure reliable fine-grained defensive-driving reasoning Future work will add duplicate annotation, video evaluation, broader baselines, and subgroup analysis The original images will not be redistributed; instead, the public release will contain the image manifest, source-image identifiers, annotations, taxonomy and annotation guidelines BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original source images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads BDHazardVLM is not a validated vehicle-safety component The annotations are intended for research on multimodal reasoning and driving safety, not identification, surveillance, or safety-critical deployment The original images will not be redistributed BDHazardVLM is a focused 500-image diagnostic benchmark rather than a representative sample of all Bangladesh roads <ref

Improvements for AI systems

Improve hazard reasoning by implementing a structured diagnostic approach that explicitly separates coarse recognition from fine-grained reasoning, as noted in Section 5: coarse hazard recognition does not imply reliable fine-grained defensive-driving reasoning. This allows for targeted model refinement based on where failures occur across the taxonomy.
Enhance risk timing and severity estimation by incorporating dynamic ego-state information (speed, acceleration) into the evaluation protocol, addressing the limitation that current image-level labels are not physical estimates of time-to-collision or dynamic collision probability. This would allow models to better assess whether the risk is immediate, latent, both, or none based on motion.
Improve defensive action selection by moving beyond general safety language toward context-specific recommendations by training models specifically on the required vocabulary: monitor occlusion, avoid overtaking, keep distance, etc., instead of merely recommending slow down and prepare to brake.
Address hallucination and bias susceptibility by developing a mechanism to measure actual unsupported reasoning rather than self-reported flags: These results measure self-reported trap recognition rather than actual hallucination frequency. This requires integrating separate, evidence-based human adjudication for the final raw model responses.
Mitigate category mode collapse by introducing regularization techniques during training that prevent models from assigning disproportionate weight to locally salient vehicle classes, which were observed when Qwen2.5-VL-7B assigns the rickshaw/easybike hazard category to 426 of 500 images (85.2%), although this category appears in only 205 ground-truth images.