SafeSceneReason: A Multimodal Reasoning Benchmark Connecting Industrial Hazards with Accident Knowledge

SafeSceneReason is a multimodal industrial-safety reasoning benchmark and companion training corpus that connects workplace scenes with knowledge from occupational accident investigations. The paper introduces two complementary data-construction pipelines: a scene-centric pipeline that converts annotated workplace images into executable safety scene graphs and generates deterministic answers through program execution over objects, relations, and safety rules; and a report-centric pipeline that extracts figures and contextual evidence from accident reports and constructs multimodal questions using evidence graphs, explicit information boundaries, multi-step reasoning paths, and iterative verification.

The resulting resource contains 110,581 verified scene-centric question–answer pairs and 13,114 refined report-centric question–answer pairs, covering perception, spatial and quantitative reasoning, compliance assessment, evidence synthesis, causal analysis, and mitigation-oriented decision making. The scene-centric component uses industrial and construction datasets with annotations for workers, PPE, machinery, tools, activities, and hazardous conditions, normalizing YOLO- and COCO-format annotations into safety scene graphs. The report-centric component begins with more than 80,000 publicly available accident reports collected from OSHA, NIOSH FACE, and CSB, retaining reports with sufficiently informative figures and contextual evidence for multimodal data construction.

Evaluation of eight representative vision–language models—GPT-5.6-Sol, GPT-5.5, Gemini-3.1-Pro, Gemini-3.5-Flash, Claude-Fable-5, Kimi-K2.6, Qwen3.6-27B, and Qwen3.5-9B—together with two safety-domain adaptations of Qwen3.5-9B, reveals substantial performance differences and persistent weaknesses in comparative, technical, and multi-evidence reasoning. The 1,000-example test set is balanced across answer positions and spans eight reasoning categories. Results show that CoT-SFT Qwen3.5-9B obtains the highest micro accuracy (89.0%), slightly exceeding GPT-5.6-Sol (88.7%) and GPT-5.5 (87.9%). GPT-5.5 achieves the best macro accuracy (89.5%), followed by GPT-5.6-Sol (89.3%) and CoT-SFT (88.7%). Gemini-3.1-Pro and Gemini-3.5-Flash form a second tier at 80.3% and 78.7%, while Claude-Fable-5 and Kimi-K2.6 obtain 74.1% and 70.7%. The two untuned open Qwen models reach only 40.9% and 37.4%, producing a 51.3-point spread among base models.

Among base models, GPT-5.6-Sol leads quantitative (87.8%), spatial/structural (86.6%), causal (93.3%), and hazard-identification (89.3%) tasks, while GPT-5.5 leads comparative reasoning (86.1%), evidence synthesis (93.1%), compliance assessment (94.6%), and mitigation/decision making (96.3%). Safety-domain supervision substantially improves Qwen3.5-9B: answer-only SFT raises it from 37.4% to 75.1% micro accuracy, while CoT SFT reaches 89.0% micro and 88.7% macro accuracy, corresponding to gains of 51.6 and 46.8 points over the base model. The largest category improvements occur in comparative reasoning (+67.6 points), quantitative reasoning (+63.4), and compliance assessment (+60.8).

Failure mode analysis reveals three key findings: local recognition does not guarantee multi-evidence composition; domain difficulty is not determined by representation frequency; and supervision reduces reasoning errors but leaves calibration failures. The paper concludes that strong general visual understanding does not yet guarantee reliable industrial-safety reasoning, and that high aggregate accuracy alone is insufficient for safety-critical use. Future work should broaden coverage to temporal workplace activities, video and sensor evidence, additional industries and accident repositories, and expert-centered evaluation of explanations and recommended controls.

Improvements to AI Systems:

1. Integrate executable scene-graph reasoning into vision-language models (VLMs): Train models to parse visual inputs into structured scene graphs (objects, relations, spatial layouts, safety rules) and execute programmatic queries over them, rather than relying solely on end-to-end pixel-to-text mapping. This enables deterministic, verifiable reasoning for spatial, quantitative, and compliance tasks.

2. Add explicit multi-step reasoning and evidence-graph construction for multimodal QA: Implement a two-stage pipeline where the model first extracts contextual evidence from both image and text (e.g., accident reports), then builds an evidence graph with explicit information boundaries and iterative verification before generating answers. This improves causal analysis and evidence synthesis.

3. Apply chain-of-thought (CoT) supervised fine-tuning on domain-specific corpora: Fine-tune smaller open models (e.g., Qwen3.5-9B) with CoT rationales derived from the 110K+ scene-centric and 13K+ report-centric QA pairs. This yields a 51.6-point micro-accuracy gain over base models, closing the gap with frontier models while being more deployable.

4. Develop calibration-aware training for safety-critical decisions: Add a calibration loss or post-hoc temperature scaling specifically for compliance, mitigation, and decision-making categories, since the paper shows supervision reduces reasoning errors but not calibration failures. This ensures confidence scores align with correctness in high-stakes outputs.

5. Create a multi-evidence composition module: Train a separate attention or fusion layer that explicitly combines local object detections (e.g., PPE, machinery) with global contextual cues (e.g., activity, hazard conditions) before answering. This addresses the failure mode where local recognition alone fails to support multi-evidence reasoning.

What the Improved AI System Can Do:

• Reason over industrial scenes with verifiable accuracy: It can answer questions like Is the worker wearing a hard hat while operating a grinder near a combustible material? by executing safety rules on a scene graph, not just pattern-matching.

• Synthesize evidence from accident reports and images: It can extract figures, tables, and text from OSHA/NIOSH reports, link them to visual evidence, and produce multi-step causal chains (e.g., The missing guardrail led to a fall, which caused the injury) with explicit evidence boundaries.

• Achieve near-frontier performance on a small footprint: A 9B-parameter model fine-tuned with CoT reaches 89.0% micro accuracy, outperforming GPT-5.6-Sol (88.7%) and GPT-5.5 (87.9%) on scene-centric tasks, making it viable for edge or on-premise deployment.

• Provide calibrated confidence for safety decisions: It will output well-calibrated probabilities for compliance assessments and mitigation recommendations, so users can trust when the system is uncertain and intervene appropriately.

• Handle comparative and quantitative reasoning robustly: It can compare multiple objects (e.g., Which scaffold is closer to the power line?) and perform arithmetic over scene graph attributes (e.g., counting workers without PPE) with 87.8% and 86.6% accuracy, respectively, up from near-random in base models.

Industrial-safety understanding requires more than detecting workers, equipment, and personal protective equipment. Models must also assess compliance, identify hazardous interactions, explain potential accident mechanisms, and recommend preventive actions. Existing safety datasets primarily focus on visual perception or isolated violation recognition and provide limited supervision for evidence-grounded reasoning. We introduce SafeSceneReason, a multimodal industrial-safety reasoning benchmark and companion training corpus that connects workplace scenes with knowledge from occupational accident investigations. SafeSceneReason combines two complementary data-construction pipelines. The scene-centric pipeline converts annotated workplace images into executable safety scene graphs and generates deterministic answers through program execution over objects, relations, and safety rules. The report-centric pipeline extracts figures and contextual evidence from accident reports and constructs multimodal questions using evidence graphs, explicit information boundaries, multi-step reasoning paths, and iterative verification. The resulting resource contains 110,581 verified scene-centric question--answer pairs and 13,114 refined report-centric question--answer pairs, covering perception, spatial and quantitative reasoning, compliance assessment, evidence synthesis, causal analysis, and mitigation-oriented decision making. Evaluation of representative proprietary and open-source vision--language models reveals substantial performance differences and persistent weaknesses in comparative, technical, and multi-evidence reasoning, demonstrating that strong general visual understanding does not yet guarantee reliable industrial-safety reasoning.