HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluation
cs.CV, cs.AI
Submitted: 2025-05-16
Updated: 2026-09-07
Comments: Accepted at Transactions on Intelligent Systems and Technology (Manuscript ID: TIST-2026-02-0123.R2)
Code: https://github.com/VectorInstitute/HumaniBench
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Although recent large multimodal models (LMMs) show impressive progress on vision language tasks, their alignment with human centered (HC) principles such as fairness, ethics, inclusivity, empathy,
Terminology
Abstract
Although recent large multimodal models (LMMs) show impressive progress on vision language tasks, their alignment with human centered (HC) principles such as fairness, ethics, inclusivity, empathy, and robustness is often overlooked. Existing LMM benchmarks are largely accuracy-agnostic. We present HumaniBench, a unified framework for characterizing HC alignment across realistic, socially grounded visual contexts. It contains 32,000 expert-verified image-question pairs from real-world news imagery, each mapped to one or more HC principles through explicit metrics. Comparing 15 state of the art LMMs reveals consistent trade-offs: proprietary systems lead on ethics, reasoning, and empathy, while open-source models show superior visual grounding and resilience. All models show persistent gaps in fairness and multilingual inclusivity. Chain-of-thought prompting and test-time scaling yield 8to 12 % gains on several HC dimensions. HumaniBench enables fine-grained analysis of alignment trade-offs not captured by conventional multimodal benchmarks. https://vectorinstitute.github.io/humanibench/
Sources
- MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
- The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence
- Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Images
- VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
- Red Teaming Visual Language Models
- Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
- MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs
- HERM: Benchmarking and Enhancing Multimodal LLMs for Human-Centric Understanding
- AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models
- MVP-Bench: Can Large Vision--Language Models Conduct Multi-level Visual Perception Like Humans?
- SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
- GPT-4o System Card
- Qwen2.5-VL Technical Report
- Phi-4 Technical Report
- Gemma 3 Technical Report
- CogVLM2: Visual Language Models for Image and Video Understanding
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models
- Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
- The Llama 3 Herd of Models
- DeepSeek-VL: Towards Real-World Vision-Language Understanding
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models