CHOQOLATE: Organizing Concept Bottleneck Latent Spaces with Choquet Integrals
cs.CV
Submitted: 2026-09-29
Updated: 2026-10-07
Code: https://github.com/yossigandelsman/clip_text_span
License: http://creativecommons.org/licenses/by-sa/4.0/
The gist: Concept Bottleneck Models (CBMs) built on vision-language models such as CLIP represent a latent space as human-understandable concepts.
Terminology
Abstract
Concept Bottleneck Models (CBMs) built on vision-language models such as CLIP represent a latent space as human-understandable concepts. These representations are unfaithful: related concepts are entangled, so individual scores do not reflect their intended meaning. We propose CHOQOLATE, an interpretable-by-design layer based on 2-additive Choquet integrals, which merges correlated concepts into compact nodes. Across four datasets, CHOQOLATE achieves a favorable accuracy-interpretability trade-off, with weight-sparse and semantically coherent nodes. A closed-form gradient derivation, backed by experiments, explains why Choquet layers drive this organization without explicit supervision. Choquet weights also map directly to Shapley values, which enables test-time intervention. On standard bias-mitigation benchmarks, suppressing spurious concepts after training performs on par with methods that require group annotations or retraining, while needing neither.
Sources
- Hoeffding Concept Bottleneck Models with Applications to Overhead Images
- Debiasing Vision-Language Models via Biased Prompts
- If Concept Bottlenecks are the Question, are Foundation Models the Answer?
- Concepts Worth Having: Refining VLM-Guided Concept Bottleneck Models with Minimal Annotations
- The Platonic Representation Hypothesis
- Enhancing Concept Localization in CLIP-based Concept Bottleneck Models
- Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- Common-Sense Bias Modeling for Classification Tasks
- Places: An Image Database for Deep Scene Understanding
Related papers
- Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories
- AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories
- Benchmarking the Robustness of Foundation Models for Mammography under Domain Shift
- MambaX-Net: Dual-Input Mamba-Enhanced Cross-Attention Network for Longitudinal MRI Segmentation
- TeleOCR: Navigating Document Parsing Across Digital and Camera-Captured Documents
- A Survey on Efficient Vision-Language-Action Models