Do Vision Encoders Exhibit Human-like Color Thresholds?

summary

The gist

Understanding and characterizing human color perception is a longstanding research goal <ref:2607.16540#pg2> One of the most traditional approaches is looking for the human color discrimination…

In short

Researchers tested over 50 vision encoders against human color discrimination thresholds using a region-overlap metric (mIoU). The study found weak alignment between model representations and human perception across all architectures. Self-supervised models showed the best alignment, while results are highly dependent on hue region and saturation level.

Key concepts

Human Color Discrimination Thresholds
These are the minimum differences in color that a human observer can perceive as distinct. The study uses these thresholds as a benchmark to see how well computer vision models understand what humans can see in terms of color separation.
Model Representations Geometry
This refers to the shape and structure of the colors learned by deep vision encoders inside their internal feature spaces. The researchers analyzed whether these learned color patterns resemble the natural, anisotropic shapes (like ellipses) that describe how humans perceive color differences.
mIoU (Mean Intersection over Union)
This is a metric used to measure the overlap between the chromatic discrimination thresholds derived from a specific vision model and the actual human discrimination ellipses. A higher mIoU indicates better alignment between what the AI model thinks about color and what humans actually perceive.

Terminology used across episodes

This episode discusses

The paper

Do Vision Encoders Exhibit Human-like Color Thresholds? · Read on arXiv

Computer Vision Center, Spain · Universitat Autonoma de Barcelona, Spain · Image Processing Lab, Universitat de Valencia, Spain · Arab Academy for Science, Technology, and Maritime Transport

Understanding and characterizing human color perception is a longstanding research goal. One of the most traditional approaches is looking for the human color discrimination thresholds, the minimum chromatic differences perceptible to human observers. In recent years, deep neural networks have become the standard networks for computer vision tasks. In particular, deep vision encoders, foundation models trained on large-scale visual data, map images into latent feature representations. Despite the widespread use of deep vision encoders, few studies have investigated whether their internal representations exhibit human-like discrimination thresholds. In this work, we present a large-scale exploratory study probing the chromatic sensitivity of more than 50 pretrained vision encoders, including convolutional networks and vision transformers, against human discrimination thresholds. Using controlled chromatic stimuli at multiple chroma levels, we compare model-derived chromatic discrimination thresholds with human discrimination ellipses through a region-overlap metric (mIoU). Our analysis reveals generally weak alignment between model representations and human perceptual thresholds across all model families, with the best mIoU < 0.25. Moreover, we find that self-supervised encoders consistently outperform supervised ones, while language-supervised models show the most polarized behavior, occupying both the top and bottom of the ranking. These findings suggest that human-like chromatic sensitivity does not emerge naturally from current large-scale visual training objectives for any of the analyzed architectures.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Do Vision Encoders Exhibit Human-like Color Thresholds?".

Jane: Here's the gist of the paper. Understanding and characterizing human color perception is a longstanding research goal <ref:2607.16540#pg2> One of the most traditional approaches is looking for the human color discrimination thresholds, the minimum chromatic differences perceptible to human observers <ref:2607.16540#pg2>.

Tom: In this work, we present a large-scale exploratory study probing the chromatic sensitivity of more than 50 pretrained vision encoders, including convolutional networks and vision transformers, against human discrimination thresholds <ref:2607.16540#pg2>.

Jane: Our analysis reveals generally weak alignment between model representations and human perceptual thresholds across all model families, with the best mIoU.

Tom: Pruning redundant reasoning tokens The study systematically evaluates the chromatic representation geometry of deep vision encoders against human discrimination behavior <ref:2607.16540#pg2>.

Paper summary: Tom: So, wrapping up this discussion on "Do Vision Encoders Exhibit Human-like Color Thresholds?", the paper confirms that while deep vision encoders develop structured and anisotropic chromatic similarity geometries in their latent spaces, these representations only partially align with human perceptual models.

Jane: The alignment varies systematically across chromaticity regions and saturation levels, and it depends strongly on the training paradigm; self-supervised and supervised models outperformed language-supervised and generative approaches in terms of this alignment.

Lu: The overall message is that even the best performing models remain far from human-level chromatic discrimination when tested against these specific human perceptual thresholds.

Meng: It means we can't just assume high performance on a benchmark automatically translates to accurate color perception, especially since the alignment depends so heavily on what kind of training signal you feed the AI.

Tom: Exactly. The implication is that for any application where precise chromatic discrimination is critical, we need to consider both the model architecture and the specific training objective alongside just looking at raw performance metrics.

Jane: It's a caution to keep in mind as we move forward with these large foundation models, reminding us that internal feature representations don't always map perfectly onto established human psychophysical data.

Conclusion: Tom: So we’re done with looking at the mechanics of how these vision encoders are trained, and now we’re just looking at what this whole study actually means for color science.

Jane: The title is pretty direct, right? "Do Vision Encoders Exhibit Human-like Color Thresholds?" It’s asking if the AI's internal understanding of color matches what a person can actually see when they look at two colors side by side.

Lu: Basically, they took fifty different types of vision models, from older convolutional networks to the newest vision transformers, and compared how well their internal representations line up with actual human color discrimination thresholds.

Meng: And the main finding is that the alignment isn't great across the board; it’s weak generally. They found that while these models develop some structured geometry in their latent space, they don't perfectly mirror how humans perceive those differences.

Tom: So, if we think about this for a practical application—say, an autonomous vehicle needing to detect subtle color changes—this study suggests we can't just trust the raw output of these massive models when it comes to precise color discrimination.

Jane: Exactly. It shows that even the best performing models still have a gap between their internal math and real human perception, and this gap changes depending on the specific shade of blue you’re looking at or how saturated it is.

Lu: The study points out that the alignment varies wildly based on the training signal; self-supervised methods actually kept their color relationships better than language-supervised ones did.

Meng: That’s important because if we are deploying these models, choosing the right training method matters more than just picking the biggest model size.

Tom: It really highlights that achieving high scores in a test doesn't automatically mean you’ve captured human sensory reality; there's still a distance to cover.

Jane: And it leaves us wondering what that distance looks like when we move beyond simple flat color patches and look at complex, real-world textures. That’s where the next part of this line gets interesting.

More episodes

← Home