Probing Human-Machine Perceptual Alignment with Semantically Ambiguous Images

arXiv:2603.24730 · cs.CV · Submitted 2026-03-25 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Probing Human-Machine Perceptual Alignment with Semantically Ambiguous Images".

Jane: Here's the gist of the paper. The paper presents a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries <ref:2603.24730#pg2>.

Tom: Framework Overview The framework is presented with three components: (1) stimulus images generated along a continuum of semantic ambiguity, (2) perceptual measurements along the same continuum using psychophysical methodology, and (3) bias and sensitivity outputs allowing comparisons within and between human and machine observers <ref:2603.24730#pg2>.

Jane: Stimulus Generation Along a Continuum The stimulus generation process utilizes Stable Diffusion v1.5 with the standard latent diffusion model formulation combined with classifier-free guidance <ref:2603.24730#pg2>.

Tom: To obtain a smooth semantic-morph continuum, the authors linearly interpolate CLIP embeddings using a mixing-ratio parameter (α) where lerp cond(α) = (1 − α) embduck + α embrabbit <ref:2603.24730#pg2>.

Title and authors: Tom: Let's look at this paper's title again: "Probing Human-Machine Perceptual Alignment with Semantically Ambiguous Images." It’s straightforward, but it sets a very specific goal.

Jane: It’s basically asking the question of where humans and machines agree or disagree when they have to guess what an image is.

Lu: The authors are Vasha DuTell and Ahna R. Girshick, working out of UC Berkeley and MIT, so they bring a strong background in both perception science and deep learning architectures.

Meng: I don't know much about the specific architecture background right now, but I’m curious how much of their work is focused on the actual implementation side versus the theory behind it.

Tom: They are clearly grounded in both, because they are using CLIP embeddings for synthesis and then testing against a whole range of classifier architectures including ConvNeXt-Base, DenseNet-one hundred twenty-one EfficientNet-B0, and ViT-B/sixteen <ref:2603.24730#pg3,ConvNeXt-Base, DenseNet-121, EfficientNet-B0>.

Jane: And the implication is that we need to reconcile two different response formats—like a human making a choice versus an AI model outputting probabilities.

Lu: That reconciliation is the central tension they are addressing; it’s about bridging that gap between 2AFC tasks and one-hot outputs for machine classifiers <ref:2603.24730#pg1>.

Meng: If I were deploying this, I'd be concerned about the computational cost of running all those different classifier architectures at every step just to measure bias.

Tom: The paper addresses that by aggregating softmax probabilities over relevant semantic categories to infer a model’s "choice" instead of just using one single metric.

Jane: And they use that aggregated choice, along with the human judgments, to fit a psychometric function for both humans and models at every guidance scale.

Lu: So they're not just looking at one fixed point; they are mapping out the entire relationship between the ambiguity of the image and how we perceive it versus how a model processes it.

Meng: That sounds like a lot of parameter tuning just to get one set of summary numbers, which is something I’d try to avoid in production if possible.

The paper's summary: Tom: So the summary boils down to them using controlled generative semantic ambiguity with human psychophysics and machine classifier evaluation in a novel approach.

Jane: They create stimuli by linearly interpolating between text embeddings, like mixing "a duck" with "a rabbit" in varying proportions, which allows them to produce everything from clearly duck-like to ambiguous hybrids.

Lu: This interpolation is done using the lerp function cond(alpha) equals one minus alpha times the duck embedding plus alpha times the rabbit embedding.

Meng: So they are generating these continuous images based on a mixing ratio that directly relates to how much ambiguity we introduce into the visual stimulus.

Tom: And then for each of those images, they use classifier-free guidance to control how strongly the generation adheres to that conditioning prompt using a guidance scale, or GS.

Jane: This GS allows them to explicitly test how adherence changes and if it shifts the perceived content away from what was intended in a specific way.

Lu: By measuring judgments from human observers via the two-alternative forced choice task and machine classifiers via aggregated softmax probabilities, they can estimate perceptual bias and sensitivity at each guidance scale.

Meng: The paper shows that this approach lets them extract parameters like Bias equals PSE minus zero point five and Sensitivity equals beta one.

Tom: And the main implication they find is that human biases are only slightly toward rabbit in the duck-rabbit condition, while classifiers show much more rabbit bias across different architectures.

Jane: The sensitivity generally increased with the guidance scale for both humans and models, which suggests that this scale controls the sharpness of semantic transitions in generated images.

Lu: They also test this across a different pairing, the elephant-rabbit condition, finding that classifier bias was reduced there and the effect of the guidance scale was less pronounced.

Meng: So it seems like these results aren't universal; perceptual bias and sensitivity really depend on the specific semantic ambiguity being probed.

The paper's improvements: Tom: The authors suggest a few key improvements to this framework, starting with incorporating a guidance scale control mechanism into image synthesis to precisely modulate perceptual ambiguity.

Jane: This is about giving researchers a better tool to control whether the guidance scale can be used to optimize stimuli for specific human or machine perception thresholds.

Lu: They also propose developing classifier-free guidance methods where the guidance scale can be dynamically varied during synthesis to explicitly test how classifier-based guidance can shift perceived content away from the intended concept.

Meng: If we could dynamically vary it, that would be a useful diagnostic tool to see how model adherence changes with prompt strength, which is something I’d love for debugging pipelines.

Tom: That directly tackles the problem of understanding and mitigating bias in generative pipelines by seeing how the model sticks to the concept as we change the guidance strength.

Jane: They also suggest implementing a framework for measuring human perceptual bias and sensitivity that is directly comparable across different machine learning architectures.

Lu: This would allow researchers to compare how different models organize semantic meaning differently, for example, observing how AlexNet shows occasional deviations compared to vision transformers.

Meng: That would be valuable because it moves us past just using one benchmark and lets us see which specific model class has inherent structural biases in its representation.

Tom: And they suggest utilizing psychometric functions fitted across a continuum of image generation parameters, like the mixing-ratio alpha, to generate a comprehensive map of visual semantic spaces for both human and machine systems.

Jane: This provides empirical support by showing that perceptual decision boundaries do not necessarily align with the midpoint of CLIP interpolations.

Lu: It guides future work toward refining representations closer to human alignment, which is a really important direction for representation learning research.

Conclusion: Tom: So to wrap up, the main implication is that human perception might be more aligned with the semantic structure captured by CLIP embeddings.

Jane: And classifiers trained for discrete categorization might develop sharper or shifted decision boundaries based on how they are trained.

Lu: It points to differences in the internal geometry of visual representations between human and machine systems leading to systematic shifts in decision boundaries, which is a key finding here.

Meng: From my side, I see it as a way to use controlled ambiguity to probe and align visual representations across these different system types.

Tom: It’s about using this framework to bridge human perception, discriminative models, generative systems, and we’re leaving that as the big takeaway for this paper.

Jane: So they're showing how controlled semantic ambiguity can be used to probe and align visual representations between human perception, discriminative models, generative systems.

Lu: It’s a really useful demonstration of how these different components interact when you put them together in this kind of controlled experiment.

University of California, Berkeley · Massachusetts Institute of Technology

cs.CV

Submitted: 2026-03-25

Updated: 2026-10-29

Comments: Accepted by NeurIPS 2026 BiAlign Workshop

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Importance score: 90/100

The gist: The paper presents a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images…

Key concepts

CLIP Embedding Space
This is a mathematical space where different images are represented by numerical vectors. Images with similar meanings, like a duck and a rabbit, are placed closer together in this space. The framework uses this space to create smooth transitions between concepts for testing perception.
Psychophysical Measurements
These are scientific methods used to measure how humans perceive stimuli. In this study, participants made choices based on ambiguous images. Researchers used these measurements to calculate metrics like 'Bias' and 'Sensitivity,' quantifying how easily humans can tell the difference between similar concepts.
Classifier-Free Guidance
This is a technique used in AI image generation that helps control the output quality. It allows researchers to adjust the strength of the guidance, which directly influences how sharp or clear the semantic transition is between two concepts in an image.

Terminology

Summary

The paper presents a psychophysically-informed framework that interpolates between concepts in the CLIP embedding space to generate continuous spectra of ambiguous images, allowing us to precisely measure where and how humans and machine classifiers place their semantic boundaries <ref:2603. p. 024730#pg2>.

Framework Overview

The framework is presented with three components: (1) stimulus images generated along a continuum of semantic ambiguity, (2) perceptual measurements along the same continuum using psychophysical methodology, and (3) bias and sensitivity outputs allowing comparisons within and between human and machine observers <ref:2603.24730#pg2>.

Stimulus Generation Along a Continuum

The stimulus generation process utilizes Stable Diffusion v1.5 with the standard latent diffusion model formulation combined with classifier-free guidance <ref:2603. p. 124730#pg2>. To obtain a smooth semantic-morph continuum, the authors linearly interpolate CLIP embeddings using a mixing-ratio parameter (α) where lerp cond(α) = (1 − α) embduck + α embrabbit <ref:2603. p. 124730#pg2>. This interpolation allows for the production of stimuli ranging from duck-like to rabbit-like hybrids across five α levels and six guidance scales <ref:2603. p. 224730#pg2>.

Psychophysical Measurements

Human observers were measured using a two-alternative forced choice (2AFC) task, and machine classifiers were evaluated by aggregating softmax probabilities over relevant semantic categories to infer a model’s “choice” <ref:2603.24730#pg2>. The framework fits a logistic psychometric function with equal upper and lower asymptotes using the psignifit 4 toolbox to extract summary parameters like Bias = PSE − 0. p. 25 and Sensitivity = β1 <ref:2603. p. 224730#pg2>.

Bias and Sensitivity Findings

Across the duck-rabbit condition, grand-average human biases ranged from 3 to -0.03, with an average of -0.015, indicating that human participants were only slightly biased towards rabbit <ref:2603. p. 324730#pg2>. In contrast, classifier models showed much more rabbit bias across architectures <ref:2603. p. 324730#pg2>. Sensitivity generally increased with guidance scale for both humans and models, suggesting that guidance scale may act as a control over the sharpness of semantic transitions in generated images <ref:2603. p. 324730#pg2>.

Comparison with Elephant-Rabbit Condition

The elephant-rabbit condition tests whether observed human–model differences generalize beyond a canonical bistable-inspired example <ref:2603.24730#pg2>. In this second condition, qualitative patterns were similar to the duck-rabbit condition, but overall classifier bias was reduced in the elephant-rabbit condition, and the effect of guidance scale was less pronounced <ref:2603. p. 924730#pg2>. This suggests that perceptual bias and sensitivity depend on the specific semantic ambiguity being probed <ref:2603. p. 924730#pg2>.

Discussion and Future Directions

The results suggest that human perception may be more aligned with the semantic structure captured by CLIP embeddings, whereas classifiers trained for discrete categorization may develop sharper or shifted decision boundaries <ref:2603.24730#pg2>. The paper points to differences in the internal geometry of visual representations between human and machine systems, leading to systematic shifts in decision boundaries <ref:2603. p. 324730#pg2>. Future work should extend this analysis to a broader range of models, including self-supervised systems such as DINO <ref:2603. p. 324730#pg2>. The framework demonstrates how controlled semantic ambiguity can be used to probe and align visual representations, bridging human perception, discriminative models, and generative systems <ref:2603. p. 424730#pg2>.

References

[1] Tal Boger and Chaz Firestone. Visual anagrams reveal highlevel effects with ‘identical’stimuli. p. 4 Current Biology, 35(19): R910–R912, 2025 <ref:2603. p. 424730#pg2>

[2] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. p. 4 arXiv preprint arXiv:1809. p. 411096, 2018 <ref:2603. p. 424730#pg2>

[3] Ryan Burgert, Xiang Li, Abe Leite, Kanchana Ranasinghe, and Michael Ryoo. Diffusion illusions: Hiding images in plain sight. p. 4 In ACM SIGGRAPH 2024 Conference Papers, pages 1–11, 2024 <ref:2603. p. 424730#pg2>

[4] Arturo Deza and Talia Konkle. Emergent properties of foveated perceptual systems. p. 4 arXiv preprint arXiv:2006. p. 407991, 2020 <ref:2603. p. 424730#pg2>

[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. p. 4 arXiv preprint arXiv:2010. p. 411929, 2020 <ref:2603. p. 424730#pg2>

[6] Jenelle Feather, Alex Durango, Ray Gonzalez, and Josh McDermott. Metamers of neural networks reveal divergence from human perceptual systems. p. 4 In Advances in Neural Information Processing Systems, pages 10078–10089, 2019 <ref:2603. p. 424730#pg2>

[7] Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. p. 4 Nature neuroscience, 14(9):1195–1201, 2011 <ref:2603. p. 424730#pg2>

[8] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. p. 4 In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016 <ref:2603. p. 424730#pg2>

[9] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. p. 4 In International conference on learning representations, 2018 <ref:2603. p. 424730#pg2>

[10] Daniel Geng, Inbum Park, and Andrew Owens. Visual anagrams: Generating multi-view optical illusions with diffusion models. p. 4 In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24154–24163, 2024 <ref:2603. p. 424730#pg2>

[11] Arnulf BA Graf, Felix A Wichmann, Heinrich H Bulthoff, ¨ and Bernhard Scholkopf. Classification of faces in man and machine. p. 4 Neural Computation, 18(1):143–165, 2006 <ref:2603. p. 424730#pg2>

[12] Anne Harrington, Vasha DuTell, Mark Hamilton, Ayush Tewari, Simon Stent, William T Freeman, and Ruth Rosenholtz. Coco-periph: bridging the gap between human and machine perception in the periphery. p. 4 In The Twelfth International Conference on Learning Representations, 2023 <ref:2603. p. 424730#pg2>

[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. p. 4 In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016 <ref:2603. p. 424730#pg2>

[14] Aaron Hertzmann. Visual indeterminacy in gan art. p. 4 In ACM SIGGRAPH 2020 Art Gallery, page 424–428, New York, NY, USA, 2020 <ref:2603. p. 424730#pg2>

[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. p. 1 arXiv preprint arXiv:2207. p. 412598, 2022 <ref:2603. p. 424730#pg2>

[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. p. 4 Advances in neural information processing systems, 33:6840–6851, 2020 <ref:2603. p. 4

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Incorporate a guidance scale control mechanism into image synthesis to precisely modulate perceptual ambiguity, allowing researchers to control whether guidance scale may act as a control over the sharpness of semantic transitions in generated images. This enables the creation of stimuli that are optimized for specific human or machine perception thresholds.

  2. Develop classifier-free guidance methods where the guidance scale can be dynamically varied during synthesis to explicitly test how classifier-based guidance can shift perceived content away from the intended concept. This offers a diagnostic tool to understand and mitigate bias in generative pipelines by seeing how model adherence changes with prompt strength.

  3. Implement a framework for measuring human perceptual bias and sensitivity that is directly comparable across different machine learning architectures, as shown by the finding that Bias and Sensitivity are not fixed properties of a model, but depend on the specific semantic ambiguity being probed. This allows researchers to compare how different models organize semantic meaning differently, such as observing how AlexNet shows occasional deviations compared to vision transformers.

  4. Utilize psychometric functions fitted across a continuum of image generation parameters (like the mixing-ratio alpha) to generate a comprehensive map of visual semantic spaces for both human and machine systems. This provides empirical support by demonstrating that perceptual decision boundaries do not necessarily align with the midpoint of CLIP interpolations, guiding future work toward refining representations closer to human alignment.

  5. Create robustness testing protocols where the effect of guidance scale is mapped against different semantic pairings, such as the elephant-rabbit condition, to determine how the magnitude of classifier bias depends on the specific label pairing. This identifies which semantic ambiguities are more sensitive or robust across various model architectures and training objectives.

Sources

Related papers