Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom.
Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.
Tom: We're looking at the paper: "Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision".
Jane: Here's the gist of the paper. Directional confusions reveal divergent inductive biases through rate-distortion geometry in human and machine vision; This asymmetry structure is a sensitive, interpretable signature of inductive bias that accuracy-based evaluation cannot recover; The research investigates how humans and modern vision models exhibit systematically different directional confusion structures even when matched for task, perturbation, and accuracy level; These organizational differences predict generalization geometry independently of accuracy; The findings establish that the organization of directional errors carries information about generalization that is independent of accuracy; Asymmetry Dissociation Humans and artificial neural networks reach similar categorical accuracy while making systematically different kinds of errors, differing not in how often they fail, but in who gets mistaken for whom; Humans exhibit broad but weak asymmetries across many class pairs, whereas deep vision models show sparser, stronger directional collapses into a few dominant categories; This dissociation is invisible to accuracy alone; Groups that are significantly more asymmetric than humans show no significant difference from humans in accuracy, whereas groups that match humans on asymmetry are significantly more accurate than humans; This confirms a double dissociation between asymmetry organization and accuracy; Rate-Distortion Formalism The study adopts a rate–distortion (RD) framework that treats each system as an effective noisy communication channel whose behavior is summarized by its stimulus–response confusion matrix; The central object is the information–error trade-off, which traces a rate–distortion (RD) frontier that characterizes how efficiently and how gracefully a system generalizes under compression; This framework accommodates asymmetric confusion matrices directly, without forcing behavioral data into a symmetric similarity space; The RD frontier is shaped by the full structure of the distortion matrix—including its asymmetric component—so systems with different asymmetry organizations are expected to trace qualitatively different frontiers, even at matched accuracy levels; Generative Simulation Linkage Generative simulations confirm that broad–weak and sink-like asymmetry organizations produce opposite effects on RD geometry even at matched accuracy; Broad–weak asymmetries should expand the RD frontier by preserving information across many class distinctions, while concentrated sink-like asymmetries should collapse it by funneling probability mass into a few dominant responses; This dissociation persists even after controlling for overall accuracy; The breadth–strength decomposition is predictive, showing that the component model showed strong and slice-consistent effects on AUC residuals, indicating that residual AUC depends on how directionality is distributed (breadth–and–weak vs sparse–and–strong mixtures), not merely “more” or “less” asymmetry; Predictive Signatures
Paper summary: Tom: So, to recap, the core thesis of "Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision" is that the way humans and AI get confused directionally reveals different inductive biases >
Jane: They are showing that humans and modern vision models can achieve similar accuracy scores while making systematically different kinds of errors, which means it's not just about how often they fail, but who gets mistaken for whom >
Lu: The paper quantifies this asymmetry in confusion matrices and links that structure directly to the geometry of the information error trade-off, which is how a system generalizes under compression >
Meng: So they use this rate distortion framework to look at how efficiently things work when you try to compress the information from these systems >
Lalam: They find that humans show broad but weak asymmetries across many class pairs, whereas deep vision models show sparser, stronger directional collapses into a few dominant categories >
Tom: That's the key difference they found—broad versus concentrated sink-like failures—and this dissociation is invisible when you only look at accuracy numbers >
Conclusion: Jane: So, looking at the full title, "Directional Confusions Reveal Divergent Inductive Biases Through Rate-Distortion Geometry in Human and Machine Vision," it suggests that the way we fail directionally gives us clues about how we are fundamentally built as learners >
Tom: Right. The authors argue that this isn't just about getting a better score on a test; it’s about the structure of those errors telling us something deep about the system’s learning mechanism >
Lu: What this means practically is that if you want to build an AI or study human vision, you can use this asymmetry organization as a predictor for how well it will generalize in the future, even before you've achieved peak accuracy on your current benchmark >
Meng: So it implies that we might need metrics that look beyond just overall accuracy and start looking at these directional error patterns to really understand the differences between human and machine learning approaches >
Lalam: And the paper confirms that robustness training can reduce the overall asymmetry, but it doesn't erase this organizational difference between broad weakness and concentrated strength >
Tom: So, in short, they show that there are different ways to learn vision that have different failure modes, and this is a structure you can measure >
Windreich Department of AI and Human Health at Icahn School of Medicine at Mount Sinai · Department of Computing at Imperial College London
cs.CV, cs.IT, math.IT, q-bio.NC
Submitted: 2026-04-23
Updated: 2026-11-13
Code: https://github.com/bethgelab/model-vs-human
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Directional confusions reveal divergent inductive biases through rate-distortion geometry in human and machine vision; This asymmetry structure is a sensitive…
Key concepts
- Directional Confusion Structures
- This refers to the specific patterns of errors where one direction is mistaken for another in a classification task. The study examines how humans and AI models organize these directional mistakes, showing they are not just about the frequency of failure but *who* gets confused with whom.
- Rate-Distortion (RD) Geometry
- This framework treats vision systems like communication channels, mapping input data to output responses. The RD frontier shows the best trade-off between how much information is preserved and how much distortion occurs during generalization, revealing how efficiently a system learns.
- Asymmetry Dissociation
- This concept highlights that humans and AI models differ in the *organization* of their errors, not just the overall accuracy. Humans show broad asymmetries across many classes, while deep models show concentrated, strong collapses into a few dominant categories. This structural difference is invisible to simple accuracy metrics.
- Generative Simulation Linkage
- This uses simulations to test how different asymmetry organizations affect the RD geometry. Broad-weak asymmetries expand the information frontier by maintaining distinctions across many classes, whereas sink-like (concentrated) asymmetries collapse it by funneling probability mass into a few dominant responses.
Terminology
Summary
Directional confusions reveal divergent inductive biases through rate-distortion geometry in human and machine vision; This asymmetry structure is a sensitive, interpretable signature of inductive bias that accuracy-based evaluation cannot recover; The research investigates how humans and modern vision models exhibit systematically different directional confusion structures even when matched for task, perturbation, and accuracy level; These organizational differences predict generalization geometry independently of accuracy; The findings establish that the organization of directional errors carries information about generalization that is independent of accuracy;
Asymmetry Dissociation
Humans and artificial neural networks reach similar categorical accuracy while making systematically different kinds of errors, differing not in how often they fail, but in who gets mistaken for whom; Humans exhibit broad but weak asymmetries across many class pairs, whereas deep vision models show sparser, stronger directional collapses into a few dominant categories; This dissociation is invisible to accuracy alone; Groups that are significantly more asymmetric than humans show no significant difference from humans in accuracy, whereas groups that match humans on asymmetry are significantly more accurate than humans; This confirms a double dissociation between asymmetry organization and accuracy;
Rate-Distortion Formalism
The study adopts a rate–distortion (RD) framework that treats each system as an effective noisy communication channel whose behavior is summarized by its stimulus–response confusion matrix; The central object is the information–error trade-off, which traces a rate–distortion (RD) frontier that characterizes how efficiently and how gracefully a system generalizes under compression; This framework accommodates asymmetric confusion matrices directly, without forcing behavioral data into a symmetric similarity space; The RD frontier is shaped by the full structure of the distortion matrix—including its asymmetric component—so systems with different asymmetry organizations are expected to trace qualitatively different frontiers, even at matched accuracy levels;
Generative Simulation Linkage
Generative simulations confirm that broad–weak and sink-like asymmetry organizations produce opposite effects on RD geometry even at matched accuracy; Broad–weak asymmetries should expand the RD frontier by preserving information across many class distinctions, while concentrated sink-like asymmetries should collapse it by funneling probability mass into a few dominant responses; This dissociation persists even after controlling for overall accuracy; The breadth–strength decomposition is predictive, showing that the component model showed strong and slice-consistent effects on AUC residuals, indicating that residual AUC depends on how directionality is distributed (breadth–and–weak vs sparse–and–strong mixtures), not merely “more” or “less” asymmetry;
Predictive Signatures
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
1
No longer treat confusion patterns as unstructured by analyzing directional asymmetry through breadth and strength decomposition to reveal distinct inductive biases. This allows systems to identify if their failures stem from distributed errors (human-like) or concentrated sink-like collapses (ANN-like).
2
Implement a mechanism that uses the rate distortion framework to predict generalization geometry independently of accuracy. This allows for the identification of systems that might match human accuracy but possess fundamentally different failure regimes based on whether their asymmetry is broad or sink-like.
3
Develop robustness training signals specifically designed to penalize sink-like collapse rather than just reducing overall asymmetry magnitude. This aims to recover the human-like distributed error structure, as robustness-oriented training can reduce global asymmetry toward the human range while leaving the organizational dissociation intact.
Abstract
Humans and modern vision models can reach similar classification accuracy while making systematically different kinds of mistakes -- differing not in how often they err, but in who gets mistaken for whom and in which direction. We show that these directional confusions reveal distinct inductive biases that are invisible to accuracy alone. Using matched human and deep vision model responses on a natural-image categorization task under 12 perturbation types, we quantify asymmetry in confusion matrices and link it to the shape of the information-error trade-off. We characterize this trade-off geometry using three signatures extracted from the rate-distortion (RD) frontier: slope beta, curvature kappa and efficiency (AUC). We find that humans exhibit broad but weak asymmetries, whereas deep vision models show sparser, stronger directional collapses. Robustness training reduces global asymmetry but fails to recover the human-like breadth-strength profile of graded similarity. Mechanistic simulations further show that different asymmetry organizations shift the RD frontier in opposite directions, even when matched for performance. Together, these results position directional confusions and RD geometry as compact, interpretable signatures of inductive bias under distribution shift.
Sources
- Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity
- Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
- Same Compression Principle, Different Geometry: Rate-Distortion Signatures Dissociate Biological and Artificial Visual Systems
- Improved Baselines with Momentum Contrastive Learning
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Very Deep Convolutional Networks for Large-Scale Image Recognition
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs