More Accurate, Less Human: Gestalt Grouping in Vision Models

The paper introduces a behavioral battery to test whether vision models organize visual information according to Gestalt grouping principles the way humans do. The authors argue that high benchmark accuracy does not establish whether models perceive visual structure like humans, since identical answers can arise from different perceptual organizations. They evaluate 45 models across five training families: supervised encoders, self-supervised encoders, contrastive vision–language encoders, open-weight VLMs, and closed foundation models.

The battery covers two Gestalt principles: Closure and Similarity. Closure is tested via silhouette recognition (shape-only object recognition on filled silhouettes, 16-way forced choice, using public per-trial responses of ten observers). Similarity is tested three ways: mark-color odd-one-out on Tableau-10 colors (scored against crowdsourced perceptual kernels), color-series counting (scored against the published 6–12 categorical-color capacity band), and object odd-one-out on THINGS natural images (scored against the test–retest ceiling of 0.863). All tasks reuse published stimuli and human data, requiring no new user studies.

Three metrics quantify behavioral alignment: Behavioral Agreement (B), the rate at which the model reproduces the human reference response; Behavioral Error Consistency (κ), a chance-corrected covariance between model correctness and per-item human accuracy; and Behavioral Effect Replication (a(k)), whether accuracy as a function of series count bends where human capacity does.

Key findings: Behavioral agreement is multi-dimensional, with cross-task rank correlations spanning ρ = 0.15 to 0.62, meaning which model looks aligned depends on which grouping task is measured. Accuracy dissociates from human-likeness on identical trials: 34 of 45 models answer more accurately than human observers, but only four match human–human error consistency. The two measures rank models discordantly in 37% of pairs (ρ = 0.38). Contrastive vision–language encoders show both directions: SigLIP passes human accuracy while sharing little of the human error pattern, whereas CLIP nearly reaches the human–human ceiling at human-level accuracy. Among open VLMs, the ordering inverts: every one that exceeds human accuracy falls below LLaVA-1.5-7B, which never reaches human accuracy yet holds the tier's highest consistency.

Only at the frontier do the two axes meet: four closed models pair above-human accuracy with κ at or beyond the ceiling in point estimate, led by Claude-Opus-4.6. Of the 34 models above human accuracy, 30 stay below the consistency ceiling at a median 78% of it. Grouping is a model-family property: the seven open VLMs sit at the bottom of both kernels, with none clearing the position-guessing floor on semantic odd-one-out and only three clearing it on the color kernel. The 23 closed foundation models invert this as a block, with ten exceeding the best encoder on semantic grouping and eight on the color kernel. Training objectives determine organization: self-supervised training leads the color kernel, contrastive vision–language leads semantic grouping and shape, while masked-image training holds the two lowest shape consistencies. Behavioral Effect Replication fails in every family, with series scores ranging more than threefold across encoder objectives and unconstrained by the human 6–12 band.

The paper concludes that behavioral grounding in human vision is measurable, task-indexed, and earned by training recipe, not by accuracy or scale. It cautions against treating vision models as drop-in human proxies for design evaluation, since their grouping of chart marks can depart substantially from human behavior. The methodology generalizes to any cognitive theory with published behavioral data, and extending the battery to proximity, continuity, and other grouping laws is the immediate next step. Limitations include that no stimulus is a complete chart (chart-context effects remain untested), the battery covers only two Gestalt principles, and residual nulls (unparseable answers) are excluded rather than penalized, staying under 3% per closed-model battery.

Improvements to AI systems:

1. Add a perceptual organization audit layer to vision encoders. After training, run the behavioral battery (silhouette closure, color odd-one-out, color-series counting, object odd-one-out) and report three scores: Behavioral Agreement, Error Consistency, and Effect Replication. Use these as a separate evaluation track alongside accuracy benchmarks, so model selection is not based on accuracy alone.

2. Introduce a human-alignment fine-tuning objective for contrastive vision–language models. For models like CLIP that already approach the human consistency ceiling, fine-tune on the behavioral battery’s error patterns (e.g., aligning model errors with human per-item error rates) to push consistency above 0.863 while maintaining accuracy. This would create models that both exceed human accuracy and share human perceptual failures.

3. Implement a task-indexed behavioral fingerprint for each model. Store the four behavioral scores as a vector. When deploying a vision model for human-centric tasks (e.g., chart readability, UI design evaluation), require the fingerprint to match human ranges on the relevant principle (e.g., color kernel for charts). Reject models whose fingerprint falls outside human bands, regardless of benchmark accuracy.

4. Add a grouping-law constraint to self-supervised and masked-image training. Since self-supervised leads color kernel but masked-image lags shape consistency, modify masked-image pretraining to include a loss term that penalizes deviations from human closure and similarity judgments on the battery’s stimuli. This directly shapes internal representations toward Gestalt organization.

5. Create a human-error-aware decoding for open-weight VLMs. For VLMs like LLaVA that underperform on semantic odd-one-out, add a post-hoc correction layer that re-ranks candidate answers using human perceptual kernels (e.g., Tableau-10 color distances, THINGS semantic similarity) before final output. This lifts consistency without retraining.

6. Build a behavioral ceiling checker for closed foundation models. Before using a closed model as a human proxy in design evaluation, automatically run the battery and flag models whose κ falls below 78% of the human ceiling (as median for above-human-accuracy models). This prevents deployment of models with high accuracy but low perceptual alignment.

What the improved AI system can do:

• Predict human perceptual errors on unseen visual grouping tasks (closure, similarity) with consistency at or above 0.863, even while exceeding human accuracy on the same trials.

• Self-report its own behavioral alignment across multiple Gestalt principles, allowing users to select models per task (e.g., color grouping for data visualization, shape closure for icon recognition).

• Avoid silent failures in human-centric applications: when asked to evaluate chart readability or icon clarity, the system will refuse or flag outputs if its grouping behavior deviates from human norms, rather than giving misleadingly accurate answers.

• Adapt its internal representations during training to match human error patterns, so that its mistakes are human-like, enabling more realistic simulations of human perception in cognitive modeling and user testing.

• Provide a multi-dimensional alignment score (not a single number) so that practitioners can trade off accuracy vs. human-likeness per application, and know exactly which grouping law is or isn’t matched.

Human vision organizes what it sees into wholes: same-colored points group into series, similar marks cohere into categories, and shapes complete into recognizable objects. These are the Gestalt operations that visualization design builds on. Whether vision models organize visual content this way has not been systematically tested. We introduce a behavioral battery that scores models against human data from prior perception studies on four grouping tasks: mark-color odd-one-out, color-series counting, silhouette recognition, and object odd-one-out. We apply it to 45 models across five training families: supervised, self-supervised, and contrastive vision-language encoders, open-weight VLMs, and closed foundation models. The battery reveals that agreement with human responses captures aspects of perceptual organization that conventional performance metrics fail to distinguish, with several closed models exhibiting substantially lower alignment than their benchmark accuracy would suggest. Scoring against published perception data therefore gives visualization research a reusable yardstick, requiring no new user study, for auditing whether the models now entering visualization pipelines organize what they see the way their human audience does.