Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: We're looking at the paper: "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes".
Title and authors: Tom: We've talked about the title, "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes," and who came up with it—O’Reilly, Cox, Leontidis, Hanheide, Bosilj.
Jane: It’s a pretty direct title; it tells you exactly what they are trying to solve: getting the numbers right for counting objects when everything is dense and varied.
Lu: They are proposing this new vision transformer based approach because existing multi-class density estimators often degrade in less-dense scenes, which is where most current methods fail.
Meng: So, it’s targeting that gap where detectors struggle in the most congested settings while other methods aren't robust enough elsewhere.
Lalam: The paper sets up this problem by showing how the ratio of total classes in a sample changes across different datasets, illustrating the distribution of class presence within a single sample.
Tom: They show that for datasets like VisDrone-DET and iSAID, you see a more uniform class frequency distribution compared to biodiversity samples from Hicks et al., which are extremely skewed.
Jane: That skew is important because it shows why methods trained on those skewed sets don't generalize well when you move to more varied real-world scenes.
Lu: The paper argues that multi-class density estimation needs to model classes nonexclusively, which better reflects the crowded and visually ambiguous contexts we see in the field.
Meng: So they’re moving away from assuming perfect separation between classes, which makes sense when things are overlapping.
Lalam: This approach addresses the discreteness assumptions inherent to detection-augmented methods by modeling classes nonexclusively, a concept they introduce into their architecture.
The paper's summary: Tom: So what’s the actual mechanism they propose? They combine a Twins-SVT pyramid vision transformer backbone with a multiscale CNN decoder to extract features for these density maps.
Jane: That means they use the transformer structure to get rich, contextual features from the image, and then a CNN part to decode those into spatial density maps at different resolutions.
Lu: They introduce the Category Focus Module as an extension of the Multiscale Aware Module from MRCNet, which adds dilated convolutions to increase the receptive field of their decoder.
Meng: So they’re essentially using a mechanism that lets them look at features at multiple scales simultaneously to get more comprehensive density predictions.
Lalam: The density estimation head uses a similar multiscale convolutional decoding scheme, leveraging hierarchical features from the pyramid ViT backbone for scale variance robust feature extraction capabilities.
Tom: And they introduce self-attention specifically to multi-class density estimation within that decoder structure, which is a key change for this task.
Jane: They’re essentially using the transformer's attention mechanism to understand how different classes relate to each other in the same image, not just counting them separately.
Lu: Similar to DSACA and CCTwins, they employ a segmentation-based secondary task, which helps reduce intercategory interference during training.
Meng: So they’re baking the class awareness into the training process rather than trying to patch it on later during inference.
Lalam: And for stability, they use a smoothly differentiable softplus activation function to ensure positive outputs while maintaining gradient flow for training.
The paper's improvements: Tom: The main improvement they claim is that this approach outperforms previous state-of-the-art multi-class density estimation methods on urban datasets like VisDrone and iSAID.
Jane: Specifically, they report a thirty-three percent, forty-three percent, and sixty-four percent reduction in mean absolute error of MAE on those testing evaluations.
Lu: This is significant because it bridges the gap between methods that work well on sparse data and those that struggle in the most congested settings.
Meng: I’m interested in how this translates to deployment; if you get a sixty-four percent reduction, does that mean my pipeline runs significantly faster or more reliably under real traffic conditions?
Tom: They also found it outperforms YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples.
Jane: That comparison against YOLO11 is striking; they’re showing superior performance even in the less challenging environments compared to a leading object detector.
Lu: The Category Focus Module is another key improvement because it improves overall model performance, particularly when dealing with smaller count ranges.
Meng: That makes sense if we think about scenarios where you might be counting very few objects in a specific area; that’s where the accuracy matters most for deployment.
Lalam: They also achieved a level of class awareness without needing to impose constraining assumptions on the density estimation head during inference time, which is a big deal for practical application.
Conclusion: Tom: So to wrap up, this paper, "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes," shows that combining vision transformers with multiscale CNNs can tackle dense counting problems effectively.
Jane: The main implication is that for real-world applications in crowded scenes—whether it’s traffic or ecological monitoring—we now have a method that gives us much more reliable counts.
Lu: It gives us scale variance robust feature extraction capabilities by fusing the features from the Twins-SVT backbone with the multiscale CNN decoder structure.
Meng: The main thing I'm taking away is that there’s a solid, measurable performance gain on established benchmarks, even if I still need to see how it handles things that break at scale in production.
Lalam: They also highlighted the use of the smoothly differentiable softplus activation function for directly explainable density estimations modeling positive counts.
Tom: So we have a model that’s more robust to different densities, better at handling class overlap, and gives us concrete error reductions on established data sets.
Jane: It’s definitely a step forward in building systems that can actually count things accurately when the scene is chaotic, which is what this paper is all about.
Villanelle O’Reilly, Jonathan Cox, Georgios Leontidis, Marc Hanheide, Petra Bosilj, James M. Brown
Interdisciplinary Institute, University of Aberdeen, UK · Department of Advanced Computing Sciences, Maastricht University, The Netherlands
cs.CV
Submitted: 2025-10-02
Updated: 2026-04-15
Comments: 8 pages, 4 figures, 5 tables
Journal ref: 2026 International Joint Conference on Neural Networks (IJCNN)
DOI: 10.1109/IJCNN69854.2026.11729521
Code: https://github.com/ultralytics/ultralytics
Project page: https://captain-whu.github.io/iSAID
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 85/100
The gist: Density map estimation enables accurate object counting in heavily occluded, and densely packed scenes where detection-based counting fails The proposed model combines a Twins-SVT pyramid vision…
Key concepts
- Twins-SVT pyramid vision transformer backbone
- This is the core feature extractor of the model. It's a type of vision transformer designed to capture rich spatial and contextual information from images at multiple resolutions (pyramid). This structure helps the model understand complex relationships between objects, which is crucial for accurately counting items in dense scenes where individual object detection becomes unreliable.
- Category Focus Module
- This auxiliary task helps the model learn better by forcing it to focus on specific object categories during training. It suppresses interference between different classes, allowing the main density estimation head to perform more effectively without needing extra constraints during testing.
- Smoothly differentiable softplus activation
- This mathematical function is used in the final layer of the model. It ensures that the output for object counts is always a positive number, which is necessary for counting. Crucially, it allows the model to be trained effectively because it remains smooth and differentiable during training, enabling gradient flow.
- Multiscale CNN decoder
- This part of the architecture takes the features from the backbone and processes them across different scales using a Convolutional Neural Network (CNN). By fusing information from these different scales, the model gains robust feature extraction capabilities that are resilient to variations in object density and size.
Terminology
Summary
Density map estimation enables accurate object counting in heavily occluded, and densely packed scenes where detection-based counting fails The proposed model combines a Twins-SVT pyramid vision transformer backbone with a multiscale CNN decoder that leverages hierarchical features for robust counting across a wide range of densities
Model Architecture and Components
The proposed architecture utilizes the Twins-SVT backbone combined with a multiscale CNN decoder to extract features The Category Focus Module is introduced as an auxiliary segmentation task to suppress inter-category interference at training time This module improves the density estimation head without the need for constraining assumptions added by the application of the auxiliary task at inference time, as required in previous methods The model employs a smoothly differentiable softplus activation to guarantee numerically stable positive output for counting constraints without sacrificing gradients at training time
Training Objectives and Loss Functions
The method optimizes the parameters using a combined loss function L = (omega × Lmask) + L2 The primary loss for density estimation is the L2 loss, which computes the class-pixel-wise squared error incorporating spatial and class-localisation error, as well as count error into the optimisation problem The auxiliary masking task employs a percategory cross-entropy loss where Q gt and Q est refer to the ground-truth and estimated mask data
Experimental Validation and Performance
Training and evaluation on the VisDrone and iSAID benchmarks demonstrate a leap in performance versus the previous state-of-the-art multi-class density estimation methods, attaining a 33%, 43%, and 64% reduction to MAE in testing evaluation The method outperforms YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples
Ablation and Robustness Analysis
Ablation studies confirm that the Category Focus Module improves overall model performance, especially at smaller count ranges The final-layer activation function is found to have a significant impact on the performance, with selecting leaky ReLU as a philosophical middle ground noticing a sharp drop in low count ranges with the leaky variant The method's applicability to ecological monitoring tasks highlights its potential for realworld applications beyond traditional urban scenarios
Conclusion and Contributions
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Increased accuracy in dense scenes by employing a vision transformer based approach, as it is
outperforming YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples.
The system can achieve a33%, 43%, and 64% reduction to MAE
on benchmarks like VisDrone and iSAID. -
Enhanced class awareness through the Category Focus Module which
improves the density estimation head without the need for constraining assumptions added by the application of the auxiliary task at inference time, as required in previous methods.
This allows for acontinuous model designed for such contexts
that avoidsthe discreteness assumptions inherent to detection-augmented methods.
-
Robustness to scale and density variance is improved by fusing features from the Twins-SVT backbone with a multiscale CNN decoder, leveraging hierarchical features for
scalevariance-robust feature extraction capabilities.
This ensuresscalevariance-robust density predictions
across a wide range of densities. -
Direct explainability of counts is guaranteed by using the smoothly differentiable softplus activation function, which provides a
numerically stable positive output, applying our counting constraint without sacrificing gradients at training time.
The system can producedirectly explainable density estimations modelling positive counts.
-
Reduced inter-category interference is achieved by introducing a class-aware density head that can
simultaneously predict several classes of density maps
and the Category Focus Module whichextracts features from several layers of the backbone, learning at multiple scales with its Multiscale Aware Module-style design.
Abstract
Density map estimation enables accurate object counting in heavily occluded, and densely packed scenes where detection-based counting fails. In multi-class density estimation, class awareness can be introduced by modelling classes non-exclusively, better reflecting crowded and visually ambiguous contexts. However, existing multi-class density estimators often degrade in less-dense scenes, while state-of-the-art detectors still struggle in the most congested settings. To bridge this gap, we propose the first vision-transformer-based approach to multi-class density estimation. Our model combines a Twins-SVT pyramid vision transformer backbone with a multiscale CNN decoder that leverages hierarchical features for robust counting across a wide range of densities. Further to that, the method adds an auxiliary segmentation task with the Category Focus Module to suppress inter-category interference at training time. The module improves the density estimation head without the need for constraining assumptions added by the application of the auxiliary task at inference time, as required in previous methods. Training and evaluation on the VisDrone and iSAID benchmarks demonstrates a leap in performance versus the previous state-of-the-art multi-class density estimation methods, attaining a 33%, 43%, and 64% reduction to MAE in testing evaluation. The method outperforms YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples. Code, and trained weights available at https://github.com/LCAS/gnr mcdest.
Sources
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- CCTrans: Simplifying and Improving Crowd Counting with Transformer
Related papers
- TEA: Text Encoder Alignment for Robust Concept Erasure in Text-to-Image Models
- Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues?
- Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment
- PVRF: All-in-one Adverse Weather Removal via Prior-modulated and Velocity-constrained Rectified Flow
- DLWM: Diverse Latent World Models for Efficient Multimodal Reasoning
- Out of Sight, Still in Mind: Token Compression for Omni-LLMs