Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes

summary

The gist

Density map estimation enables accurate object counting in heavily occluded, and densely packed scenes where detection-based counting fails The proposed model combines a Twins-SVT pyramid vision…

In short

This research addresses accurate object counting in crowded and occluded scenes where traditional detection methods struggle. The proposed model uses a Twins-SVT backbone and a multiscale CNN decoder, enhanced by a Category Focus Module, to estimate density maps. This approach achieves significant accuracy improvements over existing methods on benchmarks like VisDrone and iSAID by leveraging hierarchical features for robust counting.

Key concepts

Twins-SVT pyramid vision transformer backbone
This is the core feature extractor of the model. It's a type of vision transformer designed to capture rich spatial and contextual information from images at multiple resolutions (pyramid). This structure helps the model understand complex relationships between objects, which is crucial for accurately counting items in dense scenes where individual object detection becomes unreliable.
Category Focus Module
This auxiliary task helps the model learn better by forcing it to focus on specific object categories during training. It suppresses interference between different classes, allowing the main density estimation head to perform more effectively without needing extra constraints during testing.
Smoothly differentiable softplus activation
This mathematical function is used in the final layer of the model. It ensures that the output for object counts is always a positive number, which is necessary for counting. Crucially, it allows the model to be trained effectively because it remains smooth and differentiable during training, enabling gradient flow.
Multiscale CNN decoder
This part of the architecture takes the features from the backbone and processes them across different scales using a Convolutional Neural Network (CNN). By fusing information from these different scales, the model gains robust feature extraction capabilities that are resilient to variations in object density and size.

Terminology used across episodes

This episode discusses

The paper

Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes · Read on arXiv

Villanelle O’Reilly, Jonathan Cox, Georgios Leontidis, Marc Hanheide, Petra Bosilj, James M. Brown

Interdisciplinary Institute, University of Aberdeen, UK · Department of Advanced Computing Sciences, Maastricht University, The Netherlands

Density map estimation enables accurate object counting in heavily occluded, and densely packed scenes where detection-based counting fails. In multi-class density estimation, class awareness can be introduced by modelling classes non-exclusively, better reflecting crowded and visually ambiguous contexts. However, existing multi-class density estimators often degrade in less-dense scenes, while state-of-the-art detectors still struggle in the most congested settings. To bridge this gap, we propose the first vision-transformer-based approach to multi-class density estimation. Our model combines a Twins-SVT pyramid vision transformer backbone with a multiscale CNN decoder that leverages hierarchical features for robust counting across a wide range of densities. Further to that, the method adds an auxiliary segmentation task with the Category Focus Module to suppress inter-category interference at training time. The module improves the density estimation head without the need for constraining assumptions added by the application of the auxiliary task at inference time, as required in previous methods. Training and evaluation on the VisDrone and iSAID benchmarks demonstrates a leap in performance versus the previous state-of-the-art multi-class density estimation methods, attaining a 33%, 43%, and 64% reduction to MAE in testing evaluation. The method outperforms YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples. Code, and trained weights available at https://github.com/LCAS/gnr mcdest.

DOI: 10.1109/IJCNN69854.2026.11729521

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes".

Title and authors: Tom: We've talked about the title, "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes," and who came up with it—O’Reilly, Cox, Leontidis, Hanheide, Bosilj.

Jane: It’s a pretty direct title; it tells you exactly what they are trying to solve: getting the numbers right for counting objects when everything is dense and varied.

Lu: They are proposing this new vision transformer based approach because existing multi-class density estimators often degrade in less-dense scenes, which is where most current methods fail.

Meng: So, it’s targeting that gap where detectors struggle in the most congested settings while other methods aren't robust enough elsewhere.

Lalam: The paper sets up this problem by showing how the ratio of total classes in a sample changes across different datasets, illustrating the distribution of class presence within a single sample.

Tom: They show that for datasets like VisDrone-DET and iSAID, you see a more uniform class frequency distribution compared to biodiversity samples from Hicks et al., which are extremely skewed.

Jane: That skew is important because it shows why methods trained on those skewed sets don't generalize well when you move to more varied real-world scenes.

Lu: The paper argues that multi-class density estimation needs to model classes nonexclusively, which better reflects the crowded and visually ambiguous contexts we see in the field.

Meng: So they’re moving away from assuming perfect separation between classes, which makes sense when things are overlapping.

Lalam: This approach addresses the discreteness assumptions inherent to detection-augmented methods by modeling classes nonexclusively, a concept they introduce into their architecture.

The paper's summary: Tom: So what’s the actual mechanism they propose? They combine a Twins-SVT pyramid vision transformer backbone with a multiscale CNN decoder to extract features for these density maps.

Jane: That means they use the transformer structure to get rich, contextual features from the image, and then a CNN part to decode those into spatial density maps at different resolutions.

Lu: They introduce the Category Focus Module as an extension of the Multiscale Aware Module from MRCNet, which adds dilated convolutions to increase the receptive field of their decoder.

Meng: So they’re essentially using a mechanism that lets them look at features at multiple scales simultaneously to get more comprehensive density predictions.

Lalam: The density estimation head uses a similar multiscale convolutional decoding scheme, leveraging hierarchical features from the pyramid ViT backbone for scale variance robust feature extraction capabilities.

Tom: And they introduce self-attention specifically to multi-class density estimation within that decoder structure, which is a key change for this task.

Jane: They’re essentially using the transformer's attention mechanism to understand how different classes relate to each other in the same image, not just counting them separately.

Lu: Similar to DSACA and CCTwins, they employ a segmentation-based secondary task, which helps reduce intercategory interference during training.

Meng: So they’re baking the class awareness into the training process rather than trying to patch it on later during inference.

Lalam: And for stability, they use a smoothly differentiable softplus activation function to ensure positive outputs while maintaining gradient flow for training.

The paper's improvements: Tom: The main improvement they claim is that this approach outperforms previous state-of-the-art multi-class density estimation methods on urban datasets like VisDrone and iSAID.

Jane: Specifically, they report a thirty-three percent, forty-three percent, and sixty-four percent reduction in mean absolute error of MAE on those testing evaluations.

Lu: This is significant because it bridges the gap between methods that work well on sparse data and those that struggle in the most congested settings.

Meng: I’m interested in how this translates to deployment; if you get a sixty-four percent reduction, does that mean my pipeline runs significantly faster or more reliably under real traffic conditions?

Tom: They also found it outperforms YOLO11 in less busy scenes, exceeding it by an order of magnitude in the most crowded testing samples.

Jane: That comparison against YOLO11 is striking; they’re showing superior performance even in the less challenging environments compared to a leading object detector.

Lu: The Category Focus Module is another key improvement because it improves overall model performance, particularly when dealing with smaller count ranges.

Meng: That makes sense if we think about scenarios where you might be counting very few objects in a specific area; that’s where the accuracy matters most for deployment.

Lalam: They also achieved a level of class awareness without needing to impose constraining assumptions on the density estimation head during inference time, which is a big deal for practical application.

Conclusion: Tom: So to wrap up, this paper, "Getting the Numbers Right Modelling Multi-Class Object Counting in Dense and Varied Scenes," shows that combining vision transformers with multiscale CNNs can tackle dense counting problems effectively.

Jane: The main implication is that for real-world applications in crowded scenes—whether it’s traffic or ecological monitoring—we now have a method that gives us much more reliable counts.

Lu: It gives us scale variance robust feature extraction capabilities by fusing the features from the Twins-SVT backbone with the multiscale CNN decoder structure.

Meng: The main thing I'm taking away is that there’s a solid, measurable performance gain on established benchmarks, even if I still need to see how it handles things that break at scale in production.

Lalam: They also highlighted the use of the smoothly differentiable softplus activation function for directly explainable density estimations modeling positive counts.

Tom: So we have a model that’s more robust to different densities, better at handling class overlap, and gives us concrete error reductions on established data sets.

Jane: It’s definitely a step forward in building systems that can actually count things accurately when the scene is chaotic, which is what this paper is all about.

More episodes

← Home