Spatially-Aware Class-Agnostic Object Counting

summary

The gist

Generalised object counting aims to estimate the number of instances of an arbitrary object category from a single image…

In short

UpCount aims to count arbitrary objects in images without needing manual bounding boxes or text prompts. It achieves this by using a Masked Autoencoder (MAE) pretrained Vision Transformer to build rich visual representations. These features are refined through a multi-scale pyramid and spatial refinement modules, allowing the model to handle complex objects and perform accurate counting on unseen data.

Key concepts

Masked Autoencoder Pretraining
This involves training a Vision Transformer (ViT) encoder by reconstructing missing parts of images. The model learns robust visual features by predicting masked patches, which helps it understand the underlying structure of objects in the image before counting.
Multi-scale Pyramid and Spatial Refinement
The extracted features from different layers of the ViT are organized into a pyramid structure. This structure is then spatially refined using Dense Prediction Transformers (DPT) and FeatUp, which helps recover fine spatial details and structural cues that might be lost during the initial feature extraction process.
Proposal-Verification Counting Head
This final module identifies repeated visual patterns by comparing semantic ViT tokens. It proposes regions of interest and verifies their count based on these matches, ultimately producing a density map and the final object count without needing external guidance.

Terminology used across episodes

This episode discusses

The paper

Spatially-Aware Class-Agnostic Object Counting · Read on arXiv

Singapore University of Technology and Design · The University of Canterbury · The Australian National University

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: We're looking at the paper: "Spatially-Aware Class-Agnostic Object Counting".

Jane: Here's the gist of the paper. Generalised object counting aims to estimate the number of instances of an arbitrary object category from a single image, but many recent methods can struggle on structurally complex objects due to limited spatial modelling<ref:2607.16826#pg2> UpCount strengthens the visual representation by extracting multi-layer features from a ViT-B/16 encoder and reassembling them into a refined multi-scale pyramid that is spatially refined using Dense Prediction Transformers and FeatUp, yielding features with improved structural and spatial sensitivity<ref:2607.16826#pg2> A proposal–verification counting head then identifies repeated patterns and produces a density map for the final count<ref:2607.16826#pg2> On FSC-147, UpCount achieves 12.39 MAE and 100.89 RMSE on the test set, and it transfers effectively to vehicle counting on CARPK<ref:2607.16826#pg2> Pruning redundant reasoning tokens Object counting in computer vision aims to accurately quantify any particular object in an image, and is applicable to logistics, surveillance, urban planning and more<ref:2607.16826#pg2> Specialised counters such as ones for crowd-counting

1: and vehicles

2: exist<ref:2607.16826#pg2> These specialised counters can effectively count large quantities of objects but are limited to specific object classes, a limitation that the human counting ability does not possess<ref:2607.16826#pg2> Other general counters can count arbitrary object classes, but require additional text prompts or exemplars<ref:2607.16826#pg2> Exemplars are particularly inefficient as they require humans to manually produce annotations like bounding boxes to isolate the desired object to count<ref:2607.16826#pg2> How the training signal is built The authors propose a two-stage training scheme for UpCount<ref:2607.16826#pg2> 1) Masked Autoencoder Pretraining involves adapting the ViT-B/16 encoder to FSC-147 using masked image reconstruction<ref:2607.16826#pg2> The encoder is initialized from the full ImageNetMAE checkpoint and further pretrained using only the FSC147 training split<ref:2607.16826#pg2> The reconstruction loss is averaged across all patches<ref:2607.16826#pg2> 2) Supervised Counting Fine-Tuning uses densitymap supervision to optimize the complete architecture including all ViT blocks<ref:2607.16826#pg2> The supervised objective is given by L = Ldensity + λcLcount + λlogLlog-count + λvLverify<ref:2607.16826#pg2> Where the gains come from The core innovation lies in strengthening the visual representation at the very first stage of the pipeline<ref:2607.16826#pg2> Specifically, they replace the original encoder with an MAE-pretrained ViT-B/16 and extract features from multiple intermediate Transformer blocks, which are then reassembled into a multi-scale pyramid and spatially refined (Dense Prediction Transformers (DPT)

5: + FeatUp

6: )<ref:2607.16826#pg2> This design preserves fine structural cues while retaining highlevel semantics, improving zero-shot counting of structurally complex objects, while also eliminating the exemplar boxes or text prompts at inference time<ref:2607.16826#pg2> The final prediction is made using a proposal–verification counting head that detects repeated visual patterns and outputs a density map and count<ref:2607.16826#pg2> Performance Comparison UpCount achieves strong performance on FSC-147 with a validation MAE/RMSE of 13.62/60.07 and a test MAE/RMSE of 12.39/100.89<ref:2607.16826#pg2> On CARPK, UPCount reaches 6.27 MAE and 8.79 RMSE, substantially improving over exemplar-guided methods<ref:2607.16826#pg2> Tab II presents the results on the official CARPK test set following fine-tuning on its training set<ref:2607.16826#pg2> UpCount achieves an MAE of 6.27 and an RMSE of 8.79 without requiring exemplar boxes during inference<ref:2607.16826#pg2> These results indicate that the proposed representation transfers effectively from general class-agnostic counting to the more specialized vehiclecounting domain<ref:2607.16826#pg2> Qualitative Results Fig.

Tom: 2 outlines representative qualitative results in Fig.

Jane: 2, where UpCount produces coherent density maps on FSC-147 for cluttered scenes and structurally complex objects, and maintains stable car counts on CARPK<ref:2607.16826#pg2> The model performs reference-less classagnostic counting and does not require exemplar boxes, category labels, or textual queries at inference<ref:2607.16826#pg2> It successfully eliminates the need for external guidance during the final prediction stage<ref:2607.16826#pg2> The model's performance on CARPK shows substantial improvement over previous methods that relied on visual exemplars<ref:2607.

Title and authors: Tom: So, we're looking at this paper called "Spatially-Aware Class-Agnostic Object Counting." It tackles that general object counting problem where you want to estimate how many of *any* object are in a picture without needing to know what those objects are beforehand.

Jane: That sounds really broad for counting, Tom. What's the big challenge they're trying to solve here?

Tom: The paper points out that most existing methods struggle with structurally complex objects because their spatial models are too limited, so this work introduces a new approach called UpCount designed specifically to keep track of those spatial structures better.

Lu: The core idea is strengthening the visual representation right at the start by using a pre-trained Vision Transformer and then putting those features through a multi-scale pyramid refinement process.

Meng: I wonder how they manage to do that without just relying on some huge, fixed model like you might see in standard counting setups? What's actually happening with the initial feature extraction?

Jane: Well, they adapt a Vision Transformer B eighteen and then extract features from several intermediate blocks before reassembling them into this pyramid structure.

Tom: And that isn't just one layer of features; they use Dense Prediction Transformers and something called FeatUp to spatially refine those features.

Lu: That FeatUp part is key because it learns a joint bilateral upsample, which helps recover the fine spatial structures that get lost during the tokenization process.

Meng: So, if I'm deploying this, does all that feature extraction and reassembly add a huge computational cost for real-time counting?

Tom: The paper shows they achieve performance on FSC-one hundred forty-seven with a mean absolute error of twelve point three nine and a root mean square error of one hundred point eighty-nine but they also show it transfers well to vehicle counting on CARPK, getting a mean absolute error of six point two seven and a root mean square error of eight point seventy-nine <ref:2607.16826#pg3>.

Jane: That transfer from one dataset to another is really interesting for us because it suggests the representation learned isn't just specific to one object type or scene layout.

Tom: It seems like the main innovation is that they replace the standard encoder with this MAE-pretrained Vision Transformer, which gives it a good foundation before they start refining it.

Title and authors: Lu: They build this up in two stages, first a masked autoencoder pretraining phase using image reconstruction loss on FSC-one hundred forty-seven data, and then a supervised counting fine-tuning phase using density map supervision <ref:2607.16826#pg1>.

Meng: Two stages sounds intensive. What does the supervised objective look like when they are optimizing the whole architecture?

Tom: The objective function they use is quite comprehensive, combining a density map loss, a count loss with some lambda value, and also a log likelihood term plus a verification loss to make sure it's accurate.

Jane: It's all about making sure the final prediction isn't just guessing; it has to produce both the map and the actual count based on what it sees.

Tom: The final step is using this proposal-verification counting head that looks for repeated visual patterns, which then outputs a density map and a final count.

Lu: That head computes non-local similarities between the semantic Vision Transformer tokens and aggregates matches to find those repeated patterns, which is how they get the density map and the count.

Meng: So, what's the trade-off here? They claim it eliminates exemplar boxes or text prompts at inference time, but does that mean it can still fail on really weird visual patterns?

Jane: The paper says it performs reference-less class agnostic counting and doesn't require any external guidance like exemplar boxes or category labels during the final prediction stage.

Tom: They say this method successfully eliminates the need for those external guides during the final prediction stage, which is a big deal for deployment.

Lu: However, they also flag that their method is still limited by its initial spatial modeling capabilities, as noted in the abstract when they mention how general methods struggle with structurally complex objects.

Meng: So what's the real limitation of this specific UpCount architecture? Where does it stop working if an object is really weirdly shaped?

Jane: The paper itself states that while it preserves fine structural cues, its reliance on the initial feature extraction and reassembly still means it might not be perfect for every kind of complex structure.

Title and authors: Tom: Looking at the CARPK results, they achieved a mean absolute error of six point twenty-seven and a root mean square error of eight point seventy-nine which is substantially better than methods that relied on visual exemplars <ref:2607.16826#pg3>.

Lu: The transfer to vehicle counting on CARPK demonstrates that this representation actually generalizes well from the initial FSC-one hundred forty-seven task to a more specialized domain <ref:2607.16826#pg1,to vehicle counting on CARPK>.

Meng: From a deployment standpoint, if we're moving this to a production environment, what about the training time? How long does it take for this entire two-stage training scheme to complete on a new set of data?

Tom: The paper doesn't give us exact minutes or hours, but they show that the pretraining is done using only the FSC-one hundred forty-seven training split, which suggests it's optimized for that specific task first <ref:2607.16826#pg1>.

Jane: It sounds like the initial training is focused on getting a very strong visual foundation before they fine-tune with those density maps to get the final counting accuracy.

Lu: The paper implies that this two-stage process is what allows them to build up that structural and spatial sensitivity needed for this class-agnostic approach.

Meng: And if I'm looking at the full picture, what is the biggest implication for people who just use these tools without reading dense papers?

Jane: The main point is that you can now count objects in images without having to manually label examples or write specific text prompts for every new object type.

Tom: So, to wrap up this discussion on "Spatially-Aware Class-Agnostic Object Counting," it's a method that builds a spatially aware representation from multiple ViT blocks and refines it with density maps.

Jane: It moves us toward counting systems that are more flexible because they don't need those external guides to work correctly.

Tom: Yeah, the performance numbers on FSC-one hundred forty-seven and CARPK show a solid improvement over methods that rely on exemplars for counting <ref:2607.16826#pg1>.

Lu: Overall, this work shows how you can improve general object counting by focusing heavily on preserving spatial structure within the learned visual features.

Meng: For deployment, it seems promising because it removes the need for those manual annotation steps at inference time.

Lalam: I'm confident that this method, UpCount, is robust and its performance transfer to different counting tasks shows a strong underlying representation.

The paper's summary: Tom: So, we just talked about how UpCount strengthens its visual representation using that pre-trained Vision Transformer and the multi-scale pyramid refinement process to handle complex objects better than standard methods.

Jane: Right, and now we're looking at what they actually did in this paper called "Spatially-Aware Class-Agnostic Object Counting." It boils down to taking a general counting problem—counting *any* object without knowing what it is—and making the visual feature extraction much smarter spatially.

Lu: The summary says their core innovation is replacing the original encoder with a Masked Autoencoder pre-trained Vision Transformer B eighteen and then extracting features from multiple intermediate blocks.

Meng: So, they're using that pre-trained model just to get a really good starting point, right? What does that give them over just using a standard ViT from scratch?

Tom: It gives them a foundation trained on ImageNet MAE, which means it already understands general visual structure before they even start counting. Then they take those features and reassemble them into this multi-scale pyramid.

Jane: And the refinement happens there, using Dense Prediction Transformers and FeatUp to spatially refine those features. That FeatUp part is what helps them recover fine spatial structures that get lost during tokenization.

Lu: Exactly, it’s about preserving those fine structural cues while still keeping the high level semantics from the transformer blocks. They are essentially building a hierarchical representation.

Meng: But what's the final output? How does this all translate into a count? Does it still need some kind of human input at the end?

Tom: The final part is a proposal-verification counting head, which identifies repeated visual patterns and outputs both a density map and the actual count. They don't need any text prompts or exemplar boxes for that inference stage anymore.

Jane: That’s huge because it means this system can do reference-less class agnostic counting, which used to require you to manually label examples for every object category.

Lu: The training signal they build is a two-stage thing, starting with masked autoencoder pretraining on FSC one four seven data and then supervised counting fine tuning using density map supervision.

Meng: Two stages sounds heavy for deployment. So, what’s the actual performance comparison they show? I need to know if this is just better on a small test set or if it's actually useful outside of research labs.

Tom: They show strong results on FSC one four seven with a mean absolute error of twelve point three nine and a root mean square error of one hundred point eight nine.

Jane: And they show that it transfers well to vehicle counting on CARPK, getting an absolute error of six point two seven and an rmse of eight point seven nine there. That transfer is pretty interesting.

Lu: It shows the representation learned actually generalizes from a general class agnostic task to a more specialized domain like vehicles.

Meng: So, for someone shipping models, what’s the real cost here? Does this complex architecture make it too slow for real time?

Tom: The paper focuses on showing that they achieve zero exemplar boxes or text prompts at inference time, which eliminates a whole class of deployment headaches.

Jane: It means you can deploy a system that counts things without needing to painstakingly create and manage bounding box annotations for every new thing you want it to count.

Lu: The limitation they point out is still the initial spatial modeling capability; while they preserve cues, the reliance on that first feature extraction stage means it might not be perfect for every structurally complex object imaginable.

Meng: So, if I'm looking at this for production, what’s the practical warning? Where does this system stop working reliably?

Tom: It stops working reliably where the initial visual representation isn't strong enough to capture a specific structural feature accurately.

Jane: Overall though, it moves us toward systems that are more flexible because they don't need those external guides to work correctly when counting things around us.

The paper's improvements: Tom: We've talked about how UpCount uses that pre-trained Vision Transformer to build a strong spatial foundation and then refines it with multi-scale pyramids and density maps for counting.

Jane: So now we're looking at what they suggest as the actual improvements in this approach. It’s less about just the final score, and more about *how* they are building that image representation from the ground up.

Lu: The paper highlights that replacing the standard encoder with an MAE-pretrained Vision Transformer and extracting features from intermediate blocks is a core improvement because it preserves fine structural cues while retaining high level semantics.

Meng: Preserving those cues, right? That’s important for complex objects, but how do they handle the loss of detail that happens when you compress that information into a pyramid structure?

Lu: They address that by introducing the Dense Prediction Transformers and FeatUp modules to spatially refine those features. This is essentially a hierarchical representation built from different resolutions.

Jane: It’s like having multiple lenses looking at the same scene, and they use that joint bilateral upsample in FeatUp to recover those fine spatial structures lost during the initial tokenization process.

Tom: That sounds like a really smart way to handle the trade off between high level meaning and low level detail. A clean ablation on that would show exactly how much counting accuracy improves when you add that refinement step.

Lu: And they also introduce a proposal-verification counting head, which is another improvement in the final prediction stage. This head computes non-local similarities between semantic tokens to find repeated visual patterns.

Jane: So, instead of just predicting a single number based on the whole image, it proposes regions and verifies them against those learned visual patterns before outputting the final density map and count.

Tom: That verification step is what moves it beyond simple density estimation; it’s actively looking for evidence of repeated instances. It makes the model much more robust to noise in the input data.

Meng: But what’s the caveat? If you use this architecture, does it become too slow when you're trying to deploy it on something like a drone or a fast vehicle camera?

Lu: The paper notes that while this design improves zero shot counting of structurally complex objects without needing exemplar boxes at inference, the overall system still relies heavily on that initial spatial modeling foundation.

Jane: So, the limitation remains tied to how well that first feature extraction works; it’s not a complete solution for every single weird shape out there yet.

Tom: The performance numbers we saw earlier, like the six point two seven error on CARPK, show that even with those limitations, this architecture still transfers well across different counting tasks.

Lu: The implication is that focusing on strengthening the spatial representation at the very first stage of the pipeline is a very effective way to improve general object counting.

Jane: It suggests that for AI systems meant to count things without human intervention, improving spatial awareness early in the learning process is a better bet than just adding more layers later.

Conclusion: Tom: So, to wrap up, this paper introduces UpCount which uses a pre-trained Vision Transformer and spatial refinement modules to build a stronger visual representation for class agnostic object counting.

Jane: Exactly, it’s about improving how the model sees the world spatially so it doesn't need those manual labels or prompts to count things.

Lu: The main point is that by focusing on strengthening the visual representation at the very first stage, they can handle structurally complex objects much better than previous general methods.

Meng: For deployment, this means you get a system that works without needing those expensive human annotation steps at inference time for new object types.

Lalam: I'm quite sure that this method provides a solid and generalized way to estimate object counts across different visual scenes without requiring specific training data for every single class.

Tom: The performance numbers, like the six point two seven error on CARPK, show it transfers well from one task to another, which is a big deal for real world applications.

Jane: It really shows that improving spatial awareness early on is a solid strategy for building more flexible counting systems.

Lu: And while they acknowledge the limitation in the initial spatial modeling, the overall framework suggests a promising direction for representation learning in this area.

Meng: If we're looking at scaling this up, it means our deployment pipeline can become much simpler because we don't need to constantly feed new exemplar boxes into the system.

Lalam: I think I'm confident that the paper "Spatially-Aware Class-Agnostic Object Counting" offers a reliable path forward for making general object counting more practical.

Tom: Yeah, it’s a solid piece of work because it addresses the structural complexity issue directly with measurable results.

Jane: It gives us a good template for how to build more flexible AI systems that can count things without constant manual guidance.

Lu: Looking ahead, I think we should see more research focusing on how this spatial refinement interacts with other multimodal inputs, maybe combining it with the video understanding models we've been looking at.

Meng: I'm curious if there’s a way to make the training process itself more efficient so that pretraining doesn't take forever on huge datasets.

Lalam: The next big question will be how this spatial awareness scales when we move from 2D images to full three dee scene understanding in autonomous driving scenarios <ref:2607.16826#pg1>.

More episodes

← Home