Depth-Guided Video Object Counting in Crowded Scenes

The paper's primary objective is to advance video object counting in crowded scenes, aiming to robustly count all instances of a target category based on given text or visual prompts. The authors note that existing methods rely on RGB information, limiting their discriminative ability in crowded and occluded conditions. The paper frames the challenge as follows: "the core challenge of this task lies not only in cross-frame association and deduplication at the video level, but also in accurately distinguishing neighboring objects with highly ambiguous appearances within each frame."

The key motivating insight is that "for tightly arranged or mutually occluded objects of the same category, they still exhibit distinguishable depth differences and spatial hierarchy in 3D space. Such information can not only alleviate occlusion among adjacent objects within a single frame, but also provide geometric constraints that are less sensitive to appearance variation across frames. However, simply concatenating RGB and depth is insufficient: the key lies not in simply introducing depth, but in designing an effective fusion mechanism that can better handle occlusion, appearance similarity, and multi-scale geometric variation."

The paper proposes a unified framework consisting of depth-guided detection and depth-enhanced deduplication.

Since RGB appearance and depth have different representation properties, DG-Det introduces multi-scale RGB-D cross-attention with Depth Affinity Bias at each level of the feature pyramid. The fusion mechanism works as follows: at each scale i, we project the RGB feature Fi,t into queries Q, and the depth feature Fi,t into keys K and values V. A Depth Affinity Bias is introduced into the attention computation to suppress spurious associations between regions that are distant in 3D space but appear nearby in the image plane, defined as:

Buv(Zt) = −beta bias · phi(Zt)u − phi(Zt)v

where phi(·) is a depth normalization function (including log-inverse transformation and quantile normalization). This ensures that feature interaction between regions with different depth values is effectively suppressed.

The decoder architecture uses a global feature generation strategy based on Feature-wise Linear Modulation (FiLM) to address the mismatch between the Transformer's requirement for four input scales and the backbone's output of only three. Specifically, FiLM enables conditional affine modulation through both scaling and shifting, making it more suitable for injecting global depth priors to generate the additional coarse-scale feature.

The detector also includes an occlusion-aware design to explicitly model dense-scene occlusion, generating a continuous occlusion score oi ∈ [0, 1] for each ground-truth object during data preprocessing. The occlusion score is computed according to both the spatial coverage of neighboring objects over object i and their relative depth relationships, with the depth-ordering weight based on a Sigmoid function over depth differences—When instance j has a smaller depth value, i.e., it is closer to the camera, zij becomes larger, indicating that it is more likely to occlude instance i.

For cross-frame association, we incorporate depth into trajectory association, yielding depth-guided association and de-duplication. The matching cost combines three modalities—position, depth, and appearance—modeled probabilistically with modality-specific likelihoods and an IoU-based geometric prior, resulting in:

"Cij = wp dpos(tri, detj) + wf dfeat(tri, detj) + wd ddepth(tri, detj) + lambdaiou I[IoU(bi, bj) < tauiou] + const"

Depth consistency uses a Laplace residual model because depth estimation in real videos may be affected by occlusion, boundary ambiguity, and local distortions, causing the error distribution to typically exhibit a sharp peak with moderately heavy tails. The appearance term uses cosine similarity on the unit hypersphere: Since ui⊤uj is equivalent to cosine similarity, the above form directly captures the relation that the more consistent the feature directions are, the higher the matching probability becomes.

A final temporal-voting stage, Occlusion-Adaptive Temporal Voting, adaptively determines the minimum hit count Nmin and the frame-density threshold rho according to O¯k (the mean occlusion rate of the trajectory). Trajectories are validated using a support score that jointly evaluat[es] the absolute number of hits and the local hit density, with a trajectory deemed valid if the score ≥ 1.

The authors construct and release the RGBD-VideoCount dataset, described as a synchronized RGB-D dataset for video individual counting in crowded scenes. The dataset allows multiple categories to coexist in the same video, and also provides depth information, instance-level bounding box annotations, exemplar boxes, and video-level ROIs. It contains 2,032 finely annotated frames and 77,638 instance boxes, spanning 195 video clips across 6 categories (beverage, bag, book, plant, box, cabinet), averaging 70.51 instances per video (max 209). The data are collected using an Intel RealSense D455, with the RGB and depth streams strictly synchronized through SDK-level alignment.

The total loss combines classification, GIoU, bounding-box L1 regression, and occlusion prediction losses: L = lambda1 Lcls + lambda2 Lgiou + lambda3 Lbbox + lambda4 Locc. The model is initialized from CountGD Box with the pretrained backbone frozen, while the depth encoder, the cross-attention fusion module, and the detection head are optimized using AdamW.

The method's headline result: Extensive experiments demonstrate that our method achieves a 62.01% reduction in MAE compared to existing baselines, and also produces consistent improvements in RMSE.

• Under text-only prompts, CountVID (with SAM 2.1 tracker) achieves MAE 24.77/RMSE 28.76 on validation, while the pure RGB trackers (ByteTrack, OC-SORT, SparseTrack) range from MAE 26.74–34.74. The proposed method achieves MAE 14.55/RMSE 25.32 (validation) and MAE 13.32/RMSE 19.68 (test).

• Under text+exemplar prompts, the improvements are also consistent: CountVID achieves MAE 19.68, ByteTrack 25.51, OC-SORT 38.41, SparseTrack 25.22, versus the proposed method's MAE 13.32 (test).

• Notably, the SAM 2.1 baseline achieves an excellent AP50 but a high MAE, showing that strong single-frame detection cannot ensure accurate counting under severe occlusion.

• Comparison with RGB-D tracking models (DepthMOT and DepTR-MOT) shows that the key to the improvement lies not in naively adding depth, but in effectively modeling depth at the feature representation level and jointly exploiting occlusion cues during both detection and de-duplication.

• Using pseudo-depth (Depth Anything V2) hurts performance because the predicted depth maps suffer from 'edge smoothing,' erasing spatial boundaries between overlapping objects, whereas ground-truth depth is sparse and noisy, it keeps sharp depth changes.

• Depth coverage: With 100% depth, test MAE drops from 15.47 (RGB-only) to 7.98; partial depth (25–75%) improves MAE but can lead to RMSE fluctuations due to cross-modal inconsistency.

• Component ablation: Adding Cross-Attention Fusion reduces test MAE/RMSE from 18.07/26.80 to 14.93/20.89; adding Depth Affinity Bias further improves to 12.20/15.99; adding the Occlusion Head yields the best result of 7.98/11.45.

• Depth bias sensitivity: when beta varies within the range of 0.5–3.0, the model performance remains stable.

• Occlusion-Adaptive Voting: Compared to fixed-threshold filtering, the adaptive voting reduces MAE and RMSE by 1.91 and 2.75, respectively.

• Robustness: The method stays robust under Edge Blur (MAE 8.50) and Random Holes (MAE 8.23), but degrades under Gaussian noise (MAE 11.16) and Salt & Pepper noise (MAE 10.66), because both types of noise directly corrupt local depth structures and fine-grained depth ordering.

On FSCD-147 with pseudo-depth, incorporating this modality does not yield the expected substantial gains because the pseudo-depth is over-smoothed and blurs depth discontinuities, failing to recover instance-level depth differences in densely packed scenes.

The paper identifies glass-door beverage scenes as a failure case: the glass surface introduces reflections, refractions, and local highlights, which cause depth values acquired by the depth sensor to exhibit noise, missing regions, or structural distortions. In such scenes, neither directly captured raw depth nor model-estimated predicted depth can provide stable and reliable support for video instance counting, and the authors state that improving the robustness of the model under abnormal depth inputs... remains an important direction for future research.

The authors list three contributions: We introduce depth cues into video individual counting in crowded scenes, effectively alleviating the limitations of RGB-only methods in occluded scenarios; We propose a depth-guided RGB-D detection and de-duplication framework that better distinguishes occluded instances while reducing missed detections and duplicate counting; and We release an RGBD-VideoCount dataset with synchronized depth information and multi-category coexisting scenes to facilitate research on video counting with depth.

• Depth-guided attention with Depth Affinity Bias: Augment RGB-based detection transformers with a cross-attention fusion mechanism where depth features serve as keys/values, and inject a depth-difference bias into attention logits. This suppresses spurious correlations between objects that are near in the image plane but far in 3D space, enabling the model to separate heavily overlapping instances.

• Explicit occlusion-aware modeling: Add an auxiliary head that predicts a continuous occlusion score for each detected instance, computed from neighboring-box overlap and relative depth ordering using a sigmoid-weighted depth difference. This lets the system reason about who occludes whom rather than relying only on appearance.

• Multi-scale RGB-D feature fusion with FiLM: Use Feature-wise Linear Modulation to generate a global depth-conditioned coarse feature, reconciling backbone output scales with transformer decoder requirements and injecting global depth priors into the detection pipeline.

• Depth-enhanced tracking and deduplication: Combine bounding-box position, appearance cosine similarity, and depth consistency (modeled with a Laplace residual to tolerate sensor noise) into a probabilistic matching cost, plus an IoU-based geometric prior to avoid duplicate counts across frames.

• Occlusion-Adaptive Temporal Voting: Replace fixed hit-count thresholds with a trajectory validation criterion that adapts the minimum number of detections and local density threshold based on the trajectory's mean occlusion rate. This reduces both false-negative rejections of heavily occluded objects and false-positive trajectory fragments.

• Training with synchronized RGB-D data: Pretrain or fine-tune the system on a dataset with synchronized depth streams and instance-level annotations (e.g., RGBD-VideoCount) so the depth encoder learns to exploit sharp depth discontinuities at object boundaries, rather than over-smoothed pseudo-depth.

• Count individual objects in crowded video scenes with severe occlusion, where RGB-only methods fail — achieving up to 62% lower MAE and consistently lower RMSE on benchmarks.

• Disambiguate adjacent objects of the same category that look nearly identical but differ in depth, using geometric constraints that remain stable under appearance variation across frames.

• Reduce duplicate counting and missed detections in long video sequences by jointly associating objects using position, appearance, and depth, then validating trajectories with occlusion-aware temporal voting.

• Operate robustly under text-only prompts or text-plus-exemplar prompts, generalizing across multiple coexisting categories (e.g., beverage, bag, book, plant, box, cabinet) without category-specific retraining.

• Handle degraded depth inputs gracefully: maintaining performance under edge blur and random holes (small MAE increase), and flagging failure when depth noise is severe (e.g., Gaussian/salt-pepper) rather than silently corrupting counts.

• Provide a foundation for extending RGB-D fusion to other dense-instance tasks such as multi-object tracking, segmentation, and re-identification in cluttered 3D scenes.

Our primary objective is to advance video object counting in crowded scenes, aiming to robustly count all instances of a target category based on given text or visual prompts. Existing methods rely on RGB information, limiting their discriminative ability in crowded and occluded conditions. To address this, we propose a Depth-Guided Detector (DG-Det) along with a general post-processing pipeline. By integrating depth cues with multi-scale RGB-D cross-attention and explicit occlusion prediction, our method enhances spatial understanding and achieves robust detection in crowded and occluded scenes. Furthermore, we introduce a unified de-duplication framework to eliminate cross-frame redundant counting. To facilitate future research, we also release a new RGB-D Video Object Counting dataset featuring depth information and multiple object categories persequence. Extensive experiments demonstrate that our method achieves a 62.01% reduction in MAE compared to existing baselines, and also produces consistent improvements in RMSE. We provide the source code at https://github.com/streamer-AP/DG-Net and the dataset at https://huggingface.co/datasets/aerospace123/RGBD-VideoCount.