Auditing Metrics for Synthetic Data Utility: A Cross-Domain Study of Appearance and Geometry

The gist The SADGE metric provides a zero-shot method for estimating synthetic-data usefulness by jointly modeling appearance and geometry similarity to predict downstream performance before training

How it works

SADGE is a quantitative similarity metric that predicts the performance of synthetic image datasets for common computer vision tasks without downstream model training It addresses the bottleneck in model development by estimating whether a synthetic dataset will lead to a model that performs well on real-world data The core idea is that appearance scores measure whether synthetic images lie near the target domain, while geometric scores measure whether they preserve structurally meaningful relationships

The metric involves several key steps for each real image against a set of synthetic candidates First, it compares each real image ri ∈ R against one or more synthetic candidates sj from S using either an aligned or retrieval-based comparison mode For retrieval-based comparison, the score for ri is defined as m∗(ri, Si) = max sj∈Si m(ri, sj) (1) The similarity function m(·, ·) can be either an appearance metric (like CLIP or DINOv3 embeddings) or a geometry matching function (like MASt3R inliers or LoFTR matches)

Appearance and Geometry Similarity

The paper computes two primary similarity scores: an appearance similarity score and a geometry similarity score The appearance similarity between a real image ri and a synthetic image sj is defined as A(ri, sj) = ϕ(ri)⊤ϕ(sj)∥ϕ(ri)∥2∥ϕ(sj)∥2, which is the cosine similarity between normalized image embeddings The dataset-level appearance score is obtained by averaging these scores over the selected pairs

To incorporate structural fidelity, SADGE also computes a geometry similarity score based on dense or semi-dense correspondences between real and synthetic images This is quantified by the number of geometrically valid inliers G(ri, sj), which indicates better geometric consistency The dataset-level geometry score is G¯ = 1/N X N i=1 Gi, where Gi is the geometry score associated with the selected synthetic match for ri To stabilize raw inlier counts, they are stabilized using a logarithmic transform: G˜ = log(1 + G¯)

Fusion into the SADGE Score

The final SADGE score is produced by fusing the normalized appearance similarity metric Aˆ and the normalized geometry similarity metric Gˆ Both metrics are standardized using z-score normalization based on statistics computed on the training portion of the synthetic dataset variants The fusion function is defined as a constrained bilinear interaction model: SADGE = aGˆ + bAˆ + cGˆAˆ (6) This bilinear form is chosen because it captures complementarity while remaining monotone and low-capacity The coefficients a, b, and c are estimated by maximizing the Pearson correlation between predicted SADGE scores and target downstream scores yk (7)

Evaluation and Results

SADGE was evaluated on five public synthetic-to-real benchmark families across 15 dataset-level variants, spanning tasks like object detection, semantic segmentation, and pose estimation The results show that SADGE achieves the strongest association with downstream transfer performance under both linear and rank-based criteria Specifically, it reaches Pearson r = 0.879 and Spearman ρ = 0.768 (n = 15) The best configuration obtained is fusing DINOv3 appearance similarity with MASt3R geometric consistency through a constrained bilinear interaction

The study demonstrates that neither appearance-only nor geometry-only metrics consistently predict transfer performance on their own SADGE achieves a strong Pearson correlation of r = 0.879 with downstream task performance such as for pose estimation, semantic segmentation, and object detection Furthermore, SADGE remains best in every leave-one-dataset-out split: excluding ASD (r = 0.907), DIMO (r = 0.637), RarePlanes (r = 0.880), TUD-L (r = 0.888), and VKITTI2 (r = 0.899) This suggests that synthetic-data utility is governed by the joint structure–appearance relationship rather than by a single-factor metric

Conclusion

SADGE provides an efficient training-free ranking metric that can prioritize candidate synthetic datasets before expensive downstream model training The main finding is that the non-linear interplay between appearance and geometry dictates performance, and this fused metric remains stable under coefficient perturbations SADGE supports a ranking/evaluation claim under public protocols, not a universal claim that the same coefficients or component estimators are optimal for every future domain The limitation noted is that the metric's reliability may decrease if the target domain differs substantially from the data used for pretraining Future work will focus on expanding benchmark coverage and improving runtime

Implementation Details

The implementation uses DINOv3 for appearance similarity and MASt3R for geometry matching as the best performing pair identified by an exhaustive component sweep The runtime bottleneck is correspondence estimation with MASt3R, which is dominant in cost The final released configuration uses the constrained bilinear form with parameters a = 0.0, b = 1.8548, c = 1.3399 and specific normalization statistics This provides an efficient training-free ranking metric for synthetic-data utility The paper also provides anonymized supplementary code and evaluation scripts to reproduce the scores The authors explicitly state that SADGE is a proxy ranking metric, not a causal estimate of downstream performance

Limitations and Context

A limitation of the current study is benchmark scope: although it spans five datasets and 15 synthetic-to-real variants, coverage is constrained by the public availability of synthetic datasets SADGE may fail or become unreliable under conditions where the appearance encoder is insensitive to domain-specific artifacts relevant to the downstream task Additionally, the geometry matcher fails on textureless, transparent, reflective, repetitive, deformable, very small, or heavily occluded objects<ref:2605.

1. The SADGE metric can be integrated into a pre-training pipeline to automatically rank candidate synthetic datasets, enabling practitioners to compare candidate rendering configurations, domain-randomization schedules, rendering settings, asset libraries, filtering strategies before committing expensive downstream training resources.

2. A system utilizing SADGE can dynamically select the optimal synthesis strategy by identifying which generative process or domain randomization schedule yields the highest correlation with real-world performance metrics like object detection, semantic segmentation, or pose estimation.

3. The metric allows for a principled comparison between different synthetic data sources, as it moves beyond appearance or geometry alone to capture the utility dictated by their non-linear interplay, thus preventing the encoding of biases that fail to transfer to real-world data.

We propose SADGE, a quantitative similarity metric that predicts the performance of synthetic image datasets for common computer vision tasks without downstream model training. Estimating whether a synthetic dataset will lead to a model that performs well on real-world data remains a bottleneck in model development. Existing evaluation metrics (e.g., PSNR, FID, CLIP) primarily measure semantic alignment between real and synthetic images (Appearance Similarity Score). Less commonly, structural similarity between images is considered to assess the domain gap (Geometric Similarity Score). However, to the best of our knowledge there exists no studies that evaluate which similarity metric is the best downstream predictor for a given synthetic dataset. In this paper, we show over a wide variety of different synthetic datasets and downstream tasks that neither appearance nor geometry alone can reliably predict downstream performance; rather, it is their non-linear interplay that dictates synthetic data utility. Specifically, we measure how commonly used Appearance and Geometric Similarity metrics computed between synthetic and real images correlate with downstream performance in object detection, semantic segmentation, and pose estimation. Across five public synthetic-to-real benchmark families and 15 dataset-level variants (79k image pairs), SADGE achieves the strongest association with downstream transfer performance under both linear and rank-based criteria, reaching Pearson r=0.88 and Spearman rho=0.77. We compute for each combination of geometry-based methods and appearance-based approaches SADGE scores across all benchmark families. The best configuration is obtained by fusing DINOv3 appearance similarity with MASt3R geometric consistency through a constrained bilinear interaction, outperforming both the strongest geometry-only baseline and the strongest appearance-only baseline.