Does an Illumination Prior Help Face-Swap Detection? A Controlled Study of Temporal Self-Blended Images

The gist: Training-pair illumination delta does not measurably change frame-level detection, and no preferential robustness appears on harsh-illumination bins.

Introduction

Face-swap detectors that learn generator-specific texture artefacts transfer poorly to unseen manipulations The most effective known remedy is a training-data prior rather than an architectural one Face X-Ray [3] and Self-Blended Images (SBI) [2] synthesize fakes by blending an image with a perturbed copy of itself, forcing the classifier onto the blending boundary—a cue common to all compositing pipelines—and achieving strong cross-dataset transfer without labelled fakes The photometric-forensics literature holds that compositing two faces recorded under different illumination produces inconsistencies that physical scene constraints would forbid [4], [5] This suggests an obvious extension: build the self-blended training pair from two frames of the same video chosen to differ in illumination, so that the synthetic fake carries a photometric inconsistency in addition to a blending boundary We call this Temporal Self-Blended Images (T-SBI). The contribution of this paper is not the generator but the measurement It is easy to add such a prior and report an aggregate accuracy delta; it is much harder to show that any observed change is caused by the mechanism the prior was designed around We therefore treat T-SBI as a controlled probe and ask three separable questions: does the temporalillumination component change detection performance at all; is the illumination delta of the training pairs the active ingredient; and does the model become preferentially more robust on the image attribute—illumination—that the prior targets?

Temporal Self-Blended Image Generation and Gating

The generation of T-SBI pairs involves transferring source frame’s illumination statistics onto a self-blended copy of the target frame, preserving identity, expression, and pose so that the training signal is differential illumination rather than incidental geometric inconsistency The process includes:

**: MediaPipe FaceMesh [1] landmarks define a feathered convex-hull mask over the target face (fallback: axis-aligned ellipse). The source frame’s illumination statistics—L-channel mean, L-channel standard deviation, and a low-frequency luminance term inside the mask—are transferred onto a self-blended copy of the target frame, preserving the target’s geometry, identity, expression, and pose so that the training signal is differential illumination rather than incidental geometric inconsistency. We use luminance-only histogram matching with percentile trimming and envelope-based fitting, which suppresses three observed failure modes (flat unicolour faces, blowouts, band artefacts). Pairs are rejected if the source and target boxes differ by >1.3× in scale or >35% in centre offset. Each candidate pair is gated explicitly based on pair-level deltas, defined as ∆Lmean = µL(Is) − µL(It) and ∆Lstd = σL(Is) − σL(It), where statistics are taken over the detector’s face box. A pair is accepted if ∆Lmean ≥ 6 or ∆Lstd ≥ 3. The realized pool is itself informative: across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. The dual criterion covers a global luminance offset and a regional illumination redistribution respectively. To avoid silently discarding evenly-lit videos, a relax fallback accepts the most-different available pair when no pair from a source video clears Eq. (3). Every row records ∆Lmean, ∆Lstd, and a relax flag, enabling the quartile decomposition below. The realized pool is itself informative: across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. The realized pool is itself informative: across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. Across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. Across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. Across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. Across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.3% of pairs entered only via the relax fallback. This is not a property of the source material: over the 732 contributing videos the median per-video maximum ∆Lmean is 10.13 and 89% contain at least one pair clearing the gate. The sampler, not the corpus, is the binding constraint. Across 7,320 accepted pairs the median ∆Lmean is 5.62—below the design threshold—and 22.

1. Incorporate Temporal Self-Blended Images (T-SBI) Prior: Modify training data generation to include T-SBI pairs, which transfer the illumination statistics of one frame onto a self-blended copy of another frame from the same video, gated on a controlled luminance delta ∆L in LAB space. This forces the classifier to learn illumination inconsistency as a learnable training signal rather than just blending boundaries.

2. Implement Pair-Level Illumination Gating: Before training, filter candidate pairs based on explicit photometric differences using the criterion: A pair is accepted if ∆Lmean ≥ 6 or ∆Lstd ≥ 3. This ensures the model is trained on pairs where the illumination delta is the active ingredient, rather than relying on aggregate accuracy.

3. Utilize Imbalance-Robust Threshold Sweeps: Instead of using a fixed threshold like 0.5, employ a imbalancerobust threshold sweeps [9] approach to determine the optimal decision point, as threshold-0.5 metrics conflate calibration with capability [7], [8]. This ensures the model operates at its best performance across diverse evaluation corpora.

4. Shift Focus from Aggregate Accuracy to Attribute Validation: Validate any prior against the target attribute rather than general performance, as argued by the paper: photometric training priors must be validated against the attribute they target rather than against aggregate accuracy. This prevents overinterpretation of aggregate AUC deltas that are merely calibration-level effects.

5. Leverage Compression Robustness for Specific Scenarios: Recognize that T-SBI primarily yields a specific robustness gain: under heavy JPEG compression the TSBI variant retains frame AUC 0.780 on DFDC against 0.696 for its counterpart, consistent with a shift toward low-frequency cues rather than illumination-specific robustness. This makes the system more resilient to specific data degradation modes.

Self-blended images are widely used to train face-swap detectors, but primarily capture blending artifacts. We investigate whether adding illumination inconsistencies improves detection. Temporal Self-Blended Images (T-SBI) transfer lighting statistics between frames of the same video, with the mismatch controlled by luminance difference (ΔL). Using five training regimes and a three-seed comparison of high- and low-ΔL training, we find no evidence of illumination-specific improvements. AUC differences remain within seed variability across four datasets, and an analysis of 506,328 attribute-binned samples shows no preferential reduction in errors under harsh lighting. Instead, T-SBI shifts prediction scores, changing optimal thresholds by approximately 0.34 on FaceForensics++ and 0.30 on Celeb-DF, making comparisons at a fixed threshold misleading. However, T-SBI improves robustness to heavy JPEG compression on DFDC (AUC 0.780 versus 0.696), potentially reflecting greater reliance on low-frequency cues. These findings highlight the importance of evaluating training methods against their intended targets and accounting for threshold effects.