How Far Do Foundation Models Transfer to Infant Signals? A Cross-Dataset Transfer Audit with a Unified Need Ontology

This paper audits how far off-the-shelf audio foundation models transfer to infant cry signals across four public cry corpora. The authors ask what single-corpus evaluation practice hides and what fixes it. Across four cry corpora screened by a multi-level leakage audit (byte-level and embedding-level deduplication), they probe four frozen encoders and a handcrafted baseline under a unified five-class need ontology and shared task formulations. The audit exposes: within-domain macro-F1 swings by 0.52–0.77 for the same encoder, cross-corpus transfer is negative on average (negative-transfer ratio 0.18–0.34, significant in 17 of 30 directed cells, BH-FDR), and 349 content-identical clip groups carry conflicting labels across corpora. Transfer into the noisiest corpus is reliably positive—at matched training size and after near-duplicate removal. Frozen probes saturate at modest label budgets, while stabilized fine-tuning wins with full labels; domain-adaptive pretraining helps mainly at the full-data end. Ontology-mapped joint training wins in all four encoder-by-target settings, whereas naively merging unmapped labels costs up to 37 F1 points. The authors release the ontology, mapping code, and audit pipeline.

The paper addresses a practical question: how far do these models transfer to infant signals? Infant cry classification has progressed from MFCC-and-SVM pipelines to deep architectures, but reviews agree on the obstacle: not model capacity, but data. Public corpora contain hundreds of clips, not millions; labels are caregiver inferences from context rather than physiological ground truth; and each corpus defines its own label set, so the data cannot even be pooled without semantic decisions. Evaluation remains almost entirely within-corpus, under protocols that differ from paper to paper.

The authors demonstrate the need for an audit: "Recomputing content hashes across the raw, pre-deduplication versions of the four corpora we study, we find 349 groups of acoustically identical clips that carry different labels in different corpora — 292 of them (83.7%) disputes about whether a cry means hungry. When the same frozen encoder is probed within each corpus, macro-F1 swings by 0.57–0.80 depending on which corpus supplies the labels."

The audit defines a five-class need ontology over the four corpora, with explicit mapping rules and a multi-level leakage audit (byte-level deduplication, embedding-level near-duplicate screening, and a within-corpus train–test near-duplicate audit); measures a train×test transfer matrix for four audio foundation encoders and a handcrafted baseline under a unified held-out-test protocol, summarized with a negative-transfer ratio, cluster-bootstrap tests, and FDR control; sweeps label budgets under three adaptation arms; and measures what the ontology buys in joint training—and what skipping it costs.

The findings are diagnostic rather than triumphalist:

1. Domain gap (C1): Foundation encoders show a corpus-level domain gap of up to 0.80 macro-F1. On the hardest corpus (DAC), a handcrafted XGBoost baseline beats two of the four deep encoders and matches a third. A zero-shot general audio MLLM—probed as a sixth, training-free encoder—trails even the weakest in-domain probe.

2. Negative transfer (C2): Negative transfer is the norm (NTR 0.19–0.35, up to 0.46 cry-only) and asymmetric, with one instructive exception—transfer into the noisiest, lowest-diagonal corpus (DAC) is consistently positive in effect size, including after near-duplicate removal, though not individually significant.

3. Few-shot adaptation (C3): Frozen probes saturate at modest label budgets; DAPT significantly beats stabilized fine-tuning at 5–10-shot (the 1-shot advantage is not robust to optimization-seed variance); the 50-shot crossover to stabilized fine-tuning is a non-significant trend (paired 95% CI includes zero) that materializes only at the full-data budget; the 1-shot arm differences lie within shot-sampling noise; and naive full fine-tuning collapses.

4. Ontology joint training (C4): In the tested binary, shared-label settings, ontology-mapped joint training is best in all four encoder×target combinations, while merging unmapped labels costs up to 37 F1 points.

5. Quality–transfer correlation (C5): A pre-specified quality–transfer correlation is refuted as stated—clip-level signal quality acts only as a proxy for domain identity.

6. Label conflicts (C6): 349 content-identical clip groups carry conflicting labels across corpora.

Let D = D1,..., DK be K infant-cry corpora. Corpus Dk contains clips (x, y, g) with waveform x, raw label y ∈ Lk, and group identifier g recovered from available metadata. The raw label spaces Lk are mutually inconsistent: the same acoustic content may be labeled hungry in one corpus and discomfort in another. Each corpus is treated as a separate domain, and labels are never merged without an explicit mapping.

Given a fixed encoder f (frozen) and a fixed probe protocol, the transfer matrix T ∈ RK×K is T[i, j] = F1 probe(f, Ditrain) evaluated on Dj, where F1 is macro-F1. All entries are evaluated on the held-out test split of the target domain Dj (unified protocol): off-diagonal models are trained on source-domain clips only and never touch target-domain data.

NTR = 1 − (average off-diagonal / average diagonal). NTR > 0 means training on another corpus is on average worse than training in-domain. Because each cell is produced by the full probe pipeline, NTR is an end-to-end protocol transfer deficit, not a pure measure of representation quality.

The ontology is a pair O = (C, ϕk) of M need classes C and per-corpus maps ϕk: Lk → C ∪ ⊥, where ⊥ marks raw labels excluded from analysis. Five classes cover the label inventory of all four corpora: hunger, pain-discomfort, sleepiness, need-soothing, and neutral (non-cry vocalizations and noise). The prediction target is the caregiver-annotated behavioral need — an annotation convention, not a physiological ground truth.

Mapping rules are deterministic and released as code. Key decisions: cold hot→pain-discomfort, burping→need-soothing, unknown/dk→⊥, laugh/noise→neutral. Coverage is sparse by construction: hunger is absent from BCSD-U after deduplication, sleepiness exists only in DAC and DI-∆, and neutral exists only in BCSD-U.

Four publicly released audio encoders spanning two dominant pretraining paradigms are audited: wav2vec 2.0 base and WavLM base+ (self-supervised speech), and CLAP and AST (AudioSet-pretrained). A 252-dimensional handcrafted set (MFCCs, prosody, spectral statistics) with logistic regression or XGBoost serves as non-deep reference.

• Arm A (frozen probe): Embeddings are mean-pooled over time and fed to a balanced logistic-regression probe; the regularization constant is selected on the source-domain validation split.

• Arm B (full fine-tuning): All encoder weights are updated on target-domain labels (lr 10−4, bf16, at most 5 epochs, early stopping on validation).

• Arm C (domain-adaptive pretraining + probe): The encoder is first continued on pooled unlabeled cry audio with the masked-prediction objective (DAPT) for one epoch, then probed as in Arm A. The unlabeled pool combines the train split only of the four corpora (1,037 clips) with 3,500 clips of CryCeleb (4,537 clips in total, 283 steps).

• Arm Bs (stabilized fine-tuning, post-hoc diagnostic): After Arm B exhibited run collapse, a post-hoc arm freezes the convolutional feature encoder and fine-tunes the transformer stack at lr 10−5.

Pairs of files are matched on a dual key (content sha1, filename); 1,247 duplicates are removed before any split (781 from BCSD, 466 from DeepInfant). This is essential: BCSD's hungry set is byte-identical to DAC's, and DeepInfant V1 is a repackaging of DAC; without deduplication every cross-corpus number would be inflated by leakage. Byte-level hashing cannot catch re-encoded or trimmed copies, so an embedding-level near-duplicate audit (CLAP cosine nearest neighbors across all domain pairs, second-encoder confirmation) complements it.

Within-domain probe performance shows the same encoder varies by 0.57–0.80 macro-F1 depending on the corpus: WavLM drops from 1.000 on BCSD-U to 0.205 on DAC. AudioSet-pretrained encoders (CLAP, AST) dominate on the cleaner corpora; wav2vec2 is the only deep encoder that clearly wins on the noisy 8 kHz DAC corpus, where the handcrafted XGBoost baseline (0.271) beats WavLM (0.205) and CLAP (0.219) and essentially ties AST (0.273). On the hardest domain, generic features compete with foundation encoders.

NTR is positive for every encoder: 0.194–0.345 on the full binary setting, rising to 0.239–0.461 on the cry-only subset. Of the 30 directed off-diagonal cells (5 encoders × 6 pairs), 19 are raw-significant and 18 show significant negative transfer after Benjamini–Hochberg FDR control at q = 0.05 (paired cluster bootstrap over group id, 10,000 resamples). Transfer is asymmetric by up to 0.196 F1 for the same corpus pair in opposite directions.

Reverse transfer into DAC: Every reversed cell (off-diagonal above diagonal) involves DAC as the target. Training on the larger CC corpus and testing on DAC beats DAC in-domain training for all five encoders in effect size, by 0.020 (handcrafted) to 0.268 (WavLM) macro-F1—but none of the five reverse cells is individually significant. A size-matched control (CC subsampled to DAC's n = 315) still beats the DAC diagonal for all three recomputed encoders (mean margin +0.10/+0.20/+0.18), so size alone does not explain the exception.

Near-duplicate audit: BCSD-U↔DAC is clean (max 0.94); DAC↔CC contains 157 pairs above 0.98 (159 pairs and 312 clips in total, label agreement 99.4%), and the 15 pairs above 0.99 are 1:1 matches confirmed in a second encoder—re-encoded DAC copies inside CC that byte hashing cannot see. Removing all 312 involved clips leaves NTR positive for all three recomputed encoders (wav2vec2 0.148, CLAP 0.300, AST 0.284), and the CC→DAC exception strengthens in effect size (0.778–0.821 vs. clean DAC diagonal 0.442–0.448), so reverse transfer is not a leakage artifact.

No cross-split exact-hash hits exist, yet waveform-level matching does: CC has 42 fingerprint hit pairs across train–test, of which 13 are waveform-identical re-exports (offset 0, xcorr ≈ 1.000, different sha1—exact hashing cannot see them); BCSD-U has 5 time-shifted slice pairs (±1–2 s offsets) from the same long recording spanning train and test; DAC has 8 fingerprint-only candidate pairs. Removing the 65 suspicious test clips and recomputing the full matrix moves the diagonal by ∆ ∈ [−0.072, +0.015], so the diagonal is not systematically inflated. Cleaned NTR stays significantly positive for all five encoders (0.198–0.337 vs. 0.194–0.345 before cleaning).

The pre-specified fine-tuning arm B failed as a measurement instrument: at the full-data budget its per-seed macro-F1 is 0.7025/0.2727/0.3849 (mean 0.453, std 0.223, far above the pre-specified std gate). Judged on Arms A, C, and Bs under the inductive DAPT protocol, the low-label picture is saturation plus a low-budget DAPT advantage over fine-tuning. On paired episodes, C significantly beats Bs at 5- and 10-shot (Bs−C: −0.143 [−0.184,−0.103] at 5-shot; −0.125 [−0.172,−0.079] at 10-shot), the 1-shot DAPT advantage is not robust to optimization-seed variance (−0.052; CI [−0.124, +0.020] includes zero), and at 50-shot Bs exceeds C only as a trend (+0.022; paired 95% CI [−0.009, +0.053] includes zero). Stabilized fine-tuning clearly wins only at the full-data budget (0.822).

Mapped joint training is best in all four encoder×target combinations. The naive merge condition—pooling raw labels without the ontology—quantifies the cost of label-space mismatch: it falls below the best cross-domain transfer on DAC-test (0.639 vs. 0.786 for wav2vec2; 0.435 vs. 0.749 for CLAP) and loses 0.039–0.371 F1 to mapped joint training. The claim is scoped: in the tested binary, shared-label settings (two encoders, two target domains), explicit ontology mapping beats naive merging—we do not claim the five-class ontology is validated in full.

The point-biserial correlation between clip-level SNR and cross-domain correctness is significantly negative for all encoders (−0.04 to −0.10; Fisher combined p < 0.01; 21/30 pairs negative)—the opposite of the quality hypothesis. Two validity problems disqualify the pre-specified reading: the SNR estimator is not comparable across sampling rates (BCSD-U's 102.9 dB is a saturation artifact), and SNR is almost perfectly confounded with domain identity. C5 is refuted as stated: the pooled correlation is a domain-identity artifact—corpus identity, not clip-level signal quality, dominates cross-corpus measurements.

The authors conclude: "We audited how far off-the-shelf audio foundation models transfer to infant cry across four public corpora, under a unified five-class need ontology with a multi-level leakage audit. The answers are measurable and mostly uncomfortable." Within-domain performance swings by 0.57–0.80 macro-F1 across corpora; negative transfer is the norm (NTR 0.19–0.35; 0.24–0.46 cry-only), asymmetric, and significant in 18 of 30 directed cells after FDR control; frozen probes saturate at modest label budgets; ontology-mapped joint training is best in every encoder×target setting; a pre-specified quality–transfer correlation is refuted as stated; and 349 content-identical clip groups carry conflicting metadata labels across distributions.

The outlook: "The transfer gap, label conflicts, and reverse-transfer exception point the same way: fewer cry classifiers, more shared infrastructure—pooled unlabeled cry audio, explicit ontologies, leakage screening, unified evaluation."

Several cells are weakly powered: DI-∆ has n = 2 pain-discomfort clips; BCSD-U sits near a measurement ceiling (ntest = 30); the conflict-excluded DAC test split (n = 16) and the DAC↔BCSD-U common-label matrix are underpowered; the one-epoch DAPT budget makes Arm C a lower bound; the zero-shot MLLM comparison is deliberately asymmetric; and need labels are caregiver-annotated conventions, not physiological ground truth. The ontology mappings are author-set rules: "no independent double annotation, no label-name-normalization intermediate baseline, and no leave-one-mapping-out sensitivity analysis were run, so the C4 margin cannot be attributed to the ontology beyond 'explicit mapping beats naive merge in the tested binary settings.' group id semantics are per-corpus metadata proxies, not infant identity: the within-corpus audit found cross-group leakage the group split cannot intercept, and residual same-infant leakage beyond what four matching methods recover cannot be excluded." Removal robustness covers the recomputed encoders' C2/NTR and CC→DAC cells only.

Based on the paper’s findings, here are concrete improvements to AI systems, followed by what the improved system can do:

---

1. Add a multi-level leakage audit as a mandatory preprocessing stage.

• Implement byte-level hashing (sha1 + filename) AND embedding-level near-duplicate screening (e.g., CLAP cosine similarity with a second-encoder confirmation) before any train/test split or cross-corpus transfer.

• Add a within-corpus waveform fingerprint check (e.g., cross-correlation at offset 0) to catch re-exported or time-shifted copies that hash-based dedup misses.

2. Replace single-corpus evaluation with a transfer matrix protocol.

• For any audio foundation model, report a K×K macro-F1 transfer matrix across all available corpora, not just in-domain accuracy.

• Compute the negative-transfer ratio (NTR) with cluster-bootstrap significance tests and Benjamini–Hochberg FDR control, so users know when cross-domain training actively hurts.

3. Introduce an explicit, code-released label ontology with per-corpus mapping functions.

• Define a shared class set (e.g., hunger, pain-discomfort, sleepiness, need-soothing, neutral) and deterministic rules to map each corpus’s raw labels into it, marking excluded labels with ⊥.

• Never merge raw labels without this mapping; expose the mapping code so downstream systems can reproduce or extend it.

4. Add a “reverse-transfer check” for noisy, low-diagonal target domains.

• Before concluding that cross-corpus transfer is always negative, test the specific case where the target corpus has low in-domain performance and high label noise.

• Run a size-matched control (subsample the source to the target’s n) to separate corpus-size effects from genuine transfer benefits.

5. Implement a stabilized fine-tuning arm that freezes the convolutional feature encoder.

• When full fine-tuning collapses (high per-seed variance), fall back to freezing the CNN front-end and fine-tuning only the transformer stack at a lower learning rate (e.g., 1e-5).

• Report per-seed variance and a pre-specified standard-deviation gate to detect run collapse before interpreting results.

6. Add a domain-adaptive pretraining (DAPT) arm for low-label budgets.

• Continue pretraining the encoder on pooled unlabeled target-domain audio (e.g., masked prediction) for a short budget, then probe with a frozen linear head.

• Compare against fine-tuning at 1-, 5-, 10-, 50-shot and full-data budgets, with paired bootstrap CIs to distinguish real gains from seed noise.

7. Build a label-conflict detector across corpora.

• Automatically find content-identical clips (via byte and embedding matching) that carry conflicting labels across different datasets, and flag them for human review or exclusion.

8. Reframe quality metrics as domain-identity proxies.

• When reporting clip-level SNR or similar quality scores, always test whether they are confounded with corpus identity (e.g., via point-biserial correlation with domain).

• Do not use quality as a transfer predictor unless the domain confound is explicitly controlled.

• Guarantee leakage-free evaluation: It will never report inflated cross-corpus numbers due to duplicate or near-duplicate clips, because it automatically deduplicates at byte and embedding levels before any split.

• Provide trustworthy transfer estimates: It will output a full transfer matrix with significance tests, so users know exactly which source→target pairs are safe to use and which are actively harmful.

• Pool data across corpora without label chaos: It can merge multiple infant-cry (or any audio) datasets using an explicit ontology, avoiding the 37-point F1 drop seen with naive label merging.

• Adapt to low-label regimes robustly: It will choose between DAPT and stabilized fine-tuning based on label budget, avoiding the collapse of naive full fine-tuning and the saturation of frozen probes.

• Detect and handle label conflicts: It will automatically identify clips that are acoustically identical but labeled differently across datasets, and either exclude them or flag them for ontology revision.

• Avoid false quality-based conclusions: It will not mistakenly attribute transfer success to clip-level signal quality when the real driver is domain identity, preventing misguided data-collection or augmentation decisions.

• Scale to new corpora safely: When a new cry dataset arrives, the system will run the same audit pipeline, map its labels through the ontology, and report whether transfer from existing models is positive, negative, or neutral—before any deployment.