Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

This paper introduces the first dimension-level meta-evaluation benchmark for automated Text-to-Speech (TTS) evaluation, decomposing the concept of naturalness into 10 linguistically grounded perceptual dimensions. The authors construct a dataset of 860 utterances annotated by trained linguist raters across these dimensions, and use it to audit four neural Mean Opinion Score (MOS) predictors and four Audio Large Language Model (Audio-LLM) judges.

The paper proposes a ten-dimension evaluation schema organized into three linguistic levels, drawing on Crystal's distinction between linguistic and paralinguistic features:

Word level (fixed properties of lexical entries):

• Phonetic accuracy: Whether the sounds in each word fall within the acceptable range of realization for their lexical targets.

• Lexical stress: Whether primary stress falls on the correct syllable of each word, including appropriate vowel reduction in unstressed syllables.

Prosodic level (suprasegmental properties appropriate for syntactic and semantic properties of the utterance):

• Intonation: The pitch contour over phrases and sentences, encoding utterance type (e.g. question, statement) and discourse structure.

• Prosodic stress: Prominence on specific words or phrases relative to the rest of the utterance, determined by information structure.

• Prosodic boundary appropriateness: The chunking of the utterance into prosodic phrases aligned with syntactic structure.

• Speech rate appropriateness: The overall pace of delivery, which can fail without any corresponding failure in pitch or prosodic structure.

Paralinguistic level (speaker characteristics):

• Emotional appropriateness: The degree to which the emotional tone matches the affective content of the utterance.

• Expressiveness: Natural variation in delivery style and vocal energy, assessed relative to the system itself.

• Human plausibility: The degree to which vocal qualities fall within the range of physically realizable human production.

• Speaker identity consistency: Stability of perceived vocal characteristics across the utterance, assessed relative to the system itself.

The dataset comprises 860 samples of annotated and majority-voted audio, generated using Cartesia Sonic-3 TTS. Errors were introduced through three pathways: LLM-altered IPA transcriptions (for phonetic accuracy and lexical stress), LLM-altered plain text (for prosodic stress and prosodic boundary placement), API-enforced emotion tags (for emotional appropriateness), and acoustic manipulation via Praat (for intonation, speech rate, expressiveness, speaker identity consistency, and human plausibility). Source corpora included Harvard Sentences, EmergentTTS-Eval, and synthetically generated texts.

Each sample was annotated by three professional linguists on all ten dimensions (not just the targeted dimension), enabling measurement of cross-dimensional perceptual spillover. Inter-rater reliability (Krippendorff's α) ranged from 0.460 to 0.821 across dimensions, with the highest agreement on lexical stress (0.821) and phonetic accuracy (0.767), and the lowest on expressiveness (0.460) and intonation (0.469).

MOS predictors collapse onto acoustic signal quality. The paper finds that MOS predictors align strongly with signal-level artifacts but fail on word-level and prosodic dimensions. Specifically:

• Human Plausibility (constructed via inserted glitches) was the most consistently detected dimension, reaching significance across all tested models.

• Speech Rate Appropriateness (duration scaling) was also significant across 6 of the 8 models.

• "No MOS models show significant detection of Phonetic Accuracy, Lexical Stress, and Prosodic Boundary Placement, dimensions where TTS systems produce linguistically meaningful errors that human listeners reliably detect, but MOS predictors do not."

• Some significant correlations were negative, indicating misalignment with human perception (e.g., AudioBoxPC showed a strong negative relationship with Human Plausibility).

Audio-LLM judges show selective, prompt-dependent detection. Under a generic naturalness prompt (Condition 1), sensitivity was sparse and scattered. Gemini 3.5 Flash with transcript showed the broadest coverage, but every other model and condition reaches at most two or three significant dimensions. Beyond the Gemini family, sensitivity under C1 is sparse to absent.

Schema guidance helps selectively. Providing the full 10-dimension schema while maintaining a single aggregated score (Condition 2a) increases sensitivity to word-level dimensions for models that showed none under C1, suggesting that this capacity was present but not elicited without schema guidance. However, this benefit does not extend uniformly across all dimensions.

Output collapse is a dominant failure mode. When models were required to score each dimension independently (Condition 2b), output collapse becomes the dominant failure mode. Two of the four models assign the maximum score across nearly all dimensions and both transcript conditions, leaving correlation undefined in almost all cells.

Transcript access does not consistently increase sensitivity. Transcript availability produces inconsistent effects across conditions and models, with no clean directional pattern.

Per-dimension prompting (Condition 3) yields inconsistent results. Sensitivity remains sparse and inconsistent, with a pattern that largely differs from C1 and C2a. Step Audio 2 Mini showed no significant correlation under any transcript condition, suggesting a fundamental sensitivity limitation rather than a prompting artifact.

The paper concludes that "current automated evaluators exhibit systematic but distinct blind spots: MOS predictors align strongly with signal-level artifacts but fail on word-level and prosodic dimensions, whereas Audio-LLMs judges show selective, prompt-dependent detection that never generalises across the full dimension set. The authors argue that naturalness cannot be treated as a single scalar construct for modern TTS evaluation and that robust evaluation will require dimension-aware benchmarks." The dataset, annotation schema, and evaluation code are publicly released.

Improvements to AI systems:

1. Dimension-Aware TTS Evaluation Models – Build a multi-head neural evaluator that outputs 10 separate perceptual scores (phonetic accuracy, lexical stress, intonation, prosodic stress, prosodic boundary, speech rate, emotional appropriateness, expressiveness, human plausibility, speaker identity consistency) instead of a single MOS. This system can detect linguistically meaningful errors that current MOS predictors miss, such as misplaced lexical stress or incorrect prosodic boundaries, and provide actionable feedback to TTS developers for targeted error correction.

2. Schema-Guided Audio-LLM Judges with Anti-Collapse Regularization – Enhance Audio-LLM judges by (a) always injecting the 10-dimension schema into the prompt, even for aggregated scoring, to elicit latent sensitivity to word-level and prosodic features; (b) adding a calibration layer that penalizes output collapse (e.g., assigning maximum scores to all dimensions) by enforcing variance across dimensions and comparing against a small set of human-annotated anchors. This system can reliably score all dimensions without degenerating, and its per-dimension outputs can be used to generate natural-language explanations of TTS failures (e.g., stress on the second syllable of 'record' is incorrect).

3. Hybrid Evaluator with Signal-Artifact and Linguistic Feature Extraction – Combine a signal-processing front-end (to detect glitches, duration anomalies, and spectral artifacts) with a linguistic feature extractor (using forced alignment and prosodic analysis to measure stress placement, boundary alignment, and pitch contours) feeding into a small regression model. This system can explicitly separate signal-level quality from linguistic appropriateness, enabling TTS systems to be optimized for both acoustic fidelity and prosodic correctness, and to flag errors in real time during synthesis.

4. Cross-Dimensional Spillover-Aware Training – Train an evaluator that predicts all 10 dimensions jointly, using the paper’s finding that human raters show cross-dimensional perceptual spillover (e.g., a glitch affecting human plausibility also lowers expressiveness). By modeling these correlations, the improved system can infer unrated dimensions from partially observed ratings, reducing annotation cost and providing more consistent evaluations across diverse TTS outputs.

5. Prompt-Adaptive Audio-LLM Judge – Develop a meta-controller that selects the optimal prompting strategy (generic vs. schema-guided vs. per-dimension) based on the target TTS model and error profile, using a small validation set. This system can dynamically switch between aggregated and decomposed scoring to maximize sensitivity, avoiding the inconsistent results seen with fixed prompting, and can be used as a low-cost pre-filter before expensive human evaluation.

What the improved AI system can do:

• Provide a 10-dimensional diagnostic report for any TTS output, pinpointing specific linguistic and paralinguistic failures (e.g., intonation contour is flat for a yes/no question or speaker identity shifts mid-utterance).

• Automatically reject TTS outputs that are acoustically clean but prosodically or phonetically wrong, which current MOS predictors would pass.

• Guide TTS model fine-tuning by offering gradient-based or RLHF-style rewards per dimension, enabling targeted improvement of weak dimensions without sacrificing others.

• Serve as a scalable, human-aligned evaluation tool for TTS research and deployment, reducing reliance on expensive linguist panels while maintaining high reliability on dimensions like lexical stress and phonetic accuracy.

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions. Neither class reliably captures a breadth of linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.