Bias Analysis of L2 Speaking Assessment Systems Using Concept Activation Vectors

The paper addresses fairness in automatic second language (L2) speaking assessment systems, which are increasingly deployed in high-stakes settings such as immigration and employment certification and whose scores must reflect speaking ability rather than irrelevant attributes of the speaker. The authors note that while Transformer-based foundation models have improved the accuracy of these L2 speaking graders, but their black-box representations make fairness and interpretability analysis more difficult. A fair grader should rely on evidence of ability such as fluency, grammatical control, and content, and remain insensitive to attributes such as first language (L1) or age. These attributes are problematic because they "are nonetheless richly present in the data and can enter a grader either from the large, under-documented corpora used to pre-train its foundation model, or from imbalance in the task-specific fine-tuning data, where an over-represented group at a given proficiency can become a shortcut for predicting the score."

Concept Activation Vectors (CAVs). Building on the TCAV framework, CAVs represent a human-interpretable concept as a direction in the activation space learned by a linear classifier to separate activations of examples that exhibit the concept from those that do not. In the regression setting of speaking assessment, a linear classifier is trained on activations with binary concept labels, and the normal vector to the decision boundary is the CAV. Crucially, CAVs distinguish between whether a concept is encoded in a model's representations and whether it actually influences the model's prediction. Influence is quantified using a gradient-based sensitivity metric: the mean per-speaker gradient cosine distance, Bgr, between the concept direction and the gradient of the predicted score with respect to activations. "A value of Bgr near 1 indicates that the concept direction is close to orthogonal to the score-increasing direction... a value below 1 indicates that increasing the concept tends to increase the predicted score, while a value above 1 indicates that it tends to decrease it."

Sparse Autoencoders (SAEs). Since CAVs assume a concept can be captured by a single linear direction, which grows restrictive as Transformer representations become higher-dimensional and more entangled, the authors investigate TopK SAEs, which enforce sparsity explicitly by retaining only the K largest latent activations for each input. The SAE maps activations to a sparse higher-dimensional latent vector, and because the decoder is linear, a CAV learned in latent space can be mapped back into activation space via the decoder columns. SAE hyperparameters were chosen on four criteria: reconstruction error, sparsity, dead-feature fraction, and decoder coherence.

The primary dataset is the BULATS Speaking component, a multi-level monologic computer assisted test of business-focused L2 English, comprising 1657 training, 188 development, and 465 evaluation speakers with rich metadata including gender, age, and L1. The secondary corpus is the Speak & Improve (S&I) 2025 Corpus, with 3068 training, 438 development, and 300 evaluation speakers, where gender is inferred using an ECAPA-TDNN voice gender classifier due to the absence of demographic metadata.

Two graders are analysed: a text-based BERT grader (analysed at a 600-dimensional Layer 1 and 20-dimensional Layer 2 in its regression head) and a Whisper-based multimodal grader (analysed at the 2048-dimensional dense.in head input and 64-dimensional act.out post-ReLU representation). Concepts include proficiency thresholds (≥ A2, ≥ B1, ≥ B2, expected to align with scores as a sanity check), demographic concepts (gender, age ≤ 30), and six L1 concepts. Table I shows score gaps between concept groups, e.g., Dutch speakers score 0.82 above non-Dutch on average, while Gujarati speakers score 0.45 below.

Grader performance. Both graders are suitable for analysis: Pearson Correlation Coefficient (PCC) values lie between 0.68 and 0.79, and a large majority of responses are placed within a full grade band of the human score.

Concept recoverability. CAV classifier balanced accuracy reveals that Whisper dense.in is the only layer where every concept is linearly recoverable. The two BERT layers and the Whisper post-ReLU output have more than half L1 concepts degenerate. Proficiency concepts are usable everywhere. Since all four layers share the same class imbalance, yet recoverability differs substantially... The failure of the other layers is therefore a property of the representations they learn, not of the imbalance.

Bias detection in activation space. "Bgr < 1 consistently for proficiency concepts, and For BERT this moves further from 1 with depth: ≥ A2 drops from 0.81 at Layer 1 to 0.46 at Layer 2. L1 directions reflect the dataset's score imbalance, with Dutch speakers scoring 0.82 above non-Dutch on average and their Bgr dropping from 0.91 to 0.52 across the two BERT layers. By contrast, The Whisper grader, by contrast, stays close to Bgr = 1 for all demographic and L1 concepts and shows no clean depth trend. This is attributed to input modality: BERT scores transcripts and can access speaker attributes only as far as they surface in lexical and grammatical choices, whereas Whisper has direct access to acoustic information such as pitch and accent that correlates with gender and L1. A mild sensitivity against the female gender concept was found at BERT Layer 2 (Bgr = 1.11) despite the almost-zero gender score gap."

SAE results. In latent space the concepts become noticeably more recoverable. No concept remains degenerate. However, the dominant pattern is that Bgr shifts towards the unbiased baseline of 1. This attenuation is strongest in the lowest-dimensional representation (BERT Layer 2, 20-dimensional) and weakest in the highest (Whisper dense.in, 2048-dimensional). The L1 Dutch concept spans both extremes, preserved almost exactly at dense.in (0.96 → 0.96) yet shifting by +0.40 at BERT Layer 2 (0.52 → 0.92). The paper concludes: "Because the SAE attenuates exactly the sensitivity signal that bias analysis depends on, and most severely in the small layers where bias was clearest, the sparse representation does not improve bias measurement over probing the activations directly."

S&I replication. "Table VI reports the activation space Bgr on S&I, which reproduces the proficiency and architecture trends seen on BULATS. The BERT grader again becomes more sensitive with depth, with ≥ B1 dropping from 0.77 to 0.23 from Layer 1 to 2. Whisper shows no depth trend and stays close to the baseline of 1."

The central finding is that linear recoverability of a concept is governed by the representation it is measured in, not the concept itself, and that recoverability does not imply influence. This was demonstrated by the Whisper grader, which "encoded far more demographic information than BERT yet stayed close to the baseline of no sensitivity, while BERT grew more sensitive to proficiency with depth and reflected the dataset's score imbalance through its L1 directions. For fairness auditing, this separation of encoding from influence is the key point, since an attribute being encoded is not in itself evidence of bias. The SAE did not improve the analysis because Recoverability only needs a classifier to separate the concept, whereas the gradient metric requires the decoder-mapped direction to still align with the score gradient in activation space, which the reconstruction objective has no reason to preserve. This motivates future work on a score-aware SAE, trained to reconstruct the grader's output or to weight reconstruction error by the score gradient, so that the sparse space retains the directions the bias metric needs."

Improvements to AI systems:

1. Add CAV-based fairness auditing as a built-in pre-deployment check.

The improved AI system automatically probes each model layer with Concept Activation Vectors for protected attributes (L1, gender, age) and for proficiency thresholds (≥A2, ≥B1, ≥B2). It reports two separate numbers: concept recoverability (can the attribute be decoded from representations?) and concept influence (does moving along the attribute direction change the predicted score?). This prevents false alarms from encoded but unused attributes.

2. Use gradient-cosine sensitivity as a training regularizer.

Train the speaking grader with a fairness penalty that encourages the CAV direction of each protected attribute to be orthogonal to the score-gradient direction in activation space. Concretely, penalize (grad a f(x) times v protected) squared, while also rewarding alignment with proficiency CAVs. This directly reduces B gr deviations from 1 for demographic and L1 concepts without requiring labeled demographic data at inference time.

3. Make layer selection and concept recoverability diagnostics automatic.

The system audits multiple representational layers (e.g., BERT early vs. late, Whisper dense input vs. post-ReLU) and only reports bias metrics where the concept is linearly recoverable above a confidence threshold. This avoids the paper's observed failure mode where more than half of L1 concepts are degenerate in certain layers, which can produce misleading or unstable bias estimates.

4. Implement a score-aware sparse autoencoder for interpretable bias tracing.

Replace the standard reconstruction-only SAE with one that also reconstructs the grader’s score or weights reconstruction error by the score gradient. This preserves directions that matter for predictions, preventing the attenuation of bias signals seen in the paper (e.g., BERT Layer 2 B gr shifting from 0.52 to 0.92 for Dutch L1). The resulting sparse latent space keeps both interpretability and sensitivity.

5. Use CAV direction shift as a data-imbalance shortcut detector.

The system continuously monitors whether an L1 or demographic CAV direction becomes aligned with the score-increasing direction in later layers. For instance, if Dutch L1 B gr drops from 0.91 to 0.52 across BERT layers while Dutch speakers score +0.82 higher, the system flags a likely training-data shortcut and triggers reweighting, resampling, or adversarial debiasing of the fine-tuning data.

What the improved AI system can do:

• Provide a transparent fairness report before deployment, showing exactly which layers encode protected attributes and whether those attributes actually influence speaking scores.

• Distinguish between the model knows the speaker’s L1 and the model is biased by the speaker’s L1, reducing both false positives and false negatives in fairness audits.

• Automatically detect and mitigate L1/gender/age score shortcuts caused by imbalanced training data, without requiring demographic labels during inference.

• Preserve bias-sensitive information in sparse, interpretable representations, so auditors can trace score changes back to human-understandable concepts.

• Verify that the model is using genuine proficiency evidence (fluency, grammar, content) by checking that proficiency CAVs are consistently aligned with score gradients, while protected-attribute CAVs remain orthogonal.