From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop

The Workshop on Trustworthy Natural Language Processing (TrustNLP), co-located with major ACL conferences since 2021, has grown from 8 proceedings papers to 41 over six editions, documenting a field-wide transition from post-hoc interpretability of static models to mechanistic understanding and proactive control of generative systems. We synthesize insights from all 144 proceedings papers, classifying them along six trust dimensions grounded in established frameworks (TrustLLM, DecodingTrust). We observe co-occurrences with capability emergence. The release of the first high-impact chat models activated all trust dimensions simultaneously, while subsequent model generations shifted focus toward truthfulness and safety alignment. Analysis from the classification study reveals that truthfulness is the fastest-growing dimension (absent in 2021–2022, comprising 37% of papers by 2025–2026), fairness remains the most consistent theme, and explainability exhibits a U-shaped trajectory; declining as post-hoc methods lost relevance but resurging in 2026 through mechanistic interpretability. A cross-venue comparison with ACL, NAACL, EACL, and EMNLP (2K papers) in the same period shows that TrustNLP’s topical distribution closely follows the field average. We identify four structural insights and conclude with actionable directions for the research community.

Alongside the 5× increase in archived proceedings papers between 2021 and 2026, the research focus underwent a qualitative transformation in what “trust” means. Before the era of LLM launches (November 2022), the key questions were around fairness and interpretability: can we audit model decisions? After the launch of high-impact chat models, trust also became about reliability and trade-offs: is the LLM consistent across prompts, and how do fairness, performance, and explainability interact? By 2025–2026, with multimodal & agentic systems, trust evolved into controllability: can we constrain model behavior under adversarial conditions and mechanistically verify alignment?

We make three contributions: (1) a quantitative topic analysis of all 144 proceedings papers revealing the fastest-growing and most persistent research themes; (2) a chronological synthesis of technical contributions organized by the dominant paradigm of each phase; and (3) the identification of four structural insights and actionable directions for the research community.

We focus on TrustNLP as a representative case study as three properties make TrustNLP particularly well-suited for this role. First, its publication pipeline is integrated with the ACL ecosystem: accepted papers are peer-reviewed through ARR or direct submission and appear in the ACL Anthology alongside main-conference proceedings at NAACL, ACL, and EACL, ensuring that the work reflects the quality norms and topical pulse of the broader NLP community. Second, the workshop maintains a deliberately narrow scope centered on trustworthiness challenges in generative and interactive language systems, rather than serving as a venue for AI ethics broadly construed; this focus yields a sharper signal about how the field responds to specific capability emergence. Third, TrustNLP offers longitudinal continuity with six consecutive editions (2021–2026) with a stable core of organizers and a consistent programmatic agenda. This enables us to trace thematic evolution over time.

The study of trustworthiness in AI systems spans multiple communities, each contributing distinct methodological perspectives. Several comprehensive surveys and benchmarks have sought to systematize the evaluation of trustworthiness in large language models. DecodingTrust provides a multidimensional assessment of GPT models across toxicity, stereotype bias, adversarial robustness, out-of-distribution generalization, privacy, machine ethics, and fairness, establishing one of the first holistic evaluation protocols for proprietary models. TrustLLM proposes a unified evaluation framework that encompasses truthfulness, safety, fairness, robustness, privacy, machine ethics, transparency, and accountability, and benchmarks over a dozen open-source and proprietary models to reveal systematic gaps between the stated alignment goals and measured behavior. While these frameworks provide valuable snapshots of model trustworthiness at a single point in time, TrustNLP’s longitudinal proceedings capture how trust challenges emerge, intensify, and transform in response to capability shifts, providing a complementary, diachronic perspective that static benchmarks cannot offer.

To study how research topics have evolved across six editions, we classify all 144 archived proceedings papers by trust dimension. We derive our trust taxonomy from TrustLLM, which benchmarks six dimensions of LLM trustworthiness: Truthfulness, Safety, Fairness, Robustness, Privacy, and Machine Ethics. We adopt these as our starting point and refine them using DecodingTrust’s finer-grained distinctions to clarify scope boundaries. DecodingTrust’s Toxicity is folded into Fairness & Bias to merge commonalities where toxic content may disproportionately target particular marginalized groups. DecodingTrust’s three robustness sub-dimensions inform the scope of our Robustness category. We argue that TrustLLM’s Safety and Machine Ethics are methodologically intertwined and merge them into a single Machine Ethics & Safety dimension. In this way, we expand the scope of each class by merging overlapping class labels from the two taxonomies. Finally, neither TrustLLM nor DecodingTrust benchmarks explainability (TrustLLM proposes “Transparency” as a principle but does not operationalize it). However, explainability dominates the TrustNLP corpus in 2021–2022 and appears explicitly in the workshop CfP, so we add it as a corpus-motivated dimension. This yields six trust dimensions: Fairness & Bias, Robustness & Adversarial, Privacy, Machine Ethics & Safety, Truthfulness, and Explainability. Papers may belong to multiple dimensions.

We used three independent annotation sources to classify all 144 papers. First, we defined annotation instructions specifying the scope of each trust dimension. Labels were then obtained from three sources: (i) one author annotated all papers based on their title and abstract following the instructions; (ii) Claude Sonnet 5 (Anthropic) and Amazon Nova Lite 2.0 each independently annotated the same papers using the same instructions as a system prompt. After the first round, 60% of papers (87/144) had perfect agreement across all three annotators, and in 78% of cases (112/144) at least one LLM fully agreed with the human. Agreement is substantial to excellent (κ > 0.7) for four of six dimensions. The two most contested boundaries—Machine Ethics & Safety and Truthfulness—reflect a conceptual overlap (e.g., a paper on LLM safety degradation under attacks touches both robustness and safety). The human annotator then reviewed the remaining 40% of cases with imperfect agreement and decided the final label. A similar classification on trust papers (first filtered via keyword search), shows that TrustNLP’s topical distribution generally matches with the broader field. This is indicative that the workshop mirrors the community’s trust priorities.

Several patterns emerge from the trust dimension distribution: Truthfulness is the fastest-growing dimension, absent entirely in 2021–2022 but rising to 13–14 papers per year in 2025–2026 (38 total, 26% of the corpus). This surge tracks the LLM era: as generative models became the dominant paradigm, hallucination, factuality, and calibration emerged as first-order trust concerns. Fairness & Bias is the most stable dimension, present in every edition at 2–7 papers/year (30 total). However, its nature has shifted: early work focused on standalone bias measurement, while recent work reveals fairness–explainability trade-offs, fairness–performance trade-offs, and internal-vs-output bias mismatches. Fairness is transitioning from a standalone objective to a constraint in multi-objective optimization. Robustness & Adversarial was absent in 2021–2022 but stabilized at 5–8 papers/year from 2023 onward (27 total), driven by jailbreaks, red-teaming, and multimodal attacks. The nature of adversarial work has shifted: pre-LLM work focused on perturbing inputs, while post-LLM work involves co-constructing attacks with the model itself. Machine Ethics & Safety shows steady growth from 0 papers in 2021 to 9 in 2026 (20 total), reflecting the post large LLM (such as ChatGPT) focus around alignment, refusal behavior, and safety guardrails. Explainability peaked early (9 papers across 2021–2022), declined to 2 papers in 2025, but rebounded sharply to 13 in 2026 (29 total). The early peak reflects the pre-LLM focus on understanding model decisions; the 2026 resurgence is driven by mechanistic interpretability—sparse autoencoders, linear probes, and activation steering—representing a shift from post-hoc explanations to causal understanding of model internals. Privacy appears sporadically (concentrated in 2023 with 6 papers), suggesting that privacy remains a specialized concern within the TrustNLP community despite its prominence in the broader trustworthiness literature.

The inaugural workshops focused on fairness, explainability, and privacy. As large pre-trained models became the predominant paradigm around this time, we observed a shift toward research focused on measuring representational harms in generative models. During this period, NLP tasks were predominantly classification and ranking systems, and trust was fundamentally about transparency of decisions. Two complementary research threads defined this phase: establishing interpretability as an auditing requirement beyond simple accuracy reporting, and exposing how demographic and cross-lingual biases are structurally embedded in popular corpora and model representations. Research established interpretability as a test for auditing decisions in consequential domains such as healthcare and legal systems. Early work explored jointly bootstrapping neural relation extractors with explanation decoders to ensure that outputs were accompanied by human-readable rationales. The challenge of accountable error characterization was central: researchers argued that simply reporting accuracy is insufficient for trust; models must quantify their own uncertainty. This led to attention-based attribution frameworks that identify features responsible for both performance and fairness through intervention and weight manipulation. These attribution frameworks provided a mechanism for auditing not just what a model predicts, but why, a distinction that would become increasingly important as models grew more opaque. The emphasis on uncertainty quantification in these early papers anticipated the later focus on truthfulness & confidence calibration that would dominate the 2025 edition. A notable contribution in 2021 was a cross-linguistic study of gender bias in Wikipedia corpora across nine languages, including Arabic, German, Farsi, and Urdu. This work showed that gender bias in NLP corpora is not solely an English-language concern and that bias metrics developed for English require careful adaptation when applied across typologically diverse languages. Related work on word embeddings further demonstrated that measured bias can vary substantially depending on the choice of similarity measure and descriptive statistic, underscoring the need for careful methodological choices when evaluating demographic bias in embedding spaces. In 2022, researchers probed the Allen AI Delphi model to investigate moral reasoning. They found that models mirror the moral principles of the demographic groups involved in the annotation process, raising fundamental questions about whose values are codified in “moral” AI systems. The Encoder Marginalization framework was also introduced for dense passage retrieval, quantifying the contribution of different encoders to in-domain accuracy and providing a mechanism for identifying affecting factors during training. This early work on encoder analysis foreshadowed the mechanistic interpretability research that would emerge in 2026.

The 2023 proceedings mark the workshop’s first major expansion after the release of large LMs like ChatGPT, growing from 8 papers in 2022 to 28 in 2023. The capability emergence activated research across several trust dimensions simultaneously. While fairness remained steady (6 papers), robustness, privacy, and truthfulness all saw first-time emergence—each contributing 6 papers, making 2023 the most evenly diversified edition. With generative models deployed at scale, truthfulness emerged as a first-order concern. Work on abstractive summarization addressed the familiar problem that generated summaries may be unsupported by, or inconsistent with, the source document, placing factuality at the intersection of model training and evaluation. Other papers asked how reliability should be assessed when model behavior depends on prompt wording, when benchmark data may have appeared in training data, and when confidence must be inferred from patterns of agreement across prompts. Together, these papers show the workshop moving from general concerns about unreliable generation toward prompt-robust and contamination-aware evaluation. Robustness went from zero papers in 2021–2022 to six in 2023: work on sample attackability and minority-language robustness treated trust as resistance to perturbation and attack, including for languages and scripts outside the usual English-centered evaluation setting. Privacy similarly surged, with papers examining personal information in training corpora and pseudonymization as a way to preserve data utility while reducing exposure risk. These papers broadened the generative-model discussion by emphasizing the data, associated privacy threat models, and deployment conditions around the system, not only the surface quality of generated text. The workshop’s earlier fairness concerns extended into the generative era. Papers on abuse detection, disability bias, representational-harm metrics, and name-based effects examined how models can treat social groups unevenly and how such behavior can be measured or diagnosed. Fairness maintained its steady presence (6 papers), but the methods shifted from measuring pre-trained embeddings to auditing generative outputs. The overall lesson of 2023 is that capability emergence activated multiple trust concerns in parallel rather than concentrating attention on any single dimension.

With rapid improvements in model quality and progress in open models, the 2024 edition shifted from studying trust dimensions in isolation to understanding how they interact under deployment constraints. The diverse focus from 2023 continued: Fairness (6 papers), Robustness (5), and Truthfulness (5) remained nearly balanced, though Privacy declined from its 2023 peak to zero papers. One study compared shallow learning, LM fine-tuning, and multilingual fine-tuning for detecting machine-generated text. Another showed that detection scores are not determined only by the generated text: prompts and real texts can affect them too. Using causal diagrams, the authors identified backdoor paths and showed that some confounding bias can be partly reduced. This suggests that future detectors should account for prompt and context effects, not only classify outputs in isolation. Safety work became more proactive. Automated Adversarial Discovery framed red-teaming as a search for attacks that both fool a safety classifier and belong to previously unseen harm dimensions. In parallel, Cross-Task Defense showed that instruction tuning with refusal examples can help LLMs handle malicious long documents, such as manuals for illicit activities, while still performing benign NLP tasks. The defining contribution of 2024 was the recognition that trust dimensions conflict. Adapter modules kept text classification accuracy close to full fine-tuning while reducing training time, but their fairness effects varied by sensitive group and could amplify existing bias in some settings. Fairness did not move cleanly with explainability: methods that improved one dimensio…

Based on the paper, here are specific improvements for AI systems:

1. Internal-state trust probes as first-class evaluation components

• Add linear probes, sparse autoencoders, and activation-level monitors to standard evaluation pipelines, not just output-level tests.

• Improved system can detect latent biases, hidden knowledge remnants after unlearning, and misalignment that behavioral tests miss—e.g., flagging a model that passes safety benchmarks but harbors adversarial capabilities in its hidden states.

2. Mechanistic interpretability for causal verification of alignment

• Use activation steering, concept-tracking probes, and single-layer intervention analysis to verify why a model refuses or complies, rather than only that it does.

• Improved system can localize the safety axis in representation space, detect over-refusal caused by spurious linguistic cues, and identify domain-conditional safety (e.g., 15% compliance for human trafficking vs. 86% for surveillance) before deployment.

3. Dynamic, adversarial benchmarks that co-evolve with model capabilities

• Replace static test sets with adversarially-maintained benchmarks that update as models improve, and include multi-turn, agentic, and multimodal interaction modes.

• Improved system can resist benchmark saturation and contamination, and can evaluate trust under realistic deployment conditions (e.g., database querying, tool use, long-context reasoning) rather than single-turn prompts.

4. Unified trust framework with explicit trade-off modeling

• Implement a multi-objective optimization layer that jointly optimizes fairness, robustness, truthfulness, privacy, and explainability, with explicit cost functions for conflicts (e.g., adapters improving efficiency but harming fairness).

• Improved system can surface trade-offs to users (e.g., this model is 10% more accurate but 5% less fair on group X) and automatically adjust behavior based on deployment context.

5. Truthfulness mechanisms targeting root causes, not symptoms

• Add ghost-context detection (misattributed grounding), quantization-aware knowledge preservation, and prospective memory safeguards under cognitive load.

• Improved system can identify where factual degradation occurs in its architecture (e.g., which layers lose knowledge after quantization) and self-correct by routing around damaged components or re-anchoring to correct evidence.

6. Proactive controllability for agentic and multimodal systems

• Build in mechanistic safety constraints (e.g., refusal geometry manipulation, cross-modal attack detection like PBI-Attack) that are verifiable via internal probes, not just output filters.

• Improved system can constrain its own behavior under adversarial conditions—e.g., resisting jailbreaks that combine image and text perturbations, or refusing to act on malicious long documents—while maintaining benign task performance.

7. Cross-lingual and cross-modal trust generalization

• Extend fairness and safety evaluation to typologically diverse languages (e.g., Arabic, Farsi, Urdu) and multimodal inputs, with bias metrics adapted per language rather than translated from English.

• Improved system can detect and mitigate representational harms in non-English corpora and multimodal settings, avoiding the anglocentric blind spot identified in the paper.

8. Confidence calibration with ambiguity detection

• Use disagreement across candidate answers to detect ambiguous queries before responding, and calibrate confidence only on answerable cases.

• Improved system can avoid overconfident wrong answers on unclear questions, and can signal when it lacks sufficient evidence—critical for high-stakes domains like healthcare or legal advice.

The Workshop on Trustworthy Natural Language Processing (TrustNLP), co-located with major ACL conferences since 2021, has grown from 8 proceedings papers to 41 over six editions, documenting a field-wide transition from post-hoc interpretability of static models to mechanistic understanding and proactive control of generative systems. We synthesize insights from all 144 proceedings papers, classifying them along six trust dimensions grounded in established frameworks (TrustLLM, DecodingTrust). We observe co-occurrences with capability emergence. The release of the first high-impact chat models activated all trust dimensions simultaneously, while subsequent model generations shifted focus toward truthfulness and safety alignment. Analysis from the classification study reveals that truthfulness is the fastest-growing dimension (absent in 2021-2022, comprising 37% of papers by 2025-2026), fairness remains the most consistent theme, and explainability exhibits a U-shaped trajectory; declining as post-hoc methods lost relevance but resurging in 2026 through mechanistic interpretability. A cross-venue comparison with ACL, NAACL, EACL, and EMNLP (2K papers) in the same period shows that TrustNLP's topical distribution closely follows the field average. We identify four structural insights and conclude with actionable directions for the research community.