Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025

The paper Is the ACL Responsible NLP Checklist a Box-Ticking Exercise? A Large-Scale Analysis of EMNLP 2025 by Nusrath Jinnath and Wei Zhao from the University of Aberdeen evaluates the efficacy of the ARR Responsible NLP Research Checklist by analyzing 73,922 responses and justifications from 3,214 EMNLP 2025 papers (1,809 Main track and 1,405 Findings track). The authors curate and release the first two datasets: (a) all checklist responses and justifications from EMNLP 2025 Main and Findings tracks, and (b) checklist references linking to paper sections. Their pipeline uses Marker for PDF conversion, GPT 5.4-mini with constrained prompting for extraction, achieving 99.9% item-level accuracy on a stratified sample of 71 papers and a 90.9% section-text retrieval rate.

The main findings are: (i) 2 out of 3 best practices for responsible research are not being addressed consistently – specifically, reproducibility questions (Parts B and C) have higher compliance than ethics and societal impact questions (A2 and Part D), with Part D being over 3.5x less compliant than the rest. (ii) Over half of authors do not consider potential risks of their work – 53% of authors dismiss potential risks or social impacts, with A2 (risks) having the highest empty justification rate in the corpus at 17.1%. (iii) Nearly half of authors do not adequately justify why they answered 'NO' – 44.9% of NO justifications are poor or bad-faith, being brief (1-10 words) or empty (0 words), with 721 brief and 213 empty cases. (iv) Many authors carelessly fill out checklists, with a significant amount of logical errors – 281 papers (15.5%) in the Main track contain at least one logical contradiction where a parent question is NO but a child question is YES, with Section B dominating at 9.5% inconsistent rate; the Findings track has an even higher overall inconsistency rate of 18.1%.

Additional findings include: citation concentration scores show most checklists have a median of 2.25 sections cited per question, with 132 checklists labeled problematic (Score > 4), including one with over 40 references pointing to fewer than 5 paper sections. The co-citation matrix reveals that B1 (citing artifact creators) and C2 (experimental setup) often reference the same paper sections, indicating the checklist has become a lazy exercise of section pointers. B4 (data ethics) has the least pairings with other items and the highest N/A rate, suggesting researchers either do not handle private data or did not verify ethical compliance. E1 (AI assistance) has a significantly smaller empty rate (3.1%) compared to A2, with authors more likely to justify their use of AI than critically consider potential risks of their systems.

The Findings track comparison shows similar trends: findings submissions have less overall compliance (more NO and N/A responses), but the percentage difference is only about 2%, raising questions about the checklist's ability to distinguish paper quality. Both tracks have similar citation concentration medians (2 when rounded) and similar justification failure rates (44.6% for findings vs 44.9% for main), with A2's adequacy rating being 7% lower in findings.

The authors conclude that responsible research practice is not being promoted adequately and propose three immediate recommendations: (i) When declining a parent component, all child questions should be disabled to prevent logical contradictions; (ii) Every justification for non compliance must not be empty; (iii) A2 in particular should be given additional consideration by reviewers, beyond current standards – if risks are not discussed in the main content, reviewers should apply higher scrutiny to checklist answers. The paper also discusses implications for checklist design, noting that the current form provides no automated feedback on referencing patterns or prompts authors to review answers before submission. Limitations acknowledged include the focus on post-submission checklists (which may differ from camera-ready versions), the reliance on word count for justification quality analysis, and the need for future work on YES justification quality and manual validation of referenced sections.

Improvement 1: Constraint-Aware Checklist Validation System

• What it does: Automatically detects and blocks logical contradictions in real-time during checklist submission. If a parent question (e.g., B1: Did you cite creators?) is answered NO, the system disables all child questions (e.g., B1a, B1b) and prompts the user to either change the parent answer or provide a mandatory justification for the "NO" before proceeding.

• Specific capability: Prevents the 15.5% (Main) and 18.1% (Findings) inconsistency rates by enforcing hierarchical answer validity, reducing bad-faith submissions.

Improvement 2: Justification Quality Gate with Semantic Minimums

• What it does: Uses a fine-tuned language model (e.g., GPT-5.4-mini) to evaluate every "NO justification against two criteria: (a) non-empty (≥10 words) and (b) contextually relevant to the specific checklist item. If a justification is too brief, generic (e.g., N/A"), or off-topic, the system flags it and requires revision before submission.

• Specific capability: Reduces the 44.9% poor/bad-faith justifications and the 17.1% empty rate for A2 (risks) by forcing authors to articulate concrete reasoning, improving accountability.

Improvement 3: Risk-Impact Scrutiny Assistant for Reviewers

• What it does: Automatically cross-references an author's A2 (potential risks) and Part D (societal impact) answers against the full paper text. If the author claims no risks but the paper describes deployment scenarios, user data, or dual-use capabilities, the system generates a reviewer alert with specific sentences as evidence.

• Specific capability: Addresses the 53% of authors who dismiss risks by surfacing contradictions between checklist answers and paper content, enabling reviewers to apply higher scrutiny without manual reading.

Improvement 4: Section-Reference Coherence Checker

• What it does: For each checklist item, verifies that cited paper sections actually contain relevant content (e.g., B1 citations point to a Related Work or Methods section that mentions the artifact). Uses a retrieval model (90.9% accuracy) to map references and flag lazy patterns where multiple items cite the same section (e.g., B1 and C2 both pointing to Experimental Setup).

• Specific capability: Detects the 132 problematic checklists with high citation concentration scores and discourages box-ticking by requiring distinct, meaningful section links per item.

Improvement 5: Track-Aware Quality Differential Analyzer

• What it does: Compares checklist compliance patterns between Main and Findings tracks in real-time, identifying if a submission's low compliance (more NO/N/A) is due to genuine scope differences or carelessness. The system can adjust acceptance criteria or reviewer instructions based on track-specific baselines.

• Specific capability: Uses the observed 2% compliance gap to calibrate expectations, preventing unfair penalization of Findings papers while still flagging anomalous drops (e.g., A2 adequacy 7% lower) for targeted review.

Improved AI System Output:

An integrated submission portal that:

1. Blocks contradictory answers and disables child items dynamically.

2. Requires non-empty, context-specific justifications for all "NO" responses.

3. Alerts authors to unaddressed risks by scanning paper text.

4. Validates that each cited section genuinely supports the checklist claim.

5. Provides track-specific compliance dashboards to authors and reviewers, highlighting anomalies before final decision.

Responsible NLP practice includes a) transparency, b) ethics, and c) societal impacts. The Responsible NLP Checklist aims to push these goals, and promote responsible practice. Recently, ACL released the EMNLP 2025 Checklists to aid transparency on the current research practice, which we focus on. We curate and release the first two datasets of: a) all the checklist responses and justifications from the EMNLP 2025 Main and Finding tracks; b) checklist reference linking to paper sections. We also provide the first analysis of recent EMNLP Checklists, by examining 73,922 responses and justifications to them. For the Main track, we find that authors isolate ethics questions of the Checklist from the paper's bulk, mimicking the trend of ethics being an afterthought. We then examine NO responses. We find 44.9% of justifications are poor or bad-faith, being brief or empty. Then, we find significant issues with the checklist design and effort of authors, namely that 6% of all checklists contained logical contradictions between parent and child responses. We also find evidence of surface compliance for responsible ethics, with 53% authors dismissing potential risks or social impacts of their work, for which there should be none. We compare this to the Findings track, noticing a similar trend in both tracks. Lastly, we discuss the implications of the checklist design and provide recommendations for future checklist iterations. Including: a) enforcing a minimum word count, b) enforcing more scrutiny on the risks of appliances.