Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions
summary
The gist
Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors…
In short
The study tracked ICLR submissions with ethics flags into resubmissions to see if authors changed their research based on ethical concerns. Findings show that most authors leave flagged issues unaddressed or make only minor rhetorical changes, not substantive methodological shifts. This suggests ethics review currently shapes how stories are presented rather than redirecting the core research agenda.
Key concepts
- Longitudinal Tracking of Ethics Flags
- Researchers followed flagged submissions through subsequent public resubmissions over time. The goal was to determine if authors actually changed their methods or procedures after an initial ethics review, revealing the long-term effect of these flags on the research process.
- Substantive vs. Rhetorical Changes
- The study distinguished between changes that alter the actual technical work (substantive changes) and those that only change how the paper is written (rhetorical revisions). Only a small percentage of resubmissions showed substantive changes addressing previous ethical concerns.
- Editorial Process vs. Research Direction
- Authors often view ethics review as an 'editorial process' that shapes which research gets published, rather than a genuine reason to change the underlying research topic. Revisions are frequently made to fit publication standards rather than fundamentally altering the study's focus.
Terminology used across episodes
This episode discusses
- Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions · Paper Radio
- It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process
- OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
The paper
Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions · Read on arXiv
Massachusetts Institute of Technology · Harvard University
Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Governing AI Research Through Peer Review".
Jane: Here's the gist of the paper. Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas Page 1.
Tom: Longitudinal Tracking of Ethics Flags The study tracked flagged ICLR submissions into subsequent public resubmissions to see if authors changed their research after the original review ended Page 2.
Jane: They found that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures Page 2.
Tom: Specifically, only 17% of resubmissions exhibited procedural or methodological changes addressing a previous concern Page 3.
Title and authors: Jane: Okay, so we know authors are revising framing instead of redirecting agendas. The paper details how they did this by tracking flagged ICLR submissions into subsequent public resubmissions to see if the research changed after the original review ended. They found that only seventeen percent of resubmissions actually showed procedural or methodological changes addressing a previous concern, even though authors were revising their papers quite often.
Lu: That comparison is crucial because they weren't just looking at whether authors revised anything; they were specifically matching the change to the original ethical concern and classifying the resubmission into five categories based on what shifted. It showed that most changes were rhetorical, not technical adjustments to methods or procedures.
Meng: If only a small percentage of changes actually involved altering how they built their systems—like changing data collection protocols or modifying an algorithm—then the conference flags aren't effectively forcing the necessary engineering fixes.
Tom: I saw a breakdown in Table one showing that out of four hundred forty-six resubmissions, there were only fifty-seven that showed changes primarily in methods, and fourteen that showed changes primarily in procedures. That’s quite low when you think about the scale of AI projects we're dealing with.
Jane: Exactly. The point is that they can identify what changed between versions, but version differences alone don't prove steering occurred; you need to see a necessary consequence like a change to the methods or procedures linked to the ethical issue.
Lalam: And this leads into what they found about author intent during rebuttals. They observed that authors often respond quite thoroughly to ethics criticism during the rebuttal period, making concessions, but then they drop those concessions when rejection happens and the original reviewers don't have leverage anymore.
Lu: That pattern of making changes only to pull them back when facing final rejection tells a story about how the review process acts more like an editorial filter than a scientific steering mechanism for authors.
Tom: So, they are showing that while authors might be nice and make some tweaks during the discussion period, those tweaks don't stick if they aren't backed by a commitment to change the actual science later on.
The paper's summary: Jane: The authors propose a couple of key changes for governance. First, they strongly recommend that conferences disclose any prior ethics flags when a paper is resubmitted. They argue this creates accountability for unresolved concerns that carry over to different venues or future work.
Lu: That sounds like forcing the accountability mechanism to persist, which is smart because right now, the rejection resets the clock on that specific concern. They want to make sure authors can’t just move on and ignore a flagged issue elsewhere.
Meng: I think making that disclosure mandatory would force authors to be more rigorous about what they are actually submitting, especially if they know their past work has ethical baggage attached to it. It adds friction to the submission process.
Tom: They also suggest requiring early-stage ethics review as a prerequisite before a full paper is even submitted. That way, you catch issues in the data collection and experimental design when those things are still flexible enough to be adjusted, instead of waiting until everything is finalized.
Jane: That addresses the issue that reviewers often get stuck because by the time they review the final paper, authors have usually already chosen their problem and completed most of their experiments. Getting in earlier allows for a different kind of intervention.
Lalam: I also noticed a suggestion about reviewer credibility. They mentioned that better domain matching could make reviewers’ criticism more credible, which is something we need to think about when trying to get meaningful feedback on complex AI systems.
Lu: That ties into the idea that sometimes reviewers focus too narrowly on one aspect—like just the math or just the presentation—instead of seeing the bigger picture of how a model impacts its entire environment.
Tom: It sounds like they are pushing for a system where we get earlier, more targeted feedback, and authors have to be transparent about past concerns when they move their work around.
The paper's improvements: Jane: So wrapping up the paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions," the authors conclude that substantial revisions to a manuscript do not automatically mean that ethics review redirected the underlying research. The key implication is that conferences should review earlier, match reviews by domain, and most importantly, require disclosure of prior flags upon resubmission so rejection doesn't reset accountability for an unresolved ethical concern when the project moves elsewhere.
Tom: It’s a clear message: the current setup isn't working to steer research effectively because authors are revising their framing instead of changing what they actually study or build. The whole point is getting that disclosure requirement on resubmissions to ensure we track responsibility properly.
Lu: I think the longitudinal tracking really hammers home that simply having an ethics review process isn't enough; you need a policy change that connects those flags across different stages of the research lifecycle. It’s about making the system durable rather than just reactive during a single submission cycle.
Meng: For us shipping models, this means we need to think about how we document and track ethical concerns more rigorously so that if we have to pivot our architecture later on, that initial flag doesn't just disappear when we move to a new platform or model version.
Lalam: I agree with the focus on accountability carrying over; it’s about ensuring that when research moves from one venue to another, the ethical commitment stays with the work itself, not just the original paper submission.
Jane: We've spent this time looking at how authors react and what they suggest for governance in "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." It’s a call for a more integrated system where ethics review is part of the whole research journey, not just a checkpoint before publication.
Tom: Yeah, so we learned that paper isn't just about whether authors change their methods; it's about how we structure the review process to ensure those changes actually happen and stick. What’s next on our list?
Conclusion: Tom: So we’re wrapping up our look at "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." The big takeaway is that just revising your paper isn't enough to steer research toward better practices.
Jane: Exactly. They tracked flagged submissions and found that even when authors revise their work, only about seventeen percent actually change the methods or procedures addressing the original ethical concern. It shows how often authors just change the writing, not the actual science they did.
Lu: The longitudinal tracking is really telling there's a huge gap between what reviewers flag and what authors actually implement in their revisions. They found most changes were rhetorical, not technical adjustments to the model or data collection setup.
Meng: That’s tough for production environments. If you get a red flag about data privacy or bias, and the author just tweaks the text but doesn't change how they handle that data at scale, that's a problem for us.
Lalam: I see what they mean from my side. The authors interviewed researchers and found that authors often make concessions during rebuttal to keep their job in the review process, but then drop those concessions once rejection hits and there’s no more leverage left.
Tom: So the paper recommends forcing authors to disclose those prior ethics flags on resubmissions so accountability doesn't just vanish when the project moves venues. That sounds like a necessary policy shift for how we judge submissions going forward.
Jane: They also suggest requiring early-stage ethics review before a full submission because by the time you’re done, your experimental design is locked in and changing it becomes much harder. It’s about catching those design flaws earlier when they're cheaper to fix.
Lu: I think that idea of disclosure forcing accountability is key. If authors know their past work has an unresolved ethical issue attached to it, they have a much stronger incentive to address it now instead of ignoring it for the next project.
Meng: I agree with the early review part too because if we’re shipping something, we need that initial scrutiny on how we collect and process things before we commit to the whole pipeline.
Lalam: I’m confident in this direction. The paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions" shows us where the current system is falling short on long-term steering.
Tom: It does, and it points toward needing a structural fix in how we handle ethical feedback across different stages of research. Next up, we’re looking at RouteBalance for AI serving models—how they manage those heterogeneous stacks without breaking the schedule.
More episodes
- 2403.18642-Collective schedules: axioms and algorithms
- 2411.16152-Optimizing Winograd Convolution on ARMv8 processors
- 2505.16254-Human-AI Writing as Distributed Cognition: Reassessing Collaborative Writing Theories in the Age of LLMs
- 2506.10913-Choreographic Quick Changes: First-Class Location (Set) Polymorphism
- 2507.14812-A Black-Box Approach for Exogenous Replenishment in Online Resource Allocation
- 2507.15448-Frames over finite fields with respect to the Galois inner product and their connections to self-orthogonal and LCD codes
- 2508.21572-NewsReX: An Open-Source Multi-Framework for Neural News Recommendation
- 2509.17028-Impact of Packetization on Network Calculus Analysis
- 2510.17342-AoA Services in 5G Networks: A Framework for Real-World Implementation and Systematic Testing
- 2511.04088-Efficient and rate-optimal list-decoding in the presence of minimal feedback