Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions
Listen
Radio episode about this paper
Transcript
Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.
Tom: We're looking at the paper: "Governing AI Research Through Peer Review".
Jane: Here's the gist of the paper. Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas Page 1.
Tom: Longitudinal Tracking of Ethics Flags The study tracked flagged ICLR submissions into subsequent public resubmissions to see if authors changed their research after the original review ended Page 2.
Jane: They found that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures Page 2.
Tom: Specifically, only 17% of resubmissions exhibited procedural or methodological changes addressing a previous concern Page 3.
Title and authors: Jane: Okay, so we know authors are revising framing instead of redirecting agendas. The paper details how they did this by tracking flagged ICLR submissions into subsequent public resubmissions to see if the research changed after the original review ended. They found that only seventeen percent of resubmissions actually showed procedural or methodological changes addressing a previous concern, even though authors were revising their papers quite often.
Lu: That comparison is crucial because they weren't just looking at whether authors revised anything; they were specifically matching the change to the original ethical concern and classifying the resubmission into five categories based on what shifted. It showed that most changes were rhetorical, not technical adjustments to methods or procedures.
Meng: If only a small percentage of changes actually involved altering how they built their systems—like changing data collection protocols or modifying an algorithm—then the conference flags aren't effectively forcing the necessary engineering fixes.
Tom: I saw a breakdown in Table one showing that out of four hundred forty-six resubmissions, there were only fifty-seven that showed changes primarily in methods, and fourteen that showed changes primarily in procedures. That’s quite low when you think about the scale of AI projects we're dealing with.
Jane: Exactly. The point is that they can identify what changed between versions, but version differences alone don't prove steering occurred; you need to see a necessary consequence like a change to the methods or procedures linked to the ethical issue.
Lalam: And this leads into what they found about author intent during rebuttals. They observed that authors often respond quite thoroughly to ethics criticism during the rebuttal period, making concessions, but then they drop those concessions when rejection happens and the original reviewers don't have leverage anymore.
Lu: That pattern of making changes only to pull them back when facing final rejection tells a story about how the review process acts more like an editorial filter than a scientific steering mechanism for authors.
Tom: So, they are showing that while authors might be nice and make some tweaks during the discussion period, those tweaks don't stick if they aren't backed by a commitment to change the actual science later on.
The paper's summary: Jane: The authors propose a couple of key changes for governance. First, they strongly recommend that conferences disclose any prior ethics flags when a paper is resubmitted. They argue this creates accountability for unresolved concerns that carry over to different venues or future work.
Lu: That sounds like forcing the accountability mechanism to persist, which is smart because right now, the rejection resets the clock on that specific concern. They want to make sure authors can’t just move on and ignore a flagged issue elsewhere.
Meng: I think making that disclosure mandatory would force authors to be more rigorous about what they are actually submitting, especially if they know their past work has ethical baggage attached to it. It adds friction to the submission process.
Tom: They also suggest requiring early-stage ethics review as a prerequisite before a full paper is even submitted. That way, you catch issues in the data collection and experimental design when those things are still flexible enough to be adjusted, instead of waiting until everything is finalized.
Jane: That addresses the issue that reviewers often get stuck because by the time they review the final paper, authors have usually already chosen their problem and completed most of their experiments. Getting in earlier allows for a different kind of intervention.
Lalam: I also noticed a suggestion about reviewer credibility. They mentioned that better domain matching could make reviewers’ criticism more credible, which is something we need to think about when trying to get meaningful feedback on complex AI systems.
Lu: That ties into the idea that sometimes reviewers focus too narrowly on one aspect—like just the math or just the presentation—instead of seeing the bigger picture of how a model impacts its entire environment.
Tom: It sounds like they are pushing for a system where we get earlier, more targeted feedback, and authors have to be transparent about past concerns when they move their work around.
The paper's improvements: Jane: So wrapping up the paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions," the authors conclude that substantial revisions to a manuscript do not automatically mean that ethics review redirected the underlying research. The key implication is that conferences should review earlier, match reviews by domain, and most importantly, require disclosure of prior flags upon resubmission so rejection doesn't reset accountability for an unresolved ethical concern when the project moves elsewhere.
Tom: It’s a clear message: the current setup isn't working to steer research effectively because authors are revising their framing instead of changing what they actually study or build. The whole point is getting that disclosure requirement on resubmissions to ensure we track responsibility properly.
Lu: I think the longitudinal tracking really hammers home that simply having an ethics review process isn't enough; you need a policy change that connects those flags across different stages of the research lifecycle. It’s about making the system durable rather than just reactive during a single submission cycle.
Meng: For us shipping models, this means we need to think about how we document and track ethical concerns more rigorously so that if we have to pivot our architecture later on, that initial flag doesn't just disappear when we move to a new platform or model version.
Lalam: I agree with the focus on accountability carrying over; it’s about ensuring that when research moves from one venue to another, the ethical commitment stays with the work itself, not just the original paper submission.
Jane: We've spent this time looking at how authors react and what they suggest for governance in "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." It’s a call for a more integrated system where ethics review is part of the whole research journey, not just a checkpoint before publication.
Tom: Yeah, so we learned that paper isn't just about whether authors change their methods; it's about how we structure the review process to ensure those changes actually happen and stick. What’s next on our list?
Conclusion: Tom: So we’re wrapping up our look at "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." The big takeaway is that just revising your paper isn't enough to steer research toward better practices.
Jane: Exactly. They tracked flagged submissions and found that even when authors revise their work, only about seventeen percent actually change the methods or procedures addressing the original ethical concern. It shows how often authors just change the writing, not the actual science they did.
Lu: The longitudinal tracking is really telling there's a huge gap between what reviewers flag and what authors actually implement in their revisions. They found most changes were rhetorical, not technical adjustments to the model or data collection setup.
Meng: That’s tough for production environments. If you get a red flag about data privacy or bias, and the author just tweaks the text but doesn't change how they handle that data at scale, that's a problem for us.
Lalam: I see what they mean from my side. The authors interviewed researchers and found that authors often make concessions during rebuttal to keep their job in the review process, but then drop those concessions once rejection hits and there’s no more leverage left.
Tom: So the paper recommends forcing authors to disclose those prior ethics flags on resubmissions so accountability doesn't just vanish when the project moves venues. That sounds like a necessary policy shift for how we judge submissions going forward.
Jane: They also suggest requiring early-stage ethics review before a full submission because by the time you’re done, your experimental design is locked in and changing it becomes much harder. It’s about catching those design flaws earlier when they're cheaper to fix.
Lu: I think that idea of disclosure forcing accountability is key. If authors know their past work has an unresolved ethical issue attached to it, they have a much stronger incentive to address it now instead of ignoring it for the next project.
Meng: I agree with the early review part too because if we’re shipping something, we need that initial scrutiny on how we collect and process things before we commit to the whole pipeline.
Lalam: I’m confident in this direction. The paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions" shows us where the current system is falling short on long-term steering.
Tom: It does, and it points toward needing a structural fix in how we handle ethical feedback across different stages of research. Next up, we’re looking at RouteBalance for AI serving models—how they manage those heterogeneous stacks without breaking the schedule.
Massachusetts Institute of Technology · Harvard University
cs.CY
Submitted: 2026-09-09
Updated: 2026-11-02
Comments: 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: AI-Native Academia
Project page: https://iclr.cc/Conferences/2026/AuthorGuide
License: http://creativecommons.org/licenses/by/4.0/
Importance score: 82/100
The gist: Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors…
Key concepts
- Longitudinal Tracking of Ethics Flags
- Researchers followed flagged submissions through subsequent public resubmissions over time. The goal was to determine if authors actually changed their methods or procedures after an initial ethics review, revealing the long-term effect of these flags on the research process.
- Substantive vs. Rhetorical Changes
- The study distinguished between changes that alter the actual technical work (substantive changes) and those that only change how the paper is written (rhetorical revisions). Only a small percentage of resubmissions showed substantive changes addressing previous ethical concerns.
- Editorial Process vs. Research Direction
- Authors often view ethics review as an 'editorial process' that shapes which research gets published, rather than a genuine reason to change the underlying research topic. Revisions are frequently made to fit publication standards rather than fundamentally altering the study's focus.
Terminology
Summary
Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas Page 1. p. 0 edu
Abstract
Selective AI conferences have recently begun enforcing ethics review processes,
with the goal of steering research towards safer and more responsible practices
before publication. But do ethics flags actually steer research as intended?Where the paper says it
Longitudinal Tracking of Ethics Flags
The study tracked flagged ICLR submissions into subsequent public resubmissions to see if authors changed their research after the original review ended Page 2. They found that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures Page 2. p. 0 We first
study the longitudinal effects of ethics flags, tracing flagged ICLR (International
Conference on Learning Representations) submissions that were rejected or withdrawn into subsequent public resubmissions and tracking manuscript changes made
after the review process ends when the original reviewers no longer oversee the
project. We qualitatively classify these resubmissions into five categories based
on what changed after review and find that in 83% of 446 cases, authors leave the
flagged concern unaddressed or revise the paper without changing the implicated
methods or procedures. To answer this question, we follow
rejected and withdrawn ICLR submissions with ethics flags into public resubmissions, compare each
concern with the corresponding methods and procedures, and qualitatively classify each resubmission
into one of five categories. We find that out of 446 resubmissions, only 17% exhibit procedural or
methodological changes addressing a previous concern. We separately record changes to procedures, methods, or
both: procedures include consent, release, access, oversight, or governance; methods include data,
models, evaluations, or experimental design. Rhetorical revisions can improve a paper, but they do
not change the implicated methods or procedures and therefore do not count as research changes.Where the paper says it
Where the paper says it
Where the paper says it
Qualitative Analysis of Author Intent
Manual reading and interviews with authors revealed that while authors frequently revise papers, they rarely cite reviewer-issued ethics flags as genuine reasons to redirect the underlying research Page 4. Authors often make concessions during rebuttal but drop those concessions after rejection when the original reviewers no longer have leverage Page 4. p. 0 To answer this, we manually read reviews and rebuttals from 25 cases and interview authors about their
rebuttal processes and resubmission decisions. We find that authors often make
concessions during rebuttal while reviewers can still update their scores, but drop
those concessions after rejection, when the original reviewers no longer have leverage. We find that authors often make
concessions during rebuttal while reviewers can still update their scores, but drop
those concessions after rejection, when the original reviewers no longer have leverage. Interview participants describe publication changes as separate from research
direction changes, calling review an “editorial process” that shapes “what stories
get seen” and, in another case, saying peer reviews are “mostly to filter out papers.Where the paper says it
Where the paper says it
Policy Recommendations for Governance
The paper recommends several policy changes, especially disclosure of prior ethics flags upon resubmission so accountability for unresolved concerns carries over Page 1. One suggestion is to require authors to confidentially disclose prior ethics flags on resubmission, including the category and whether they changed the research or disputed the concern Page 5. p. 3 This is exacerbated by our
observation that ethics concerns are often concentrated in one reviewer rather than shared across the
panel. Conferences should therefore require authors to confidentially disclose prior ethics flags on
resubmission, including the category and whether they changed the research or disputed the concern. Feedback should arrive while implicated choices can still
change. By full-paper review, data collection and experiments may already be complete; conferences
could therefore require an early-stage ethics pre-submission as a prerequisite for later full-paper
submission.Where the paper says it
Where the paper says it
Conclusion on Steering Effectiveness
The results strongly suggest that substantial revisions to a manuscript do not necessarily mean that ethics review redirected the underlying research Page 6. The authors conclude that conferences should review earlier, match by domain, and above all require the disclosure of prior flags upon resubmission so rejection does not reset accountability for an unresolved ethics concern when the project moves to another venue Page 6. p. 3 The resubmission types,
review histories, and interviews explain why: concessions can matter immediately while reviewers
still influence the decision; after rejection, the original reviewers lose that leverage, and reviewer
noise may keep the concern from resurfacing. As such, conferences should review earlier, match by
domain, and above all require the disclosure of prior flags upon resubmission so rejection does not
reset accountability for an unresolved ethics concern when the project moves to another venue.Where the paper says it
Limitations of the Study
The study's limitations include focusing only on papers that are resubmissions, potentially missing cases where authors entirely abandoned their projects due to a flag Page 6. The corpus is limited to ICLR submissions, and only three of the twenty-five invited researchers responded to the qualitative interviews Page 3. p. 3 However, given the similar review–rebuttal–decision cycle across top AI
conferences, we expect the mechanisms we identify to extend beyond ICLR, although their prevalence
may differ. Additionally, only three of 25 invited researchers responded; this is reasonable for a
qualitative study, but more research is needed to make claims about broader researcher sentiments.Where the paper says it
Related Work Context
The paper situates its findings within existing literature on ethical reflection in technical research and ethics review at top conferences Page 2. It contrasts its longitudinal analysis with prior work that examined isolated review cycles or broader discussions of harms without considering how ethics reviews shape research Page 3. p. 1 These works examine isolated review cycles or ethical practice more broadly, but do not consider
whether ethics reviews shape research and in what way. 3 Longitudinal Analysis: Do Authors Change Research After Ethics Flags?Where the paper says it
References
The paper cites various works concerning ethical reflection in technical research, ethics review at top conferences, and the operationalization of AI ethics Page 5. Key references include work by Winner on artifacts having politics Page 6, Hecht et al. p. 5 Langdon Winner. Do artifacts have politics? Hullman, and Tina Eliassi-Rad. Examining responsibility and deliberation
in AI impact statements and ethics reviews. Ethics review at top conferences. In 2021, NeurIPS established new “Ethics Guidelines” and issued
several conditional acceptances and one rejection on ethical grounds [Bengio et al.Where the paper says it
Where the paper says it
Where the paper says it
--- Page 1 ---
Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions Kento Nishi∗ Alec Laprevotte† Isaiah Bullock† Mfoniso M. Andrew† Massachusetts Institute of Technology †Harvard University Correspondence: knishi@mit. p. 0 Governing AI Research Through Peer Review:
A Mixed-Methods Study of the Longitudinal
Effects of Ethics Flags Across Resubmissions
Kento Nishi∗ Alec Laprevotte†
Isaiah Bullock† Mfoniso M. Andrew†
∗Massachusetts Institute of Technology †Harvard University
Correspondence: knishi@mit.Where the paper says it
--- Page 2 ---
researchers’ responsibility for foreseeable effects Since major conferences shape which research receives attention and legitimacy they share responsibility for what they accept and promote [Hecht et al., 2021]. In practice, conferences exercise this responsibility through peer review, giving reviewers leverage to steer research towards safer and more responsible directions. p. 1 Since major conferences shape which research
receives attention and legitimacy, they share responsibility for what they accept and promote [Hecht
et al. In practice, conferences exercise this responsibility through peer review, giving reviewers
leverage to steer research towards safer and more responsible directions. In practice, conferences exercise this responsibility through peer review, giving reviewers
leverage to steer research towards safer and more responsible directions. Yet reviewers gain this leverage very late in most research projects. Yet reviewers gain this leverage very late in most research projects. By the time they evaluate a
submission, authors have usually chosen the problem, collected the data, implemented the system,
and completed most experiments. By the time they evaluate a
submission, authors have usually chosen the problem, collected the data, implemented the system,
and completed most experiments. Since reviewers can update scores in response to rebuttals and
author comments during the “discussion period,” authors are naturally incentivized to acknowledge
and address the reviewers’ concerns. Since reviewers can update scores in response to rebuttals and
author comments during the “discussion period,” authors are naturally incentivized to acknowledge
and address the reviewers’ concerns. But once a rejection decision is issued, the original reviewers no
longer influence whether or how the work is disseminated moving forward. But once a rejection decision is issued, the original reviewers no
longer influence whether or how the work is disseminated moving forward. Unfortunately, reviewer
judgments are notoriously noisy, so a new reviewer panel at the next conference may not flag the
same unresolved concern. Unfortunately, reviewer
judgments are notoriously noisy, so a new reviewer panel at the next conference may not flag the
same unresolved concern. This creates room for performative concessions during the discussion
period that are not necessarily carried into revisions for the next venue. This creates room for performative concessions during the discussion
period that are not necessarily carried into revisions for the next venue. Ideally, authors who accept reviewer criticism would implement these recommendations before
submitting their work again. Ideally, authors who accept reviewer criticism would implement these recommendations before
submitting their work again. If such steering occurs, we would expect clear changes across submissions: authors who acknowledge a privacy concern, for example, may remove identifiers, restrict data
access, add an evaluation, or change a release decision. If such steering occurs, we would expect clear changes across submissions: authors who acknowledge a privacy concern, for example, may remove identifiers, restrict data
access, add an evaluation, or change a release decision. How often, then, do authors change their
research in response to ethics flags after the original review ends?Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
Where the paper says it
--- Page 3 ---
To study the longitudinal effects of ethics flags we need rejected or withdrawn projects that authors resubmit after the original review ends Only ICLR publicly releases all submissions and reviews after decisions NeurIPS and ICML release rejected papers only when authors opt in We therefore collect 47,893 ICLR submissions and 185,194 public reviews from OpenReview (2021–2026) We search reviews for recorded instances of the “Flag for ethics review” checkbox identifying 2,498 flagged submissions Our 2021 snapshot lacks public flags so counts begin in 2022 and rise from 107 to 1,100 in 2026 across six ethics categories Of these 1857 were rejected or withdrawn so we search them for resubmissions We cannot simply match titles because authors often change a paper’s title abstract coauthors or scope Accordingly we query OpenAlex with title variants author surnames and salient abstract terms then rank candidates by title similarity author and abstract overlap and publication year We retrieve 6,618 candidates and retain 601 top matches A retrieval score is insufficient to establish project identity so we manually read each pair and retain 446 resubmissions Even after identifying a resubmission a version difference alone does not show steering so we look for a necessary observable consequence changes to the methods or procedures implicated by the ethics concern For each pair we locate the concern in the original review compare the corresponding parts of the submission and resubmission and qualitatively code each resubmission into one of five categories We record no observable change when authors leave the concern unaddressed and mainly rhetorical change when they change limitations discussion of harms or claims without changing the implicated technical work Rhetorical revisions can improve a paper but they do not change the implicated methods or procedures and therefore do not count as research changes
--- Page 4 ---
Change after the flag Count No observable change 217 Mainly rhetorical 153 Procedures + methods 57 Primarily procedures 14 Primarily methods 5 Total labeled 446 Table 1 Resubmissions In Tab. 1 we provide strong evidence that most authors leave flagged methods and procedures unchanged In Tab. p. 2 Change after the flag Count
No observable change 217
Mainly rhetorical 153
Procedures + methods 57
Primarily procedures 14
Primarily methods 5
Total labeled 446
Table 1: Resubmissions. 1, we provide strong evidence that most authors leave flagged
methods and procedures unchanged.Where the paper says it
--- Page 5 ---
First we notice that in the pool of 25 review histories authors often respond at length to ethics criticism during rebuttal including partial concessions absent from later resubmissions Yet in most cases only one of three to five reviewers raises the concern the others discuss technical issues In at least one case the reviewer emphasizing ethics scored the paper 6 while reviewers who did not flag ethics scored it 4–5 Several rebuttals frame the concern as a dispute over disciplinary conventions or required background rather than a valid reason to change the research Despite this resistance on substantive grounds authors do revise their prose for resubmission We call this pattern filtering adapting papers for publication without treating criticism as grounds to redirect research Participant 1 (P1) describes resubmissions as “mostly similar” with revisions focused on “adding another table,” “more metrics” and issues that were “mostly metric based” P1 further adds that peer reviews are “mostly to filter out papers” since many reviewers repeat objections to text the authors have already revised Participant 2 (P2) says reviewers were “very focused on a particular aspect instead of seeing a larger picture.
Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.
Improvements for AI systems
-
Disclose prior ethics flags upon resubmission to ensure accountability for unresolved concerns, as authors often change
what stories get seen
rather thanwhat they study or build.
This requires a policy change that carries accountability across venues. -
Implement early-stage ethics review as a prerequisite for later full-paper submission, allowing scrutiny of data collection and experimental design while choices can still be revised. This addresses the issue where
data collection and experiments may already be complete
by the time of full review. -
Establish domain matching for reviewers to improve feedback credibility, as authors noted that
Better domain matching could make reviewers’ criticism more credible.
This prevents reviews from focusing only on a narrow aspect, addressing P2's observation that reviewers werevery focused on a particular aspect, instead of seeing a larger picture.
Abstract
Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.
Sources
- It's Time to Do Something: Mitigating the Negative Impacts of Computing Through a Change to the Peer Review Process
- OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts
Related papers
- AI-Mediated Communication Can Steer Collective Opinion
- Epistemic Infrastructures of Science in AI Era Should Rebalance Costs of Generation and Verification
- AI out-persuades expert humans in text-based dialogue
- Evaluating Ambient Clinical Scribes in India: The Need for Multilingual Real-World Clinical Conversation Data
- AI Value Alignment for Evolving Social Norms
- Voting Biases in Decentralized Autonomous Organization (DAO) Governance