Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions

arXiv:2609.10740 · cs.CY · Submitted 2026-09-09 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: Computer Science. The day in computer science: software, systems and theory on arXiv.

Tom: We're looking at the paper: "Governing AI Research Through Peer Review".

Jane: Here's the gist of the paper. Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas Page 1.

Tom: Longitudinal Tracking of Ethics Flags The study tracked flagged ICLR submissions into subsequent public resubmissions to see if authors changed their research after the original review ended Page 2.

Jane: They found that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures Page 2.

Tom: Specifically, only 17% of resubmissions exhibited procedural or methodological changes addressing a previous concern Page 3.

Title and authors: Jane: Okay, so we know authors are revising framing instead of redirecting agendas. The paper details how they did this by tracking flagged ICLR submissions into subsequent public resubmissions to see if the research changed after the original review ended. They found that only seventeen percent of resubmissions actually showed procedural or methodological changes addressing a previous concern, even though authors were revising their papers quite often.

Lu: That comparison is crucial because they weren't just looking at whether authors revised anything; they were specifically matching the change to the original ethical concern and classifying the resubmission into five categories based on what shifted. It showed that most changes were rhetorical, not technical adjustments to methods or procedures.

Meng: If only a small percentage of changes actually involved altering how they built their systems—like changing data collection protocols or modifying an algorithm—then the conference flags aren't effectively forcing the necessary engineering fixes.

Tom: I saw a breakdown in Table one showing that out of four hundred forty-six resubmissions, there were only fifty-seven that showed changes primarily in methods, and fourteen that showed changes primarily in procedures. That’s quite low when you think about the scale of AI projects we're dealing with.

Jane: Exactly. The point is that they can identify what changed between versions, but version differences alone don't prove steering occurred; you need to see a necessary consequence like a change to the methods or procedures linked to the ethical issue.

Lalam: And this leads into what they found about author intent during rebuttals. They observed that authors often respond quite thoroughly to ethics criticism during the rebuttal period, making concessions, but then they drop those concessions when rejection happens and the original reviewers don't have leverage anymore.

Lu: That pattern of making changes only to pull them back when facing final rejection tells a story about how the review process acts more like an editorial filter than a scientific steering mechanism for authors.

Tom: So, they are showing that while authors might be nice and make some tweaks during the discussion period, those tweaks don't stick if they aren't backed by a commitment to change the actual science later on.

The paper's summary: Jane: The authors propose a couple of key changes for governance. First, they strongly recommend that conferences disclose any prior ethics flags when a paper is resubmitted. They argue this creates accountability for unresolved concerns that carry over to different venues or future work.

Lu: That sounds like forcing the accountability mechanism to persist, which is smart because right now, the rejection resets the clock on that specific concern. They want to make sure authors can’t just move on and ignore a flagged issue elsewhere.

Meng: I think making that disclosure mandatory would force authors to be more rigorous about what they are actually submitting, especially if they know their past work has ethical baggage attached to it. It adds friction to the submission process.

Tom: They also suggest requiring early-stage ethics review as a prerequisite before a full paper is even submitted. That way, you catch issues in the data collection and experimental design when those things are still flexible enough to be adjusted, instead of waiting until everything is finalized.

Jane: That addresses the issue that reviewers often get stuck because by the time they review the final paper, authors have usually already chosen their problem and completed most of their experiments. Getting in earlier allows for a different kind of intervention.

Lalam: I also noticed a suggestion about reviewer credibility. They mentioned that better domain matching could make reviewers’ criticism more credible, which is something we need to think about when trying to get meaningful feedback on complex AI systems.

Lu: That ties into the idea that sometimes reviewers focus too narrowly on one aspect—like just the math or just the presentation—instead of seeing the bigger picture of how a model impacts its entire environment.

Tom: It sounds like they are pushing for a system where we get earlier, more targeted feedback, and authors have to be transparent about past concerns when they move their work around.

The paper's improvements: Jane: So wrapping up the paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions," the authors conclude that substantial revisions to a manuscript do not automatically mean that ethics review redirected the underlying research. The key implication is that conferences should review earlier, match reviews by domain, and most importantly, require disclosure of prior flags upon resubmission so rejection doesn't reset accountability for an unresolved ethical concern when the project moves elsewhere.

Tom: It’s a clear message: the current setup isn't working to steer research effectively because authors are revising their framing instead of changing what they actually study or build. The whole point is getting that disclosure requirement on resubmissions to ensure we track responsibility properly.

Lu: I think the longitudinal tracking really hammers home that simply having an ethics review process isn't enough; you need a policy change that connects those flags across different stages of the research lifecycle. It’s about making the system durable rather than just reactive during a single submission cycle.

Meng: For us shipping models, this means we need to think about how we document and track ethical concerns more rigorously so that if we have to pivot our architecture later on, that initial flag doesn't just disappear when we move to a new platform or model version.

Lalam: I agree with the focus on accountability carrying over; it’s about ensuring that when research moves from one venue to another, the ethical commitment stays with the work itself, not just the original paper submission.

Jane: We've spent this time looking at how authors react and what they suggest for governance in "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." It’s a call for a more integrated system where ethics review is part of the whole research journey, not just a checkpoint before publication.

Tom: Yeah, so we learned that paper isn't just about whether authors change their methods; it's about how we structure the review process to ensure those changes actually happen and stick. What’s next on our list?

Conclusion: Tom: So we’re wrapping up our look at "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions." The big takeaway is that just revising your paper isn't enough to steer research toward better practices.

Jane: Exactly. They tracked flagged submissions and found that even when authors revise their work, only about seventeen percent actually change the methods or procedures addressing the original ethical concern. It shows how often authors just change the writing, not the actual science they did.

Lu: The longitudinal tracking is really telling there's a huge gap between what reviewers flag and what authors actually implement in their revisions. They found most changes were rhetorical, not technical adjustments to the model or data collection setup.

Meng: That’s tough for production environments. If you get a red flag about data privacy or bias, and the author just tweaks the text but doesn't change how they handle that data at scale, that's a problem for us.

Lalam: I see what they mean from my side. The authors interviewed researchers and found that authors often make concessions during rebuttal to keep their job in the review process, but then drop those concessions once rejection hits and there’s no more leverage left.

Tom: So the paper recommends forcing authors to disclose those prior ethics flags on resubmissions so accountability doesn't just vanish when the project moves venues. That sounds like a necessary policy shift for how we judge submissions going forward.

Jane: They also suggest requiring early-stage ethics review before a full submission because by the time you’re done, your experimental design is locked in and changing it becomes much harder. It’s about catching those design flaws earlier when they're cheaper to fix.

Lu: I think that idea of disclosure forcing accountability is key. If authors know their past work has an unresolved ethical issue attached to it, they have a much stronger incentive to address it now instead of ignoring it for the next project.

Meng: I agree with the early review part too because if we’re shipping something, we need that initial scrutiny on how we collect and process things before we commit to the whole pipeline.

Lalam: I’m confident in this direction. The paper "Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions" shows us where the current system is falling short on long-term steering.

Tom: It does, and it points toward needing a structural fix in how we handle ethical feedback across different stages of research. Next up, we’re looking at RouteBalance for AI serving models—how they manage those heterogeneous stacks without breaking the schedule.

Massachusetts Institute of Technology · Harvard University

cs.CY

Submitted: 2026-09-09

Updated: 2026-11-02

Comments: 40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: AI-Native Academia

Project page: https://iclr.cc/Conferences/2026/AuthorGuide

License: http://creativecommons.org/licenses/by/4.0/

Importance score: 82/100

The gist: Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors…

Key concepts

Longitudinal Tracking of Ethics Flags
Researchers followed flagged submissions through subsequent public resubmissions over time. The goal was to determine if authors actually changed their methods or procedures after an initial ethics review, revealing the long-term effect of these flags on the research process.
Substantive vs. Rhetorical Changes
The study distinguished between changes that alter the actual technical work (substantive changes) and those that only change how the paper is written (rhetorical revisions). Only a small percentage of resubmissions showed substantive changes addressing previous ethical concerns.
Editorial Process vs. Research Direction
Authors often view ethics review as an 'editorial process' that shapes which research gets published, rather than a genuine reason to change the underlying research topic. Revisions are frequently made to fit publication standards rather than fundamentally altering the study's focus.

Terminology

Summary

Selective AI conferences have recently begun enforcing ethics review processes with the goal of steering research towards safer and more responsible practices before publication Abstract The authors show that authors often revise the framing of their project following ethics flags rather than redirecting their underlying research agendas Page 1. p. 0

Where the paper says it

edu Abstract Selective AI conferences have recently begun enforcing ethics review processes, with the goal of steering research towards safer and more responsible practices before publication. But do ethics flags actually steer research as intended?

Longitudinal Tracking of Ethics Flags

The study tracked flagged ICLR submissions into subsequent public resubmissions to see if authors changed their research after the original review ended Page 2. They found that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures Page 2. p. 0

Where the paper says it

We first study the longitudinal effects of ethics flags, tracing flagged ICLR (International Conference on Learning Representations) submissions that were rejected or withdrawn into subsequent public resubmissions and tracking manuscript changes made after the review process ends when the original reviewers no longer oversee the project. We qualitatively classify these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures.

Specifically, only 17% of resubmissions exhibited procedural or methodological changes addressing a previous concern Page 3. p. 1
Where the paper says it

To answer this question, we follow rejected and withdrawn ICLR submissions with ethics flags into public resubmissions, compare each concern with the corresponding methods and procedures, and qualitatively classify each resubmission into one of five categories. We find that out of 446 resubmissions, only 17% exhibit procedural or methodological changes addressing a previous concern.

The authors noted that changes to methods or procedures were substantive, while rhetorical revisions did not count as research changes Page 3. p. 2
Where the paper says it

We separately record changes to procedures, methods, or both: procedures include consent, release, access, oversight, or governance; methods include data, models, evaluations, or experimental design. Rhetorical revisions can improve a paper, but they do not change the implicated methods or procedures and therefore do not count as research changes.

Qualitative Analysis of Author Intent

Manual reading and interviews with authors revealed that while authors frequently revise papers, they rarely cite reviewer-issued ethics flags as genuine reasons to redirect the underlying research Page 4. Authors often make concessions during rebuttal but drop those concessions after rejection when the original reviewers no longer have leverage Page 4. p. 0

Where the paper says it

To answer this, we manually read reviews and rebuttals from 25 cases and interview authors about their rebuttal processes and resubmission decisions. We find that authors often make concessions during rebuttal while reviewers can still update their scores, but drop those concessions after rejection, when the original reviewers no longer have leverage.

Interview participants described publication changes as separate from research direction changes, calling review an “editorial process” that shapes “what stories get seen” Page 4. p. 0
Where the paper says it

We find that authors often make concessions during rebuttal while reviewers can still update their scores, but drop those concessions after rejection, when the original reviewers no longer have leverage. Interview participants describe publication changes as separate from research direction changes, calling review an “editorial process” that shapes “what stories get seen” and, in another case, saying peer reviews are “mostly to filter out papers.

Policy Recommendations for Governance

The paper recommends several policy changes, especially disclosure of prior ethics flags upon resubmission so accountability for unresolved concerns carries over Page 1. One suggestion is to require authors to confidentially disclose prior ethics flags on resubmission, including the category and whether they changed the research or disputed the concern Page 5. p. 3

Where the paper says it

This is exacerbated by our observation that ethics concerns are often concentrated in one reviewer rather than shared across the panel. Conferences should therefore require authors to confidentially disclose prior ethics flags on resubmission, including the category and whether they changed the research or disputed the concern.

Another recommendation is to establish early-stage ethics review as a prerequisite for later full-paper submission Page 5. p. 3
Where the paper says it

Feedback should arrive while implicated choices can still change. By full-paper review, data collection and experiments may already be complete; conferences could therefore require an early-stage ethics pre-submission as a prerequisite for later full-paper submission.

Conclusion on Steering Effectiveness

The results strongly suggest that substantial revisions to a manuscript do not necessarily mean that ethics review redirected the underlying research Page 6. The authors conclude that conferences should review earlier, match by domain, and above all require the disclosure of prior flags upon resubmission so rejection does not reset accountability for an unresolved ethics concern when the project moves to another venue Page 6. p. 3

Where the paper says it

The resubmission types, review histories, and interviews explain why: concessions can matter immediately while reviewers still influence the decision; after rejection, the original reviewers lose that leverage, and reviewer noise may keep the concern from resurfacing. As such, conferences should review earlier, match by domain, and above all require the disclosure of prior flags upon resubmission so rejection does not reset accountability for an unresolved ethics concern when the project moves to another venue.

Limitations of the Study

The study's limitations include focusing only on papers that are resubmissions, potentially missing cases where authors entirely abandoned their projects due to a flag Page 6. The corpus is limited to ICLR submissions, and only three of the twenty-five invited researchers responded to the qualitative interviews Page 3. p. 3

Where the paper says it

However, given the similar review–rebuttal–decision cycle across top AI conferences, we expect the mechanisms we identify to extend beyond ICLR, although their prevalence may differ. Additionally, only three of 25 invited researchers responded; this is reasonable for a qualitative study, but more research is needed to make claims about broader researcher sentiments.

Related Work Context

The paper situates its findings within existing literature on ethical reflection in technical research and ethics review at top conferences Page 2. It contrasts its longitudinal analysis with prior work that examined isolated review cycles or broader discussions of harms without considering how ethics reviews shape research Page 3. p. 1

Where the paper says it

These works examine isolated review cycles or ethical practice more broadly, but do not consider whether ethics reviews shape research and in what way. 3 Longitudinal Analysis: Do Authors Change Research After Ethics Flags?

References

The paper cites various works concerning ethical reflection in technical research, ethics review at top conferences, and the operationalization of AI ethics Page 5. Key references include work by Winner on artifacts having politics Page 6, Hecht et al. p. 5

Where the paper says it

Langdon Winner. Do artifacts have politics?

on mitigating negative impacts through peer review Page 5, and studies examining responsibility and deliberation in AI impact statements Page 5. p. 4
Where the paper says it

Hullman, and Tina Eliassi-Rad. Examining responsibility and deliberation in AI impact statements and ethics reviews.

The bibliography also includes specific conference guidelines and ethical review process documents Page 5. p. 1
Where the paper says it

Ethics review at top conferences. In 2021, NeurIPS established new “Ethics Guidelines” and issued several conditional acceptances and one rejection on ethical grounds [Bengio et al.

--- Page 1 ---

Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions Kento Nishi∗ Alec Laprevotte† Isaiah Bullock† Mfoniso M. Andrew† Massachusetts Institute of Technology †Harvard University Correspondence: knishi@mit. p. 0

Where the paper says it

Governing AI Research Through Peer Review: A Mixed-Methods Study of the Longitudinal Effects of Ethics Flags Across Resubmissions Kento Nishi∗ Alec Laprevotte† Isaiah Bullock† Mfoniso M. Andrew† ∗Massachusetts Institute of Technology †Harvard University Correspondence: knishi@mit.

edu

--- Page 2 ---

researchers’ responsibility for foreseeable effects Since major conferences shape which research receives attention and legitimacy they share responsibility for what they accept and promote [Hecht et al., 2021]. In practice, conferences exercise this responsibility through peer review, giving reviewers leverage to steer research towards safer and more responsible directions. p. 1

Where the paper says it

Since major conferences shape which research receives attention and legitimacy, they share responsibility for what they accept and promote [Hecht et al. In practice, conferences exercise this responsibility through peer review, giving reviewers leverage to steer research towards safer and more responsible directions.

Yet reviewers gain this leverage very late in most research projects. p. 1
Where the paper says it

In practice, conferences exercise this responsibility through peer review, giving reviewers leverage to steer research towards safer and more responsible directions. Yet reviewers gain this leverage very late in most research projects.

By the time they evaluate a submission authors have usually chosen the problem, collected the data, implemented the system, and completed most experiments. p. 1
Where the paper says it

Yet reviewers gain this leverage very late in most research projects. By the time they evaluate a submission, authors have usually chosen the problem, collected the data, implemented the system, and completed most experiments.

Since reviewers can update scores in response to rebuttals and author comments during the “discussion period,” authors are naturally incentivized to acknowledge and address the reviewers’ concerns. p. 1
Where the paper says it

By the time they evaluate a submission, authors have usually chosen the problem, collected the data, implemented the system, and completed most experiments. Since reviewers can update scores in response to rebuttals and author comments during the “discussion period,” authors are naturally incentivized to acknowledge and address the reviewers’ concerns.

But once a rejection decision is issued, the original reviewers no longer influence whether or how the work is disseminated moving forward. p. 1
Where the paper says it

Since reviewers can update scores in response to rebuttals and author comments during the “discussion period,” authors are naturally incentivized to acknowledge and address the reviewers’ concerns. But once a rejection decision is issued, the original reviewers no longer influence whether or how the work is disseminated moving forward.

Unfortunately, reviewer judgments are notoriously noisy, so a new reviewer panel at the next conference may not flag the same unresolved concern. p. 1
Where the paper says it

But once a rejection decision is issued, the original reviewers no longer influence whether or how the work is disseminated moving forward. Unfortunately, reviewer judgments are notoriously noisy, so a new reviewer panel at the next conference may not flag the same unresolved concern.

This creates room for performative concessions during the discussion period that are not necessarily carried into revisions for the next venue. p. 1
Where the paper says it

Unfortunately, reviewer judgments are notoriously noisy, so a new reviewer panel at the next conference may not flag the same unresolved concern. This creates room for performative concessions during the discussion period that are not necessarily carried into revisions for the next venue.

Ideally authors who accept reviewer criticism would implement these recommendations before submitting their work again. p. 1
Where the paper says it

This creates room for performative concessions during the discussion period that are not necessarily carried into revisions for the next venue. Ideally, authors who accept reviewer criticism would implement these recommendations before submitting their work again.

If such steering occurs, we would expect clear changes across submissions: authors who acknowledge a privacy concern, for example, may remove identifiers, restrict data access, add an evaluation, or change a release decision. p. 1
Where the paper says it

Ideally, authors who accept reviewer criticism would implement these recommendations before submitting their work again. If such steering occurs, we would expect clear changes across submissions: authors who acknowledge a privacy concern, for example, may remove identifiers, restrict data access, add an evaluation, or change a release decision.

How often then do authors change their research in response to ethics flags after the original review ends? p. 1
Where the paper says it

If such steering occurs, we would expect clear changes across submissions: authors who acknowledge a privacy concern, for example, may remove identifiers, restrict data access, add an evaluation, or change a release decision. How often, then, do authors change their research in response to ethics flags after the original review ends?

To answer this question we follow rejected and withdrawn ICLR submissions with ethics flags into public resubmissions compare each concern with the corresponding methods and procedures and qualitatively classify each resubmission into one of five categories We find that out of 446 resubmissions only 17% exhibit procedural or methodological changes addressing a previous concern This raises another adjacent question if authors rarely change the research then what do they change instead We cannot answer this from tracking papers alone because comparing two versions lets us identify what authors changed but not why they changed it That is one may revise a method because they accepted criticism or because the project evolved independently it is also possible that they reject a criticism but still end up rewriting the paper for a different reason like a change in target audience

--- Page 3 ---

To study the longitudinal effects of ethics flags we need rejected or withdrawn projects that authors resubmit after the original review ends Only ICLR publicly releases all submissions and reviews after decisions NeurIPS and ICML release rejected papers only when authors opt in We therefore collect 47,893 ICLR submissions and 185,194 public reviews from OpenReview (2021–2026) We search reviews for recorded instances of the “Flag for ethics review” checkbox identifying 2,498 flagged submissions Our 2021 snapshot lacks public flags so counts begin in 2022 and rise from 107 to 1,100 in 2026 across six ethics categories Of these 1857 were rejected or withdrawn so we search them for resubmissions We cannot simply match titles because authors often change a paper’s title abstract coauthors or scope Accordingly we query OpenAlex with title variants author surnames and salient abstract terms then rank candidates by title similarity author and abstract overlap and publication year We retrieve 6,618 candidates and retain 601 top matches A retrieval score is insufficient to establish project identity so we manually read each pair and retain 446 resubmissions Even after identifying a resubmission a version difference alone does not show steering so we look for a necessary observable consequence changes to the methods or procedures implicated by the ethics concern For each pair we locate the concern in the original review compare the corresponding parts of the submission and resubmission and qualitatively code each resubmission into one of five categories We record no observable change when authors leave the concern unaddressed and mainly rhetorical change when they change limitations discussion of harms or claims without changing the implicated technical work Rhetorical revisions can improve a paper but they do not change the implicated methods or procedures and therefore do not count as research changes

--- Page 4 ---

Change after the flag Count No observable change 217 Mainly rhetorical 153 Procedures + methods 57 Primarily procedures 14 Primarily methods 5 Total labeled 446 Table 1 Resubmissions In Tab. 1 we provide strong evidence that most authors leave flagged methods and procedures unchanged In Tab. p. 2

Where the paper says it

Change after the flag Count No observable change 217 Mainly rhetorical 153 Procedures + methods 57 Primarily procedures 14 Primarily methods 5 Total labeled 446 Table 1: Resubmissions. 1, we provide strong evidence that most authors leave flagged methods and procedures unchanged.

1 in Tab. I provide strong evidence that most authors leave flagged methods and procedures unchanged Why do authors’ revisions so often stop short of the implicated research itself We cannot answer this from tracking papers alone because comparing two versions lets us identify what authors changed but not why they changed it That is one may revise a method because they accepted criticism or because the project evolved independently; it is also possible that they reject a criticism but still end up rewriting the paper for a different reason like a change in target audience

--- Page 5 ---

First we notice that in the pool of 25 review histories authors often respond at length to ethics criticism during rebuttal including partial concessions absent from later resubmissions Yet in most cases only one of three to five reviewers raises the concern the others discuss technical issues In at least one case the reviewer emphasizing ethics scored the paper 6 while reviewers who did not flag ethics scored it 4–5 Several rebuttals frame the concern as a dispute over disciplinary conventions or required background rather than a valid reason to change the research Despite this resistance on substantive grounds authors do revise their prose for resubmission We call this pattern filtering adapting papers for publication without treating criticism as grounds to redirect research Participant 1 (P1) describes resubmissions as “mostly similar” with revisions focused on “adding another table,” “more metrics” and issues that were “mostly metric based” P1 further adds that peer reviews are “mostly to filter out papers” since many reviewers repeat objections to text the authors have already revised Participant 2 (P2) says reviewers were “very focused on a particular aspect instead of seeing a larger picture.

Green — the paper says it, and the page is printed. Red — no passage carries it, so treat that sentence as unverified. The same check runs on your own text: check a summary against its source, $5.

Improvements for AI systems

  1. Disclose prior ethics flags upon resubmission to ensure accountability for unresolved concerns, as authors often change what stories get seen rather than what they study or build. This requires a policy change that carries accountability across venues.

  2. Implement early-stage ethics review as a prerequisite for later full-paper submission, allowing scrutiny of data collection and experimental design while choices can still be revised. This addresses the issue where data collection and experiments may already be complete by the time of full review.

  3. Establish domain matching for reviewers to improve feedback credibility, as authors noted that Better domain matching could make reviewers’ criticism more credible. This prevents reviews from focusing only on a narrow aspect, addressing P2's observation that reviewers were very focused on a particular aspect, instead of seeing a larger picture.

Abstract

Selective AI conferences have recently begun enforcing ethics flags and related review requirements, with the goal being to steer research towards safer and more responsible practices before publication. But do these requirements actually steer research as intended? In this paper, we show that authors more often revise how projects are presented following ethics flags than redirect their underlying research agendas. We first study the longitudinal effects of ethics flags by following rejected and withdrawn ICLR submissions with ethics flags into later public resubmissions, tracking manuscript changes after the ICLR review ends, when the original reviewers no longer oversee the project. We qualitatively code these resubmissions into five categories based on what changed after review and find that in 83% of 446 cases, authors leave the flagged concern unaddressed or revise the paper without changing the implicated methods or procedures. Then, we ask: if authors rarely change the research in response to ethics flags, what do they change instead? To answer this, we manually read reviews and rebuttals from 25 cases and directly interview authors about their rebuttal processes and resubmission decisions. We find that authors often concede concerns during rebuttal when reviewers can update their assessments, but drop those concessions after rejection when they do not regard the criticism as a sound reason to change the research. Interview participants describe publication changes as separate from changes to research direction, calling review an "editorial process" that shapes "what stories get seen" and, in another case, saying peer reviews are "mostly to filter out papers." Authors more readily change what they publish than what they study or build; we therefore recommend policy changes, especially disclosure of prior ethics flags upon resubmission so accountability carries over.

Sources

Related papers