Broader Impacts at NeurIPS 2020
This is a brief overview of the use of broader impacts statements at NeurIPS 2020, starting with a little background about the use of introduction of the broader impact criterion by the National Science Foundation (NSF).
Background
In 1997, the NSF updated its review criteria to involve two main elements used to evaluate research proposals: intellectual merit and broader impacts. Intellectual merit is defined as the proposed activity’s potential to “advance knowledge and understanding within its own field or across different fields” and broader impacts is the potential to “benefit society or advanced desired societal outcomes.” The NSF officially states that it weights the intellectual merit and broader impact criteria equally, but it is not clear that this is true in practice. (link to review guidelines, more info on history of BI inclusion)
The broader impact criterion is included because the NSF believes that tax payers should benefit from the funded work in some meaningful way. It was originally added to address the area of broadening participation of underrepresented groups within STEM. However, a 2008 letter discussing broader impacts added several additional criteria, which diluted the emphasis specifically on participation:
- Advance discovery and understanding while promoting teaching, training, and learning
- Broadening the participation of underrepresented groups
- Enhance infrastructure for research and education
- Broaden dissemination to enhance scientific and technological understanding
- Provide benefits to society
Broader impacts may therefore be accomplished through several avenues of the proposal, including through the research itself (e.g. cancer treatment), through activities that are directly related to the research project (e.g. research opportunities for students from underrepresented groups), or through activities that are supported by, but are complementary to, the project (e.g. outreach activities funded by the grant).
Use at NeurIPS 2020
While the broader impacts criterion in the context of an NSF grant is meant to emphasize the potential benefits of the proposed research to participants and to society, broader impacts statements at NeurIPS 2020 were focused on evaluating both benefits and harms of the work that has already been conducted:
In order to provide a balanced perspective, authors are required to include a statement of the potential broader impact of their work, including its ethical aspects and future societal consequences. Authors should take care to discuss both positive and negative outcomes. (source)
The focus is primarily on the benefits/harms to society as the research itself has already been conducted and therefore any harms to research participants has already happened; concerns about the ethics of the research itself or about publishing the work for ethical reasons have a dedicated ethical review process associated with them.
The broader impacts statement was likely introduced after an article written about changes to the peer review process by Brent Hecht et al. Researchers had the option to “opt out” of the broader impacts statement if they didn’t think it applied to their work; 9 percent of submissions did not have a broader impact statement. For those who wrote impact statements, the call referred the authors to two sources: Suggestions for Writing NeurIPS 2020 Broader Impacts Statements, and A Guide to Writing the NeurIPS Impact Statement.
How the broader impacts statements were evaluated was somewhat unclear. Indeed, the whole thing was discussed as an “experiment.” Reviewers were instructed to follow their best judgement in evaluating the broader impacts on the basis of “yes,” “no,” or “only partially.” An FAQ for authors stated that their work would not be rejected solely on the basis of the broader impacts statement. Instead, as Brent Hecht has put it, the goal is to evaluate the work instead based on the quality and transparency of the broader impacts statement, rather than on what the broader impacts statement itself actually says. On the other hand, the call does suggest that a submission could be rejected for ethical considerations.
In the end, thirteen papers were flagged for ethical review, which was conducted by a team of 22 ethics researchers. Four of these were rejected, and the others were conditionally accepted assuming updates to the broader impacts statements.
Looking ahead, NeurIPS 2021 has released a paper checklist that includes explicit discussion of the negative impacts of the work (along with many other considerations).
Notes mentioning this note
There are no notes linking to this note.