Could grant proposal reviews be made available openly?

By Lilian Nassi-Calò

Photo: AJC1.

Recently there have been many discussions, including in this blog, about the peer review process, the most important step in the process of scientific journal publishing, which gives articles quality and credibility.

Many authors advocate the publication of reviews openly as comments following the article, claiming that this practice could help controlling fraud in the evaluation process, while allowing public recognition of referees.

More recently, researchers have discussed, also through social networks, what would be the impact of making the peer review process more open and transparent. Furthermore, the discussion was expanded to include the review of research grant proposals. This topic raises the attention of both the scientific community and civil society. For researchers, to ensure fairness and objectivity on the process and to the general public (taxpayers) to ensure that projects are approved on the merits and not for political reasons or to favor third parties, increasing society’s trust in scientific research.

A paper by Chris Woolston in Nature¹ analyzes the impact of two recently published articles on the opening of grant reviews and more, the possibility of changing the evaluation of proposals after publication of the results.

Scholars have difficulties to understand why some research funding proposals are approved, while others are not. Studies point out several inconsistencies and low reproducibility in the grant review process. Referees assign different weights to aspects of the proposals such as methodology, originality and feasibility of the projects.

To address these disparities, Daniel Mietchen, an evolutionary biologist at the Museum of Natural History in Berlin, Germany, sustained in a publication in PLoS Biology², among other things that approved grant applications should be made publicly available. Comments on Twitter following the publication either supported or discouraged the initiative, for several reasons. The author of the study supports his hypothesis that opening the reviews can transform science as a whole based on the fact that researchers can compare their proposals to their colleagues’, besides opening prospects for cooperation.

Mietchen’s study results from a transparency survey on financing of scientific projects designed by David Gurwitz and coworkers, also published in PLoS Biology³ performed with 27 funding agencies in five continents. The authors concluded that none of them publish the approved proposals final results and 18 of them publish only brief summaries. Generally, agencies publish extensive manuals on how to obtain financing, but refrains from reporting on the proposals – approved or not – or decisions on funding.

Just as in the evaluation process of manuscripts, the careful work of grant assessment is discarded upon the proposal approval or declining. The time employed by the referees in a barely visible and recognized work causes the so-called “reviewers fatigue”, who are increasingly demanded in view of the growing number of projects to analyze. The open availability of reviews would allow the public recognition of their work and help researchers to better prepare their proposals.

According to the article already mentioned on transparency in project evaluation, there is no need to disclose all the details of the approved projects. It would suffice to report the summary, the impact of the research, the reviewer’s names, summary of the assessments, value of the granted proposal, the agency approval rate, and a final report. However, even disclosing only part of the project data, there is some degree of concern among researchers that it may, somehow, compromise the legitimacy of the evaluation process. The authors consider, however, that peer review will soon require a new dimension in its legitimacy, which might actually be the disclosure of the reviews.

Another concern relates to the confidentiality of the proposal contents. Gurwitz and coworkers state that the funding agencies consulted in their study prefer to keep confidential certain information from the proposals until the research results are published, to prevent unscrupulous researchers to plagiarize ideas and projects, in such a competitive environment, as scientific research.

Welcome Trust, a leading biomedical research funding agency currently publishes brief summaries of many of their approved research projects. According to scientific director Kevin Moses, there is no intention to expand that information, in order to respect the confidentiality of researchers and their projects.

The availability of reviews could pave the way for other types of evaluation, for example the one that occurs after publication, such as non-anonymous post-publication review articles, as suggested by Hilda Bastian, scientific editor of PLoS Medicine and PubMed Commons, an initiative that allows to post comments on PubMed papers after publication. In her opinion, open comments and criticism can improve the quality of research by efficiently exposing mistakes and weaknesses. Lisa Langsetmo, an osteoporosis researcher at McGill University in Montreal, Canada, shares Bastian’s ideas, suggesting a two-step review, “the first being blind and anonymous and the second, post-publication, open and permanent.”

In Brazil it still prevails the blind (double or single) peer review system for journal articles and research funding proposals, and the idea of making available openly the reviewers comments has not yet raised  interest among the scientific community.

Notes

¹ WOOLSTON, C. What would happen if grant reviews were made public? Nature. 2015, vol. 517, nº 247. DOI: 10.1038/517247f

² MIETCHEN, D. The transformative nature of transparency in research funding. PLoS Biol. 2014, vol. 12, nº 12. DOI: 10.1371/journal.pbio.1002027.

³ GURWITZ, D., MILANESI, E., KOENIG T. Grant application review: The case of transparency. PLoS Biol. 2014, vol. 12, nº 12. DOI: 10.1371/journal.pbio.1002010.

References

Editorial ethics – good and bad scientific practices. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2014/09/10/editorial-ethics-good-and-bad-scientific-practices/

Editorial ethics: fraudulent arbitration. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2015/02/20/editorial-ethics-fraudulent-arbitration/

GURWITZ, D., MILANESI, E., KOENIG T. Grant application review: The case of transparency. PLoS Biol. 2014, vol. 12, nº 12. DOI: 10.1371/journal.pbio.1002010.

MIETCHEN, D. The transformative nature of transparency in research funding. PLoS Biol. 2014, vol. 12, nº 12. DOI: 10.1371/journal.pbio.1002027.

Paper proposes four pillars for scholarly communication to favor the speed and the quality of science. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2013/07/31/paper-proposes-four-pillars-for-scholarly-communication-to-favor-the-speed-and-the-quality-of-science/

PubMed Commons: NLM launches pilot version of open comments on articles. SciELO in Perspective. [viewed 15 January 2015]. Available from: http://blog.scielo.org/en/2013/12/20/pubmed-commons-nlm-launches-pilot-version-of-open-comments-on-articles/

Scientometrics of peer-reviewers – will they be finally recognized?. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2014/05/14/scientometrics-of-peer-reviewers-will-they-be-finally-recognized/

WOOLSTON, C. What would happen if grant reviews were made public? Nature. 2015, vol. 517, nº 247. DOI: 10.1038/517247f

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.