Peer review modalities, pros and cons

By Lilian Nassi-Calò

From March 2015 on, Nature will offer authors the option to keep confidential their name and affiliation during the review process of submitted articles. This modality, called double-blind peer review, differs from the single-blind process, in which only the identity of the reviewers is undisclosed, even after the publication of the article.

Peer review is seen by most researchers as the most effective and efficient mechanism to ensure the quality, reliability, integrity and consistency of the academic literature. The limitations and shortcomings of the process, especially regarding fraud and plagiarism, although recognized, do not diminish its large use, even because a more efficient method is still unknown.

The steady increase of the number of journals and articles worldwide, driven mainly by online publication, has not been accompanied to the same extent by the number of researchers, which led to the saturation of the thorough peer review work. Thus, it is increasingly difficult to get good reviews within the deadlines prescribed by the journals – and wished by the authors.

Recently, new forms of peer review are being considered as alternatives to single and double-blind methods. Fully open review, where the identity of authors and reviewers is known by both; open reviews published at the end of the article, enabling post-publication discussions; and the replacement of peer review by post-publication review are among the alternatives that gained attention as ways of evolution of the original peer review process.

A detailed study by Mulligan, et al. in 2009¹, with 4,037 researchers around the world with recognized activity as reviewers, found that the vast majority sees peer review as essential to scholarly communication, and nine out of ten said that the process added quality to their own manuscripts, double-blind being considered the most effective form of review by 76% of respondents. The single-blind system is also considered effective by 45% of respondents, and 47% of them support the post-publication review as a way to complement the process. Only 15%, however, believe that peer review can be replaced by download and usage statistics, and a still small but growing number believe in open review (20%) and open and published peer review (25%).

The main arguments in favor of double-blind review are the elimination of subjective judgment and authorship and affiliation biases, preventing less renowned institutions and authors from non English countries speaking countries not having the same opportunities to publish their articles. In addition, it encourages honest opinions and allows the reviewer to focus on the quality of the manuscript. The reverse is also true, i.e., in the double-blind process a prominent researcher or someone belonging to a renowned institution would not have their articles approved solely based on this fact. Moreover, researchers believe that it is not possible to keep complete anonymity, since the subject, self-citations, or the style end up giving strong indications of authorship. Others sustain that knowing the author is important to better understand the article content and helps detect plagiarism.

Researchers who indicate the single-blind review as the most effective (45% of the researchers interviewed by Mulligan, et al.) point out the same features to defend its use: it eliminates bias, encourages honest opinions and focuses on the quality of the manuscript. They possibly refer to these advantages over open review and open and published review. The reasons to consider single-blind as a not effective review process are that less prestigious institutions and authors from developing countries are hampered in this mode, besides allowing competitors to delay publication on purpose.

Open review is considered effective by 20% of researchers, mainly because it ensures honest and less corrosive comments than in single or double-blind review, bringing reviewer and author into a fruitful scientific debate. Opponents believe that it encourages reviewers to be uncritical, it can exclude young reviewers and gives the author the opportunity to influence the reviewer, besides encouraging the dispute between them.

The open and published review (identified as effective by 25% of the researchers) has the advantage of ensuring that reviewers are honest and thoughtful about writing their comments, it improves the quality of the reviews, and encourages the debate between author and reviewer, also involving other researchers, if third-party comments are admitted. Another advantage is to publicly recognize the important work of the reviewers. As disadvantages it was pointed out that some cultures may consider that the publication of reviews may hurt hierarchical relationships or that the reports would not be as critical as they should.

Another type combines single or double-blind peer review with post-publication review, and it has 47% of respondents who rank it as effective, especially considering that this mode encourages dialogue, broadens the scope of the comments beyond one or two reviewers and enables article inconsistencies to be pointed out as soon as they are identified, and published along the paper. Those who consider it ineffective argue that without proper editorial control the discussion may be extend indefinitely, while others think that debates of this nature shall occur preferentially at conferences.

Finally, replacement of peer review by usage statistics, considered effective by only 15% of the researchers, has as main advantage the fact of being faster, objective, and does not prevent equally valid negative outcomes to be published. On the other hand, many researchers who disagree with this methodology are primarily concerned with the delay in generating significant statistics and that papers will not be improved, transforming science into a popularity contest. In addition, there is no evidence that subjective differences in download habits are considered in this analysis. Ultimately, a popular article is not necessarily a good article.

Overall, 91% of the 4,037 researchers surveyed by Mulligan, et al. consider that peer review improved the quality of their last published paper. The same percentage of authors, 91%, said the section that most benefits from the review is the discussion, while 85% of authors from non English speaking countries/regions, such as Asia and Latin America, say that peer review greatly improved writing and readability of their work. This index, however, even in North America, Oceania and Europe is between 60% and 70%, showing that review can improve the writing of authors whose native language – or the second language is English.

The SciELO program, in its continuing effort to improve the quality, credibility, visibility, use and impact of journals it publishes recommends the editors the evaluation of manuscripts to be conducted efficient and transparently with the support of online systems. These systems, in addition to organizing the functions of all the actors involved in the process (authors, editor in chief, associate editors, reviewers, administrators, etc.), it allow the editors and the Advisory Committee of SciELO to follow workflows and improve the efficiency the process, besides generating statistics and indicators.

According to data extracted from the manuscripts management system Scholar One on February 27, it was possible to obtain the distribution of peer review modalities adopted by SciELO Brazil journals. Of the 79 journals registered in the system, 23 adopt single-blind review (29%); 53 double-blind review (67%); and three journals adopt triple-blind review, where only the editor-in-chief knows the identity and affiliation of the authors and reviewers, but the associated editors do not. According to the subject area, the single-blind methodology is preferred by journals in the areas of Life Sciences, Engineering and the Exact and Earth Sciences; double-blind review is adopted preferably by Agricultural Sciences, Applied Social Sciences, Health Sciences, Social Sciences, and Linguistics, Literature and Arts. The triple-blind review, still incipient in Brazilian journals and in other countries, is adopted by two Social Sciences and one Health Sciences journal.

Preferences identified in SciELO Brazil journals largely coincide with those of Mulligan’s, et. al., in which double-blind review is ranked as effective by 76% of respondents and single-blind review by 45%. Regarding the subject areas, the authors also found that Arts and Humanities, Social Sciences and Health Sciences tend to prefer the double-blind method, while the Exact and Earth Sciences, Agricultural Sciences, and Engineering preferably use single-blind review.

Nature, in its announcement on the choice of double-blind review², indicates that results of a survey conducted by the editors of their own group of journals with young researchers around the world confirm the trend in favor of this type of review, revealing the perception that author bias affect single-blind review. This bias is also reflected in fewer citations authors from developing countries receive in prestigious journals.

As reported in this blog, the peer review system is saturated by the increasing number of online publications and the disproportionate number of researchers to review them. Paradoxically to what we present in this post, the two models proposed to meet the demand involve open reviews. One proposes a hybrid model involving post-publication review and the second reports an online service for registration and publication of reviews, in order to give the reviewers credit and recognition.

The question what is the most appropriate system of peer review has no single answer. Each journal should strive to best meet its community of authors, editors and readers and evaluate the pros and cons of each type. It is consensus, however, that simple measures such as training new generations of reviewers is imperative for the sustainability of the peer review system.

Notes

¹ MULLIGAN, A., HALL, L., and RAPHAEL, E. Peer Review in a changing world: an international study measuring the attitudes of researchers. J. Am. Soc. Inf. Sci. Technol. 2013, vol. 64, nº 1, pp. 132-161. DOI: 10.1002/asi.22798

² Nature journal offers double-blind review. Nature. 2015, vol. 518, nº 7539. DOI: 10.1038/518274b.

References

Article analyses saturation of peer reviewers. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2015/01/22/article-analyses-saturation-of-peer-reviewers/

Controversial Article in The Journal “Science” exposes the weaknesses of Peer-Review in a set of Open Access Journals. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2013/11/05/controversial-article-in-the-journal-science-exposes-the-weaknesses-of-peer-review-in-a-set-of-open-access-journals/

Editorial ethics: fraudulent arbitration. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2015/02/20/editorial-ethics-fraudulent-arbitration/

MENEGHINI, R. and PACKER, A. L. Is there science beyond English? EMBO reports. 2007, vol. 8, nº 2, pp. 112-116. DOI: 10.1038/sj.embor.7400906.

MENEGHINI, R., PACKER, A. L., NASSI-CALO, L. Articles by Latin American authors in prestigious journals have fewer citations. PLoS One. 2008, vol. 3, nº 11. DOI: 10.1371/journal.pone.0003804

MULLIGAN, A., HALL, L. and RAPHAEL, E. Peer Review in a changing world: an international study measuring the attitudes of researchers. J. Am. Soc. Inf. Sci. Technol. 2013, vol. 64, nº 1, pp. 132-161. DOI: 10.1002/asi.22798

Nature journal offers double-blind review. Nature. 2015, vol. 518, nº 7539. DOI: 10.1038/518274b.

Non-native English-speaking authors and editors evaluate difficulties and challenges in publishing in international journals. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2014/05/19/non-native-english-speaking-authors-and-editors-evaluate-difficulties-and-challenges-in-publishing-in-international-journals/

Paper proposes four pillars for scholarly communication to favor the speed and the quality of science. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2013/07/31/paper-proposes-four-pillars-for-scholarly-communication-to-favor-the-speed-and-the-quality-of-science/

PubMed Commons: NLM launches pilot version of open comments on articles. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2013/12/20/pubmed-commons-nlm-launches-pilot-version-of-open-comments-on-articles/

Reproducibility of research results: the tip of the iceberg. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2014/02/27/reproducibility-of-research-results-the-tip-of-the-iceberg/

Scientometrics of peer-reviewers – will they be finally recognized?. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2014/05/14/scientometrics-of-peer-reviewers-will-they-be-finally-recognized/

The SciELO journals are being improved by the adoption of classic workflows relating to the online management of article submissions. SciELO in Perspective. [viewed 01 March 2015]. Available from: http://blog.scielo.org/en/2014/07/30/the-scielo-journals-are-being-improved-by-the-adoption-of-classic-workflows-relating-to-the-online-management-of-article-submissions/

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.