Article analyses saturation of peer reviewers

By Lilian Nassi-Calò

The peer review process is a key step of the publishing process that gives quality and credibility to scientific articles. In Brazil and in many countries, however, this system faces three major difficulties: the difficulty of finding good referees, the evaluation time and the quality of the reviews.

The subject has been discussed on this blog several times and many authors propose alternatives to the classic model of double-blind review (in which the identity of authors and referees is known only by the editor), in order to improve the quality of reviews and, at the same time, publicly give credit to the reviewers. One such model is premised on the joint publication of the article followed by comments from the referees. This model aims to curb that false or forged papers go beyond the control of the review process and get published.

With the increasing number of articles and journals worldwide, due to the migration from print publication to digital, the academic peer review system has been showing signs of collapse. Referees that provide consistent quality reviews, within the requested timeframe, have been burdened with more articles to review than they can handle. According to Martijn Arns, director and researcher at the Research Institute Brainclinics in Nijmegen, the Netherlands, the most dangerous consequence of this process of saturation is that articles are sent to referees who are not experts in the area, resulting in inadequate or superficial review, as reported in an article published in Nature¹ in December 2014.

In this article the author shows his concern about the sustainability and quality of the peer review system, attested by his own experience and that of his colleagues in receiving inadequate reviews, showing a lack of understanding of the substance of the article by the referee. The growing number of online journals in recent decades has contributed to the aforementioned overloading of the referees. In addition, the editorial policy of open access journals consists on the publication of articles with sound methodology and “certified accuracy and validity, regardless of the assessment of its meaning.” This recommendation may sound to referees as a sign to approve articles only based on the analysis of the Methods and Statistics sections, without considering the reasons for the study and its wider context.

The motivation imposed by the ‘publish or perish’ science culture these days certainly contributes to the increasing number of articles. The open access journal PLoS ONE, for example, has published more than 105,000 articles since 2006, and the publisher Frontiers, over 20,000 since 2007. If each article is evaluated by two reviewers, this represents more than 250,000 referees just for these publishers.

As the articles growth rate (estimated at 113% between 2000 and 2013 in Scopus database) is much higher than the increase in the number of researchers (2.8% in the EU and 1.5% in the US), we can conclude that the reviewers demand tends to increase. One consequence of this increased demand is the assignment of articles to referees who are not experts in the area, which has the capacity to evaluate the methods and results, but cannot judge the relevance and contribution of this work to the area. Moreover, it is also up to peer reviewers to check whether the citations to the literature are suitable and in fact corroborate the author’s reasoning. The citation of obsolete or inappropriate articles is detrimental to the advancement of science, for misconceptions are perpetuated.

Arns proposes a model which was called hybrid to ensure the quality of the peer review process. According to this model, one third of the articles, comprising those with negative results, methodological studies or confirmation of previous experiments, could be published without going through peer review and subjected to post-publication review. That would leave the referees more time to assess most relevant articles, in which the pre-publication review prevents the unwary readers of exposure to ‘miracle cures’ or absurd statements. Thus, it would be able to prevent that the expansion of online publications result in saturation of the reviewers with consequent impairment of the quality of the reviews.

As already mentioned, there are models that propose public recognition of the evaluators as a way to give credit to this important phase of the scientific publishing process and also to improve the quality of reviews.

The startup Publons² was created to allow registration of the contribution of peer reviewers on a web portal and to encourage researchers to post online their experiences as referees. In an article in Nature in October 2014³, Richard Van Noorden reports the experience of the company created in New Zealand in 2013 by Andrew R.H. Preston and Daniel Johnston. Nature also heard two service users, the researchers Yogendra Kumar Mishra, who studies nanomaterials at the University of Kiel, in Germany, and Malcolm Jobling, a marine biologist at the University of Tromsø, in Norway. Mishra and Jobling stand out for their high ranking of reviews recently published in Publons.

The name Publon means the fundamental unit of scientific research that justifies being published, as well as the quantum is the fundamental unit of any physical entity involved in an interaction.

According to Preston, peer review is a specialized and valuable work, which is discarded soon after the publication of the article wasting, thus, ideas, suggestions and comments. The purpose of Publons is to gather and store this content, promoting discussions. In addition, it aims to recognize and make peer review a measure of scientific production, as well as publications and citations.

The use of the portal is quite simple: researchers create a login at the Publons webpage and register their review works before and/or after publication. The pre-publication review of the texts is not revealed until the journal authorizes its publication in blind, double blind or open format. According to Preston, to be selected by an editor to review a work is the translation of the recognition of this researcher as an expert in the area, and this simple fact deserves to be recorded and posted. Many reviewers record this experience in their CV and may use it to, for example, request to take part at editorial committees of journals.

To get an idea of the volume of reviews conducted by top reviewers, Jobling reviewed more than 125 manuscripts only this year and recorded 39 reviews in Publons in the last three months. His average is more than 100 manuscripts per year. Mishra revises an average of 5 articles per month, and recorded 22 in Publons in the last three months. The average time taken to evaluate a paper is variable, between three and twelve hours, according to Jobling, who declines three times more manuscripts than he accepts to review. Mishra says that, unlike many colleagues, he also reviews articles from low Impact Factor journals. He says that the time spent in reviewing articles varies greatly, depending on the level of the article and the journal to which it was submitted.

Publons ensures that many reviews posted in their portal are not open. Therefore, the credit to the referee is done, but the text of the review is not shown. The website also works with open reviews, such as those from Peer J and GigaScience, and sites like F1000, that automatically import the reviews of their papers into Publons. In Jobling’s opinion, reviews should be opened only with the consent of the journal, the authors and reviewers. Jobling keeps his anonymous, while Mishra prefers not to publish, just register them in Publons.

According to Jobling, however, the full recognition of peer reviewers’ work is still awaited. Preston, however, believes that the record of post-publication review is developing promisingly, and more reviewers must adhere to the initiative, creating a profile of inter-publisher reviewers, in favor of the public recognition of this important work.

Notes

¹ ARNS, M. Open access is tiring out peer reviewers. Nature. 2014, vol. 515, nº 467, pp. 467. DOI: 10.1038/515467a. Available from: http://www.nature.com/polopoly_fs/1.16403!/menu/main/topColumns/topLeftColumn/pdf/515467a.pdf

² Publons. 2014. Available from: http://publons.com/

³ VAN NOORDEN, R. The scientists who get credit for peer review. Nature. 2014. DOI: 10.1038/nature.2014. Available from: http://www.nature.com/news/the-scientists-who-get-credit-for-peer-review-1.16102

References

About us. Publons. 2014. Available from: http://publons.com/about/?utm_content=menu

Controversial Article in The Journal “Science” exposes the weaknesses of Peer-Review in a set of Open Access Journals. SciELO in Perspective. [viewed 30 November 2014]. Available from: http://blog.scielo.org/en/2013/11/05/controversial-article-in-the-journal-science-exposes-the-weaknesses-of-peer-review-in-a-set-of-open-access-journals/

F1000. 2014. Availale from: http://f1000research.com/

Guide for GigaScience reviewers. GigaScience. 2014. Available from: http://www.gigasciencejournal.com/about/reviewers

In the beginning it was just plagiarism – now its computer-generated fake papers as well. SciELO in Perspective. [viewed 30 November 2014]. Available from: http://blog.scielo.org/en/2014/03/31/in-the-beginning-it-was-just-plagiarism-now-its-computer-generated-fake-papers-as-well/

Paper proposes four pillars for scholarly communication to favor the speed and the quality of science. SciELO in Perspective. [viewed 30 November 2014]. Available from: http://blog.scielo.org/en/2013/07/31/paper-proposes-four-pillars-for-scholarly-communication-to-favor-the-speed-and-the-quality-of-science/

Policies & Procedures: Open Peer Review. Peer J. 2014. Available from: http://peerj.com/about/policies-and-procedures/#open-peer-review

Scientometrics of peer-reviewers – will they be finally recognized?. SciELO in Perspective. [viewed 30 November 2014]. Available from: http://blog.scielo.org/en/2014/05/14/scientometrics-of-peer-reviewers-will-they-be-finally-recognized/

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.