Peer review: bad with it, worse without it

By Lilian Nassi-Calò

Adapted photo from the original: Quinn Dombrowski.

Peer review of scientific articles is the evaluation of research findings or project proposals regarding its competence, significance and originality by independent qualified experts who are researching and publishing work in the same field (peers).

Science advances based on the framework of accumulated knowledge of mankind, and relies on the dissemination of results in journals or other types of publications. Since the establishment of the first scientific journals in the 17^th Century, results should be disseminated and presented openly to review by their peers.

Since then, however more regularly and with greater emphasis after World War II, peer review has been employed in the evaluation of research results and the exclusion of this step leads to the discredit of the publication and the authors who publish on it, at least for most of the international scientific community. A study conducted in 2009 by the British agency Sense about Science¹ with over 4,000 researchers shows that 84% consider that without peer review, scientific communication would be totally out of control, undermining the credibility of the information published.

It is difficult for someone not belonging to academy understanding the complexity of the steps involved in scientific publication. In many countries, however, civil society has taken knowledge of science by reading newspapers and on the Internet, and shows signs of starting to understand the importance of having access to certified information to build opinions and make decisions on day-by-day issues such as technology, environmental protection, climate change, health, law, economics and many others. Thus, the concept of information going through some kind of certification to gain validity and reliability becomes intuitive and increasingly accepted.

Several studies show that peer review is seen as one of the pillars – if not the most important – of scientific communication. The authors are unanimous in saying that the implicit trust that accompanies peer review helps to separate the chaff from the wheat among the ever-increasing amount of scientific literature available, online and printed. Despite the difficulties in going through the review process, the authors believe that the process indeed improves the quality of the manuscript. The reviewers evaluate the manuscript and make recommendations to the journal editor whether the article should be accepted as is, reviewed before publication or rejected, assessing quality, originality and importance, and making suggestions for improvement. This makes researchers want to be published in refereed journals that have a sound evaluation mechanism. These concepts were summarized by Donald Kennedy in an Science editorial in 2004², where he states that the review of a scientific paper usually involves adding qualifications and limitations to the conclusions.

Peer review, moreover, is particularly important for young researchers in the beginning of their careers, because it allows them, as referees, to develop methodological and writing skills, power of synthesis and critical judgment, resulting in a virtuous circle that build better authors.

The conclusions on peer review reached by the aforementioned studies do not mean, however, that the process is completely reliable and error-free. Au contraire, peer review, by definition, is an extremely specialized work, can be time consuming, not transparent or too critical, not free from biases and may not detect unethical behavior as plagiarism or fabricated results. Thus, criticisms arise as well as alternative mechanisms to replace it. Other recent posts in this blog analyze in detail these aspects.

A recent case drew the attention of the international scientific community, when in November 2014 the reputed publisher of almost three hundred open access journals based in the UK, BioMed Central, found evidence of attempted manipulation of the peer review process of several journals it publishes. After meticulous investigation, it was found that 43 articles had been published based on fake reviews produced by forged referees. The affected papers are being retracted, in agreement with the authors’ institutions. Most importantly, however, is to discover and reveal the scheme behind the fraud.

Apparently, the manipulation of reviews included the intermediation by firms that offer language polishing services and pre-submission assistance to authors. It is not known, however, to what extent the authors were aware about the fake reviews. It is possible that they have hired these agencies in bona fide and ended harmed. BioMed Central, however, obtained from the authors names of agencies that, besides performing legitimate services, offer favorable reviews upon payment of fees.

The problem seems to have assumed greater proportions beyond BioMed Central, given the fact that the Committee on Publication Ethics (COPE) issued a statement saying that it “has become aware of systematic, inappropriate attempts to manipulate the peer review processes of several journals across different publishers.” COPE reports that these journals are reviewing their manuscripts to determine which articles should be retracted.

The so-called “peer review of crisis” triggered by fraud notices, poor ethical behavior and by the increased number of publications and low quality reviews is, in specialists’ opinions, part of the process, and rather than harm its reputation, it actually strengthens it.

Peer review, however, remains the mechanism endorsed by the scientific community and, more recently, by other sectors of society, as that which provides the documents reliability, quality and originality. The ethical misconduct by some should not undermine the reputation of the process, which is fundamentally based on trust between publishers, authors and reviewers.

It will be published in this space a series of posts on this subject, which aims to portray the view of the actors of the editorial chain on how to strengthen and improve the process; to show how organizations, associations and other institutions work to contribute to the improvement of evaluation through guidelines, tutorials and good practice manuals; new products and services available to publishers, authors and reviewers in order to make peer review a more fair and transparent process; and interviews with Brazilian editors and authors on the subject.

Notes

¹ Peer Review Survey 2009. Sense about Science. 2009. Available from: http://www.senseaboutscience.org/data/files/Peer_Review/Peer_Review_Survey_Final_3.pdf

² KENNEDY, D. Intelligence Science: Reverse Peer Review? Science. 2004, vol. 303, n°5666, p. 1945. DOI: 10.1126/science.303.5666.1945

³ COPE statement on inappropriate manipulation of peer review processes. Committee on Publication Ethics. Available from: http://publicationethics.org/news/cope-statement-inappropriate-manipulation-peer-review-processes

References

350 years of scientific publication: from the “Journal des Sçavans” and Philosophical Transactions to SciELO. SciELO in Perspective. [viewed 22 March 2015]. Available from: http://blog.scielo.org/en/2015/03/05/350-years-of-scientific-publication-from-the-journal-des-scavans-and-philosophical-transactions-to-scielo/

Article analyses saturation of peer reviewers. SciELO in Perspective. [viewed 24 March 2015]. Available from: http://blog.scielo.org/en/2015/01/22/article-analyses-saturation-of-peer-reviewers/

Editorial ethics – good and bad scientific practices. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2014/09/10/editorial-ethics-good-and-bad-scientific-practices/

Editorial ethics: fraudulent arbitration. SciELO in Perspective. [viewed 14 January 2015]. Available from: http://blog.scielo.org/en/2015/02/20/editorial-ethics-fraudulent-arbitration/

MOYLAN, E. Innapropriate manipulation of peer review. BioMed Central blog. 2015. Available from: http://blogs.biomedcentral.com/bmcblog/2015/03/26/manipulation-peer-review/

NICHOLAS, D., and et al. Peer review: still king in the digital age. Learned Publishing. 2015, vol. 28, nº 1, pp. 15-21. DOI: 10.1087/20150104

PATEL, J. Who reviews the reviewers. BioMed Central blog. 2014. Available from: http://blogs.biomedcentral.com/bmcblog/2014/11/26/who-reviews-the-reviewers/

Peer Review and the Acceptance of New Scientific Ideas. Discussion paper from a Working Party on equipping the public with an understanding of peer review. Sense about Science. 2004. Available from: http://www.senseaboutscience.org/data/files/resources/17/peerReview.pdf

Peer review modalities, pros and cons. SciELO in Perspective. [viewed 24 March 2015]. Available from: http://blog.scielo.org/en/2015/03/27/peer-review-modalities-pros-and-cons/

Scientometrics of peer-reviewers – will they be finally recognized?. SciELO in Perspective. [viewed 24 March 2015]. Available from: http://blog.scielo.org/en/2014/05/14/scientometrics-of-peer-reviewers-will-they-be-finally-recognized/

Sense about Science. Peer Review and the Acceptance of New Scientific Ideas. London: Sense About Science, 2004. Available from: http://www.senseaboutscience.org/data/files/resources/17/peerReview.pdf

Sense about Science. The nuts and bolts. A guide for early career researchers. London: Sense About Science, 2012. Available from: http://www.elsevier.com/__data/assets/pdf_file/0006/228597/Peer-review_The-nuts-and-bolts.pdf

Trust and Authority in Scholarly Communications in the Light of the Digital Transition. CICS/CIBER. 2013. Available from: http://ciber-research.eu/download/20140115-Trust_Final_Report.pdf

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.