The editors’ role on peer review: how to identify bad referees

By Lilian N. Calò

Adapted photo from the original: Pasquale Paolo Cardo.

Peer review is one of the critical stages of scientific publishing, which aims to ensure the reliability and quality of publications. The current saturation of the pre-publication peer review process, as well as the alternative models presented to try to prevent the problem from getting worse has been the subject of frequent discussion in this blog^1-3. It is well known among the scientific community that the current model of peer review delays a great deal the publication of research results, although it does not always confer quality to publications.

Many editors will agree that it is increasingly difficult to obtain quality reviews in the time frame advocated by the journal’s editorial process. The probable causes of this behavior are several, from the ever increasing number of articles to evaluate, in clear disagreement with the number of researchers available to meet the demand; the lack of mechanisms for rewarding this activity in the academic midst, the low number of specialists in certain niches of science, or simply the lack of willingness to undertake a highly specialized task that demands time and effort, however, brings little or no reward and most of the time it is simply discarded after approval of the manuscript.

In addition to the challenges posed by the peer review process, reliability and transparency are not always assured. Most often protected by anonymity, reviewers may act in an unethical way, compromising the process that should, in the first place, evaluate the articles for their scientific merit. Referees, moreover, are themselves researchers, who dedicate themselves to studies in the same area of ​​the papers they assess. It is understandable that conflicts of interest may arise which, strictly speaking, should be stated by the reviewer at the same time as he/she declines to carry out a particular evaluation. However, often, it is not what occurs.

Recent studies using theoretical models of peer review have shown that the results are sensitive to the unethical behavior of the referees. According to the model proposed by Thurner and Hanel⁴, reviewers who choose to act on their own benefit rather than for the sake of science tend to reject manuscripts that they judge to be of superior quality than their own work – and are called selfish referees. At the other end of their model, these authors propose the existence – theoretically – of impartial reviewers, who are constantly increasing the quality standard of manuscripts they accept for publication, while selfish referees favor mostly substandard manuscripts. Thus, theoretically, a scenario in which there were only impartial referees would result in no articles being published, as the standards for publication would be too high, whereas a scenario with only selfish reviewers would result in the publication of quality articles below the average of submitted articles. Of course, the real situation lies between these two extremes.

A recently paper posted on PeerJ Preprints⁵, authored by Rafael D’Andrea and James O’Dwyer at the University of Illinois, USA, takes the Thurner and Hanel model and expands it by testing how the results are altered when referees adopt less extreme behaviors, as well as the role of journal editors in mitigating the negative impacts of certain referees’ unethical conduct.

D’Andrea and Dwyer extended Thurner and Hanel’s model by introducing two referee categories to the already existing impartial (with fixed standards) and selfish (indifferent). These are impartial referees with moving standards and conscientious selfish referees. The impartial referees with fixed standards accept for publication manuscripts with minimum quality standards and reject the others. The impartial with moving standards have similar behavior, only that their quality standard is based on the average quality of the papers accepted for publication in the previous review cycle, and this standard is therefore constantly updated, which justifies its denomination. The unethical indifferent selfish referees accept articles whose quality is less than the average of the submitted articles, whereas conscientious selfish referees follow this rule, but also obey to minimum standards of quality.

The results have led to the conclusion that selfish referees, both indifferent and conscientious have a pronounced effect on reducing the quality of published literature. The main cause of this impact is their willingness to accept low quality articles, not their tendency to reject quality articles that they perceive as a threat or competition to their own scientific output. The only way to ensure the quality of publications is to keep referees committed to rejecting articles that do not meet the minimum quality criteria advocated by journals, regardless of their inclination to sabotage work of better quality than their own. In addition, the performance of selfish referees has a strong impact on the rejection rates of articles in general and in particular of above-average quality manuscripts.

What can journal editors do to mitigate, since it is not possible to neutralize the impact of the action of unethical reviewers? D’Andrea and Dwyer concluded that in the event that one of the referees recommends the publication of the manuscript, and the other its rejection, consulting other referees to break the tie helps to preserve the quality of the evaluation process. On the other hand, the authors observed that this strategy tends to increase the proportion of rejected manuscripts with above average quality, which is not positive for the science.

The practice of drawing lists of selfish referees (blacklisting) consists of keeping records of those reviewers who have high rates of disagreement among referees. This reasoning is based on the fact that two impartial referees never disagree with each other, but an impartial and a selfish one may disagree. Thus, a high level of disagreement may indicate a selfish referee. In the theoretical model reported in the article, the effect of blacklisting was overestimated, because in reality, each journal can maintain its own list of unethical referees, but they are not likely to share them. In spite of this, removing suspect referees from the sample increases the quality of published articles and reduces the rejection rate of good articles.

It should be noted that the authors emphasize that this editorial strategy ends up having an undesirable effect, since it overloads good referees with more articles to review and releases from the task those with unethical behavior, giving them more time for their own research.

An interesting observation in the evaluation model studied, which seems counterintuitive with what we observe in real, is that sending rejected articles back to the authors for review had no effect on improving their quality, although the strategy diminished the proportion of above-average quality rejected manuscripts. On the other hand, immediate approval or rejection, without peer review, of exceptionally good or exceptionally bad manuscripts has almost no impact on the quality of what is published or on the rejection rate, except, as expected, to ensure that great articles will be published and those very bad will not be.

This study has adopted some extreme approximations, such as considering that the referees’ only bias was their selfish interest in sabotaging articles of superior quality regarding their own scientific output; assuming that editors have no bias at all; considering that the quality standards of all journals are uniform, and finally, to judge that the manuscripts are evaluated based only on scientific merit and intrinsic value for publication. However, the authors believe that more sophisticated models accounting for these omissions should reach the same qualitative conclusions regarding the impact of the referees indifference and the editors’ potential capacity to mitigate them.

The authors concluded that the current format of pre-publication peer review offers little or no incentive for the selfless behavior of peer reviewers and provides little guarantee of efficiency and transparency in the process. This observation is in line with numerous other studies that share the notion that a thorough review of the current format of peer review is necessary to restore credibility, efficiency and transparency, while providing scholarly communication the desired celerity.

Notes

1. VELTEROP, J. Is the reproducibility crisis exacerbated by pre-publication peer review? [online]. SciELO in Perspective, 2016 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2016/10/20/is-the-reproducibility-crisis-exacerbated-by-pre-publication-peer-review/

2. NASSI-CALÒ, L. Adoption of open peer review is increasing [online]. SciELO in Perspective, 2017 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2017/01/10/adoption-of-open-peer-review-is-increasing/

3. PACKER, A.L., SANTOS, S. and MENEGHINI, R. SciELO Preprints on the way [online]. SciELO in Perspective, 2017 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2017/02/22/scielo-preprints-on-the-way/

4. THURNER, S. and HANEL, R. Peer-review in a world with rational scientists: Toward selection of the average. Eur. Phys. J. B. [online]. 2011, vol. 84, pp. 707-711 [viewed 19 June 2017]. DOI: 10.1140/epjb/e2011-20545-7. Available from: http://epjb.epj.org/articles/epjb/abs/2011/23/b110545/b110545.html

5. D’ANDREA​, R. and O’DWYER J.P. Can editors protect peer review from bad reviewers? [online] PeerJ Preprints. 2017, 5:e3005v3 [viewed 19 June 2017]. DOI: 10.7287/peerj.preprints.3005v3. Available from: http://peerj.com/preprints/3005v3/

References

D’ANDREA​, R. and O’DWYER J.P. Can editors protect peer review from bad reviewers? [online] PeerJ Preprints. 2017, 5:e3005v3 [viewed 19 June 2017]. DOI: 10.7287/peerj.preprints.3005v3. Available from: http://peerj.com/preprints/3005v3/

NASSI-CALÒ, L. Adoption of open peer review is increasing [online]. SciELO in Perspective, 2017 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2017/01/10/adoption-of-open-peer-review-is-increasing/

NASSI-CALÒ, L. Peer review modalities, pros and cons [online]. SciELO in Perspective, 2015 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2015/03/27/peer-review-modalities-pros-and-cons/

NASSI-CALÒ, L. Peer review: bad with it, worse without it [online]. SciELO in Perspective, 2015 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2015/04/17/peer-review-bad-with-it-worse-without-it/

PACKER, A.L., SANTOS, S. and MENEGHINI, R. SciELO Preprints on the way [online]. SciELO in Perspective, 2017 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2017/02/22/scielo-preprints-on-the-way/

THURNER, S. and HANEL, R. Peer-review in a world with rational scientists: Toward selection of the average. Eur. Phys. J. B. [online]. 2011, vol. 84, pp. 707-711 [viewed 19 June 2017]. DOI: 10.1140/epjb/e2011-20545-7. Available from: http://epjb.epj.org/articles/epjb/abs/2011/23/b110545/b110545.html

VELTEROP, J. Is the reproducibility crisis exacerbated by pre-publication peer review? [online]. SciELO in Perspective, 2016 [viewed 19 June 2017]. Available from: http://blog.scielo.org/en/2016/10/20/is-the-reproducibility-crisis-exacerbated-by-pre-publication-peer-review/

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.