Peer review: journal recommendation to reviewers

By Lilian N. Calò

Adapted photo from the original: frankieleon.

Considered as one of the pillars of scholarly communication, peer review has undergone a profound analysis and detailed scrutiny of its methodology. The current model, which presupposes that all scientific output must undergo pre-publication peer review, is being reviewed and very soon, the emergence of more sustainable, fast, and efficient alternative models of evaluation is previewed^1-3.

We are already witnessing this transition. Journals already offer authors the option of open evaluation, with the possibility of publishing the revision history right after the article, and others leave open to the scientific community the evaluation of relevance and impact, pre-judging only methodological soundness and a certain degree of innovation. Preprints repositories open up numerous perspectives for scholarly communication in scenarios where researchers, funding agencies, professionals and society benefit from the immediate and open availability of research results.

In this scenario, it is increasingly important for journal editors to guide reviewers on what is expected of their work. Clear and precise instructions on how to conduct peer review make it possible to carry out this important step of publishing research results in an appropriate manner considering the journals’ editorial policy, their scope and target audience, and to promote uniformity on its procedure.

Recommendations for reviewers on the particularities of the peer review process conducted by the publisher or journal, however, are little practiced. The analysis of the instructions to authors of health sciences journals of the SciELO Brazil collection held in mid-2016, and published in this blog⁴, showed that none of these journals includes recommendations to reviewers on how to conduct peer review. The literature confirms this deficiency in guidelines, as shown in the study by Hirst and Altman⁵. The authors conducted a broad search in 2010 and 2011 on the health literature databases for journal recommendations to referees concerning peer review, editorial policy, and issues related to publication standards. They found none. The authors also conducted a search on the sites of the 116 journals of the McMaster list, reviewed by the American College of Physicians⁶ for recommendations to reviewers and out of those, only 41 journals (35%) provided such guidelines in the instructions to authors session.

In order to investigate the presence of such recommendations and their content in renowned journals, we searched the journal portals and the main publishers for instructions to reviewers on how to conduct peer review in accordance to the editorial policy of their journals. We selected the following journals: PLoS ONE, Molecular Biology of the Cell (abbreviated as Mol Biol Cell), Proceedings of the National Academy of Science of the USA (PNAS), F1000Research (F1000), PeerJ, Europhysics Letters and Cell. The portal of the Royal Society of Chemistry was included because it aggregates 44 journals on chemistry, biology, biochemistry and biomedicine, toxicology, photochemistry, food sciences, materials science and environment, among others. In addition, we have researched the recommendations of the following publishers: Nature Publishing Group, BMJ Publishing Group, Elsevier, Taylor & Francis and Willey Blackwell.

The reviewers’ recommendations were evaluated for 20 criteria: preliminary screening by monitoring editor, editor-in-chief or editorial committee; information about accepting/declining carrying out an evaluation; acceptable time frame to perform an evaluation; conflict of interests; confidentiality; acknowledging the strengths as well as the flaws of the article to be addressed; being cautious when requesting additional work; the 5 criteria for publication; confidential comments to the editor; peer review modality; blinding; adoption of a standard peer review form; agreement with open data policy; editing of the reviewers’ comments by the editor-in-chief; and transference of reviews and manuscripts to other journals.

All the surveyed journals and publishers follow the first criterion, which is the screening of submitted articles by the monitoring editor, editor-in-chief, or editorial committee before forwarding it to the reviewers. The Royal Society of Chemistry, however, does not explicitly report this step on the page dedicated to the referees, but this does not mean that it does not adopt this practice.

All journals and publishers consulted contain explicit instructions on how reviewers should report as soon as possible whether they accept or decline to conduct a particular review within the time period stipulated by the journal/publisher. All of them, except Europhysics Letters, ask the referees to declare any identified conflict of interests when receiving a manuscript to evaluate. As for confidentiality, all journals and publishers, except F1000, which adopts open post-publication peer review, require reviewers to keep confidential any information about the manuscript and its authors throughout the evaluation process, at least until publication of the article.

In an attempt to conduct peer review constructively, most journals and publishers orient their reviewers to recognize the manuscript’s strengths, while pointing out the flaws to be addressed. From our selection, only PLoS ONE, F1000 and the journals of the Royal Society of Chemistry do not suggest this specifically. In the same way, the request for new experiments and/or analyses, according to experienced editors, should be made sparingly, when it is really necessary. Guidance in this regard, however, is made only by the Nature Publishing Group, PeerJ, PNAS, and Mol Biol Cell.

The journals and publishers consulted are unanimous in stipulating and requiring the following criteria as minimum for publication of manuscripts: originality of the contribution and agreement with the journal’s objectives and scope; adequate and sound methodology; rigorous analysis of the results and conclusions supported by the data presented; correct use of the English language; compliance with the ethical experimental criteria and research integrity.

During the evaluation process, it is possible that the reviewer might need to make confidential comments to the editor, which are not brought to the attention of the authors. The publications consulted vary considerably in position on this topic. PLoS ONE specifically does not accept these comments, whereas Cell, the Nature Publishing Group and Willey Blackwell accept. The others do not mention whether or not they accept these comments, and presumably, they judge on a case-by-case basis.

As for the peer review modality, all journals and publishers consulted conduct pre-publication evaluation, except F1000, which conducts post-publication evaluation. The most common process is simple blind, that is, the reviewers know the identity of the authors, but not the opposite, which occurs in PLoS ONE, PNAS, PeerJ, Cell, the Royal Society of Chemistry, Elsevier, and Taylor and Francis. Only Willey Blackwell adopts the double blind assessment, that is, reviewers and authors are unaware of each other’s identity and the journal Mol Biol Cell does not specify the blinding adopted. F1000 and the BMJ group carry out open evaluation, that is, the reviewers sign the evaluations when sending them to the authors. However, reviewers at PeerJ, the Nature Publishing Group and the journals of the Royal Society of Chemistry are encouraged, but not required, to inform their names to authors in an attempt to make the peer review process more transparent and accountable.

From the journals consulted, PLoS ONE, Cell, Europhysics Letters, and the publishers Royal Society of Chemistry, Elsevier, Taylor and Francis (only part) and Willey Blackwell use standard per review forms. PNAS, F1000, PeerJ, and the BMJ Publishing Group do not adopt them; the remainder does not specify.

As for the adoption of open data policy, only PLoS ONE, F1000, Cell, Elsevier and Willey Blackwell require authors to agree to make available research data in open access repositories. The others do not mention any recommendation in this regard.

It is known that despite the evaluation by the reviewers, it is the editor-in-chief’s responsibility to make the final decision on the acceptance or rejection of the manuscript for publication. However, many journals adopt the policy of not intervening in the content of the reviews sent to the authors. In our selection, PLoS ONE, F1000, PeerJ, the BMJ Publishing Group, and the Nature Publishing Group declare that they do not, as a rule, edit reviewers’ comments, and other journals/publishers do not report their policy accordingly.

Peer review work is extremely meticulous and demands time and effort from researchers. In order not to waste this valuable work in cases where the manuscript is not approved for publication in a particular journal, it is not uncommon practice within an editorial group, with the agreement of the authors, to transfer the manuscript and the reviews to another journal. PLoS ONE, Mol Biol Cell, Cell, Europhysics Letters, the Royal Society of Chemistry, the BMJ Publishing Group, and the Nature Publishing Group adopt this practice, while others do not report this possibility in the session recommendations to reviewers. Only in the case of F1000 this does not apply because the journal conducts post-publication peer review.

The set of recommendations for referees provided by the journals and publishers listed in this post is extremely useful and allows the evaluation to be conducted in a homogeneous, balanced and fair manner, meeting the objectives and editorial policy of the publication. An editorial published in Mol Biol Cell in 2011 by David G. Drubin, presenting the recommendations to the journal’s reviewers, is entitled “Anyone can knock down a barn, but it takes a good carpenter to build one.” The author explains the title: “Authors pour their hearts, souls, and creative energies into performing experiments and reporting the results in manuscripts, yet reviewers often seem more intent on kicking down the barn than they are on trying to help the carpenter with its design and construction, or they demand the addition of an entire new wing to the original structure”⁷.

Drubin claims that publications represent the results of a researcher’s entire career, and ensures them recognition and funding, so for a researcher’s scientific legacy, peer review is of the utmost importance.

The dissemination of good peer review practices benefits the entire scientific community, therefore, providing clear policies and procedures, norms, and rules on how to conduct it should be encouraged, as it has been done by the SciELO Program, which has among its priorities increasing the efficiency, transparency, and quality of manuscripts assessment processes.

Notes

1. SPINAK, E. What will peer review be like in 2030? [online]. SciELO in Perspective, 2017 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2017/07/26/what-will-peer-review-be-like-in-2030

2. NASSI-CALÒ, L. Adoption of open peer review is increasing [online]. SciELO in Perspective, 2017 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2017/01/10/adoption-of-open-peer-review-is-increasing

3. VELTEROP, J. Is the reproducibility crisis exacerbated by pre-publication peer review? [online]. SciELO in Perspective, 2016 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2016/10/20/is-the-reproducibility-crisis-exacerbated-by-pre-publication-peer-review

4. NASSI-CALÒ, L. Instructions to authors of Health Science journals: what do they communicate? [online]. SciELO in Perspective, 2016 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2016/08/03/instructions-to-authors-of-health-science-journals-what-do-they-communicate

5. HIRST, A. and ALTMAN, D.G. Are Peer Reviewers Encouraged to Use Reporting Guidelines? A Survey of 116 Health Research Journals. PLoS one [online]. 2012, vol. 7, no. 4, e35621, ISSN: 1932-6203 [viewed 16 August 2017]. DOI: 10.1371/journal.pone.0035621. Available from: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0035621

6. Journals Reviewed List [online]. McMaster University [viewed 16 August 2017]. Available from: https://hiru.mcmaster.ca/hiru/journalslist.asp

7. DRUBIN, D. G. Any jackass can trash a manuscript, but it takes good scholarship to create one (how MBoC promotes civil and constructive peer review). Molecular Biology of the Cell [online]. 2011, vol. 22, no. 5, pp. 525-527, ISSN: 1939-4586 [viewed 16 August 2017]. DOI: 10.1091/mbc.E11-01-0002. Available from: http://www.molbiolcell.org/content/22/5/525

References

DRUBIN, D. G. Any jackass can trash a manuscript, but it takes good scholarship to create one (how MBoC promotes civil and constructive peer review). Molecular Biology of the Cell [online]. 2011, vol. 22, no. 5, pp. 525-527, ISSN: 1939-4586 [viewed 16 August 2017]. DOI: 10.1091/mbc.E11-01-0002. Available from: http://www.molbiolcell.org/content/22/5/525

HIRST, A. and ALTMAN, D.G. Are Peer Reviewers Encouraged to Use Reporting Guidelines? A Survey of 116 Health Research Journals. PLoS one [online]. 2012, vol. 7, no. 4, e35621, ISSN: 1932-6203 [viewed 16 August 2017]. DOI: 10.1371/journal.pone.0035621. Available from: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0035621

Journals Reviewed List [online]. McMaster University [viewed 16 August 2017]. Available from: https://hiru.mcmaster.ca/hiru/journalslist.asp

NASSI-CALÒ, L. Adoption of open peer review is increasing [online]. SciELO in Perspective, 2017 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2017/01/10/adoption-of-open-peer-review-is-increasing

NASSI-CALÒ, L. Instructions to authors of Health Science journals: what do they communicate? [online]. SciELO in Perspective, 2016 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2016/08/03/instructions-to-authors-of-health-science-journals-what-do-they-communicate

NASSI-CALÒ, L. Peer review procedure [online]. SciELO in Perspective, 2017 [viewed 20 September 2017]. Available from: https://docs.google.com/spreadsheets/d/1-kLCDWKp54zy7MPzlN6S3qUWjEsf0rOkQbz5q4XDp08/edit?usp=sharing

SPINAK, E. What will peer review be like in 2030? [online]. SciELO in Perspective, 2017 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2017/07/26/what-will-peer-review-be-like-in-2030

VELTEROP, J. Is the reproducibility crisis exacerbated by pre-publication peer review? [online]. SciELO in Perspective, 2016 [viewed 16 August 2017]. Available from: http://blog.scielo.org/en/2016/10/20/is-the-reproducibility-crisis-exacerbated-by-pre-publication-peer-review

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.