Open peer review: Publishing peer review reports influences referee behavior?

By Lilian Nassi-Calò

Image: Jon Eric Marababol.

The term open peer review (OPR) covers a range of peer review practices or models that aim to increase the transparency, efficiency, and accountability of the review process. The opening of peer review, in fact, can mean a wide variety of interactions between reviewers, editors and authors, resulting in many combinations, as shown in a recent post on this blog¹, based on the work of Ross-Hellauer.

Under the horizon of open science, numerous journals have been experimenting with different models of open peer review. Their objectives, among others, are to obtain better review reports and more constructive criticism, to encourage the participation of the scientific community in the evaluation process, to recognize the work of the referees, and provide due visibility to the review reports that are part of the scientific record of a study and that, in the previous model, were simply discarded after the approval of the manuscript.

Many editors are reluctant to adopt open review practices for many reasons. Not knowing the reaction to this practice by authors and reviewers of the scientific community in their area may be one of them. Reports of successful experiences from other journals on the adoption of open peer review models may encourage them to adopt it.

A paper recently published in Nature Communications² aimed to report a pilot study conducted with five Elsevier publisher journals in different areas of knowledge on the effects of publishing the peer review evaluation reports of 9,220 articles submitted between 2010 and 2017. The authors emphasize that publishing peer review reports is one of the most important – and less problematic – aspects of open peer review, since it does not involve financial resources or complex technologies.

The study evaluated several parameters, such as the willingness of reviewers to evaluate the articles, the length of the recommendations, the response time, the style and tone of the review reports and the final decision (accepted, minor revision, major revision or rejected). The reviewers, as well as the authors of the articles submitted during the study, were informed about the nature of the study at the time of the invitation to review the manuscripts and could have then declined. Should they accept the invitation, they were also offered the option of publishing their name and institutional affiliation in the review report.

The review reports of the accepted articles were published in open access as supplementary material with an independent DOI, thus making them citable and linked to the article on the publisher’s website, regardless of the subscription model of the journal (open access or subscription-based).

Results

Out of the more than 62,000 referees invited by the five journals involved in the pilot study, only about 22,000 (35.8%) agreed to be referees and were informed about the publication of the review reports. Regarding the reviewers’ profile, the authors observed that senior researchers refused most evaluations, while young researchers, with or without a doctorate, were more willing to review. In this analysis, no gender differences were observed.

In order to test the hypothesis that the refusal of senior researchers was not due to publication of the review reports, the authors compared the results with those of five similar Elsevier journals, which constituted of the control group and conducted conventional peer review. The results were similar; that is, in the control group, younger researchers were also the more likely to accept performing evaluations.

The analysis of the reviewer recommendations varied little during the study, which received proportionally more rejection and request for major revisions. Younger non-academic researchers issued more positive recommendations and fewer rejections. However, among the small number of reviewers who chose to have their name published in the review report (8.1%), the recommendations were more positive, with a higher percentage of minor revisions, fewer rejections and more articles accepted.

The authors attribute this correlation to the fact that the reviewers who wrote positive evaluations later chose to reveal their identities to positively signal their reputation to the academic community. Also in this regard, the results were checked against the control group to confirm that the outcome was not influenced by the publication of the review reports. In any case, this particular result underscores one of the immediate positive aspects of open peer review associated to the open identities of the reviewers.

The time to perform the review before and during the pilot study was not practically changed, remaining within the average of approximately 30 days. However, reviewers with a doctoral degree showed a tendency to slightly increase the time to perform a review. Again, these results were confronted with the control group of Elsevier journals that were not part of the pilot study and were duly confirmed.

One of the most anticipated aspects of the research was whether the content of the review report would be substantially altered due to the perspective of publishing the reports. To assess this hypothesis the authors conducted polarity analyzes – i.e., whether the tone of the review report was primarily positive or negative –, and a subjectivity analysis –whether the style was predominantly objective or subjective. The results showed that there were essentially no changes in the review report content during the pilot study, except for somewhat harsher and objective comments in open peer review, and that, once again, younger non-academic peer reviewers led this trend.

Conclusions

The study evaluated 62,790 individual observations linked to 9,220 articles submitted and 18,525 reports from referees between 2010 and 2017 in five journals from different disciplines of the Elsevier group. Including the control group from five other journals that did not participate in the pilot study, the authors analyzed a total of 138,117 observations and 21,647 manuscripts.

The main conclusions of this study suggest that the publication of review reports does not influence or compromise the peer review work. The authors were unable to detect any significant effect on the readiness to perform the evaluation, on the content and outcome of the recommendations, or on the time required to evaluate.

It should be noted, however, that while the publication of review reports was widely accepted by the reviewers of the pilot study, the same did not occur with the disclosure of their identity, an option accepted by only 8.1% of them, suggesting that this aspect of anonymity is still important for open peer review. This behavior, the authors suggest, could provide protection for possible retaliation or unforeseen consequences in the academic sphere, especially in very competitive areas.

Despite the sizeable sample and the fact that the study included journals from different areas of knowledge (without, however, identifying them), the authors emphasize that the results, however, cannot be generalized. Other studies indicate that researchers in the areas of human and social sciences are more resistant to adopting innovations in the scope of open science. On the other hand, the peer review policies chosen by the journals may eventually attract different communities of authors who are likely to adopt new forms of openness and transparency in scholarly communication, forming communities with different tendencies within the same area of knowledge.

The second major paradigm shift in scholarly communication after open access publishing has only just begun and, same as the first, it causes uncertainty and apprehension, in order to later persuade and surprise us.

Notes

1. SPINAK, E. Sobre as vinte e duas definições de revisão por pares aberta… e mais [online]. SciELO em Perspectiva, 2018 [viewed 27 March 2019]. Available from: https://blog.scielo.org/blog/2018/02/28/sobre-as-vinte-e-duas-definicoes-de-revisao-por-pares-aberta-e-mais/

2. BRAVO, G., et al. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Communications [online]. 2019, vol. 10, 322 [viewed 27 March 2019]. DOI: 10.1038/s41467-018-08250-2. Available from: https://www.nature.com/articles/s41467-018-08250-2

References

BRAVO, G., et al. The effect of publishing peer review reports on referee behavior in five scholarly journals. Nature Communications [online]. 2019, vol. 10, 322 [viewed 27 March 2019]. DOI: 10.1038/s41467-018-08250-2. Available from: https://www.nature.com/articles/s41467-018-08250-2

FRANCIS, R. Open peer review: What do reviewers think? [online]. F1000 Research blog, 2016 [viewed 27 March 2019]. Available from: https://blog.f1000.com/2016/09/22/open-peer-review-what-do-reviewers-think/

SPINAK, E. Sobre as vinte e duas definições de revisão por pares aberta… e mais [online]. SciELO em Perspectiva, 2018 [viewed 27 March 2019]. Available from: https://blog.scielo.org/blog/2018/02/28/sobre-as-vinte-e-duas-definicoes-de-revisao-por-pares-aberta-e-mais/

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.