Peer review – on structures and content

By Ernesto Spinak

Image: Nic McPhee.

Throughout the history of scholarly communication (at least since the 18^th century when the Royal Society appointed a Committee of Papers), the peer review process was instrumental in selecting manuscripts to ensure that they met some quality criteria. The practice of peer review, despite its limitations and multiple criticisms, has remained stable for a long time until the introduction of the Internet.

The possibilities of online communication, together with the Open Access movement first, and now Open Science, has given rise to many ideas about how to design more efficient, fast, reliable, and unbiased peer review procedures, overcoming the economic and human resources limitations that are the system’s most frequent criticisms. On this topic, we published in recent months two related posts. One that looks at ideas on how peer review could be in 10 or 15 years¹, and another on the more than twenty different views that editors and researchers have about open evaluations².

In order to organize the discussion of the topic, it is interesting to analyze a taxonomy of variants of the peer review taken from a work by Bo-Christer Björk³. Note that in this taxonomy classical aspects of evaluation as single-blind or double-blind are not considered, because none of the important new proposals seem to incorporate them.

Considering that in the posts already published mentioned above many variants of peer review were analyzed, in this note we will see in more detail a different proposal of peer review. The model that we analyze, whose example is that of open access mega-journals (OAMJs), performs a minimal review by the editorial staff, and then transfer the post-publication stage, the rest of the evaluation, to the public (we could also consider it as a partial review). It is the model used by PLoS ONE, among others.

In conventional journals the evaluation process occurs prior to publication and is based on a set of criteria expressed in one way or another in the “journal recommendations to reviewers”. In the classic peer review process, these criteria consist of four elements:

1. Novelty and originality: the work makes an intellectual breakthrough by contributing innovatively to knowledge, whether by new methods, new empirical results, or new theoretical interpretations.
2. Meaning and importance: the work adds something to the corpus of knowledge, producing an impact that improves understanding or practice.
3. Relevance: the topic is within the interest of the journal readers.
4. Solidity and rigor (technical or scientific rigor) that is related to methodological precision, consistency and integrity, quality of argumentation, or logic in data interpretation.

New forms of publishing incorporate alternative approaches to evaluation, and as we have already mentioned, one of these forms are those used by OAMJs, most notably PLoS ONE (the first in its genre) since 2006, and Scientific Reports, launched by Nature in 2011. These two initiatives are joined by a dozen other institutions, including the American Institute of Physics (AIP Advances), BMJ Open, F1000 and PeerJ, covering the biological sciences. The data that are provided in this post were taken from references ⁴ and ⁵, particularly “Let the community decide? The vision and reality of soundness-only peer review in open-access mega-journals (2018)”⁴ the most exhaustive study to date on mega-journals, in which 31 senior editors representing 16 editorial organizations that publish OAMJs were interviewed.

OAMJs have five main characteristics: large volumes of publications, broad thematic scope, open access publishing model, APC funding, and an innovative approach to peer review. The innovative approach requires that editors and reviewers only assess soundness and technical rigor (the formal structure) and do not take into account neither originality nor significance or relevance (i.e., content). That is, from the previous table, they only consider item 4, not taking into account the first three topics. The evaluation of these other criteria moves “downstream” to be judged by the academic community as soon as the article is published. The indicators at the article level (citations, downloads and altmetrics) will be the tools used for post-publication peer review.

The separation of the three content criteria from the review phase has been highly discussed in scientific publishing with opinions for and against. Among the arguments in favor is the idea that by reviewing only the article structure, the traditional gatekeeper’s role of journal editors is being altered. Editorial boards and referees are no longer the only reviewers who decide the limits of the discipline’s paradigms. Moreover, this new type of assessment provides greater efficiency to the publishing process by reducing the time of the presentation-rejection spiral that many articles go through before they can be published in a journal. Finally, it could also be considered a “democratization” of science. As Gary Ward, CEO of PLoS, said, “…if the paper is well written and the conclusions are consistent, let the community decide on the impact”⁶.

The results of the study we have cited suggest that criteria based only on the formal solidity and rigor of the document can influence editorial decisions and they, indeed, do so. However, the editors also consider that post-publication evaluations on the novelty, significance and relevance, however, contain problematic interpretation aspects. While altmetric measures for post-publication evaluation of an article are integral parts of the OAMJ model, publishers are not sure on how effective they are in practice. Particularly, editors had mixed views on which altmetrics could be useful to measure an article’s importance or significance.

My opinion

It must be recognized that, despite the different criticisms on the limitations of the current peer review process, attitudes and behaviors on this subject are deeply rooted in the DNA of the scientific community and are difficult to change.

Nevertheless, the slogan “let the community decide” is strongly attractive (it is even politically correct), and the intention to implement this paradigm so far has been highly controversial and remains problematic.

Notes

1. SPINAK, E. What will peer review be like in 2030? [online]. SciELO in Perspective, 2017 [viewed 30 May 2018]. Available from: https://blog.scielo.org/en/2017/07/26/what-will-peer-review-be-like-in-2030/

2. SPINAK, E. Sobre as vinte e duas definições de revisão por pares aberta… e mais [online]. SciELO em Perspectiva, 2018 [viewed 30 May 2018]. Available from: https://blog.scielo.org/blog/2018/02/28/sobre-as-vinte-e-duas-definicoes-de-revisao-por-pares-aberta-e-mais/

3. BJÖRK, B.C. and HEDLUND, T. Emerging new methods of peer review in scholarly journals. Learned Publishing [online]. 2015, vol. 29, no. 2, pp. 85-91 [viewed 30 May 2018]. DOI: 10.1087/20150202. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1087/20150202

4. SPEZI, V., et al. Let the community decide? The vision and reality of soundness-only peer review in open-access mega-journals. Journal of Documentation [online]. 2018, vol. 74, no. 1, pp.137-161 [viewed 30 May 2018]. DOI: 10.1108/JD-06-2017-0092. Available from: https://www.emeraldinsight.com/doi/full/10.1108/JD-06-2017-0092

5. SPEZI, V., et al. Open-access mega-journals: The future of scholarly communication or academic dumping ground? A review. Journal of Documentation [online]. 2017, vol. 73, no. 2, pp.263-283 [viewed 30 May 2018]. DOI: 10.1108/JD-06-2016-0082. Available from: https://www.emeraldinsight.com/doi/full/10.1108/JD-06-2016-0082

6. ADAMS, C. PLOS ONE [online]. SPARC. 2011 [viewed 30 May 2018]. Available from: https://sparcopen.org/our-work/innovator/plos-one/

References

ADAMS, C. PLOS ONE [online]. SPARC. 2011 [viewed 30 May 2018]. Available from: https://sparcopen.org/our-work/innovator/plos-one/

BJÖRK, B.C. and HEDLUND, T. Emerging new methods of peer review in scholarly journals. Learned Publishing [online]. 2015, vol. 29, no. 2, pp. 85-91 [viewed 30 May 2018]. DOI: 10.1087/20150202. Available from: https://onlinelibrary.wiley.com/doi/abs/10.1087/20150202

SPEZI, V., et al. Let the community decide? The vision and reality of soundness-only peer review in open-access mega-journals. Journal of Documentation [online]. 2018, vol. 74, no. 1, pp.137-161 [viewed 30 May 2018]. DOI: 10.1108/JD-06-2017-0092. Available from: https://www.emeraldinsight.com/doi/full/10.1108/JD-06-2017-0092

SPEZI, V., et al. Open-access mega-journals: The future of scholarly communication or academic dumping ground? A review. Journal of Documentation [online]. 2017, vol. 73, no. 2, pp.263-283 [viewed 30 May 2018]. DOI: 10.1108/JD-06-2016-0082. Available from: https://www.emeraldinsight.com/doi/full/10.1108/JD-06-2016-0082

SPINAK, E. What will peer review be like in 2030? [online]. SciELO in Perspective, 2017 [viewed 30 May 2018]. Available from: https://blog.scielo.org/en/2017/07/26/what-will-peer-review-be-like-in-2030/

SPINAK, E. Sobre as vinte e duas definições de revisão por pares aberta… e mais [online]. SciELO em Perspectiva, 2018 [viewed 30 May 2018]. Available from: https://blog.scielo.org/blog/2018/02/28/sobre-as-vinte-e-duas-definicoes-de-revisao-por-pares-aberta-e-mais/

WAKELING, S., et al. Open-Access Mega-Journals: A Bibliometric Profile. PLoS ONE [online]. 2016, vol. 11, no. 11, e0165359 [viewed 30 May 2018]. DOI: 10.1371/journal.pone.0165359. Available from: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0165359

About Ernesto Spinak

Collaborator on the SciELO program, a Systems Engineer with a Bachelor’s degree in Library Science, and a Diploma of Advanced Studies from the Universitat Oberta de Catalunya (Barcelona, Spain) and a Master’s in “Sociedad de la Información” (Information Society) from the same university. Currently has a consulting company that provides services in information projects to 14 government institutions and universities in Uruguay.

Translated from the original in Spanish by Lilian Nassi-Calò.