Peer-review as a research topic in its own right

By Lilian Nassi-Calò

Over the last ten years, more and more researchers in all fields of knowledge have been devoting themselves to the subject of scholarly communication as a research topic in its own right. The number of articles and reviews which focus on the different aspects of scholarly communication, as well as the challenges which have been dictated by the digital age, is significant and worthy of consideration. One of the most studied topics is the assessment of the peer-review process. This includes both its qualitative and quantitative aspects, its ability to detect and curb unethical practices, and an examination of the approaches used in the evaluation process along with how technology can facilitate and improve the whole process.

Recent postings to this blog have discussed the pros and cons of the various approaches used in peer-review and the major challenges which impact on the process. In spite of the challenges, such as over-worked peer-reviewers, and output which is either fraudulent or of poor quality, the peer-review process is firmly entrenched as the most important stage in the process leading to the  publishing of research output in the opinion of all those who play a role in the publishing chain. This process is seen as one which gives quality, reliability and uniqueness to the article in question.

More than a dozen articles report the results of the surveys carried out with authors, publishers and referees into the various aspects of the peer-review process. Amongst the most recently published works, two stand out because of the high level of sampling which was carried out – both pieces of work refer to surveys which had more than 3,500 responses – and one of them stands out because it covers other issues concerning scholarly communication, in addition to the peer-review process.

The first of these articles, co-authored by Mulligan, Hall and Raphael¹ describes a study carried out in 2009 by the organization Sense about Science on behalf of the Alfred P. Sloan Foundation. Online questionnaires were sent to some 40,000 researchers worldwide, who had been randomly selected from the Thomson Reuters Web of Science database. Around ten percent  – 4,037 authors – responded to the questionnaire which looked at different aspects of the peer-review process, such as the general level of satisfaction, the approaches adopted (blind and other forms of review), the speed of the process, the subjective evaluation of the results of the peer-review of their own articles by a subset of the respondents, and suggestions for improvement of the process and other aspects.

The article, co-authored by Nicholas, et al², examines a piece of research dealing with the topic of reliability in scholarly communication in the digital age which was carried out in 2013 by the Center for Information and Communication Studies based at the University of Tennessee (USA) and Ciber Research Limited, located in the UK, and also funded by the Sloan Foundation.  Online questionnaires were sent to authors using the six major journal publishers in which they both publish and for which they act as referees.  Three thousand six hundred and fifty (3,650) participants responded to the survey regarding issues concerning the establishment of parameters of reliability in scientific literature, reading and citation habits, use of social media and respondents’ opinion of alternative indexes for measuring scientific impact, along with other issues.  Peer-review cuts right across the board and has an influence on all of these topics, and the results of all these studies are in harmony with those of the 2009 study mentioned above. The studies are also in agreement with other authors as far as their major role of gatekeeper in the scholarly communication process is concerned, in spite of the fact that there is a certain lack of agreement as to the ways in which the process can be improved.

The 2009 study focuses in more detail on the opinions of the scientific community on peer-review from the perspective of the researcher, the author and the referee, and seeks to answer the question of the degree of satisfaction of those who were interviewed about the current status of the peer-review process, their expectations, concerns, and suggestions as to how it can be improved. The 2013 article, however, is less detailed and more comprehensive since its objective is to assess if and how digital publishing changed deeply rooted concepts such as authority and reliability of the channels of scientific literature which academic researchers use, cite and where they publish.

It attempts to analyze the preferences and habits of researchers both as producers as well as users of scientific information, especially in relation to their behavior in the digital world and their use of social media and open access publishing.

Peer-review – pros and cons

Both studies, along with many others, show that the peer-review process stands firm as the central pillar which governs reliability in scientific research. It deals with a practice which is familiar, trustworthy and traditional which shows little variation across academic disciplines, age and seniority of researchers and geographic region. In addition, 91% of those interviewed in Mulligan and collaborators’ study stated that the process led to a significant improvement in their own articles especially in terms of the argument they were putting forward. It is interesting to note that the suggestions from the referees are so important in the opinion of the authors that they assert that it is worth the effort submitting one’s articles to renowned journals, even if they are not accepted, if only to benefit from review by eminent referees.

The main criticisms relate to the following aspects: the tardiness of the peer-review process on the part of the referees, the major reason given by researchers for considering that the process  is unsustainable; the quality of the referees which leads to poor quality articles or cases of plagiarism getting through the peer-review process; biased or prejudiced referees; and the lack of transparency in the process.

As far as the way in which peer-review should be carried out, the authors’ preference mainly comes down to double blind (76%) followed by single blind (45%). See the post published previously on this blog which gives a detailed analysis of this topic³.

Impact of the peer-review process on academic activities

The peer-review process has an influence upon major academic activities – use/reading, citation and publication/dissemination – with the greatest impact being on the final two.  Refereed journals are the preferred source of literature used by researchers. In addition, the Impact Factor of these journals was considered to be important by the majority of respondents as an indication of the quality of the information contained therein. In addition to evaluation and assessment indexes, researchers also access their own preferred sources on the Internet or in non- peer-reviewed journals in order to search for ideas and innovative topics for research.

Many authors incorrectly associate open access journals with low quality or the absence of peer-review. Researchers are unaware of the fact that reputable publishers also publish quality open-access journals and, with the exception of journals published by predatory publishers, there is no reason to have less trust in the information published in open access. In fact, authors stated that they included citations from open access sources, provided they had been refereed.

Following the trend in literature usage habits, researchers are even more conservative when it comes to citations. The principal reasons for choosing/ trusting a citation include: the author or the author’s institution is known to the researcher; the journal or source is known; and, the article cited is a seminal work in the subject area or is endorsing the methodology used by the author. In addition, authors prefer to cite articles from refereed journals which are considered reliable and quality sources. Social media are generally considered as informal communication and tend to not be cited. Corroborating this view of social media, the study by Mulligan, Hall and Raphael shows that barely 15% of the researchers interviewed considered that usage statistics, downloads and sharing in social media could, in principle, be used as an alternative to peer review.

In choosing a journal in which to publish or disseminate results, authors search primarily for important ones in their area of research.  Next in terms of importance is peer review, followed by publisher reputation and citation level, or journal Impact Factor, in that order. However, 75% of researchers say they feel confident in citing articles from refereed journals because they believe the information contained therein is arbitrated. Journal publications submitted for peer review are also fundamental for consolidating an academic career or in obtaining research funding. This is a unanimous consideration, independent of research area, even in the social sciences where books, monographs and works presented at congresses make up a significant portion of the academic output.

Suggestions for improvement

In order to maintain the sustainability of peer review, it is important to consider the factor of reviewer motivation. It is important to note that 86% of the 3,597 survey participants interviewed by Mulligan, Hall and Raphael who act as peer reviewers stated that they enjoyed acting as reviewers and intended to continue with this work despite not receiving any form of immediate compensation. This result may appear to be in conflict with the crisis in peer review reported recently. The majority of respondents in the two major recent studies on peer review, however, stated they consider peer review to be sustainable even though there is a lack of available reviewers. They consider this activity fundamental to practicing their professions. However, many believe that training and motivating young researchers is imperative for the continuation of the research process.

The majority of respondents declined at least one peer review assignment in the previous  12 months, the average being 2.2 per year. Among the most cited reasons for declining are: the article is outside of the researcher’s expertise; the researcher is busy with issues of his/ her own research; and lastly, is busy with previously received materials for review. Almost half of the respondents replied that they thought it useful to supplement assessments with some form of post-publication comments. Others mention using preprint repositories prior to peer review to accelerate the dissemination of research results. These articles are, in a way, assessed by the comments received and by the endorsement of the moderators, despite not having formally passed through the peer review process. The article repository ArXiv of Cornell University in the fields of physics and mathematics, and the open access biomedical journal PeerJ are good examples of this form of assessment.

Social media, primarily reference managers such as Mendeley and alternative metrics at the article level such as Altmetric, can be used as means of assessing the quality of published research. However, in the opinion of 85% of the respondents, they are not a substitute for formal assessment, and 47% believe they are complementary to formal assessment. Nicholas, et al., reports that researchers draw a sharp line between formal methods of communication (refereed journals) and informal methods (social media). For most respondents in the survey, with the exception of a few of the younger ones who are just beginning their careers, the only way to trust materials disseminated via social media is when they are linked to some formal source, for example, tweets, Facebook pages or blogs pointing to peer reviewed journals.

However, it remains unclear amongst researchers what controls should be exercised on research data that is made available publicly. Would it be possible, or even advisable, that some kind of prior assessment be undertaken? It seems premature for researchers to have formed an opinion in this regard considering that the topic of open data is a recent phenomenon.

In this connection, initiatives such as PaperCritic, which aims to provide feedback to authors about their works through comments posted on its website, should be mentioned. It also gives researchers the opportunity to comment on the articles of others. Similarly, Peerevaluation aims to complement existing quantitative metrics with a series of innovative quality and open review method indicators with suggestions for improving articles.

Finally, the majority of respondents in the research by Nicholas, et al., believes that the Impact Factor (IF) is maintaining a close correlation with peer review, both as an indicator of reliability and quality. Many authors are of the opinion that the paradigm shift from a journal based metric to an article based metric will lead to a weakening of the IF and its close relationship with quality of research. In fact, the IF is gaining strength, and has an important role primarily in developing countries and emerging economies in determining where to publish or what to cite.

It appears that, in the opinion of the scientific community, distinguishing quality information and literature in all areas of knowledge must involve peer review journals. The transformation of the scientific community, much discussed in some media, is a future topic for discussion. The dissatisfaction that there may be with peer review is the reason to improve the system and strengthen it further in the opposite direction to the supposed crisis or its overload.

Notes

¹ MULLIGAN, A., HALL, L., and RAPHAEL, E. Peer Review in a changing world: an international study measuring the attitudes of researchers. J. Am. Soc. Inf. Sci. Technol. 2013, vol. 64, nº 1, pp. 132-161. DOI: http://dx.doi.org/10.1002/asi.22798

² NICHOLAS, D., and et al. Peer review: still king in the digital age. Learned Publishing. 2015, vol. 28, nº1, pp. 15-21. DOI: http://dx.doi.org/10.1087/20150104

³ Peer review modalities, pros and cons. SciELO in Perspective. [viewed 29 March 2015]. Available from: http://blog.scielo.org/en/2015/03/27/peer-review-modalities-pros-and-cons/

References

ARNS, M. Open access is tiring out peer reviewers. Nature. 2014, vol. 515, nº 467, pp. 467. DOI: http://dx.doi.org/10.1038/515467a

Article analyses saturation of peer reviewers. SciELO in Perspective. [viewed 29 March 2015]. Available from: http://blog.scielo.org/en/2015/01/22/article-analyses-saturation-of-peer-reviewers/

Could grant proposal reviews be made available openly?. SciELO in Perspective. [viewed 29 March 2015]. Available from: http://blog.scielo.org/en/2015/03/20/could-grant-proposal-reviews-be-made-available-openly/

MULLIGAN, A., HALL, L., and RAPHAEL, E. Peer Review in a changing world: an international study measuring the attitudes of researchers. J. Am. Soc. Inf. Sci. Technol. 2013, vol. 64, nº 1, pp. 132-161. DOI: http://dx.doi.org/10.1002/asi.22798

NICHOLAS, D., and et al. Peer review: still king in the digital age. Learned Publishing. 2015, vol. 28, nº1, pp. 15-21. DOI: http://dx.doi.org/10.1087/20150104

NICHOLAS, D., and et al. Trust and Authority in Scholarly Communications in the Light of the Digital Transition: setting the scene for a major study. Learned Publishing. 2014, vol. 27, nº 2, pp. 121–134. DOI: http://dx.doi.org/10.1087/20140206

ONITILO, A. A., and et al. A core-item reviewer evaluation (CoRE) system for manuscript peer review. Accountability in Research. 2014, vol. 21, nº 2, pp. 109–21. DOI: http://dx.doi.org/10.1080/08989621.2014.847664

Open-Data: liquid information, democracy, innovation… the times they are a-changin’. SciELO in Perspective. [viewed 05 April 2015]. Available from: http://blog.scielo.org/en/2013/11/18/open-data-liquid-information-democracy-innovation-the-times-they-are-a-changin/

PARK, I.-U., PEACEY, M. W., and MUNAFÒ, M. R. Modelling the effects of subjective and objective decision making in scientific peer review. Nature. 2014. vol. 506, nº 7486, pp. 93–6. DOI: http://dx.doi.org/10.1038/nature12786

Peer review modalities, pros and cons. SciELO in Perspective. [viewed 29 March 2015]. Available from: http://blog.scielo.org/en/2015/03/27/peer-review-modalities-pros-and-cons/

Peer review: bad with it, worse without it. SciELO in Perspective. [viewed 29 March 2015]. Available from: http://blog.scielo.org/en/2015/04/17/peer-review-bad-with-it-worse-without-it/

Trust and Authority in Scholarly Communications in the Light of the Digital Transition. CICS/CIBER. 2013. Available from: http://ciber-research.eu/download/20140115-Trust_Final_Report.pdf

VESNIC-ALUJEVIC, L. Peer review and scientific publishing in times of web 2.0. Publishing Research Quarterly. 2014, vol. 30, nº 1, pp 39-49. DOI: http://dx.doi.org/10.1007/s12109-014-9345-8

External Links

Altmetrics – <http://altmetrics.org/>

ArXiv – <http://arxiv.org/>

Mendeley – <http://mendeley.com>

PaperCritic – <http://www.papercritic.com/>

Peer J – <https://peerj.com/>

PeerEvaluation – <http://www.peerevaluation.org/>

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in Portuguese by Nicholas Cop Consulting.