On the immediate rejection of manuscripts without external peer review

By Lucas Massimo and Adriano Codato

Since 2013 the editors of Revista de Sociologia e Política have been collecting data that allowed analyzing basic indicators on the evaluation process of manuscript submitted to the journal. These data were published as an editorial, annually, in June of 2014, 2015 and 2016. In these texts we summarized the results to formulate a proposal for the management routine of high-impact journals, based on decisions made on preliminary analysis by the editors, without consulting anonymous independent reviewers, also known as “desk review evaluation”.

We shall begin by observing the frequency of the decisions made by the editors upon the receipt of manuscripts in the past three years, as shown in Table 1.

Table 1 – Decisions made by the editors

Source: Authors’ data based on files from Revista de Sociologia e Política.

In recent years, the editors have rejected most of the original manuscripts based on preliminary analysis: line I from Table 1 shows that this decision has been taken at about 60% of manuscripts submitted to our journal. In 2014 and 2015 the rejection based on blind peer reviews (lines II and III) stabilized at the range of 20% of the manuscripts, but has decreased from 56% to 47% of peer reviewed manuscripts. The papers accepted for publication conditioned to revision (line IV) are about 7%, and based on the average of the last two years, 11.95% of these articles submitted were approved (line V).

Table 2 presents data on the extent of the evaluation process.

Table 2 – Time spent on decision (in days)

Source: Authors’ data based on files from Revista de Sociologia e Política.

*The variation coefficient (VC) was obtained by dividing the standard deviation by the mean, which result is represented in percentage. This is a simple arithmetic indicator, which is appropriate to measure the dispersion or convergence of the time for decision-making, for the purpose of comparison between groups.

Table 2 shows that the average time for approving a manuscript (line I) in 2015 was 104.3 days, a figure slightly lower than the previous year, when this decision was taken in 117.8 days (on average). The preliminary rejection (line II) is still taken in about 20 days, and the time taken to reject an article (line III) was 105.6 days in 2015. The time to issue a review is an indicator that depends on each reviewer – which explains the wide variation on line IV, with the standard deviation being fairly close to the average – but still, the table indicates that reviews are being issued in about 45 days in the last two years.

It is quite important to note that we used the variation coefficient in time measurement. As we have already mentioned in the editorial of vol.23 n.54 (published in June 2015), the lower variation coefficients reveal a reduction of dispersion in decision-making time. The lower dispersion indicates that the evaluation process has become more uniform, and this was possible because the decisions made in all processes (regardless of who the authors of the manuscripts are) consume a similar time in our workflow. Comparing the values of VC in 2013 (when we did not use a submission management platform), as compared to the two subsequent years, we confirmed the findings announced in June 2015 (The Editors, 2015), namely, ScholarOne Manuscripts® has been an important factor for standardization and rationalization of editorial routines, bringing benefits to all those involved in science communication.

Finally, Table 3 shows some basic indicators on our review processes.

Table 3 – Indicators on the review process (2014 – 2015)

Source: Authors’ data based on files from Revista de Sociologia e Política.

Table 3 confirms that there has been a profound change in the working conditions of our referees. At the outset we clarify that the same referee may issue more than one review (that is why values in line II are slightly higher than those in line V). According to the protocol, each referee is asked to evaluate only one manuscript per semester. A reviewer may only issue more than two reviews per year if the authors resubmit the same article to a second round of assessment.

As shown in Table 1, the rate of articles approved during arbitration is ascending, i.e., over the years, the number of articles approved by the referees has been increasing. Table 3 shows another dimension of this phenomenon, which is the decrease in rejection rate by peer reviewers to evaluate manuscripts (it has dropped from 60% to only a little over 30% of all manuscripts). The number of reviews per article have also diminished, which is expected considering the increase of desk review rejections, because, strictly speaking, there is no need of four reviews to support an editorial decision. In fact, maybe this review overflow expresses the editors’ hesitation, as we mentioned in the June 2015 editorial (vol. 23 n.54), “ill-finished articles take the referees’ time and induce them to inconclusive reviews” (The Editors, 2015, p.07). Decisions taken before inviting reviewers, i.e., preliminary analysis of articles carried out by the editors, can lower the frequency of these situations (since it would be naive to assume that they will disappear).

Given its strategic importance to authors, editors and reviewers, we sought to formalize from 2015 on a desk review rejection procedure.

During the preliminary analysis, we carefully examined the structured abstract form that is required during the submission process (according to the IMRAD model – Introduction, Method, Results, Analysis and Discussion). The first aspect considered is the adherence of the article to the editorial scope of our journal. The second point to be considered is whether the full text has clear divisions, indicating the existence of a logical structure commonly used in scientific articles. The third aspect taken into account in the preliminary analysis is the objective description of the methodology and didactic presentation of empirical data, which is considered as an evidence of the authors’ zeal in finalizing the manuscript. If at least two editors decide for the preliminarily rejection, a message indicating the rejection is sent to the authors.

Of course, the editors are not experts on the subjects of all submitted articles (this is why it is a “preliminary” assessment). As noted above (Table 1), most of the articles received in recent years have been rejected during desk review analysis. Given the high amount of articles in this situation, we use a standard message to communicate the authors. Since June 2016, this message reads as follows:

Each year, the Revista de Sociologia e Política receives more articles than it can publish. Manuscripts will be sent to external reviewers for analysis, that is, to at least two anonymous referees. Besides, articles must conform to the journal’s editorial policy and represent a clear contribution to the study field. The manuscript also needs to have an analytical character rather than a descriptive nature regarding empirical data (which must be original or reworked in an unprecedented way), and the document should reveal a concise and coherent structure, according to the usual standards of scholarly communication. Despite the merits of the proposal, we consider that it has failed, at this moment, to meet all these criteria.

With this excerpt we intent to explain the reasons why the texts were preliminarily rejected, without being reviewed by anonymous referees.

We seek to mitigate this loss by putting emphasis on the gains made by all of our collaborators, assuming that even the authors’ articles rejection without peer review have more to gain from a quick response (in about 20 days) than if they would remain months awaiting for a negative response.

Ideally, for authors and editors, we would provide a detailed explanation in at least one paragraph about the reasons the articles rejected in preliminary analysis were not sent to referees. For systematic and rigorous it is our preliminary review process, we still do not have the financial, material and human resources available to provide all authors this level of detail. We hope to be able to do this soon.

Reference

OS EDITORES. Editorial. Rev. Sociol. Polit. [online]. 2015, vol. 23, nº 54, pp. 3-8. ISSN 0104-4478. DOI: 10.1590/1678-987315235401. Available from: http://ref.scielo.org/75kdzh.

About Lucas Massimo and Adriano Codato

Lucas Massimo is the editor of Revista de Sociologia e Política, and a graduate Student at the Doctorate Program in Political Sciences of Universidade Federal do Paraná (UFPR), and a tutor at the Political Science class of Centro Universitário Internacional UNINTER (Network Learning modality).

Adriano Codato is a Professor of Political Sciences at Universidade Federal do Paraná (UFPR) and the editor of Revista de Sociologia e Política (www.scielo.br/rsocp). He was a post-doctoral fellow at the Centre Européen de Sociologie et de Science Politique de la Sorbonne (CESSP-Paris). He coordinates the Observatório de Elites Políticas e Sociais do Brasil (http://observatory-elites.org/) and he is also a CNPq researcher.

Translated from the original in portuguese by Lilian Nassi-Calò.