The impact of the migration to the ScholarOne® platform on the editorial flows of a journal in the Human Sciences [Originally published as the editorial in the Revista de Sociologia e Política vol. 23 n. 54]

By Lucas Massimo and Bruno Bolognesi

In June 2014 the journal Revista de Sociologia e Política began publishing performance indicators of the assessment process for manuscripts submitted for publication. This was possible because we have been carrying out a systematic monitoring of some basic data since 2013, such as the time spent on the arbitration process, and the profiles of the authors and the peer reviewers¹. Now we are in a position to make this data public and begin doing annualized comparisons.

This initiative combines a set of measures that are being implemented with the objective of streamlining the journal routines. This matter was already discussed in other editorials of the Revista de Sociologia e Política (n.45, n.49 and n.50), so for now it suffices to say that this protocol provides authors access to objective indicators concerning how the editorial functions are being carried out – information that can be decisive in deciding to which journal research results should be submitted.

Also, the movement of these indicators gives an understanding of the effects of the measures which have been taken in recent years. Since 2014, all submissions are being assessed using the ScholarOne^® platform. This has resulted in profound changes in the way authors, editors and reviewers interact during the arbitration process. Since November 11 of 2013 we no longer receive any article proposals by email. Despite the difficulty of the procedures required by the software in the submission process, the data reported below supports the adoption of this software since it has reduced the time in assessing the manuscripts and in the workload of the peer reviewers. It has also helped standardize the workflows of the editors, resulting in a more consistent and streamlined process as a whole.

We begin by showing the data below on the type of editorial decisions taken from receipt of the original work, followed by data on the number of days taken by each decision. This is then followed by a comparison of the profiles of our contributors over the past two years. And finally, we give our perspective on how this data can be interpreted. As already put in place last year, we take care in providing aggregate data to maintain the anonymity of authors and reviewers. In addition to the editors, the SciELO Portal technical team has access to the complete data so that, as of 2014, all the procedures of the Revista de Sociologia e Política can be made available to external audit.

In 2014 we received 188 manuscripts for publication. Table 1 shows the decisions taken by the editors.

The data reflects the fact that in 2014 we introduced a new type of editorial decision – “conditional approval after peer review”. This decision is taken in situations where the reviewers recommend publication but indicate changes which do not affect the structure of the article. Thus the article forgoes a second round of review. In such cases, the authors must prepare a detailed report on how each suggested change was either accepted or rejected, and the reasons for it. The decision to publish continues to rest with the editors, but is now based on the dialogue between the authors and the reviewers which is registered in the report. This reduces the burden on the reviewers (they do not have to re-read a manuscript which was assessed favorably), and thus we reduced the time authors must wait without affecting the fairness of the arbitration process. And finally, the “report to editors” allows us to monitor whether the specified changes and the recommended suggestions were in fact followed.

The second aspect that should be mentioned is the increase in the number of rejected articles in the preliminary analysis (from 40.8% to 61%). Since the adoption of ScholarOne^® , many routine administrative procedures for the journal were automated and we could, therefore, devote more time to examining the submitted manuscripts. With more time for this task, we could be more judicious in decisions as to whether or not to refer articles for review to the peer reviewers and in determining, in a more thorough way, if the article fully met our editorial profile. This is a collective decision and takes into account the author’s clarity in presenting the use of a research method, compliance with the formal rules of presenting the text in the journal (whether it has a bilingual summary, keywords, title, etc.), and the suitability of the article to the scope of the journal.

The increase in rejections in the preliminary analysis explains why the approval rate almost doubled (it rose from 9% to 17% when taking into account all the articles approved). It should be expected that more reviewers would recommend publication if the submissions they receive for review are more in line with type of research the Revista de Sociologia e Política publishes. When we look at only the articles that were sent on to reviewers (46 articles in 2014 compared to 72 in 2013), we find that the approval rate rises to 43.4%.

The last data of relevance in Table 1 is the decrease in the number of submissions between 2013 and 2014 (it falls from 130 manuscripts to 118, a drop of 9.2%). This may be associated with the difficulty in properly processing a submission with the new method, or the newly defined scope of the journal, but it may also be a result of the ranking that the journal received in the Qualis – CAPES² system by the subject area committees – History (B1), Law (B1), Economy (B2), Philosophy (C), and Anthropology (B1). We know that this ranking is an important parameter used by authors in determining to which journal they should submit an article.

Table 2 compares the decision-making time of the editors over the past two years. The main aspect that must be stressed here is the improvement in our dispersion indicators (standard deviation and the coefficient of variation³).

We note that the times for approval and rejection in the preliminary analysis increased as seen in the means, while the standard deviation consistently dropped. Therefore, we can say that the times in 2014 are more consistent than the year before because there are fewer outliers in relation to the mean. So, if we look at the data for 2013, we are forced to conclude that it is pointless to state that our approval time is close to 70 days because few authors receive replies within that period. Table 2 shows that, as of 2014, the time was more uniform since, by using ScholarOne^®, the mean increases (117.8 days as opposed to 71.2) while the coefficient of variation falls from 195.8 to 36.1, a drop of 81.5%, which shows that the deviation in time as measured in days is closer to the mean for 2014 than it is for 2013.

Thus, we can state with greater certainty that in 2014 approved articles were processed in about four months. The consistency in the editorial procedures also allows us to stipulate goals based on a more sound position. In 2014, our goal for assessment and approval or rejection of texts received was 90 days. Henceforth, we will be working with a view to 120 days. Authors are already being informed of this when they submit their manuscripts.

Let us now turn to the profile of our contributors. Figure 1 shows the academic titles of the authors. Figure 2 shows the subject areas in which they work at their respective institutions, and Figure 3 shows the geographic region where they obtained their last degree.

The overall picture painted by this data points to a relatively homogeneous community of contributors. The Revista de Sociologia e Política is sought mostly by authors who have doctorates in Political Science and are working in programs in this same discipline in the Southeast of Brazil.

A comparison with 2013 shows that we had an increase in the order of 22% in 2014 in the number of co-authors. We went from 130 articles with 187 co-authors (1.43 authors/article) to 118 articles with 207 co-authors (1.75 authors/article). Also, we were more oriented towards authors working in Political Science and Public Policy programs, in detriment to authors in Sociology and other areas⁴. A comparison of the data also shows that the number of unsolicited submissions to our journal from authors in Latin America decreased; however we were preferred by more authors located in the Northeast region of Brazil.

Now we would like to address the profile of our reviewers. Figure 4 shows the origin, or institutional affiliation, of the peer reviewers that collaborated with us in 2013 and 2014.

In 2014 we had 247 reviewers (as opposed to 477 in 2013) and received 105 assessments (in the previous year there were 231) prepared by 98 individuals⁵. Thus the rejection rate in assessments issued remained at the same level as in 2013, at 57%.

A comparison with data from the year before the last shows a small decrease in the number of reviewers from the Southeast of Brazil, which is compensated by an increase in reviewers from the South, North and Northeast regions; however these are small variations which suggests that, from the point of view of their origin, our community of referees reflects a concentration of postgraduate programs in the subject area. What is most striking in this data is the decrease in the number of participating reviewers (from 477 to 247, or a drop of 48.2%) and the decrease in the total number of assessments issued (from 231 to 105, or a decrease of 55%). Since the rate of rejections as a proportion of the total assessments remained the same during those two years, we can see a decrease in the burden placed on the reviewers of our journal, since in 2013 the approval rate was 16.6% (excluding those eliminated in the preliminary analysis) whereas in 2014, as stated above, the rate almost tripled and reached 43.4%.

Now lets move on to the discussion of this data. What conclusions can we draw from them?

1. The use of ScholarOne^® reduced the dispersion in the indicators of assessment process times. This means that, by adopting this platform, the Revista de Sociologia e Política now has a more uniform process of peer review which in turn implies an improvement in the roles of its authors and editors. The former wins when they receive, as a whole, replies regarding editorial decisions in more or less the same timeframe. We editors have better control over the review process. As a result, we can work with more realistic goals and keep the authors informed about them.

2. The adoption of ScholarOne^® reduced the workload of the reviewers. This is surprising because we feared that the issuing of assessments would have been more time consuming given that few members of our community were familiar with this software. However, at the end of the day, we observed that this did not happen. Our reviewers were less frequently called upon because the process of screening submissions to pass on to peer review had been improved. The preparation of structured abstracts⁶ and individual profiles on the site gave the authors more work, but the manuscripts arrived at the peer review stage in a more finalized form. This result could also be associated with an increase in the total approval rate which went from 9.2% to 16.9% between 2013 and 2014.

3. The rationalization of editorial routines, at last, improved the editors’ control over the production process of the journal, as well as over the final quality of the published product. The increase in the number of rejected articles in the preliminary analysis implies that the burden of refusal of poorly structured articles was removed from the reviewers and taken on by the editors of the journal. This is important because now assessment becomes based exclusively on the merits and excellence of the content of the manuscript and not on formalities or the scope and appropriateness to the journal’s profile. Poorly completed articles take time away from reviewers and lead to inconclusive assessments. A rigorous and efficient arbitration process can prevent this from happening.

In April of 2015 SciELO announced that the journals in its collection must adopt an automated system for the management of submissions. According to Abel Packer, “ By the end of the year, all the journals in the SciELO Brazil collection should operate with a system or service capable of producing basic reports required by SciELO quantifying events (received, rejected, approved, author affiliations, editors and reviewers) (Packer, 2015)”. The implementation of these services increases the pressure on the editorial committees because they involve tough changes to the informal ways that some publishers carry on their business. Nevertheless, the results shown here suggest that the transition is a positive one for all those involved in the process of scientific publication. A transparent and well monitored system of arbitration improves the global credibility of Brazilian scientific publication. This is an essential step in the professionalization of editorial committees. The position of the Revista de Sociologia e Política is that the search for transparency and accountability on the part of journals should precede the discussion about the criteria with which public funds should be distributed to scientific journals.

Notes

1. We would like to express our gratitude to our colleague Karolina Mattos Roedder who is responsible for the executive secretariat of the journal, for her zeal in collecting and verifying the information presented in this article.

2. Qualis is a journal classification system for each field of knowledge developed and maintained by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) (Foundation for the Support and Evaluation of Graduate Education (CAPES)). The position of a journal in the Qualis ranking determines its prestige and influences the quantity and quality of submitted papers.

3. The coefficient of variation is the ratio of the standard deviation to the mean, and is expressed as a percentage. It serves to compare the degree of dispersion of data between two groups since the standard deviation always refers to its mean, making it hard to compare with parallel findings. In our case, we are comparing the decision-making time taken by editors and reviewers in the process of peer review in 2013 and 2014.

4. In 2014, we labelled the programs aggregated in this category as belonging to “Anthropology”. From this year onwards, we will substitute this label with “Other areas”.

5. The difference between the number of assessments and the number of referees is due to reviewers who issued more than one assessment.

6. We refer to the abstract model in the IMRAD format (introduction, methods, results, and discussion) which was presented in the editorial of the Revista de Sociologia e Política, n.45, March 2013.

References

Editorial. Rev. Sociol. Polit. [online]. 2014, vol. 22, nº 49, pp. 1-3. ISSN 0104-4478. Available from: http://ref.scielo.org/cj8x6g

Editorial. Rev. Sociol. Polit. [online]. 2014, vol. 22, n º 50, pp. 3-7. ISSN 0104-4478. Available from: http://ref.scielo.org/2293ks

MASSIMO, L. Editorial. Rev. Sociol. Polit. [online]. 2013, vol. 21, nº 45, pp. 5-5. ISSN 0104-4478. Available from: http://ref.scielo.org/9rmg6y

PACKER, A. Online management of manuscripts is a required indexing criterion of SciELO. SciELO in Perspective. [viewed 23 April 2015]. Available from: http://blog.scielo.org/en/2015/04/01/online-management-of-manuscripts-is-a-required-indexing-criterion-of-scielo/

To read the editorial, go to:

http://ref.scielo.org/d7hpps

About Lucas Massimo and Bruno Bolognesi

Lucas Massimo (lucasmassimo@gmail.com e editoriarsp@gmail.com) is the editor of the Revista de Sociologia e Política, obtained his doctorate in Political Science from the Universidade Federal do Paraná and is a researcher in the Núcleo de Pesquisas em Sociologia Política Brasileira at the same university.

Bruno Bolognesi (brunobolognesi@gmail.com), is the associate editor of the Revista de Sociologia e Política, is a political scientist and researcher in the Núcleo de Pesquisas em Sociologia Política Brasileira at the Universidade Federal do Paraná, and a researcher in the Núcleo de Estudos dos Partidos Políticos Latino-americanos at the Universidade Federal de São Carlos.

Translated from the original in Portuguese by Nicholas Cop Consulting.