How to write an academic review? [Originally published in DADOS’ blog in July/2019]

By Luiz Augusto Campos

Academic journals are nowadays the main pillar of scientific dissemination, production and funding systems. They not only serve to disseminate the ideas and findings of a given research, but, above all, to set patterns of what is a legitimate academic research and, therefore, worthy of prestige and funding.

It is curious, however, that the definition of what should and should not be published depends largely on the work of an anonymous and unpaid group of people: the peer reviewers. In the double-blind peer review system, still hegemonic in the contemporary scientific world, it is the reviewers who most influence the fate of an article.

While alternative models based on open post-publication peer review (preprint) or later disclosure of peer identities as proposed by the PLOS ONE platform are advancing, it appears that the model based on anonymity will not be quickly abdicated. It is noteworthy that, in this model, the reviews also are restricted to the authors of the evaluated paper and journal editors.

For this reason, many scholars, especially at the beginning of their careers, are not quite confident about how to write a review. Without any pretense of exhaustion, the purpose of this post is to outline, in general terms, what a reviewer should consider before writing an assessment. These are a series of norms consolidated over the years in the editorial flow of DADOS and other social science journals.

Why write a review?

Since they are anonymous and private, reviews reward very little credit in terms of academic recognition. Why, then, write them up? First, a review is the main quality control instrument of scientific output and of the minimum standards of this activity. By refusing to grant a review, we are outsourcing this task to others, thus, abdicating the possibility of guaranteeing certain standards of scientific quality.

Second, providing advice is the best way to stay current on a particular field of research. Not all scholars attend every renowned congress in their discipline, but most want to have their ideas published. It is also common for neophytes in a thematic area to make efforts to publish their research. Hence the relevance of reviews as a way of accessing the freshest on a topic.

Third, today’s peer reviewer is tomorrow’s author and vice versa. This means that as authors we expect our peers to be diligent in their evaluations and, therefore, we should be reciprocal. Moreover, it is usual for editors to follow the papers of their most effective reviewers with greater diligence and speed.

Fourth, several systems for evaluating scientific output have incorporated the quality and quantity of review in their metrics. The ScholarOne flow management system, used by many journals, has a module that allows editors to rate reviews and rank reviewers. Some areas of CAPES also consider the review productivity in their scale. Especially since the building of the PLOS ONE platform, based on data from thousands of reviews, it is likely that this activity will be increasingly recognized and studied.

What is the recommended time for accepting and writing a review?

In general, each reviewer is given one month to write their assessment. However, the important thing is to inform editors as soon as possible on your availability, i.e., not to delay acceptance or refusal of the invitation. This is because unanswered invitations remain open, leaving editors in doubt about the availability/willingness of the reviewer and the need to call a second or third name. Therefore, it is mostly recommended to accept the task and request more time to perform it than to postpone it indefinitely.

What if I cannot write a review at all?

The academic routine can indeed be strenuous and, in most cases, it leads to delays in tasks with less tangible rewards. Although there is a wide range of the time each scholar devotes to his or her different activities, it is reasonable to assume that he or she writes a review once every two months, on average. But even where it is in fact impossible to heed an invitation to write a review, reviewers may be helpful by referring other colleagues and experts to perform the task. Therefore, even if unavailable to write a review, the suggestion of other names is welcome.

What makes it a good review?

A good review is one capable of summarizing the characteristics, qualities and flaws of a given academic manuscript taking into account the parameters of a given journal, in order to support the decision of its publication or not by the editors.

It should be pointed out that the reviews are subsidies to the editorial decision and not the decision itself. It is up to the editor and his/her advisers to arbitrate what to do with each paper, especially in the face of conflicting reviews. Hence the importance of avoiding overly synthetic advice such as “the paper is good and should be published” or “the paper is bad and should not be published”. Instead, try to discuss the merits and defects of the manuscript, clearly indicating why they are so assessed and how they express themselves in the paper. It is also recommended that the referee indicates ways for improvement, suggests references and highlights unclear passages.

What are the possible reviews and what do they mean?

Journals usually provide five possible deliberations in a review:

To approve: The paper could be published as it is, requiring at most a few spelling and grammar revisions, none on the content or structure.

To approve with minor revisions: The paper requires only a few revisions, which in theory can be easily performed by the authors. This encompasses issues of form, lack of some reference, or an unclear argument or excerpt. Nevertheless, it is understood that the paper should be published. Although not mandatory, it is also understood that the reviewer is open to re-evaluate a second version of the manuscript, if it happens to be resubmitted.

To approve with major revisions: The paper requires deeper revisions, but nothing that requires a drastic restructuring. This encompasses more general issues, lack of several relevant references and many unclear arguments. In any case, it is understood that the paper should be published and that the outcome it produced need not be redone. Although not mandatory, it is also understood that the reviewer is open to re-evaluate a second version of the manuscript, if it happens to be resubmitted.

To reject and resubmit: In order for the paper to become publishable and to contribute to its thematic area, its objectives and structure must be substantively modified. This decision assumes that the paper demands profound changes, but that its original intent is viable, legitimate and may result in substantive contributions in the future, hence the incentive for resubmission. Although not mandatory, it is understood that the reviewer is open to re-evaluate a second version, although that may be resubmitted to new reviewers.

To reject: In order for the paper to become publishable and to contribute to its subject area, its objectives and structure must be completely modified. This decision does not assume that the paper “is beyond salvation”, but only that the necessary changes are so drastic that they would result in another paper, if they were to be made.

Am I an overly harsh or lenient reviewer?

Strictly speaking, the vast majority of manuscripts submitted to a quality journal are rejected. So don’t punish yourself if the number of papers you approve is small or far less than the rejected ones. Also remember that your decision is not final, it is up to the editor to ratify, moderate or complement it.

What if I detect any conflict of interest?

In the hypothetical world of blind peer review, reviewers have no clue about the authors’ identity and vice versa. In real life, however, it is common for the topic, style and approach of a given paper to suggest the reviewer its authorship. When this kind of suspicion occurs, the reviewer should assess the extent to which a potential conflict of interest exists, i.e., when the motivation for a given assessment is not only the interest for knowledge advancement, which undermines objectivity and impartiality of his/her deliberation. In such cases, the reviewer should decline the invitation and communicate his/her motivation to the editorial team. Further details on potential conflicts of interest and ethics of scientific assessment can be obtained from the FAPESP Code of Good Scientific Practice¹.

Note

1. FUNDAÇÃO DE AMPARO À PESQUISA DO ESTADO DE SÃO PAULO. Code of good scientific practice [online]. FAPESP: Fundação de Amparo à Pesquisa do Estado de São Paulo. 2014 [viewed 12 September 2019]. Available from: http://www.fapesp.br/boaspraticas/FAPESP-Code_of_Good_Scientific_Practice_2014.pdf

External links

Dados – Revista de Ciências Sociais <http://dados.iesp.uerj.br/>

PLOS ONE <https://journals.plos.org/plosone/>

About Luiz Augusto Campos

Luiz Augusto Campos is a professor of Sociology and Political Science at IESP-UERJ and editor-in-chief of DADOS, one of the leading Brazilian social science journals, published for over fifty years.

Original article in Portuguese

http://dados.iesp.uerj.br/como-redigir-um-parecer/

Translated from the original in Portuguese by Lilian Nassi-Calò.