Output and impact of Brazilian research: confronting international and national contexts

By Rogério Mugnaini

Image: dashu83.

For at least 30 years, it is possible to observe warnings in the literature on the limitation of the coverage of international information sources when it comes to scientific output indicators from countries excluded from mainstream science, such as Latin American countries. Should the absence of a significant part of the local literature in global sources not be enough, Beigel¹ notes that the very design of the evaluation systems around the world, on which research funding and the researcher’s performance depend on, exclude the knowledge generated in the peripheral circuits by reinforcing the pressure to publish in mainstream journals.

To measure the amount of information, let us take the Brazilian case described by Mugnaini, Digiampietri and Mena-Chalco², by identifying about 400,000 original and review articles published between 1998 and 2012 in journals indexed in the Web of Science (WoS) and/or SciELO. The study showed that, while at the beginning of the time period the journals indexed only in WoS published 73.1% of the national production, in 2012 this percentage decreased to 54.7%, thus, revealing the growing increase of SciELO coverage – even considering its selectivity, which means that a large number of national journals are not considered.

Looking more closely at the core journals (according to Bradford’s Zones) to the publications in each area, the Mugnaini, Digiampietri, and Mena-Chalco² study shows that Agrarian Sciences, Human Sciences, and the various Social Sciences publish fully in SciELO journals. Considering the intermediate Bradford’s Zone, it was noted that of these areas, only Agrarian Sciences presented a recent internationalization effort, which resulted in 14% of articles in WoS journals. In the peripheral Bradford’s Area, one can observe the greater internationalization, meaning that the percentage of publication occurs in a dispersed set of journals, increasing significantly for each of these areas. On the other hand, in the hard sciences, internationalization is a phenomenon that can be observed already in the core Bradford’s Zone, signaling that there is a more restricted set of WoS journals used with significant publication frequency.

While these findings reinforce the importance of evaluations covering national scientific output to be carried out as exhaustively as possible, it should be borne in mind that impact indicators (based on citation) are designed to be complementary. This consensus, established in the specialized literature in bibliometric studies, and assimilated by a large part of the international evaluation processes – which led, for example, to the Leiden Manifesto – has also been established in Brazil about two decades ago.

However, when it comes to the use of impact indicators in large-scale evaluation exercises, it is limited exclusively to qualifying scientific production according to the Impact Factor of the journal where it was published, i.e., what is at stake is not Brazilian research’s impact. Moreover, an observed consequence is the consolidation of citation indexes of worldwide scope, such as Web of Science (WoS), which is the pioneering and most used initiative, and more recently Scopus, whose use has been gradually increasing. Thus, SciELO, created in the 1990s, has become an important complementary source, whose potential for impact assessment begins to show its potential. This potential was envisioned by the very creator of the Science Citation Index, who at the time warned that scientists from these countries would still send their best work to international journals³.

In this sense, analyzing the role of national journals in the dissemination of applied science in Spain, Sanz, Aragón and Mendez⁴ found that these journals usually served as a warning to researchers and engineers on recent advances in the field, often through state of-art reviews and reports. The authors also found that some Spaniards went so far as to avoid publishing in them, because they felt it would weigh negatively in the evaluation of their curricula. They came to inquire about a possible lack of citation to these journals, because they were devoted to local problems, but rejected the idea that they should therefore be discarded, since the citation flow in the world literature concentrates in a small portion of core international journals.

The graph below makes it possible to verify that, as observed in the Spanish case, the scientific output of Engineering in Brazil shows that only a third of its articles in SciELO journals (dark red) are cited, in contrast to articles published in WoS journals, whose percentage of cited articles is 69%. Areas in which these percentages present a very similar profile are Computer Science, Economics and Business, and Mathematics. However, Human Sciences presents a low percentage of articles cited in both contexts (29% for SciELO and 14% for WoS).

Another important observation is that the volume of articles cited in the international context (dark blue) is significantly larger, as a reflection of the publication effort, but also of the greater number of foreign journals

Source: Mugnaini, Noyons e Packer⁵

Distribution of cited and not cited articles in the corpus of SciELO and WoS journals (citation window of up to two years), according to area of the cited article.

However, a slightly different scenario is shown when one observes the volume of citations received by articles published only in SciELO journals: the percentage of citations originating from the domestic flow (Brazilian articles in SciELO journals, in dark red) is quite significant not only for the Social and Human Sciences, but for Agrarian and Biological Sciences areas; and differently from what could be expected, the international impact of articles in SciELO journals not only has a significant international impact (mainstream flow, blue tones), but most come from foreign authors (observed in areas such as Exact Sciences in general, and many areas of Health Sciences and Biology).

These findings allow us to inquire whether it still makes sense to speak about “Lost science in the third world”⁶ and to envisage a reality in which we no longer must qualify the science produced according to the journal in which it was published, but rather assess it by itself.

Source: Mugnaini, Noyons e Packer⁵

Distribution of citations (citation window of up to two years) to Brazilian articles published in SciELO journals, according to the area of the cited article, indexing database and presence of Brazilian author(s) on the citing article.

Notes

1. BEIGEL, F. Publishing from the periphery: Structural heterogeneity and segmented circuits. The evaluation of scientific publications for tenure in Argentina’s CONICET. Current Sociology [online]. 2014, vol. 62, no. 5, pp. 743-765, ISSN: 1461-7064. DOI: 10.1177/0011392114533977. Available from: http://journals.sagepub.com/doi/abs/10.1177/0011392114533977?journalCode=csia

2. MUGNAINI, R., DIGIAMPIETRI, L.A. and MENA-CHALCO, J.P. Comunicação científica no Brasil (1998-2012): indexação, crescimento, fluxo e dispersão. Transinformação. 2014, vol. 26, no. 3, ISSN: 0103-3786. DOI: 10.1590/0103-3786201400030002. Available from: http://ref.scielo.org/v23s7m

3. SANZ, E., ARAGON, I. and MENDEZ, A. The function of national journals in disseminating applied science. Journal of Information Science [online]. 1995, vol.21, no.4, pp. 319-323. DOI: 10.1177/016555159502100408. Available from: http://journals.sagepub.com/doi/pdf/10.1177/016555159502100408

4. GARFIELD E. Quantitative analysis of the scientific literature and its implications for science policymaking in Latin America and the Caribbean. Bulletin of PAHO. 1995, vol. 29, no.1, pp.87-95.

5. MUGNAINI, R., NOYONS, E. and PACKER, A.L. Fluxo de citações inter-nacional: fontes de informação para avaliação de impacto científico no Brasil. In: Encontro Brasileiro de Bibliometria e Cientometria, Rio de Janeiro, 2018.

6. GIBBS, W.W. Lost science in the third world. Scientific American. 1995, vol. 273, no. 2, pp. 92-99.

References

BEIGEL, F. Publishing from the periphery: Structural heterogeneity and segmented circuits. The evaluation of scientific publications for tenure in Argentina’s CONICET. Current Sociology [online]. 2014, vol. 62, no. 5, pp. 743-765, ISSN: 1461-7064. DOI: 10.1177/0011392114533977. Available from: http://journals.sagepub.com/doi/abs/10.1177/0011392114533977?journalCode=csia

FONSECA, C. Avaliação dos programas de pós-graduação: do ponto de vista de um nativo. Horiz. antropol. [online]. 2001, vol. 7, no. 16, pp. 261-275, ISSN: 1806-9983. DOI: 10.1590/S0104-71832001000200014. Available from: http://ref.scielo.org/9dpqcy

FRAME, D. Problems in the use of literature-based S&T indicators in developing countries. In: MORITA-LOU, H. (ed.) Science and technology indicators for development. Boulder: Westview Press, 1985.

GARFIELD E. Quantitative analysis of the scientific literature and its implications for science policymaking in Latin America and the Caribbean. Bulletin of PAHO. 1995, vol. 29, no.1, pp.87-95.

GIBBS, W.W. Lost science in the third world. Scientific American. 1995, vol. 273, no. 2, pp. 92-99.

HICKS, D. et al. The Leiden Manifesto for research metrics. Nature. 2015, vol. 520, no. 7548, pp. 429.

MUGNAINI, R. and DIGIAMPIETRI, L. A. The Brazilian National Impact: Movement of Journals Between Bradford Zones of Production and Consumption. In: International Conference on Scientometrics and Informetrics, Istambul, 2015.

MUGNAINI, R. Ciclo avaliativo de periódicos no Brasil: caminho virtuoso ou colcha de retalhos? In: ENANCIB – Encontro Nacional de Pesquisa em Ciência da Informação, João Pessoa, 2015.

MUGNAINI, R., DIGIAMPIETRI, L.A. and MENA-CHALCO, J.P. Comunicação científica no Brasil (1998-2012): indexação, crescimento, fluxo e dispersão. Transinformação. 2014, vol. 26, no. 3, ISSN: 0103-3786. DOI: 10.1590/0103-3786201400030002. Available from: http://ref.scielo.org/v23s7m

MUGNAINI, R., NOYONS, E. and PACKER, A.L. Fluxo de citações inter-nacional: fontes de informação para avaliação de impacto científico no Brasil. In: Encontro Brasileiro de Bibliometria e Cientometria, Rio de Janeiro, 2018.

SANZ, E., ARAGON, I. and MENDEZ, A. The function of national journals in disseminating applied science. Journal of Information Science [online]. 1995, vol.21, no.4, pp. 319-323. DOI: 10.1177/016555159502100408. Available from: http://journals.sagepub.com/doi/pdf/10.1177/016555159502100408

About Rogerio Mugnaini

With specialization in bibliometric and scientometric indicators, evaluation of national scientific output and scientific policy, he has developed research on scientific journals since 2003, when he developed the Bibliometrics Module of the SciELO Project. He is a Professor at the University of São Paulo, in the Department of Information and Culture of the School of Communications and Arts (Escola de Comunicações e Artes, ECA), where she coordinates the Center for Research and Technology in Scientific Production. He was a member of committees, both at the university level (Vice-President of the Commission for Accreditation of the Support Program for Scientific Publications of the University of São Paulo between 2010 and 2016) and at the national level (Capes Triennial 2013 and Quadrennial 2017, area of Communications and Information). Member of the National Association of Research and Graduate Studies in Information Science (Associação Nacional de Pesquisa e Pós-Graduação em Ciência da Informação, ANCIB) and the International Society of Scientometrics and Informetrics (ISSI).

Translated from the original in Portuguese by Lilian Nassi-Calò.