Impact beyond academia: SciELO’s presence in public policy

By Evandro Coggo Cristofoletti, Sergio Salles-Filho, Yohanna Juk, Bernardo Cabral, Karen Esteves, Fernandes Pinto, Carlos Graziani, and César Antonio Pereira

Introduction

Image: Ed Us via Unsplash

Funding agencies, universities, and Science, Technology, and Innovation (ST&I) policymakers have been discussing ways to strengthen the development of indicators, metrics, and assessments of research’s social relevance, including its impact on public policy. For the SciELO Program, measuring the impact of research on public policy is a keyway to demonstrate the social value of the research results published in SciELO-indexed journals. As one of the leading scientific publishing platforms globally, SciELO has the opportunity to highlight how indexed research transcends the academic environment and contributes effectively to the formulation of policies that impact society, offering metrics that complement traditional bibliometric indices

Methodology

Assessing the impact of research on public policy is no trivial task. Besides involving various methodological and practical challenges, there is also the difficulty related to the availability of data that allows for the production of consistent metrics and indicators. Qualitative or mixed approaches, based on case studies of policies influenced by research—through questionnaires, interviews, ethnographies or documentary research—continue to be widely used and recognized in literature. At the same time, there is a growing number of initiatives that seek to identify elements of research (mentions or use of articles, authors, institutions and funding agencies, among others) in public policy documents. Although still exploratory and incipient, this approach has the advantage of allowing work with large volumes of data and the application of bibliometric and scientometric techniques for the development of impact metrics. Among the tools used in this context, a highlight is Overton, a database that aggregates public policy documents from different countries, allowing these documents to be linked to academic publications cited or used in their preparation.

Considering this approach, we carried out a data collection and processing approach based on XML files made available by SciELO, totaling approximately 6 GB, organized by country and subdivided into multiple folders. Initially, we extracted the “publisher-id” (id_scielo) identifier from all XML files and, when available, the articles’ DOI, covering the folders referring to Chile, Ecuador, Paraguay, SciELO Brazil, and SciELO Public Health. The other folders contained only the id_scielo. Then, using the Xylose library, we obtained additional DOIs through the ArticleMeta API. In total, we identified 620,411 DOIs (from a universe of about 1.2 million records), which were used as input for searching Overton for policy documents that mentioned or cited these publications; Overton tracks and indexes documents produced by political organizations or those that aim to influence public policy, notably government sources, multilateral organizations, and think tanks

Based on this procedure, we downloaded the data returned by Overton and organized it into a consolidated spreadsheet. This spreadsheet brings together, for each policy document identified, the articles from the SciELO database mentioned, as well as metadata about the policy itself—including the source, country of origin, and organization responsible for its development. Based on this dataset, we were able to generate initial analyses, which we present below, offering a preliminary overview of the scope and connections between research indexed in SciELO and public policy documents mapped by Overton.

Results

General results

Using the initial input of 620,411 DOIs, a total of 47,231 unique DOIs (i.e., excluding any citation repetition) were identified in policy documents. However, when we consider the data with repetition (since a single article can be cited in more than one policy document and/or a policy document can cite more than one article from the identified sample), we have a total of 100,401 mentions to articles. Excluding repetitions of policy documents from this total number of mentions, we have 47,796 unique policy documents that mention an article from the database. This means that many articles are mentioned more than once, which is an indication of a positive impact on policies. We were able to identify that 4,435 articles were present in more than one policy document.

Countries that cite articles indexed in SciELO

Most of the political organizations that mention articles indexed in SciELO are International Governmental Organizations (IGOs), such as the World Health Organization (WHO) and UNESCO (United Nations Educational, Scientific and Cultural Organization), among others (see next item), totaling 14,565 policy documents (about 34%). The second largest share comes from Brazil (considering government, think tanks, and other organizations), with 6,599 policy documents (15.42%), followed by the US, with 4,247 (9.92%). This is a great finding, given that the database indexes more than 4 million policy documents from the US and about 300,000 from Brazil. Meaning there’s a good impact on national documents. Following, Table 1. lists the 20 countries with the most policy documents (totaling 39,508 documents in total, about 92% of the total).

Table 1. Countries of origin of policy documents

Organizações que citam DOIs do SciELO

When we look at organizations, we see that the WHO leads in terms of policy documents citing articles indexed in SciELO (3,588 policy documents), followed by the Institute of Applied Economic Research (IPEA) (3,503), Guidelines in PubMed (1,892 policy documents), and the Brazilian Government (with 1,809 policy documents). This indicates a significant presence in the area of health, but also in economics and public policy.

Table 2. The 20 organizations with the most policy documents

Sources in Brazil

When we break down Brazilian sources, we have IPEA, the Brazilian Government, and Empresa Brasileira de Pesquisa Agropecuária (Brazilian Agricultural Research Corporation, Embrapa) as the most prominent.

However, there is a diversity of institutions, such as Banco do Brasil (Bank of Brazil) and Agência Nacional de Vigilância Sanitária (Brazilian Health Regulatory Agency, Anvisa), reflecting the thematic scope of the articles and the indexing scope of SciELO, and its ability to cater to different areas and specializations of knowledge, which is also reflected in the diversity of use in various arenas and policy segments.

Conclusion

The results presented show that research published in SciELO has achieved significant relevance in public policy documents, with emphasis on a significant number of Brazilian documents. Despite the wide variety of public policy areas citing research published in SciELO, initial analyses reveal an impact on policies in areas such as health and economics. However, it is important to recognize some limitations of the study, such as its dependence on Overton database coverage, which favors countries in the Global North (with issues such as indexing for each country and language), as well as the challenge of identifying how these articles are actually being used. Future investigations may explore complementary approaches, locate specific journals or areas, and gain a deeper understanding of the profile of policy documents compared to the articles mentioned in these documents.

External links

SciELO Article Meta

SciELO ARTICLEMETA Restful API – GitHub

About Evandro Coggo Cristofoletti

Postdoctoral researcher in Research Management at Unicamp. PhD in Science and Technology Policy from Unicamp. Member of the Innovation Policy Analysis Group (GAPI-DPCT) and the Laboratory for Studies on the Organization of Research and Innovation at Unicamp (Lab-GEOPI). Conducts research on higher education and science and technology policy, focusing on the relationship between universities and society. He is interested in the following topics: social and political impact of research, universities and sustainable development, the relationship between universities and the public sector, university extension programs, neoliberalism, and universities.

About Sergio Salles-Filho

Agricultural Engineer (UFRRJ, 1981), holds a master in Agricultural Sciences (Unesp, 1985), and PhD in Economics (Unicamp, 1993). Full Professor in the Department of Science and Technology Policy at IG/Unicamp, he was director of FCA (2010-2013) and IG (2017-2021), as well as superintendent of FINEP (2001-2003). Visiting Researcher at MIoIR/UK (2013-2014), he has received the Zeferino Vaz Award three times. He was deputy evaluation coordinator at FAPESP (2010-2020). He works in economics, planning, and management of science, technology, and innovation, with more than 110 publications, 50 supervisions, and coordination of projects on policy evaluation and strategic planning in ST&I.

About Yohanna Juk

Postdoctoral researcher at the InSySPo (Innovation Systems, Strategies and Policies), a Project of the Department of Science and Technology Policy at Unicamp, funded by FAPESP (SPEC). She holds a PhD and master’s degree in public policy and a bachelor’s degree in economics from UFPR. She conducts research on the design, implementation, and evaluation of science, technology, and innovation policies. She is interested in the following topics: gender inequality in science, with a focus on equity, diversity, and inclusion in research funding; the use of evidence in public policy; and the evaluation of ST&I programs and institutions through bibliometrics, altmetrics, and impact measurement methodologies.

About Bernardo Cabral

Adjunct professor at the Faculty of Economics of the Universidade Federal da Bahia (UFBA), currently on secondment to the Fundação Oswaldo Cruz (Fiocruz). He holds a PhD in Industrial and Technological Economics from the Universidade Federal do Rio de Janeiro (UFRJ) (2018) and a bachelor’s degree (2010) and master’s degree (2012) in Economics from the Universidade Federal da Bahia. He also completed postdoctoral studies at the Department of Science and Technology Policy at the Universidade Estadual de Campinas (DPCT/UNICAMP) (2024). He is a researcher and deputy leader of the Foresight Studies Center (NEF) and a collaborator at the Laboratory for Studies on the Organization of Research and Innovation at Unicamp (Lab-GEOPI).

About Karen Esteves Fernandes Pinto

Postdoctoral researcher at the Genomics and Bioenergy Laboratory at Unicamp. She works on two research fronts: (1) Research and Innovation Research, focusing on the economic and socio-environmental impacts of scientific production, especially financed by public funds; and (2) Innovation and Entrepreneurship Ecosystems, analyzing the activities of Unicamp’s Innovation Agency (INOVA) and its role in mapping business activities linked to the university. Her work seeks to understand how academic research and university-business interactions contribute to sustainable development and the strengthening of innovation in Brazil.

About Carlos Graziani

Carlos Graziani holds a degree in Systems Analysis and Development from the Faculdade de Ciências Sociais Aplicadas de Extrema (2014). He has experience in network design, configuration, and implementation; web development with PHP, MySQL, and jQuery; and is support analyst.

Translated from the original in Portuguese by Lilian Nassi-Calò.