Study analyzes the use of social networks in the assessment of scientific impact

By Lilian Nassi-Calò

Photo: Jason Howie.

In recent years, the use of social networks in science communication has been increasing on a large scale, and specific platforms have been created for interaction and information sharing among researchers. Despite the increasing interest of the academic community on social networks as a tool for scientific communication, little is known about the usage profile of these tools, and how traditional measures of scientific impact based on citations (offline indexes, impact off-line) correlate with the new impact measure (online index, impact online).

A paper presented at the 47^th Hawaii International Conference on System Sciences (HICSS) in 2014¹ by researchers from the University of St. Gallen in Switzerland examined whether and how scientific impact can be measured through social media data analysis, and how this approach is related to the traditional metrics. According to the authors, network centrality measures² based on analysis of social media had not been considered in the context of the impact assessment. The results of the exploratory work, performed at a single institution with a small number of researchers, indicate that these measures correlate with traditional impact metrics and can be used to complement them.

Social network sites (SNS) are defined as “Web-based services that allow individuals to (1) construct a public or semi-public profile within a bounded system; (2) articulate a list of other users with whom they share a connection, and (3) view and traverse their list of connections and those made by others within the system”. Examples of SNS are Academia.edu, ResearchGate, and Mendeley since their sites are directed to the scientific community and, in addition to the aforementioned functions, they also allow uploading and sharing articles, endorse the work of colleagues or find related literature.

Traditionally, scientific impact is measured by bibliographic metrics such as publications and citations in refereed journals. Among these are the Impact Factor (Web of Knowledge), the Scimago Journal Rank (Scopus, Elsevier), and the h-Index, the first two focusing on publication and the last on the individual researcher. These metrics incorporate numerous flaws and limitations, as has been widely debated in the scientific community, including in this blog. Citations measures effectively reflect the value of the work and its ability to pass through the editorial process and peer review. However, the impact of a publication also refers to the degree of influence it exerts and in this case, citations are only part of the measure of this influence among the scientific community and society. In certain disciplines, other forms of publication as books (in arts and humanities), reports and technical manuals (engineering), presentations and conference proceedings (mathematics, computer science) outweigh the journal articles, but are not detected by traditional bibliometrics. In addition, these metrics foster self-citation culture and citation cartels, neglecting its context, which is, how and why certain articles are cited.

Despite the limitations, the traditional metrics have advantages, too: they allow comparisons between journals, disciplines and institutions and are easy to calculate and understand. However, it is agreed that scientific impact cannot be measured only by citations. The authors demonstrate through this study that the emergence of the Internet as a space for scientific communication complements the analysis of scientific impact, enriching and diversifying its assessment.

There are currently under development alternative scientific impact metrics based on social media such as, for example, Altmetric, which appears very promising. Social networks store a lot of information and promote linkages between researcher communities and general stakeholders. The analysis of these interactions allows us to evaluate the impact of publications in a broader aspect than traditional metrics. Moreover, altmetrics and webometrics can be applied at the article, journal or individual researcher levels. Altmetric was adopted by SciELO Brazil to monitor the performance of articles on social networks.

The Altmetrics Manifesto makes a compilation of the goals and scope of this initiative. Its authors define impact as being formed by four pillars: use (access and download); peer-review (experts’ opinion); citations; and the altmetrics component (storage, links, bookmarks and sharing). The main advantage, they say, is the celerity with which an article is assessed by means of social media. Rather than waiting two years or more to count citations, in only one week sharing via Twitter, Facebook, LinkedIn and other SNS may provide the specific impact of an article, and not of the journal where it was published.

The study by Hoffmann and colleagues included 55 scholars of a public administration school at the University of St. Gallen, in Switzerland. The authors gained access to the ResearchGate profile of the researchers of this institution between September 2012 and February 2013 and evaluated three indicators in order to analyze the relationship between online (social media) and offline scientific impact (citations):

1. Seniority: formal position, honors, awards, participation in editorial boards.
2. Impact of publications: classical bibliometric measures as h-Index, journal Impact Factor, besides online impact on SNS, such as shares, downloads and bookmarks.
3. Centralization of the network: measure the extent that a researcher is connected to other members of the scientific community, an indication of its prominence and influence.

The results can be summarized as follows:

• When using ResearchGate to networking, researchers tend to follow more colleagues at their own institution rather than establishing new contacts;
• In scholars networks, assistant professors occupy more central positions (i.e., establishing connections with more colleagues), followed by full professors, doctors and post-docs;
• The online activity (online communication) of scholars strongly correlates with measures of centrality, but not with offline impact or seniority;
• Online and offline impact measures are strongly related. However, while off-line measures are related to seniority, online impact is related to the centrality of positions that actors assume in the network.
• Seniority is correlated with measures of centrality, i.e. the offline social capital is also expressed by the online universe.

Performance evaluation of scientific research, institutions and journals until recently relied almost exclusively on bibliometric measures, mainly citations. Insofar as new technology tools have emerged, alternative metrics have been proposed to assess scientific impact. This study points out the limitations of traditional metrics such as disregarding aspects of the interactions between the actors of the networks, and the formation of social capital for individual members. As social media facilitate the analysis of the researchers’ networking, one would expect that measures of centrality of the network could provide data on evaluation of scientific impact.

This study led to the conclusion that the scholars at the Swiss University use social networks more in the Facebook than in the Twitter approach, i.e., they do not follow many of their colleagues, preferring to interact with their offline contact community, such as colleagues from the same institution, or collaborators. Thus, social networks are intended to corroborate, rather than to establish contacts.

As expected, scholars at the beginning of their careers – and therefore younger researchers – occupy the central positions of social networks and are more active in online communities, possibly because of their career projection desire and to establish strong collaborative networks, which may result in a greater impact in the future. Regarding the impact of publications, the almetrics score correlates with seniority, rather than with traditional offline metrics as h-Index.

This study contributes to the debate – just started – about scientific impact assessment and altmetrics. In this small sample, online impact correlates better with seniority than with offline impact, and online impact is closely related to the centrality of the network. Thus, social networks based metrics could help to elucidate the dynamics of online and offline scientific impact measures.

According to the manifesto’s authors “Altmetrics are in their early stages; many questions are unanswered. But given the crisis facing existing filters and the rapid evolution of scholarly communication, the speed, richness, and breadth of altmetrics make them worth investing in” (PRIEM et al 2010).

Notes

¹ HOFFMANN, C.P., LUTZ, C., and MECKEL, M. Impact Factor 2.0: Applying Social Network Analysis to Scientific Impact Assessment. In: 47th Hawaii International Conference on System Science, Hilton Waikoloa Village, 2014. DOI: 10.1109/HICSS.2014.202

² Centrality, in network analysis, refers to indicators which identify the most important vertices within a graph. Applications include identifying the most influential person(s) in a social network, key infrastructure nodes in the Internet or urban networks, among others.
Source: Centrality. Wikipedia. [viewed 26 January 2015]. Available from: http://en.wikipedia.org/wiki/Centrality

References

Altmetrics, Alternative metrics and Alternative measurements: new perspectives on the visibility and impact of scientific research. SciELO in Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2013/08/14/altmetrics-alternative-metrics-and-alternative-measurements-new-perspectives-on-the-visibility-and-impact-of-scientific-research/

Article downloads: An alternative indicator of national research impact and cross-sector knowledge exchange – Originally published on the Elsevier newsletter “Research Trends Issue 36″. SciELO in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/03/24/article-downloads-an-alternative-indicator-of-national-research-impact-and-cross-sector-knowledge-exchange-originally-published-on-the-elsevier-newsletter-reseatch-trends-issue-36/

BIK, H.M, and GOLSTEIN, M.C. An Introduction to Social Media for Scientists. PLoS Biol. 2013, vol. 11, nº 4. DOI: 10.1371/journal.pbio.1001535

BOYD, D., and ELLISON, N.B. Social Network Sites: Definition, History, and Scholarship. Journal of Computer-Mediated Communication. 2007, vol. 13, nº 1, pp. 210–230. DOI: 10.1111/j.1083-6101.2007.00393.x

Declaration recommends eliminate the use of Impact factor for research evaluation. SciELO in Perspective. [viewed 23 January 2015]. Available from: http://blog.scielo.org/en/2013/07/16/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation/

HOFFMANN, C.P., LUTZ, C., and MECKEL, M. Impact Factor 2.0: Applying Social Network Analysis to Scientific Impact Assessment. In: 47th Hawaii International Conference on System Science, Hilton Waikoloa Village, 2014. DOI: 10.1109/HICSS.2014.202

Interview with Euan Adie, CEO of altmetric.com. SciELO in Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2013/08/29/interview-with-euan-adie-ceo-of-altmetric-com/

Interview with Vincent Larivière. SciELO in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2013/08/16/interview-with-vincent-lariviere/

Paper investigates: is your most cited work your best work?. SciELO in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/11/24/paper-investigates-is-your-most-cited-work-your-best-work/

PRIEM, J. Scholarship: Beyond the paper. Nature. 2013, vol. 495, nº 7442, pp. 437–440. DOI: 10.1038/495437a

PRIEM, J., and et al. Altmetrics: a manifesto. Altmetrics.org, 2010, pp. 1–5. Available from: http://altmetrics.org/manifesto/

Rise of the Rest: The Growing Impact of Non-Elite Journals – Originally published on Google Scholar Blog on October 8, 2014. SciELO in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/10/13/rise-of-the-rest-the-growing-impact-of-non-elite-journals-originally-published-on-google-scholar-blog-on-october-8-2014/

Study proposes a taxonomy of motives to cite articles in scientific publications. SciELO in Perspective. [viewed 26 January 2015]. Available from: http://blog.scielo.org/en/2014/11/07/study-proposes-a-taxonomy-of-motives-to-cite-articles-in-scientific-publications/

What can alternative metrics – or altmetrics – offer us?. SciELO in Perspective. [viewed 31 January 2015]. Available from: http://blog.scielo.org/en/2014/08/07/what-can-alternative-metrics-or-altmetrics-offer-us/

External links

Academia.edu – <http://www.academia.edu>

Altmetric – <http://www.altmetric.com/>

Mendeley – <http://www.mendeley.com/>

ResearchGate – <http://www.researchgate.net/>

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.