What can alternative metrics – or altmetrics – offer us?

The prestigious Spanish journal El Profesional de la Información (EPI)¹, has just published in its July / August issue four articles dedicated to the topic of “alternative metrics”, which are also known as altmetrics.

EPI is a specialist journal in the field of information science which has been published since 1992 in Spanish and/or English. It falls into the category of a hybrid open access journal, and as far Spanish language journals are concerned, it is among the top journals as measured by the Impact Factor in both the Journal Citation Reports and Scopus. Three of the articles which we will look at in this post are available in open access.

The issue begins with an introduction written by Euan Adie, the founder of Altmetric.com. The next article presents the new possibilities offered by altmetrics for data exploration, and the following articles reflect on their possible use in evaluating research. This group of articles concludes with a case study which applies altmetrics to those high impact authors writing in the journal EPI itself.

Euan Adie², who provides the first article in this issue –“Taking the alternative mainstream”³–was one of the invited panelists at the SciELO 15 Years Conference, where he gave a presentation entitled “Altmetrics in practice”. In the same way as he did in the presentation that he gave in São Paulo, his journal article discusses the reasons forthe useof altmetrics to measure the influence that scientific works have on society, and takes the view that altmetrics are complementary to the traditional system of counting citations, since they can be used in much broader contexts. There are many types of impact measurement, not solely bibliometric indicators, since science has repercussions upon the economy, politics, research institutions and upon the public in general.

The main parties interested in these altmetrics are authors themselves, which makes sense, since it is to be hoped that they will want some indication as to whether what they have written has caught the attention of the public and to what extent. From a practical point of view, the editors of academic journals who provide these altmetrics are giving authors a personal service.

The challenge presented by this new tool is that altmetrics data is very transitory and difficult to collect and, at the present time, does not yet have clear interpretations. It lacks a generally accepted theoretical framework.

The following article “Altmétricas para la evaluación de la investigación y el análisis de necesidades de información“(Altmetrics: measuring the impact of research output and analyzing information requirements) addresses the topic that directly concerns us, namely just how good are the tools which quantify the presence of research on the social web as a complement to the more traditional forms of evaluation by its impact. The major problem is one of the nature and limits of human knowledge. It is also necessary to standardize the procedures for the collection of data in order to guarantee the consistency of altmetrics indicators so that they can be compared, and build a conceptual framework within which to interpret them.

The major altmetrics services providers are:

• altmetric.com from Macmillan Publishers; this is the major provider of altmetrics with more than 4,000 clients. This service is also used by SciELO for its Brazilian journal collection.
• Impactstory.org with the support of the Sloan Foundation.
• PlumAnalytics.com, recently acquired by Ebsco.
• There are also a number of other smaller providers in existence or in the process of being set-up.

Impactstory.org and Altmetric.com are in the process of merging with the objective of creating a supplier with a much greater coverage.

The “raw”data which is obtained can be interpreted in five levels or categories:

• Downloads as a measure of the dissemination of a particular article.
• The number of times a particular article is included in a social bookmark (CiteUlike, Mendeley) as a measure of interest generated.
• Comments in Twitter would be the greatest measure of interest in a particular article. They are immediate and transitory, but also of social interest.
• Interest in a particular article increases if it is recommended by specialist services, as for example the recommendations in the medical field on the site Faculty of 1000.
• Finally, the traditional way of measuring impact continues to be citation analysis.

Mendeley is one of the principle sources of altmetric data, broken down by discipline. But unlike systems where citations are cumulative, the coverage offered by Mendeley is variable since users modify their bibliographies and delete those they consider obsolete over time. There are still no longitudinal studies that analyze the coverage of altmetric data over time. Another problem with the numbers from Mendeley is that more than half of the users are doctoral students, with graduate students and researchers a distant second. This means that the numbers from Mendeley come from a relatively small and biased group in relation to the total academic community. The representation of Mendeley users goes down sharply in the social sciences, and to less than 30% in the humanities.

The data collection procedures of aggregators are not standardized, necessitating the establishment of standards such as those being produced by NISO (2013)⁴. Under the umbrella term of “altmetrics”, a wide variety of metrics and indicators are taken into consideration which try to measure the social dissemination of scientific research on the web. Qualitative studies are needed on the use of tools such as Twitter, Facebook and others which will help us to find out how and to what extent scientists use these social tools.

The third EPI article “New data, new possibilities: Exploring the insides of Altmetric.com” reaffirms concepts discussed earlier and presents an interesting table with the 16 principle sources of information which are used by altmetric.com to gather data. They are: Blogs, News, Reddit, Facebook, Google Plus, Pinterest, Twitter, Stack Exchange, CiteUlike, Connotea, Mendeley, F1000, YouTube, LinkedinGroups, Research Highlights, and Miscellaneous.

Altmetric.com asserts that it makes great efforts to resolve ambiguities; however the information is still far away from being exhaustive, precise and accurate. In any case, these techniques are applied only to sources in English, not to those in other languages such as Spanish. It is also important to note that social networks such as as academia.edu and ResearchGate, which are used by many researchers, are omitted. According to the EPI study being discussed here, five sources (Twitter, Mendeley, Facebook, CiteULike and blogs) account for 95% of the information gathered, leaving the remaining sources with a marginal presence.

The final altmetrics article in EPI “Presencia en redes sociales y altmétricas de los principales autores de la revista El Profesional de la Información” (Presence in social networks, and altmetrics of major authors publishing in the journal El Profesional de la Información), analyzes the presence and impact of the 47 most published authors in EPI during the period 2009-2013, and their public profiles in Google Scholar, Linkedin and Mendeley. It confirms the real difficulties involved in calculating altmetrics not only at the author level but also at the article level, without taking into account the problem of detecting affiliations, the lack of title standardization, and so on.

My thoughts

The tools of altmetrics are in an experimental phase, and which have two issues yet to be resolved. The first one is technical, referring to collection of data that has broader coverage, in various languages, and the ability to resolve ambiguities. This will be resolved once there are accepted international standards, in the same manner that other metrics such as Counter for the measurement of the use of electronic resources were accepted.

The second problem is more complex and has to do with epistemology, that is, what theoretical base can be established to interpret the data obtained by computer algorithms, and to deduce, from that point on, consistent, credible and innovative indicators. If we analyze the history of the development of classic bibliometrics and scientometrics, we will see that it took close to 20 years of theoretical discussions by eminent specialists to arrive at a consensual set of tools and interpretations. In spite of this, the Impact Factor, the most famous indicator, still raises suspicions and resistance from the academic community.

Notes

¹ El Profesional de la Información. 2014, vol. 23, nº 4, July-August. Available from: http://elprofesionaldelainformacion.metapress.com/app/home/issue.asp?referrer=parent&backto=journal,1,96;homemainpublications,1,1;

² Euan Adie – http://www.scielo15.org/euan-adie/

³ Taking the alternative mainstream –http://elprofesionaldelainformacion.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,1,12;journal,1,96;homemainpublications,1,1

⁴ NISO to Develop Standards and Recommended Practices for Altmetrics. 20 junio 2013 – http://www.niso.org/news/pr/view?item_key=72efc1097d4caf7b7b5bdf9c54a165818399ec86

References

ADIE, E. Taking the alternative mainstream. EPI. 2014, vol. 23, nº 4, pp. 349-351. Available from: http://elprofesionaldelainformacion.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,1,12;journal,1,96;homemainpublications,1,1;

BORREGO, A. Altmétricas para la evaluación de la investigación y el análisis de necesidades de información. EPI. 2014, vol. 23, nº 4, pp. 352-358. Available from: http://elprofesionaldelainformacion.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,2,12;journal,1,96;homemainpublications,1,1;

GARCÍA, NR., et al. New data, new possibilities: Exploring the insides of Altmetric.com. EPI. 2014, vol. 23, nº 4, pp. 359-366. Available from: http://elprofesionaldelainformacion.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,3,12;journal,1,96;homemainpublications,1,1;

SALINAS, DT., and GUISADO, YM. Presencia en redes sociales y altmétricas de los principales autores de la revista El profesional de la información. EPI. 2014, vol. 23, nº 4, pp. 367-372. Available from: http://elprofesionaldelainformacion.metapress.com/app/home/contribution.asp?referrer=parent&backto=issue,4,12;journal,1,96;homemainpublications,1,1;

External link

Faculty of 1000 – http://f1000.com/

Counter. http://www.projectcounter.org/

About Ernesto Spinak

Collaborator on the SciELO program, a Systems Engineer with a Bachelor’s degree in Library Science, and a Diploma of Advanced Studies from the Universitat Oberta de Catalunya (Barcelona, Spain) and a Master’s in “Sociedad de la Información” (Information Society) from the same university. Currently has a consulting company that provides services in information projects to 14 government institutions and universities in Uruguay.

Translated from the original in Spanish by Nicholas Cop Consulting.