Euan Adie
The emergence of a new concept usually generates many questions – and this is the case of Altmetrics. What does the term Altmetrics mean? Do these new metrics aim to supplant or just complement the classical bibliometric indicators that are based on citations? What is it that changes the way we understand the impact of research on academia and on society? How are these new metrics used? How is the concern or the need to be selective in the makeup of these metrics expressed? What controls and/or corrective mechanisms does altmetrics use to address manipulations of these metrics? How are these metrics perceived by systems of research evaluation at institutions and research funding agencies?
In this exclusive interview, Euan Adie, CEO of the company altmetric.com, replies to these and other questions for the public on the blog SciELO in Perspective, a blog committed to bringing to its readers discussions that are taking place in the world relating to new ways of evaluating research, and to opening up new horizons for its impact and scope.
1. To begin with, can you tell us about the emergence of the concept of Altmetrics , and why this particular name?
Sure! The ‘alt’ refers to alternative: an alternative to only counting citations for use as metrics. Altmetrics are essentially about taking a broader view of outputs and impacts.
The outputs part refers to getting credit for things like speaking at a conference, or contributing to a database, or writing software. The impact part is about recognizing that scholarly research influences different groups in all sorts of ways that aren’t captured by citations alone.
There are lots of other potentially interesting data sources, like news outlet or social media network mentions, blogs, reference managers, policy documents and download counts: why aren’t we as producers & consumers of research making better use of them?
A specific type of altmetrics is article level metrics: these are metrics for one type of output (articles) and the kind of data you might see on publisher sites.
2. Does it deal with alternative metrics which aim to supplant or just complement classic citation based bibliometric metrics?
That’s a good question. Citations are still a very important measure – and altmetrics are complementary to rather than a replacement for them – but they only measure one thing, which is that another scholar found the article useful in their own work. This misses other forms of impact, be they clinical, economic or otherwise public.
That said, within the altmetrics community there’s broad support for suggesting that altmetrics could be one alternative to using the impact factor (IF) in areas that the IF isn’t suited for (and there are a lot of these).
One key thing to note is that altmetrics don’t measure quality, in the same way that citations don’t measure quality. People can cite your paper because it’s terrible. If you need to know how good a paper is then you should be reading it! Or getting experts to read it for you: databases like F10001 .com or the Cochrane Collection2 are interesting in this respect.
3. The following statement made by Ian Mulvany of eLife3 concerning the Altmetric Explorer developed by your company appears on the altmetric.com website :
“Altmetric Explorer has the potential to change the way we understand the impact that research has within academia and more importantly the wider impact that it has in society.”
Could you comment on the way our understanding of the impact that research has within academia and in society will change?
Yes, this is less about the numbers and more about just surfacing information that funders and authors might never otherwise be exposed to.
Say I write a paper about the best ways to care for autistic children in a paediatric hospital ward. That might have an impact on the way paediatricians and nurses practice and improve the quality of care for hundreds of children: but I’d never know this just from citation counts. I might research a rare disease; my peer group is small, but it has an important impact on people and families dealing with that disease. I could have written a paper that gets a lot of interest from certain parts of the public – we see this a lot with, say, the interest in Japan in research on the effects of ionizing radiation on health (after Fukushima…).
Altmetrics doesn’t guarantee that you’ll see evidence of any of the above either. It does allow the possibility though, while citations don’t – they are only ever going to reflect impact amongst people who write papers. We should be open to evidence that a paper has reached the right audience wherever it comes from, be it a Reddit4 thread, a nurse’s blog or a magazine story.
The tricky bit is judging what is meaningful & important and what’s not.
4. The altmetric.com services which operate at the article level can be acquired by individuals. What uses are researchers making of these metrics? Do you think that Altmetrics will find their way into researchers’ resumés?
Yes! Though I think as anecdotal evidence rather than as hard metrics for now – you might discover and note on your CV that your papers have been picked up by x news outlets including the NY Times, y blogs including the one best known in your field etc.
We see people tweeting about the relative score (in the top x% of articles by attention) quite a lot too.
Not so much “I was tweeted twenty times!” or “I have an Altmetric score of fourteen!” – these don’t mean as much. It’s important for people to have context for any numbers.
Most of the uses we see are from authors looking at their own papers rather than readers using altmetrics for, say, context or discovery.
5. Can article level metrics be extended to be applied at the journal level? What is your experience? Are you aware of any studies which look at the correlation between Altmetrics and the journal Impact Factor? Are you considering working with Altmetrics at the journal level?
We’ve been asked about it a few times but I think it’s something that we need to see more demand for from end users. With an article it’s fairly easy to drill down into the data and get a feel for why it has been getting attention; it’s much more difficult to do this for a journal so you’d have to take the numbers at face value more which isn’t necessarily a good thing.
I haven’t seen any formal studies correlating altmetrics and the IF but if you look at the data by eye the top journals by altmetrics are similar to the top journals by IF: Nature, Science, The Lancet and so on. This is partly because these journals have a much wider reach than other titles and so the work is exposed to more people, naturally it’s then going to get more attention.
The differences are interesting. We see a lot of attention around PLoS One, and around preprints in some areas (notably physics, where there’s a lot of buzz around articles on arXiv)
6. As far as your company altmetric.com is concerned, does the harvesting of article related data rely on having the agreement of the publishers to do this? How does altmetric.com ensure that all the conversations relating to a particular article have been identified?
We certainly miss things. One of the biases we’re currently struggling with is that we pick up a lot of information from Europe, North America and Japan, but not much from South America or the rest of Asia. We get a lot of feedback from authors telling us that we’ve missed a particular blog post or page, this is really helpful as it improves the service for everybody.
7. Bibliographic citation based metrics are published by international multidisciplinary indexes such as WoS, Scopus, Google Scholar and, regionally, by SciELO. These indexes limit journal coverage because of the selection criteria that they have established. In the case of altmetric.com and Altmetrics in general, in what way is there a concern or need for selectivity?
We’ll happily track most things, but the selectivity comes when we’re choosing what to use for the context calculations or show to end users. Our concerns here are around publishers going rogue and suddenly trying to game the system to make their journals look good.
Certainly some criteria like age don’t apply. We track journals that’ve just launched – there’s no need to wait for eighteen months to see the first citations come in.
8. Do you know if Altmetrics are being used in research evaluation systems by institutions or by the agencies which fund research?
Sure, we’ve spoken to many and have a couple as customers. In each case it’s early days and I don’t think anywhere is looking at a broad spectrum of altmetrics systematically (though some may be using, say, F1000 reports).
The type of use we hear about is centered more around the actual collected data – the tweets, the news stories etc. – rather than just the numbers. For example, a funder may look at tweets from a government department or NGOs as evidence that an article they’ve funded has reached the correct audience.
9. Some bibliometric indicators such as the Impact Factor are subject to manipulations to increase their value. Some of these manipulations are quite acceptable whereas others range from dubious to completely unacceptable. What control and /or correction mechanisms do Altmetrics have? To face the social bots, for example?
We get asked this a lot! And rightly so. We see these kinds of manipulations all the time, but interestingly they’re not usually gaming per se, it’s not being done to inflate scores or anybody’s standing. Instead it’s a consequence of how spammers work on the web or on social networks: to make accounts look legitimate they’ll replicate content from random accounts and sometimes those accounts belong to scientists.
We sometimes hear criticism noting that you could ask ten friends to tweet about your paper to get better altmetrics – we’re not all that concerned with this kind of thing, as far as I’m concerned that’s legitimate attention.
Beyond that scale we (and the rest of the altmetrics community) take things more seriously. We use a combination of automated and manual approaches to flag up and then deal with manipulation. One of the good things about having a large database and spectrum of different data sources is that we can build up a good view of what kinds of attention patterns are to be expected and which are unusual in some way.
10. A paper published in the August 9th issue of Science concludes that rating systems are socially influenced in the sense that new ratings are influenced by previous ones, so related metrics may be biased. What is your opinion of this report?
I think it’s a great study, and I suspect the finding holds true in lots of other areas too. It is just looking at Reddit (or something very much like Reddit) though, so it’s difficult to generalise from.
The motivation to rate varies greatly from site to site, indeed even within the same site. Sometimes ratings are public, sometimes private. Sometimes users strongly identify with other people or groups on the site, sometimes they don’t.
On Reddit I’m maybe voting to indicate that I found a link interesting. On Twitter if I retweet somebody it might be because I want it shared more widely, or because it makes me look good somehow (people tweet things they haven’t read, just as they cite papers they haven’t read). If I star something it could be because I liked it, but also because I want to save it to read later, or to give whoever wrote it a virtual pat on the back.
This is the strength and the weakness of altmetrics… on the one hand you’ve got a lot of different measures to pick from and they’re indicative of a broad array of different types of attention. The downside is that you need so much information about where the data came from and what it means. This is partly why we stick to measuring attention – which is fairly uncontroversial and easily understood – rather than quality.
Notes:
1 Faculty of 1000 (F1000): online publication launched in 2002 to evaluate the quality of academic articles in Biomedicine based on the opinions of experts- http://f1000.com/.
2 Cochrane Library: collection of systematic reviews of controlled randomized trials of medical interventions with the objective of disseminating the resutls and conclusions –http://cochrane.bireme.br/portal/php/index.php.
3 eLife: Open Access, peer-reviewed academic journal in Biomedicine and Life Sciences – http://www.elifesciences.org/.
4 Reddit: Social news website where users submit and vote on links to content on the Web. – http://www.pt.reddit.com/.
Related Posts:
Como citar este post [ISO 690/2010]:
Pingback: News Roundup: Week of 2 September 2013 | Altmetric.com