Scientometrics of peer-reviewers – will they be finally recognized?

Since bibliometrics and scientometrics began to be studied as disciplines in their own right, concerns have surfaced about how to measure and evaluate the role of the peer-reviewer as well as the expressions of thanks and acknowledgements which are all part of the process which goes towards the preparation of a paper which is submitted for publication in a learned journal.

On 7^th April, it was announced that ORCID (Open Researcher and Contributor ID) had commissioned CASRAI (Consortia Advancing Standards in Research Administration Information) to undertake a project to establish a standard methodology for acknowledging the work of peer-reviewers in the form of persistent identifiers which will allow retrieval and measurement. But before describing the project in detail, it is necessary to present the key players and outline the importance of this project.

Key Players

ORCID is a not-for-profit institution whose objective is the creation and maintenance of an international database of researchers in all subject areas, assigning them each a unique digital identifier. In this way it is possible to link research outputs with authors, grant agencies and the professional activities undertaken by authors. The ORCID identifier, in the same way as the ResearchID, can be integrated into a Web of Science¹ record, and many scientific journals are beginning to request that authors register with ORCID and include their personal identifier when submitting papers for publication.

CASRAI (Consortia Advancing Standards in Research Administration Information) is a not-for-profit organization established in Ontario, Canada, whose goal is the standardization of the data that researchers and funding institutions produce, with the objective of facilitating the interchange of information throughout the life-cycle of research activities.

The Project

The project owes its origin to the age-old problem of how to acknowledge peer-review activities in the ORCID database and how the data should also incorporate grant applications and workflows that occur as papers submitted for publication go through the peer-review process. To address this need, ORCID requested that CASRAI develop a standardization project, and this led to the creation of a working group which invited various members representing learned societies, publishers and international funding agencies. This working group must deliver a draft document for external assessment this coming June.

Among the major issues to be resolved by the working group are the following: (i) the evaluation of technology suitable for the identification and definition of the data elements which allow acknowledgement, (ii) the identification of options for the assignment of a persistent document identifier for the revision per se, and (iii) recommendations for the metadata structure that the citation should have.

These discussions are important because the peer-review community has always been concerned about how its role and the impact on professional development of the service it provides can be better recognized. A key problem requiring resolution is what should be the best way to accommodate the wide variety of peer-review activities, such as reviewing a conventional journal article, a grant request for a project, etc., and then how to register these activities under a common citation standard. Moreover, information relating to this activity should be registered by means of a persistent identifier which will allow the peer-reviewer to be linked with the documents worked on. In the future, this will permit the retrieval and compilation of scientometric or altmetric statistics, or similar ones by combining different variables with the names of the peer-reviewers.

F1000Research is an open access journal that is jointly managing the project with ORCID and which has begun to include the reviewer identifier in the articles published on its site. It should also be noted that the review process is completely open, with the names and reports of the peer-reviewers being published along with the article. The reports of the peer-reviewers are also assigned a DOI (Digital Object Identifier) so that they receive the credit they deserve for their work.

Source: http://blog.f1000research.com/2014/04/07/project-to-include-referee-reports-in-orcid-profiles/

The importance of the project

Peer-review demands many hours of work that, by and large, is not publicly recognized. The quality of the peer-review service is fundamental to the process of research, especially as it relates to establishing reliability in scholarly communication. As Alice Meados states (2014), this is because in the last decade the scientific community, which has been witness to rapid changes with no end in sight, has not changed its perception of trust in the peer-review procedures, regardless of whether the journals are open access or commercial, or if the peer-review is double blind or open, if altmetric values are complied, or if plagiarism detection software is used.

A recent study undertaken by the University of Tennessee² concludes that peer-review journals continue to be the most reliable and preferred vehicle for scholarly communication within the research community. The supposed lack of peer-review was one of the arguments of the researchers surveyed, who stated not wanting to publish in open access journals unless there was a rigorous peer-review process.

Increasingly, the community asks itself how this work of peer-review can be better recognized, both to encourage the participation of reviewers and to evaluate impact on professional development. And it is with these long standing requirements that the working group organized by CASRAI will have many things to consider.

If the ORCID-CASRAI project is successful, then the important work of peer-review, which is almost always anonymous, with no recognition other than being included in the lists that the journals publish every year or two, will now be able to be included in the lists of acknowledgments.

But to be successful, as stated by Pablo de Castro y Tomás Baiget³ in the INCYT list, the project must have sufficient application and acceptance. It will require the collaboration of the stakeholders, that is, institutions, publishers and companies, regardless of whether publications are open access or commercial. Finally, the great challenge for ORCID is to get authors to systematically maintain the information that will be collected from their accounts, and if this work will be done in a synchronized and automatic manner with other institutional systems.

As reported to us, the SciELO Program will incorporate a similar initiative in the course of this year. It would be good if the journals published in the SciELO network were to begin asking authors and reviewers to register in ORCID and include this data in the papers that they send for publication.

Notes

¹ ResearchID and ORCID integration – http://wokinfo.com/researcherid/integration/

² Trust an authority in scholarly communications in the light of the digital transition: Final report University of Tennessee and CIBER Research Ltd. December 2013. http://ciber-research.eu/download/20140115-Trust_Final_Report.pdf

³ Buscan poder reconocer el trabajo de los evaluadores. 9 de abril 2014. Comunicaciones de Pablo de Castro y Tomás Baiget en la lista de Indicadores en Ciencia y Tecnología http://listserv.rediris.es/cgi-bin/wa?A0=INCYT

References

LAW, A. ORCID & CASRAI Kick-off New Standards Project on ‘Peer Review Services’. CASRAI. 7 April 2014. Available from: http://casrai.org/about/announcements/orcid-%26-casrai-kick-off-new-standards-project-on-%E2%80%98peer-review-services%E2%80%99

HAAK, L. Acknowledging Peer Review Activities. ORCID. 8 April 2014. Available from: http://orcid.org/blog/2014/04/08/orcid-and-casrai-acknowledging-peer-review-activities

LAWRENCE, R. Project to include referee reports in ORCID profiles. F1000Research. 7 April 2014. Available from: http://blog.f1000research.com/2014/04/07/project-to-include-referee-reports-in-orcid-profiles/

MEADOWS, A. In (Digital) Scholarly Communications We Trust?. the scholarly kitchen. 7 April 2014. Available from: http://scholarlykitchen.sspnet.org/2014/04/07/in-digital-scholarly-communications-we-trust/?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3 A+ScholarlyKitchen+%28The+Scholarly+Kitchen%29

External Links

ORCID – http://orcid.org/

CASRAI – http://casrai.org/about

About Ernesto Spinak

Collaborator on the SciELO program, a Systems Engineer with a Bachelor’s degree in Library Science, and a Diploma of Advanced Studies from the Universitat Oberta de Catalunya (Barcelona, Spain) and a Master’s in “Sociedad de la Información” (Information Society) from the same university. Currently has a consulting company that provides services in information projects to 14 government institutions and universities in Uruguay.

Translated from the original in Spanish by Nicholas Cop Consulting.