How to assess research proposals?

By Lilian Nassi-Calò

Photo: Oliver Tacke.

The peer review of research proposals (grants) aims to judge the merit of projects and researchers and enable the best to be contemplated. The high number of candidates and proposals, however, has caused saturation of the reviewers, who find themselves immersed in increasing numbers of projects, not knowing the best way to assess them.

In a post previously published in this blog, the possibility of making reviews on grant proposals openly available has been discussed, as a way to help researchers devise better proposals, while allowing public recognition of referees and helping to prevent fraud in the appraisal process. This alternative comes from the successful experience of journals which made peer-reviewers comments openly available along with the published article.

Recently, Ewan Birney, director of the European Molecular Biology Laboratory’s European Bioinformatics Institute at Hinxton, UK, asked his Twitter followers for practical suggestions on how to identify the best candidates from hundreds of research grants submissions received by his institution. To his surprise, the scientific community responded enthusiastically with many suggestions which, in turn, led to other comments on Twitter. The experience was reported in Nature¹, which is also receiving comments on its page.

Birney² started the debate on Twitter asking about a proxy for quality other than the journal title to assess the candidates’ competence, whose articles combined added up to about 2,500 overall. Yoav Gilad³, a geneticist at the University of Chicago, IL, US, advised him to read the 2,500 abstracts or the papers, even if it meant including more referees in the assessment process. Birney said that he considered it not feasible, although correct. Birney thinks, like many, that the journal’s title or its Impact Factor (IF) does not necessarily reflect the individual quality of the papers. Moreover, his task is even more difficult, because it includes assessing proposals that do not fall exactly within his area of ​​expertise. “Of course, even if I was using journal as proxy here it wouldn’t help me – everyone here has published ‘well’”.

The discussion continues on Twitter with a suggestion from Stephen Curry⁴, a structural biologist at the Imperial College in London about asking candidates to identify their four most relevant publications and justify their choices in a single page report. Richard Sever⁵, co-founder of the Cold Spring Harbor Laboratory (CSHL) bioRxiv biomedical articles’ repository and assistant-director of CSHL Press considered it a good idea, pointing out, however, that this method could actually select candidates good at writing one page summaries.

The biggest concern, according to Birney, in using citation based metrics, as suggested by many researchers, lies in the fact that they vary considerably between disciplines and may not be comparable in a heterogeneous sample. Hugo Hilton⁶, an immunologist at the University of Stanford at CA, US, expressed his concern, as a candidate, that the selection processes are subject to not totally clear criteria and classic biases as the prestige of journals where applicants publish. It is worth, at this point, mentioning the Declaration on Research Assessment (DORA) of 2012⁷, in which members of the American Society for Cell Biology pledged not to use the IF to evaluate researchers in grant proposals, career promotions and hiring, precisely to avoid distortions. Up to now the Declaration was signed by over 150 prominent scientists and 80 academic organizations.

Birney says that the referees should have a certain degree of autonomy to assess the proposals and there is no problem if all of them do not follow exactly the same procedures in their assessments. “I would prefer subjective but unbiased opinions, and five of them with different criteria than trying to unify the criteria so we all agree with the same answers.” However, he points out, transparency in the process is essential.

Despite being aware of the problems in using journals prestige as a proxy for quality, Birney believes that its use is unavoidable due to the large volume of proposals and candidates. He also advises candidates to highlight their achievements clearly in the proposal, rather than just pinpoint journal titles from their publications list.

The paper on Nature received several comments suggesting ways to speed up the evaluation process and come up with shortlists. It is also possible to registered users to submit their views on the topic⁸. Join the discussion you too!

Notes

1. CHAWLA, D.S. How to judge scientists’ strengths. Nature. 2015, volº 527, nº 279. DOI: 10.1038/527279f

2. Ewan Birney: http://twitter.com/ewanbirney

3. Yoav Gilad: http://twitter.com/Y_Gilad

4. Stephen Curry: http://twitter.com/Stephen_Curry

5. Richard Sever: http://twitter.com/cshperspectives

6. Hugo Hilton: http://twitter.com/Hilton_HG

7. SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Declaration recommends eliminate the use of Impact factor for research evaluation. SciELO in Perspective. [viewed 22 November 2015]. Available from: http://blog.scielo.org/en/2013/07/16/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation/

8. <http://www.nature.com/foxtrot/svc/login?type=commenting>

References

CHAWLA, D.S. How to judge scientists’ strengths. Nature. 2015, volº 527, nº 279. DOI: 10.1038/527279f

MALHOTRA, V. and MARDER, E. Peer review: The pleasure of publishing – originally published in the journal eLife in January/2015. SciELO in Perspective. [viewed 21 November 2015]. Available from: http://blog.scielo.org/en/2015/05/11/peer-review-the-pleasure-of-publishing-originally-published-in-the-journal-elife-in-january2015/

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Could grant proposal reviews be made available openly?. SciELO in Perspective. [viewed 21 November 2015]. Available from: http://blog.scielo.org/en/2015/03/20/could-grant-proposal-reviews-be-made-available-openly/

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Declaration recommends eliminate the use of Impact factor for research evaluation. SciELO in Perspective. [viewed 22 November 2015]. Available from: http://blog.scielo.org/en/2013/07/16/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation/

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Paper proposes four pillars for scholarly communication to favor the speed and the quality of science. SciELO in Perspective. [viewed 21 November 2015]. Available from: http://blog.scielo.org/en/2013/07/31/paper-proposes-four-pillars-for-scholarly-communication-to-favor-the-speed-and-the-quality-of-science/

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Peer-review as a research topic in its own right. SciELO in Perspective. [viewed 21 November 2015]. Available from: http://blog.scielo.org/en/2015/04/24/peer-review-as-a-research-topic-in-its-own-right/

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Scientometrics of peer-reviewers – will they be finally recognized?. SciELO in Perspective. [viewed 21 November 2015]. Available from: http://blog.scielo.org/en/2014/05/14/scientometrics-of-peer-reviewers-will-they-be-finally-recognized/

External links

bioRxiv – <http://biorxiv.org/>

San Francisco Declaration on Research Assessment – <http://am.ascb.org/dora/>

About Lilian Nassi-Calò

Lilian Nassi-Calò studied chemistry at Instituto de Química – USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME/PAHO/WHO and a collaborator of SciELO.

Translated from the original in portuguese by Lilian Nassi-Calò.