{"id":2077,"date":"2015-11-25T16:13:40","date_gmt":"2015-11-25T18:13:40","guid":{"rendered":"http:\/\/blog.scielo.org\/en\/?p=2077"},"modified":"2015-11-25T16:19:30","modified_gmt":"2015-11-25T18:19:30","slug":"how-to-assess-research-proposals","status":"publish","type":"post","link":"https:\/\/blog.scielo.org\/en\/2015\/11\/25\/how-to-assess-research-proposals\/","title":{"rendered":"How to assess research proposals?"},"content":{"rendered":"

By Lilian Nassi-Cal\u00f2<\/strong><\/p>\n

\"Photo:<\/a>

Photo: Oliver Tacke<\/a>.<\/p><\/div>\n

The peer review of research proposals (grants) aims to judge the merit of projects and researchers and enable the best to be contemplated. The high number of candidates and proposals, however, has caused saturation of the reviewers, who find themselves immersed in increasing numbers of projects, not knowing the best way to assess them.<\/p>\n

In a post previously published in this blog, the possibility of making reviews on grant proposals openly available has been discussed, as a way to help researchers devise better proposals, while allowing public recognition of referees and helping to prevent fraud in the appraisal process. This alternative comes from the successful experience of journals which made peer-reviewers comments openly available along with the published article.<\/p>\n

Recently, Ewan Birney, director of the European Molecular Biology Laboratory’s European Bioinformatics Institute at Hinxton, UK, asked his Twitter followers for practical suggestions on how to identify the best candidates from hundreds of research grants submissions received by his institution. To his surprise, the scientific community responded enthusiastically with many suggestions which, in turn, led to other comments on Twitter. The experience was reported in Nature1<\/sup>, which is also receiving comments on its page.<\/p>\n

Birney2<\/sup> started the debate on Twitter asking about a proxy for quality other than the journal title to assess the candidates\u2019 competence, whose articles combined added up to about 2,500 overall. Yoav Gilad3<\/sup>, a geneticist at the University of Chicago, IL, US, advised him to read the 2,500 abstracts or the papers, even if it meant including more referees in the assessment process. Birney said that he considered it not feasible, although correct. Birney thinks, like many, that the journal\u2019s title or its Impact Factor (IF) does not necessarily reflect the individual quality of the papers. Moreover, his task is even more difficult, because it includes assessing proposals that do not fall exactly within his area of \u200b\u200bexpertise. “Of course, even if I was using journal as proxy here it wouldn\u2019t help me – everyone here has published \u2018well\u2019\u201d.<\/p>\n

The discussion continues on Twitter with a suggestion from Stephen Curry4<\/sup>, a structural biologist at the Imperial College in London about asking candidates to identify their four most relevant publications and justify their choices in a single page report. Richard Sever5<\/sup>, co-founder of the Cold Spring Harbor Laboratory (CSHL) bioRxiv biomedical articles\u2019 repository and assistant-director of CSHL Press considered it a good idea, pointing out, however, that this method could actually select candidates good at writing one page summaries.<\/p>\n

The biggest concern, according to Birney, in using citation based metrics, as suggested by many researchers, lies in the fact that they vary considerably between disciplines and may not be comparable in a heterogeneous sample. Hugo Hilton6<\/sup>, an immunologist at the University of Stanford at CA, US, expressed his concern, as a candidate, that the selection processes are subject to not totally clear criteria and classic biases as the prestige of journals where applicants publish. It is worth, at this point, mentioning the Declaration on Research Assessment (DORA) of 20127<\/sup>, in which members of the American Society for Cell Biology pledged not to use the IF to evaluate researchers in grant proposals, career promotions and hiring, precisely to avoid distortions. Up to now the Declaration was signed by over 150 prominent scientists and 80 academic organizations.<\/p>\n

Birney says that the referees should have a certain degree of autonomy to assess the proposals and there is no problem if all of them do not follow exactly the same procedures in their assessments. “I would prefer subjective but unbiased opinions, and five of them with different criteria than trying to unify the criteria so we all agree with the same answers.” However, he points out, transparency in the process is essential.<\/p>\n

Despite being aware of the problems in using journals prestige as a proxy for quality, Birney believes that its use is unavoidable due to the large volume of proposals and candidates. He also advises candidates to highlight their achievements clearly in the proposal, rather than just pinpoint journal titles from their publications list.<\/p>\n

The paper on Nature received several comments suggesting ways to speed up the evaluation process and come up with shortlists. It is also possible to registered users to submit their views on the topic8<\/sup>. Join the discussion you too!<\/p>\n

Notes<\/h3>\n

1. CHAWLA, D.S. How to judge scientists\u2019 strengths. Nature<\/em>. 2015, vol\u00ba 527, n\u00ba 279. DOI: 10.1038\/527279f<\/a><\/span><\/p>\n

2. Ewan Birney: http:\/\/twitter.com\/ewanbirney<\/a><\/p>\n

3. Yoav Gilad: http:\/\/twitter.com\/Y_Gilad<\/a><\/p>\n

4. Stephen Curry: http:\/\/twitter.com\/Stephen_Curry<\/a><\/p>\n

5. Richard Sever: http:\/\/twitter.com\/cshperspectives<\/a><\/p>\n

6. Hugo Hilton: http:\/\/twitter.com\/Hilton_HG<\/a><\/p>\n

7. SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Declaration recommends eliminate the use of Impact factor for research evaluation<\/em>. SciELO in Perspective. [viewed 22 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2013\/07\/16\/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation\/<\/a><\/p>\n

8. <http:\/\/www.nature.com\/foxtrot\/svc\/login?type=commenting<\/a>><\/p>\n

References<\/h3>\n

CHAWLA, D.S. How to judge scientists\u2019 strengths. Nature<\/em>. 2015, vol\u00ba 527, n\u00ba 279. DOI: 10.1038\/527279f<\/a><\/span><\/p>\n

MALHOTRA, V. and MARDER, E. Peer review: The pleasure of publishing \u2013 originally published in the journal eLife in January\/2015<\/em>. SciELO in Perspective. [viewed 21 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2015\/05\/11\/peer-review-the-pleasure-of-publishing-originally-published-in-the-journal-elife-in-january2015\/<\/a><\/p>\n

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Could grant proposal reviews be made available openly?.<\/em> SciELO in Perspective. [viewed 21 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2015\/03\/20\/could-grant-proposal-reviews-be-made-available-openly\/<\/a><\/p>\n

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Declaration recommends eliminate the use of Impact factor for research evaluation<\/em>. SciELO in Perspective. [viewed 22 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2013\/07\/16\/declaration-recommends-eliminate-the-use-of-impact-factor-for-research-evaluation\/<\/a><\/p>\n

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Paper proposes four pillars for scholarly communication to favor the speed and the quality of science<\/em>. SciELO in Perspective. [viewed 21 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2013\/07\/31\/paper-proposes-four-pillars-for-scholarly-communication-to-favor-the-speed-and-the-quality-of-science\/<\/a><\/p>\n

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Peer-review as a research topic in its own right<\/em>. SciELO in Perspective. [viewed 21 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2015\/04\/24\/peer-review-as-a-research-topic-in-its-own-right\/<\/a><\/p>\n

SCIENTIFIC ELECTRONIC LIBRARY ONLINE. Scientometrics of peer-reviewers \u2013 will they be finally recognized?<\/em>. SciELO in Perspective. [viewed 21 November 2015]. Available from: http:\/\/blog.scielo.org\/en\/2014\/05\/14\/scientometrics-of-peer-reviewers-will-they-be-finally-recognized\/<\/a><\/p>\n

External links<\/h3>\n

bioRxiv – <http:\/\/biorxiv.org\/<\/a>><\/p>\n

San Francisco Declaration on Research Assessment – <http:\/\/am.ascb.org\/dora\/<\/a>><\/p>\n

 <\/p>\n

\"lilian\"<\/a>About\u00a0Lilian Nassi-Cal\u00f2<\/h3>\n

Lilian Nassi-Cal\u00f2 studied chemistry at Instituto de Qu\u00edmica <\/em>\u2013 USP, holds a doctorate in Biochemistry by the same institution and a post-doctorate as an Alexander von Humboldt fellow in Wuerzburg, Germany. After her studies, she was a professor and researcher at IQ-USP. She also worked as an industrial chemist and presently she is Coordinator of Scientific Communication at BIREME\/PAHO\/WHO and a collaborator of SciELO.<\/p>\n

 <\/p>\n

Translated from the\u00a0original in portuguese<\/a>\u00a0by Lilian Nassi-Cal\u00f2.<\/p>\n","protected":false},"excerpt":{"rendered":"

The peer review of research proposals (grants) aims to judge the merit of projects and researchers and enable the best to be contemplated. The director of an institution in the United Kingdom shared on Twitter his struggle in evaluating the numerous proposals received and started a discussion forum from which ideas and suggestions emerged. …<\/span> Read More →<\/span><\/a><\/span><\/p>\n","protected":false},"author":22,"featured_media":2078,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[3],"tags":[10,41,31,32,7],"class_list":["post-2077","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analysis","tag-impact-factor","tag-public-policies","tag-research-evaluation","tag-research-policy","tag-scholarly-communication"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2077","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/users\/22"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/comments?post=2077"}],"version-history":[{"count":3,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2077\/revisions"}],"predecessor-version":[{"id":2081,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/posts\/2077\/revisions\/2081"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/media\/2078"}],"wp:attachment":[{"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/media?parent=2077"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/categories?post=2077"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.scielo.org\/en\/wp-json\/wp\/v2\/tags?post=2077"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}