{"id":2455,"date":"2017-01-16T15:50:35","date_gmt":"2017-01-16T17:50:35","guid":{"rendered":"http:\/\/blog.scielo.org\/en\/?p=2455"},"modified":"2017-04-12T10:31:50","modified_gmt":"2017-04-12T13:31:50","slug":"five-things-to-consider-when-designing-a-policy-to-measure-research-impact-originally-published-in-the-conversation","status":"publish","type":"post","link":"https:\/\/blog.scielo.org\/en\/2017\/01\/16\/five-things-to-consider-when-designing-a-policy-to-measure-research-impact-originally-published-in-the-conversation\/","title":{"rendered":"Five things to consider when designing a policy to measure research impact [Originally published in The Conversation]"},"content":{"rendered":"

By Andrew Gunn, Researcher in Higher Education Policy, University of Leeds and Michael Mintrom, Professor of Public Sector Management, Monash University<\/strong><\/p>\n

\"the_conversation_logo\"<\/a>This year will see the Australian government\u00a0pilot1<\/sup>\u00a0new ways to measure the impact of university research.<\/p>\n

As recommended by the\u00a0Watt Review2<\/sup>, the Engagement and Impact Assessment will encourage universities to ensure academic research produces wider economic and social benefits.<\/p>\n

This fits into the\u00a0National Innovation and Science Agenda3<\/sup>, in which taxpayer funds are targeted at research that will have a beneficial future impact on society.<\/p>\n

Education Minister\u00a0Simon Birmingham said4<\/sup>\u00a0the pilots will test<\/p>\n

\u201chow to measure the value of research against things that mean something, rather than only allocating funding to researchers who spend their time trying to get published in journals\u201d.<\/em><\/p>\n

This move to measure the non-academic impact of research introduces many new challenges that were not previously relevant when evaluation focused solely on academic merit.\u00a0New research5<\/sup>\u00a0highlights some of the key issues that need to be addressed when deciding how to measure impact.<\/p>\n

1. What should be the object of measurement?<\/h3>\n

Research impact evaluations needs to trace out a connection between academic research and \u201creal world\u201d impact beyond the university campus. These connections are enormously diverse and specific to a given context. They are therefore best captured through case studies.<\/p>\n

When analysing a case study the main issues are: what counts as impact, and what evidence is needed to prove it? When considering this, Australian policymakers can use recent\u00a0European examples6<\/sup>\u00a0as a benchmark.<\/p>\n

For instance, in the UK\u2019s\u00a0Research Excellence Framework\u00a0(REF) \u2013 which assesses the quality of academic research \u2013 the only impacts that can be counted are those directly flowing from academic research submitted to the same REF exercise.<\/p>\n

To confirm the impact, the beneficiaries of research (such as policymakers and practitioners) are required to provide written evidence. This creates a narrow definition of impact because those that cannot be verified, or are not based on submitted research outputs, do not count.<\/p>\n

This has been a cause of frustration for some UK researchers, but the high threshold does ensure the impacts are genuine and flow from high quality research.<\/p>\n

2. What should be the timeframe?<\/h3>\n

There are unpredictable time lapses between academic work being undertaken and it having impact. Some research may be quickly absorbed and applied, whereas other impacts, particularly those from basic research, can take decades to emerge.<\/p>\n

For example,\u00a0a study looking at time lags in health research7<\/sup>\u00a0found the time lag from research to practice to be on average 17 years. It should be noted, though, that time lapses vary considerably by discipline.<\/p>\n

Only in hindsight can the value of some research be fully appreciated. Research impact assessment exercises therefore need to be set to a particular timeframe.<\/p>\n

Here, policymakers can learn from previous trials such as one conducted by\u00a0Australian Technology Network and Group of Eight in 20128<\/sup>. This exercise allowed impacts related to research that occurred during the previous 15 years.<\/p>\n

3. Who should be the assessors?<\/h3>\n

It is a long established convention that academic excellence is decided by academic peers. Evaluations of research are typically undertaken by panels of academics.<\/p>\n

However, if these evaluations are extended to include non-academic impact, does this mean there is now a need to include the views of end-users of research? This may mean the voices of people outside of academia need to be involved in the evaluation of academic research.<\/p>\n

In the 2014 UK REF, over 250 \u201cresearch users\u201d (individuals from the private, public or charitable sectors) were recruited to take part in the evaluation process. However, their involvement was restricted to assessing the impact component of the exercise.<\/p>\n

This option is an effective compromise between maintaining the principle of academic peer review of research quality while also including end-users in the assessment of impact.<\/p>\n

4. What about controversial impacts?<\/h3>\n

In many instances the impact of academic research on the wider world is a positive one. But there are some impacts that are controversial – such as fracking, genetically modified crops, nanotechnologies in food, and stem cell research – and need to be carefully considered.<\/p>\n

Such research may have considerable impact, but in ways that make it difficult to establish a consensus on how scientific progress impacts \u201cthe public good\u201d. Research such as this can trigger societal tensions and ethical questions.<\/p>\n

This means that impact evaluation needs to also consider non-economic factors, such as: quality of life, environmental change, and public health. Even though it is difficult placing dollar values on these things.<\/p>\n

5. When should impact evaluation occur?<\/h3>\n

Impact evaluation can occur at various stages in the research process. For example, a funder may invite research proposals where the submissions are assessed based on their potential to produce an impact in the future.<\/p>\n

An example of this is the European Research Council\u00a0Proof of Concept Grants, where researchers who have already completed an ERC grant can bid for follow-on funding to turn their new knowledge into impacts.<\/p>\n

Alternatively, impacts flowing from research can be assessed in a retrospective evaluation. This approach identifies impacts where they already exist and rewards the universities that have achieved them.<\/p>\n

An example of this is the\u00a0Standard Evaluation Protocol\u00a0(SEP) used in the Netherlands, which assesses both the quality of research and its societal relevance.<\/p>\n

A novel feature of the proposed Australian system is the assessment of both\u00a0engagement and impact9<\/sup>, as two distinctive things. This means there isn\u2019t one international example to simply replicate.<\/p>\n

Although Australia can learn from some aspects of evaluation in other counties, the Engagement and Impact Assessment pilot is a necessary stage to trial the proposed model as a whole.<\/p>\n

The pilot – which will test the suitability of a wide range of indicators and methods of assessment for both research engagement and impact – means the assessment can be refined before a planned national rollout in 2018.<\/p>\n

Notes<\/h3>\n

1<\/sup>\u00a0AUSTRALIA RESEARCH COUNCIL. 2017 pilot to test impact, business engagement of researchers [online].\u00a0Australian Research Council<\/em>, Australian Government, 2016. [viewed\u00a016 January 2017]. Available from:\u00a0http:\/\/www.arc.gov.au\/news-media\/media-releases\/2017-pilot-test-impact-business-engagement-researchers<\/a><\/p>\n

2<\/sup>\u00a0SHAW, C. Watt report suggests financial incentives for measuring research\u00a0impact [online].\u00a0The Conversation<\/em>, 2015. [viewed\u00a016 January 2017]. Available from:\u00a0https:\/\/theconversation.com\/watt-report-suggests-financial-incentives-for-measuring-research-impact-51815<\/a><\/p>\n

3<\/sup>\u00a0MAZZAROL, T. Will the National Innovation and Science Agenda deliver Australia a world class National Innovation System? [online].\u00a0The Conversation<\/em>, 2015. [viewed\u00a016 January 2017]. Available from:\u00a0https:\/\/theconversation.com\/will-the-national-innovation-and-science-agenda-deliver-australia-a-world-class-national-innovation-system-52081<\/a><\/p>\n

4<\/sup>\u00a0DEPARTMENT OF EDUCATION AND TRAINING MEDIA CENTRE, THE. 2017 pilot to test impact, business engagement of researchers [online].\u00a0The Department of Education and Training Media Centre<\/em>, 2016. [viewed\u00a016 January 2017]. Available from:\u00a0https:\/\/ministers.education.gov.au\/birmingham\/2017-pilot-test-impact-business-engagement-researchers<\/a><\/p>\n

5<\/sup>\u00a0GUNN, A. and MINTROM, M. Evaluating the non-academic impact of academic research: design considerations.\u00a0Journal of Higher Education Policy and Management<\/em>\u00a0[online], 2017, vol. 39, n.1, pp. 20-30. [viewed\u00a016 January 2017]. DOI:\u00a010.1080\/1360080X.2016.1254429<\/a><\/p>\n

6<\/sup>\u00a0GUNN, A. and MINTROM, M. Higher Education Policy Change in Europe: Academic Research Funding and the Impact Agenda.\u00a0European Education<\/em>\u00a0[online], 2016, vol. 48, n.4, pp. 241-257. [viewed\u00a016 January 2017]. DOI:\u00a010.1080\/10564934.2016.1237703<\/a><\/p>\n

7<\/sup>\u00a0MORRIS, Z.S., WOODING, S. and GRANT, J. The answer is 17 years, what is the question: understanding time lags in translational research.\u00a0Journal of the Royal Society of Medicine<\/em>\u00a0[online], 2011, vol. 104, n. 2, pp. 510-520. [viewed\u00a016 January 2017]. Available from:\u00a0http:\/\/journals.sagepub.com\/doi\/full\/10.1258\/jrsm.2011.110180<\/a><\/p>\n

8<\/sup>\u00a0AUSTRALIAN TECHNOLOGY NETWORK and GROUP OF EIGHT, Guidelines for completion of case studies in ATN\/Go8 EIA impact assessment trial,\u00a0Group of Eight<\/em>\u00a0[online], 2012. [viewed\u00a016 January 2017]. Available from:\u00a0https:\/\/www.go8.edu.au\/sites\/default\/files\/docs\/eia_trial_guidelines_final_mrb.pdf<\/a><\/p>\n

9<\/sup>\u00a0TAYLOR, S. When measuring research, we must remember that \u2018engagement\u2019 and \u2018impact\u2019 are not the same thing [online],\u00a0The Conversation<\/em>, 2016. [viewed\u00a016 January 2017]. Available from:\u00a0https:\/\/theconversation.com\/when-measuring-research-we-must-remember-that-engagement-and-impact-are-not-the-same-thing-56745<\/a><\/p>\n

References<\/h3>\n

AUSTRALIA RESEARCH COUNCIL. 2017 pilot to test impact, business engagement of researchers [online]. Australian Research Council<\/em>, Australian Government, 2016. [viewed\u00a016 January 2017]. Available from: http:\/\/www.arc.gov.au\/news-media\/media-releases\/2017-pilot-test-impact-business-engagement-researchers<\/a><\/p>\n

AUSTRALIAN TECHNOLOGY NETWORK and GROUP OF EIGHT, Guidelines for completion of case studies in ATN\/Go8 EIA impact assessment trial, Group of Eight<\/em> [online], 2012. [viewed\u00a016 January 2017]. Available from: https:\/\/www.go8.edu.au\/sites\/default\/files\/docs\/eia_trial_guidelines_final_mrb.pdf<\/a><\/p>\n

DEPARTMENT OF EDUCATION AND TRAINING MEDIA CENTRE, THE. 2017 pilot to test impact, business engagement of researchers [online]. The Department of Education and Training Media Centre<\/em>, 2016. [viewed\u00a016 January 2017]. Available from: https:\/\/ministers.education.gov.au\/birmingham\/2017-pilot-test-impact-business-engagement-researchers<\/a><\/p>\n

GUNN, A. and MINTROM, M. Evaluating the non-academic impact of academic research: design considerations. Journal of Higher Education Policy and Management<\/em> [online], 2017, vol. 39, n.1, pp. 20-30. [viewed\u00a016 January 2017]. DOI: 10.1080\/1360080X.2016.1254429<\/a><\/span><\/p>\n

GUNN, A. and MINTROM, M. Higher Education Policy Change in Europe: Academic Research Funding and the Impact Agenda. European Education<\/em> [online], 2016, vol. 48, n.4, pp. 241-257. [viewed\u00a016 January 2017]. DOI: 10.1080\/10564934.2016.1237703<\/a><\/span><\/p>\n

MAZZAROL, T. Will the National Innovation and Science Agenda deliver Australia a world class National Innovation System? [online]. The Conversation<\/em>, 2015. [viewed\u00a016 January 2017]. Available from: https:\/\/theconversation.com\/will-the-national-innovation-and-science-agenda-deliver-australia-a-world-class-national-innovation-system-52081<\/a><\/p>\n

MORRIS, Z.S., WOODING, S. and GRANT, J. The answer is 17 years, what is the question: understanding time lags in translational research. Journal of the Royal Society of Medicine<\/em> [online], 2011, vol. 104, n. 2, pp. 510-520. [viewed\u00a016 January 2017]. Available from: http:\/\/journals.sagepub.com\/doi\/full\/10.1258\/jrsm.2011.110180<\/a><\/span><\/p>\n

SHAW, C. Watt report suggests financial incentives for measuring research\u00a0impact [online]. The Conversation<\/em>, 2015. [viewed\u00a016 January 2017]. Available from: https:\/\/theconversation.com\/watt-report-suggests-financial-incentives-for-measuring-research-impact-51815<\/a><\/p>\n

TAYLOR, S. When measuring research, we must remember that \u2018engagement\u2019 and \u2018impact\u2019 are not the same thing [online], The Conversation<\/em>, 2016. [viewed\u00a016 January 2017]. Available from: https:\/\/theconversation.com\/when-measuring-research-we-must-remember-that-engagement-and-impact-are-not-the-same-thing-56745<\/a><\/p>\n

External links<\/h3>\n

European Research Council – Proof of Concept – https:\/\/erc.europa.eu\/proof-concept<\/a><\/p>\n

Research Excellence Framework – http:\/\/www.ref.ac.uk\/<\/a><\/p>\n

Research Excellence Framework – Information for research users – http:\/\/www.ref.ac.uk\/about\/users\/<\/a><\/p>\n

VSNU – Quality Assurance Research: Standard Evaluation Protocol – http:\/\/www.vsnu.nl\/en_GB\/sep-eng.html<\/a><\/p>\n

Read the original post:<\/h3>\n

https:\/\/theconversation.com\/five-things-to-consider-when-designing-a-policy-to-measure-research-impact-71078<\/a>\"The<\/p>\n

 <\/p>\n

\"cc-by-nd0\"<\/a>This text was published\u00a0under a Creative Commons License \u2014 Attribution-NoDerivatives<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

The move of the Australian government to measure the impact of university research on society introduces many new challenges that were not previously relevant when evaluation focused solely on academic merit. …<\/span> Read More →<\/span><\/a><\/span><\/p>\n","protected":false},"author":5,"featured_media":205,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[3],"tags":[10,41,48,7],"class_list":["post-2455","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-analysis","tag-impact-factor","tag-public-policies","tag-research-funding-agency","tag-scholarly-communication"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t