# Impact and the REF - panel discussion # ## RepoFringe 3rd August 2017 ## * Chairing: Keith McDonald SFC (KMc) * Panel: Catriona Firth, HEFCE (CF) Pauline Jones, University of Edinburgh (PJ) Anne-Sofie Laegran, University of Edinburgh (ASL) * Taking notes: Pauline Ward, EDINA KMc introduced the panel members. The conference heard presentations from each of the panel, CF, ASL and then PJ. ### Catriona Firth presentation ### Key points: - REF2014 was the first to measure benefit to society as impact. Impact templates were used. Units of assessment were used (i.e. groupings of disciplines). Case studies were assessed in terms of reach and outcomes, i.e. number of people reached and how much they benefitted. Over 250 non-academic users of research were involved in measuring impact. - Different types of impact were measured eg impact on policy, on industry. - Interesting analysis commissioned by HEFCE looked at the submitted case studies and links between research and impact. Lord Stern was commissioned to review REF2014 and impact: - deemed it a success. - However recommended burden should be lowered for institutions, - need to tackle gaming of the system. - Wanted to shift focus from individual to the institution. - Gain a more rounded view of impact. - Interdisciplinary research could be measured more effectively. - All research active staff should be put forward for the REF. - Outputs should be determined as an average rather than four outputs. - Outputs should not be portable in the same way. HEFCE have received almost 400 responses to their consultation on the Stern proposals: - Clear support for broadening the definition of impact and research and for greater support for public engagement. - Concerns over the minimum of one impact case study, could skew results, and reveal individual score of researcher/case study. - Separation of rigour from significance and impact less favourably received. - Questions around longevity of impact. - Institutional level assessment of impact: giving flexibility to HEIs to demonstrate strength which might not fit into one discipline. Mixed response - lots of support for interdisciplinarity but need to avoid conflating with institutional impact. Risk of creating clash in the relationship between unit of assessment and HEI. HEFCE planning to publish initial decisions in a few weeks, and summary of responses a short while later. Website, logo and twitter account to be launched thereafter. ### Anne-Sofie Laegran presentation ### Key points: - Impact is not new, what's new is that we're demonstrating it. - Increased awareness now , cultural change among academics. - Public engagement has great benefit to society but is difficult to capture. - Training on Pathways to Impact as well as how to generate and measure impact. ASL's team base some of their training on questions asked by RCs: who will benefit from the research, how can this be measured, and how can the beneficiaries engage with the research? Stimulates researchers to consider who are decision-makers - in order to effect change, need to engage with policy makers, museums etc. Partnerships eg with NGOs in order to influence government for benefit of the public. Knowledge exchange should be embedded in research (technology transfer, public engagement, policy influencing) these outcomes are impact. Important to distinguish the process from the result. Challenges encountered during preparation for REF2014 - to demonstrate change and benefit as opposed to just activity, then how research contributed to this change. Also tricky to find evidence, especially retrospectively. For REF2020 there has been more work to gather evidence along the way. ASL presented examples of UoE impact case studies submitted to REF2014: - One UoE case with an exhibition demonstrated impact through large number of public visiting the exhibition and value of artist's work on open market increased. - Scottish Government testimonial that youth crime had dropped as a result of a policy change as a result of UoE research. (Difficult for the HEI to make this claim - Scottish Government the authority on the question). - 'One of the most influential thinkers on the financial crisis', UoE academic published in London Review of Books, retweeted by Stephanie Flanders very influential economist . - Music The Skoog musical instrument designed to be accessible for children with disabilities, sold to numerous institutions and used at the Cultural Olympiad London 2012. These cases demonstrate activity including writing and production of technology, but the impact was what happened afterwards. The importance of open access for impact is open to question because impact tends to not be linked to a specific publication or dataset. Doesn't mean it won't grow in the future. Research should be open. Social media can be important but to be considered measurable impact social media impact needs to be very influential or have a great reach. Of itself does not demonstrate impact. But can help track and understand impact - see who the audience is. Metrics - numeric evidence was undoubtedly important but context has to be provided also to explain why the metrics are relevant. We do not have standard metrics for impact. Altmetrics can be a good tool to track impact but on their own the figures do not show impact, will not replace case studies. Too easy to manipulate. Impact was 20% of the REF score, but determines 26% of SFC funding for 2014-2021. A 4* case study can attract up to £120k additional funding to a HEI per annum, whereas a 2* case study attracts zero. See www.ref.ac.uk for case studies. ### Pauline Jones presentation ### How do we get ready for REF 2020? Since REF2014 results there have been many developments to follow. Open Access is important to the public and widely recognised as a good thing. Implementation of Open Access policy in Spring 2016 has fed into preparation for next REF. Academics are already looking ahead to next REF and planning for it. Support for Research Information Systems has to be continuous because all HEIs need a RIS to facilitate the institution's submission to the REF. New things coming out of Stern proposals present some challenges for institutions in terms of planning. Which researchers will be counted as independent researchers and therefore submitted to REF? Decoupling of the REF i.e. removal of the four outputs will be a significant change. Non-portability of publications recognises the university's investment. But need to consider carefully contribution of researcher. Will it be possible to have less complexity and less burden in the next REF? Sceptical because of extra burden of open access compliance. Institutional impact case studies - 50% of UoE case studies for last REF had interdisciplinarity. Environment statements - consideration potentially of a whole-university point of view (might reduce burden for us, our 31 environment statements some duplication). Systems - room to ease the burden by improving our systems and processes. REF2027 or REF2028 - will there be a REF? There are some signs there is likely to be an open data requirement. We need to start thinking about that now, as with the open access for publications, long process to prepare and get ready. This is role for the repository community. Open access monographs may be included, seems a welcome development. Metrics - The Metric Tide by James Wilsdon about the metrics supporting REF and other aspects of institutional management, emphasised need for context - metrics on their own, except perhaps for REF Environment aspect, need to be complemented by more qualitative data. We have to be agile. What's really changed is tha t REF managers and policy officers have to work with CRIS systems to make the systems work better. For REF 2028 need to work together to understand challenges of making research more open and pathway to benefits to the public. Supporting the systems we have such as ResearchFish to be more interoperable so that HEIs can push data to it more easily. ### Questions ### * Q: How can we ensure research more broadly defined is open and can lead to impact? Need to ensure material is available online. And discoverable. Need to be clear when outputs are found eg distinguish between peer-reviewed or working papers. Data could be bigger part of the evidence gathering. Not clear how data translates to an impact case study. Using PURE need to distinguish between eg media clippings need to think about how work with press offices to ensure we have clarity. Potential for repositories to produce a holistic view of a researcher's output. Demonstrate academic and wider impact - can we capture that alongside outputs? * Q: A common theme was around the burden of REF exercise, but it could be argued REF of itself can act as a positive driver of change...? Has created a more positive attitude towards engaging with the community. REF drives strategic investment in universities. Lots of engagement that was already happening has received additional investment, increased sharing of knowledge. REF and RAE have driven professionalisation of research and support for researchers more generally. Has had a measurable effect in driving up the number of highly-cited publications from the UK. * Q: Please comment on the academic environment, data perspective, data, capability of a departmental research group. Research support often very devolved across an institution. * Q: The relationship with the academics changes with the conception of research tha t REF has ushered in. UoE now has much more involvement and has to invest. Is driving greater pre-evaluation of research, changes the way departments engage with researchers. Some is seen as burdensome but also this increased activity has been positive for HEIs insofar as they are engaging. * Q: Why are publications given very different relative value as outputs? Considering the publications are often not linked directly to impact. As an example, often when academics give evidence at Parliament, journalists and politicians not necessarily reading the publications but they are part of accountability, part of demonstrating the authority of the academic to comment on a given issue, their peer-reviewed work is demonstrated as rigorous and in-depth. Wider impact on society - how this is defined, is being considered by HEFCE. * Q: The hard-to-read - another strand, growing body of evidence. Where there are lay summaries, it provides a hook for the lay audience into the work eg John Tennant work on lay summaries, services such as Kudos. There's a real link between that impact and the ability of the public to access and make the research more easily understandable. Any piece of research no matter how complex should have a lay summary to explain why it has been funded, what its wider significance is. This can be of great benefit to other academic audiences too, beyond the author's discipline, industry. Important to communicate more widely. Repositories are useful. Just depositing is not enough, just as publishing in a journal is not enough. You have to reach out to your target audiences. Is incredibly important - needs to be more support and training for academics to help them do this. Public engagement/ impact officers have been doing this work and finding academic staff engaging very positively with it, seeing wider benefits of translating work into a lay summary. Need to write in as transparent language as possible.