Outreach & Policy

Response to the Higher Education Funding Council on the Research Excellence Framework for assessing the quality of research in UK Higher Education institutions

Submitted 15 December 2009

We are pleased to see that the revised proposals move away from reliance on metrics (in particular, bibliometrics), and that it is recognised that these cannot substitute for expert review. We also appreciate the need to control the costs of the exercise, both centrally and for HEIs.

Significant elements of the proposed model appear not yet to have been worked out in detail, which is understandable given the change of direction and plans for further consultation and pilot studies as set out in the timetable. We hope and expect that these exercises will be meaningful, and that the HE community and other interested parties will continue to be involved in the shaping of REF as it progresses towards implementation. The Geological Society stands ready to assist in any way which is helpful to HEFCE and to the academic Earth science community.

2. What comments do you have on the proposed approach to assessing outputs? If you disagree with any of these proposals please explain why.

Comments are especially welcomed on the following proposals:

• that institutions should select research staff and outputs to be assessed
• for the categories of staff eligible for selection, and how they are defined
• for encouraging institutions to submit – and for assessing – all types of high-quality research outputs including applied and translational research
• for the use of citation information to inform the review of outputs in appropriate UOAs (including the range of appropriate UOAs, the type of citation information that should be provided to panels as outlined in Annex C, and the flexibility panels should have in using the information)
  and on the following options:
  • whether there should be a maximum of three or four outputs submitted per researcher
  • whether certain types of output should be ‘double weighted’ and if so, how these could be defined.

Sampling

It is not clear what is meant by ‘sampling’ of outputs for detailed review, in this context. If sampling is taken up – a matter over which it is suggested the panels and sub-panels may have some control – what will this mean in operational terms?

Would panels look only at a proportion of outputs from each submitted unit, and take this as a representative sample? Given the small population size in some cases, it is hard to see how ‘robust sampling techniques’ could be deployed in this way. The randomising element this would introduce into the process would presumably be quite unacceptable to most departments.

Or is the suggestion that a sample be selected for detailed review from across the UOA, to test for consistency with some other method to be applied to the whole population? If so, what would that method be? If it is to be (biblio)metric-led, this would run counter to HEFCE’s acceptance that such methods are neither sufficiently robust nor rich to constitute the central element of REF. Furthermore, any such attempt at post-hoc justification – on the dubious basis that peer review does not produce significantly different summary results to detailed review for the sample – would do nothing to counter the deleterious behaviour-influencing aspects of a bibliometric-led system.

We hope that we have got the wrong end of the stick here, and that there is a third, less troubling, explanation. In any case, this must be made clear.

Maximum number of outputs

Exceptional individuals within submitted units may have made contributions across multiple categories of endeavour which are eligible for assessment (original research, building on such research, economic and social impact, etc), as well as within multiple disciplines. Presumably HEFCE will wish to recognise such achievements, particularly given the emphasis on fostering excellence, and on interdisciplinary research. There is some concern that it would be difficult to demonstrate such wide-ranging contributions within a maximum of three outputs. We would therefore urge HEFCE to allow a maximum of 4 outputs, despite the potential additional cost this will impose, and to find other ways to limit the burden of work on panels.

Double weighting

The opportunity and means for publishing in monograph or similar form are by no means uniform within disciplines, let alone between them. There are also likely to be contentious definitional issues. It is hard to see how a demonstrably fair approach can be developed in this area, so we would be somewhat uneasy about introducing such a proposal.