Revamp provider-level TEF

01 March 2019

The Russell Group is calling on Dame Shirley Pearce to recommend dropping the controversial subject-level TEF, as part of her independent review for the Government. Instead of spending further time and resource on an exercise facing insurmountable challenges, we are urging the Review Panel and Department for Education to prioritise a “revamp” of provider-level TEF, to ensure it provides prospective students with meaningful and accurate information.

In our submission to the Review Panel's call for evidence (attached), we argue that the Teaching Excellence and Student Outcomes Framework (TEF), administered at subject-level, “should not be taken forward” on the basis that it will produce misleading results for students. The subject-level pilot process has been an important and useful exercise, involving ten Russell Group universities. However, despite best efforts to overcome the statistical and other weaknesses inherent in the subject-level TEF model, they have not been resolved. These can be seen below.

Commenting, Russell Group Chief Executive Dr Tim Bradshaw said:

“At this juncture we think it’s right to drop subject-level TEF, primarily because it is unhelpful to students. A great deal of effort has gone into overcoming its statistical problems through the pilot process, but it is now clear they cannot be resolved. We fear this exercise risks producing seriously misleading results.

“Efforts would be better spent supporting prospective students to use the wealth of existing information that is available more effectively as they choose their universities and courses. We are proposing a new information interface to pull together a wide range of public datasets and information from universities’ own websites and prospectuses. This is something the Government should look at as part of the ongoing redesign of Unistats.

“Alongside this, we would like to see the Government overhaul and revamp provider-level TEF as a priority. The UK has a world-leading higher education system and we need a TEF that is truly reflective of this.”

Weaknesses in the subject-level TEF model include the following:

• While the numbers of students on individual courses are too small to enable any meaningful analysis, aggregating students into large enough groups to enable comparison at subject-level undermines the validity of the exercise. It means treating disparate groups of students as though they are the same.
• Indeed, the subject groupings currently being piloted aggregate a range of different courses together which do not share similar approaches to teaching and learning. Many of the subject groupings contain courses which are often taught in different departments or even different faculties. For example, the ‘Creative Arts’ grouping combines Music, Drama and Art. This will, in turn, lead to false comparisons being made and so risks misleading prospective applicants rather than providing them with useful information. 
• Even though courses have been grouped together for the purpose of assessment, as a result of the benchmarking methodology the data being used in subject-level TEF is based on such small numbers there is a risk that outcomes could be determined by random year-on-year fluctuations as opposed to genuine variations in quality.
• Despite this, many of the metrics at some providers will be suppressed as student numbers are too small: 87% of providers have non-reportable metrics in at least one subject area. This means, for some institutions, awards will be made based only on partial data, whereas for others, awards will be based on the full suite of data. This risks misleading prospective applicants who will not be comparing “like with like” and won’t be able to distinguish between ratings which have been awarded based on very different sets of data.  
• The metrics of providers offering certain subjects are clustered together within a very small range. For example, the performance of graduates in entering highly skilled employment and further study for medicine, dentistry and nursing is extremely high. This effect is likely to lead to a preponderance of golds in some subjects and, potentially, of bronzes in others. It is difficult to see how this helps applicants differentiate between providers. In cases where the majority of providers could receive a bronze for a particular subject, this could create a real incentive to close courses if student demand drops off.

In addition, the subject-level methodology has been designed predominantly for students studying single honours courses. The framework being piloted does not capture the experiences of students who are studying joint or multiple honours programmes, those undertaking modules in other departments and other faculties, or those studying courses which cross discipline boundaries (like natural sciences or liberal arts degrees).  Universities are increasing their provision of inter-disciplinary courses to meet student and employer demand, and subject-level TEF risks discouraging these innovative new programmes. 

Provider-level TEF, by contrast, still has the potential to become a genuinely useful tool to help prospective students make more informed decisions, providing key reforms are now made to:

• Address the significant flaws in the benchmarking methodology by placing substantially more weight on absolute values alongside benchmark and sector average scores. This would recognise and incentivise high performance more effectively and would better reflect prospective students’ understanding of TEF ratings.
• Ensure institutions subject to NSS boycotts, or action which could skew NSS results, are not penalised in the assessment process. This is especially important as some universities have no published NSS metrics at all for TEF4.
• Replace the overly simplistic medal rating system with a “profile approach” which could involve providing much more helpful information about institutional strengths and weaknesses on a dashboard where informed comparisons can be made.