Drivers and benefits of research policy and information systems within the UK

It is no surprise that the need for research information comes from both internal and external drivers. Perhaps the most important external driver for information within the UK context is the Research Excellence Framework (REF), an initiative of the four UK higher education funding bodies to assess the performance of UK researchers.

With effect from 2015-16, the funding bodies will use the REF outcomes to allocate research funding. Goals of the assessment also include greater accountability to the public for research investment, benchmarking information and establishing reputational yardsticks.

The evolution of research management

Research management has emerged relatively recently within Higher Education Institutions (HEIs) with little structure or design, counter to long-standing systems such as those in Finance or Human Resources. Consequently, there is little similarity in roles and responsibilities across the sector. In recent years, however, the need to manage research — or rather, the need to manage research strategically — has come to the fore. Increasingly institutions have developed focused research strategies, invested more heavily in research managers, and sought to increase reporting of outputs and outcomes from research activity. Forward-thinking institutions now recognize the importance of managing research through skilled people, working in partnership with academic colleagues.

Within this context, management information systems are playing an increasingly important role. Ten or fifteen years ago research systems were poorly planned, internally developed and largely bespoke pieces of software operating in isolation. As investment in research grew and the complexity of managing public and private grant funding increased, institutions have focused upon purchasing (or developing in-house) products to manage research operations — that is to say, the processing of research activity, such as grant application development, cost accounting and budgeting. Most recently, institutions have recognized the need to progress from operational research management to strategic research information systems or “information warehouses.” The thirst for management information within the most agile institutions is seemingly endless.

What needs to be understood are the drivers behind this need for data and recognition of who is using the data and for what purposes.

The drivers of research intelligence

The Higher Education Statistic Agency acts on behalf of all UK institutions as a collector of statutory information pertaining to finance, estates, students, research and enterprise. However, this information takes the form of annual submissions and little value is derived from this by institutions or policy makers. Admittedly, the data is relatively standardized and well understood by stakeholders, but in an environment as fast-paced and changeable as research, annual data is simply not timely enough. Moreover, the real value from information comes from combinations of data collated holistically, and it is disappointing that national data collection activities generally collect data in narrow silos.

The REF (previously known as the Research Assessment Exercise) is recognized worldwide for its robustness, no doubt because it strikes an appropriate balance on a number of levels. First, the assessment consists of key data metrics — such as research income, postgraduate research supervision and, most recently, citation data — but such data is interpreted by panel assessors for each of 36 subject categories (or Units of Assessment). This balance between data and expert peer review is critical to maintain the confidence of academic staff. Secondly, the collection of data for the REF is holistic in that data is collected within a weighted framework consisting of Outputs (65%), Impact (20%) and Environment (15%). These three areas provide a balanced assessment of the research activity within categories at a sufficiently granular level to be meaningful and cover the academic outputs generated by research, the health and vitality of the research environment and the public benefit — or socio-economic impact — arising from funded research. (See Figure 1.)

The impact agenda is a new feature within the REF, and there is little doubt that this has generated a new data collection industry within institutions and indeed across other stakeholders. For example, the UK Research Councils are keen to demonstrate the value generated by the research they fund and have launched several system-based initiatives to collect data from researchers across the sector. As a result, institutions are scouring their databases to find inventions, patents and collaborative R&D activities to evidence the impact of the research they support.

Internally, institutions have their own information needs and these are geared largely towards a need to understand performance. Mostly these have tended to be heavily influenced by financial drivers — such as number and values of grant applications and awards — and these are, of course, critical to the sustainability of institutions. During a challenging financial situation there is little doubt that university leaders have been anxiously monitoring grant income figures. However, more strategic organizations are beginning to recognize that a rounded, balanced view of research performance is required that encompasses all the activities undertaken by academic staff, from income generation, through to publication, supervision and external engagement. Such balance is particularly important when considering the broad range of disciplines and the difficulties of using “hard” financial indicators in the Arts, Humanities and Social Sciences. A potential framework that could be adapted by other institutions is presented in Figure 2.

Who uses research information?

Research information is required for different purposes by a number of different stakeholders. The most important objective must be to reach agreement between all parties, both academic and administrative, on a data framework that can be used consistently. Once this framework is set — and that is no easy matter to achieve — the key is to build information from the bottom up to create simple, easy to interpret performance measures. Given the nature of HEIs and academic staff in particular, the need to have access to underpinning data at all times is paramount. Without this, mistrust and a lack of confidence in data will emerge within the institution.

Broadly speaking, once a base layer of data is in place (the bottom level) and framework for performance management is agreed (the top level), three tiers of stakeholders will need to use the information:

• Senior Executive Team: Keep information in summary form, with easy to interpret charts and 5 or 10 key indicators. The information should span the entire institution. 
• Heads/Deans of Academic Faculties: Maintain information at a relatively high level, but provide access to underpinning data pertaining to that particular faculty.
• Departmental Heads/Research Managers: Provide more detailed information and use more granular measures. Restrict data to the particular academic department and, depending on access levels, provide data at the individual level to monitor performance and align with staff appraisal mechanisms.

What is research information used for?

The use of research information and performance measures is self-evidently a method for assessing the quality of the research within the institution. While drivers such as the REF are important for reputation and funding reasons, these are simply part of a performance management framework that should be present in the institution in any case. Why would an institution not wish to understand on a regular basis the quality of its research activity?

In a challenging financial situation, this information will be used in allocating scarce resources. Investment decisions should be based upon clear evidence at a time when funding organizations and institutions themselves need to understand their strengths, their distinctiveness and areas of genuine competitiveness. It is no coincidence that institutions are now increasingly focused upon benchmarking their performance externally to judge their relative performance levels — it is unfortunate that reliable and timely external data is so hard to come by.

At a more granular level, a key element of success in using research information and ensuring that it is engaged with by academic staff is to link data and performance measures to academic appraisal mechanisms. Where data is reliable, validated and linked to an agreed performance framework, research information can be key to improving quality and managing talent.

Conclusions

The profession of research managers is still developing in the UK, but has developed a growing role within institutions as they seek to manage research strategically. In tandem with this, there is an increased thirst for information and an evidence base from which to make decisions and assess performance. These are driven largely by external drivers, but arguably should form part of any institution’s “balanced scorecard” of activities as they manage their organization through challenging and changeable times for higher education.

There is much to be done within institutions to develop frameworks for managing information and to create genuine “intelligence.” Hurdles emerge in the form of poor data quality, difficulties in agreeing on succinct performance measures, and problems with engaging academic staff in activities that are perceived to work against their freedom. However, institutions should keep their eyes firmly on the prize: a holistic, reliable and easy-to-understand system that allows research to be managed strategically and for the benefit of the academic mission.

Contributor: 

Scott Rutherford

Title: 

Director of Research and Enterprise

Affiliation: 

Queen’s University Belfast

Scott Rutherford joined Queen’s University Belfast as Director of Research and Enterprise in November 2010. He provides strategic oversight and leadership for the research development and knowledge exchange activities of the university. Previously, Scott managed academic research activities at Imperial College London and oversaw a number of high-profile projects to implement research systems to support the research lifecycle, from research development to publication and commercialization. In a post at the Higher Education Funding Council for England, he played a key role in developing the Research Excellence Framework programme. Scott holds an undergraduate degree in English and Geography from the University of Keele and an MBA from Imperial College London.