«Quality assurance and assessment of scholarly research A guide for researchers, academic administrators and librarians May 2010 This ...»
Quality assurance and
assessment of scholarly research
A guide for researchers, academic administrators and librarians
This guide has been produced by The Research Information Network to provide
researchers, academic administrators and librarians with an understanding of
quality assurance processes and some of the current issues surrounding the debate
about quality assurance.
The guide is available at www.rin.ac.uk/quality-assurance
Hard copies can be ordered to distribute to colleagues, please email email@example.com About the Research Information Network The Research Information Network has been established by the higher education funding councils, the research councils, and the UK national libraries. We investigate how efficient and effective the information services provided for the UK research community are, how they are changing, and how they might be improved for the future. We help to ensure that researchers in the UK benefit from world-leading information services, so that they can sustain their position as among the most successful and productive researchers in the world. All our publications are available on our website at www.rin.ac.uk Contents
1. Introduction 4
2. Why quality assurance and assessment? 6
3. Definitions 7
4. Tools and mechanisms 8
5. Assurance and assessment of research projects and programmes 9
6. Assurance and assessment of researchers, institutions, and 16 scholarly journals
7. Conclusion 20 References and useful resources 22
1. Introduction Quality assurance and assessment play many important roles in the research community. They inform crucial decisions on the funding of projects, teams and whole institutions, on how research is conducted, on recruitment and promotion, on what is published or disseminated, and on what researchers and others choose to read. They underpin trust in the work of the research community.
This document provides an overview of some of the key issues surrounding quality assurance and assessment of scholarly research. It is intended for academic administrators, researchers and librarians who deal with elements of quality assurance and quality assessment as part of their daily work, but who wish to understand more about the broader context of that work.
The various techniques and processes involved in determining quality have evolved over time to meet the changing needs of researchers, institutions and funders. Further development will doubtless be needed in response to changing circumstances and needs, as well as to the opportunities presented by new technologies and services.
Assurance and assessment regimes, tools and techniques will thus continue to evolve in
• increases in the volume of research. The number of researchers, of the projects they undertake, and of the outputs they produce is growing year by year. This puts increasing strain on existing assurance and assessment systems • increased pressure on costs. Effective assurance and assessment systems are costly, even though some of the most costly parts – including the time spent by peer reviewers – are seldom revealed in cash terms. The need to bear down on costs will continue to exert pressure to find more streamlined systems and techniques.
• increased competition between researchers. Constraints on funding bring increased competition for research grants and contracts, falling success rates, more work for assurance and assessment systems, and more difficult decisions in determining which projects, researchers and institutions succeed in the competition. Similar pressures are evident as researchers compete for space in prestigious publications.
• new kinds of research outputs. Many assurance and assessment regimes are built around outputs that are formally published in journals, monographs and conference proceedings.
But the digital revolution leads to new kinds of outputs – multimedia presentations, working papers, pre-prints, blogs and so on – that can readily be circulated worldwide. And dataintensive research brings a new focus on data as an output in its own right. Assurance and assessment regimes will need to keep up with these new outputs and forms of communication.
New technologies and services have already changed the ways in which assurance and assessment is done. And new opportunities are arising in the form of social tagging and recommendation systems, checks on plagiarism, and facilities for online comments and ratings. New technologies are also facilitating the development of a wide range of sophisticated bibliometric and other measures that can provide useful evidence to support assurance and assessment systems. One of the key issues for the future will be how these new tools and techniques evolve alongside more traditional systems of assurance and assessment of the quality of research.
2. Why quality assurance and assessment?
Quality assurance and assessment are critical at several stages in the research process. Since resources are limited (and likely to be increasingly so) funders need effective mechanisms to ensure that they support only the best projects, researchers and institutions. It is also important that procedures are in place to ensure that research projects are undertaken properly, efficiently and effectively. And once projects have been completed, and results are being prepared for presentation, checks are needed to ensure that what is published and disseminated meets high standards. This is important in order to ensure that the records of research – sometimes referred to as the ‘minutes of science’ – provide an accurate record of what has been found. Such checks also help researchers and other readers who have limited time – and often limited expertise in particular specialised fields or projects – to identify and locate the most high-quality and significant findings of relevance to them.
This document begins with an overview of the closely-related processes of quality assurance and quality assessment. It then examines how these processes manifest themselves at key stages in the
(i) a specific research project or programme (ii) a researcher’s career (iii) a university or research institution (iv) scholarly publications
3. Definitions Quality is of course not a straightforward concept. The Oxford English Dictionary (OED) defines it as the nature or standard of something as measured against other things of a similar kind, and especially the degree of excellence it possesses. In the academic environment, there is widespread agreement that quality involves adherence to key principles such as intellectual rigour, accurate recording and honest reporting of results, and integrity in recognising the work of other researchers.
But there may be legitimate differences of view as to the quality of, for example, notably innovative or groundbreaking work; and notions of quality may in some cases be related to specific contexts such as the scope or remit of a research funder or publisher. Numerous bodies undertake quality assurance and assessment for different purposes; the varying definitions and processes they use may clearly give rise to different results.
It is important also to note that while the terms quality assessment and quality assurance are often used interchangeably, there are important distinctions between them. ’Assessment’ may be defined as the process or means of evaluating academic work. ‘Assurance’, on the other hand, implies a promise or guarantee – a statement that something is of good quality and can be trusted. The relatively modern concept of quality assurance may be defined as the set of procedures designed to ensure a desired level of quality in a service or product.
The key distinction, therefore, is that an assessment process implies a review – involving human judgements and/or quantitative scores – which may find work of varying quality, from the poor or mediocre to the excellent or outstanding. A finding at the end of the process that work is of poor quality does not mean that the process has failed, rather the reverse. Reviewers have no responsibility to improve the work they examine.
Quality assurance, by contrast, implies a process of tests or filters to ensure that products or services pass a quality threshold. If the process is external, quality assurance often also provides a kitemark to indicate that the work has passed that threshold. External processes in particular may also involve an active role for reviewers in ensuring that the work under scrutiny emerges as the best possible version of itself. If work of poor quality emerges at the end of a quality assurance process, that process has failed. It is important to note, however, that in the research environment there can be no guarantee that findings and conclusions are correct and unchallengeable. The advancement of knowledge and understanding depends on challenging, qualifying or even overturning current findings. All that an assurance process can do is to seek to ensure that research is properly conducted, and its results are reported accurately, on the basis of the best currentlyavailable techniques, knowledge and understanding.
The processes of assurance and assessment thus involve a number of different tools and mechanisms which are employed for different purposes, in different contexts, and at different stages of the research lifecycle. The rest of this document focuses, after a brief introduction to the key mechanisms, on how they are applied and with what effect.
4. Tools and mechanisms Some assurance and assessment mechanisms involve quantitative measures such as bibliometrics, while others involve monitoring, review and evaluation by people who are knowledgeable in the relevant field. The principle that judgements should be made by experts who are respected in the field – peer review – is held by most researchers to be fundamental to any effective system for assuring or assessing research quality. Peer review attracts deep and strong support across the research community. But it comes in a number of different forms, and practices vary considerably in different contexts and fields.
Peer review also attracts criticism, on the grounds that it brings delay; that it is not always effective in detecting misconduct and malpractice; that the selection of reviewers may introduce bias into the system; that the judgements made are subjective and inconsistent; that it tends toward conservatism and stifles innovation; and that it disadvantages interdisciplinary research. There are also concerns about the costs of the system (which are largely hidden, since peer reviewers are generally not paid for their time); that the burdens being placed on the reviewing community may become unsustainable as the volumes of research activity and publications continue to increase;
and that there is a need for more training of reviewers to ensure greater consistency.
Such criticisms and concerns have not, however, undermined the value attached to judgements made by respected peers in any system of assurance and assessment. Quantitative measures are generally seen as supplements, not substitutes for such judgements. For further information on the peer review system, the challenges it faces, and how it might change, see the RIN booklet Peer Review: a guide for researchers.1
1 Research Information Network (2010) Peer Review: a guide for researchers
5. Assurance and assessment of research projects and programmes The arrangements for assuring and assessing the quality of research projects varies in some particulars between different fields and disciplines, but may conveniently be divided into six main
(i) programme and project proposals (ii) monitoring and oversight during projects (iii) sharing early findings with colleagues (iv) formal publication (v) data sharing (vi) post-publication assessment and review 5.1 Programme and project proposals Research institutions and funders use a number of checks in seeking to ensure that only highquality research is funded. Many institutions implement mechanisms – through departmental management structures or through institutional bodies such as research offices – to monitor project proposals, thereby ensuring only those that pass quality thresholds are supported by the institution or submitted to external bodies for support.
When external support is sought, Research Councils, along with most research charities and institutions, evaluate project proposals using various peer review mechanisms. Panels of experts are drawn from academia as well as other parts of the public, commercial and voluntary sectors, with members selected to give the panel the necessary range of expertise to assess the proposals they are asked to consider.
Assessment and evaluation may begin even before a specific proposal has been framed. In order to address certain criticisms about the ‘conservatism’ of peer review, some funders are experimenting with new approaches to encourage innovative and cross-disciplinary proposals. The Engineering and Physical Sciences and the Natural Environment Research Councils (EPSRC and NERC) are using ‘sandpits’: 3-5 day workshops, overseen by a director and other facilitators, where researchers from a wide range of disciplines discuss issues and ideas, and then develop, test and refine proposals which may be funded at the end of the process.2 In a more conventional process, researchers prepare proposals with their colleagues. These are then assessed primarily on their scholarly quality, covering such issues as intellectual rigour, creativity, methodology and so on. But many related quality criteria may come into play, including the fit between the proposal and the funder’s priorities, the track record of the applicants, and value for money.
2 See the ESPRC http://www.epsrc.ac.uk/funding/grants/network/ideas/Pages/moresandpits.aspx and NERC http://www.nerc.
ac.uk/funding/assessment websites for more information on sandpits.
Fit with a funder’s priorities may be determined in relation to published strategies which are reviewed from time to time, and for public funding agencies agreed with government. Such strategies and priorities may increasingly include achieving economic, social or cultural impact beyond the research community. Research Councils are collectively committed to increasing the impact of the research they fund.