The Review of Peer Review 2013 Final Report
1
FOREWORD As Chair of the review panel, I would like to thank the panel members for their time, efforts and input to the analysis and discussion. Like many members of the peer community we all recognise the hard work and integrity of EPSRC staff especially as resources have got much tighter. I would like to give special thanks to Carol McAnally, who together with Tanya Cottrell and Paul Tomsen, has supported the panel during our deliberations.
2
1
INTRODUCTION
4
2
EXECUTIVE SUMMARY
5
3
MEMBERSHIP
7
4
TERMS OF REFERENCE
8
5
METHODOLOGY
9
6
SUMMARY OF THE VIEWS PRESENTED TO THE PANEL
11
6.1
SUMMARY OF DISCUSSIONS WITH EPSRC STAFF
11
6.2
RESPONSE TO INVITATIONS TO SUBMIT WRITTEN EVIDENCE
12
6.3
OVERVIEW OF RESULTS FROM SURVEY
16
7
RECOMMENDATIONS
18
8
ANNEX 1 - Panel Membership and Profiles
27
9
ANNEX 2 – Purpose, Scope and Operation
31
9.1
BACKGROUND
31
9.2
PURPOSE
31
9.3
SCOPE
31
9.4
MEMBERSHIP
32
9.5
METHODOLOGY
32
9.6
TIMESCALES
32
10
ANNEX 3– EPSRC’s Peer Review Process
34
11
ANNEX 4 [List of those invited to submit written evidence to the Review]
35
12
ANNEX 5 [Copy of on-line survey]
37
13
ANNEX 6 [Detailed Analysis of on-line survey]
40
14
ANNEX 7 EPSRC’s Organisational Structure and Budget Allocation
48
3
1.
INTRODUCTION
The Engineering and Physical Sciences Research Council (EPSRC) is the largest UK research council (RC), investing more that £850m per annum. Each RC has its own processes for allocating funds, consistent with strategy, and all rely largely on peer review to drive the quality of research funded. In October 2012 the EPSRC Council agreed to commission an independent review of its peer review processes, due to public criticism indicating a lack of trust. These concerns were exacerbated by an environment of declining funding in real terms (e.g. Spending Review 2010). The current per review processes have evolved from their original format introduced shortly after EPSRC itself was established in 1994. The peer review college was implemented in 1999. Today EPSRC has around 25 different funding schemes and an electoral college of around 4,000. Peer review is used to award funds in most cases, although the process varies in proportion to the significance of the awards. EPSRC’s principles of peer review were first published in 2005. It has sought feedback on its peer review processes periodically, the last major review being in 2008. Subsequent policy changes which have generated criticism include:2008 Pathways to impact (as implemented by EPSRC) 2010 Frequently unsuccessful applicants 2011 National importance As Chairman of the EPSRC Council, Dr Paul Golby approached prospective panel members in the first quarter of 2013 and the panel was formed in the second quarter, under the chairmanship of Professor Dame Julia Goodfellow. Key features of the panel were that it:• Be totally independent – membership of the panel was guided by the EPSRC Council, not the Executive; • Be free ranging – the panel decided what to look at and how to conduct its review; • Have unlimited access - the panel had full access to any documentation, data or people that it felt appropriate to aid its deliberations; and • Be transparent - the report to Council will be made public together with the Council’s response. Following its first meeting in July 2013 the panel agreed its terms of reference and methodology, and these were published on EPSRC’s website at http://www.epsrc.ac.uk/newsevents/news/independentreviewofpeerreview/. Subsequent meetings were held approximately monthly and the panel received written and/or verbal input from individuals, universities, companies and other institutions. The panel approached its task with an open mind, determined to discover for itself what, if any, the issues were associated with EPSRC’s peer review processes, whether in perception or reality. This report, comprising the panel’s findings and recommendations, was prepared for the EPSRC Council meeting on 2nd December 2013. 4
2.
EXECUTIVE SUMMARY
There is overwhelming support for the use of peer review on selecting research proposals of the highest scientific quality from the peer community and from EPSRC staff themselves. It is clear that the peer community care deeply about the EPSRC’s framework for peer review about which they are generally supportive but equally there is a significant level of concern over the details of the process. There is broad recognition for the work of EPSRC staff within a very tight administrative budget. The panel started with no pre-conceived ideas to the problems but responded to issues presented to it during the consultations. It did not undertake a forensic review of the peer review process. Our methods involved meetings with staff and the peer community, requests for written responses and a questionnaire which itself led to 200 pages of comments. The panel read the documentation and analysed the responses to the questionnaire. Our recommendations are intended to offer some guidance to EPSRC Council about the issues that have been raised. Perhaps more importantly we would like to see them used to continue an open debate between EPSRC and the peer community in order to strengthen the transparency and fairness of the peer review process. Recommendation 1: EPSRC should ensure that their communications and associated actions emphasise that peer review is seen as a shared endeavour. Moreover, these should emphasise the primary role that peer review plays in EPSRC decision making. Within EPSRC, Council and the Executive should ensure that those of its staff who are involved in peer review feel valued. Recommendation 2: EPSRC should think through more clearly how the strategic criteria are incorporated into the processes and documentation. In particular we recommend that the postal peer review form is changed to reflect a primary focus on quality of science and that the panel process in the use of strategic criteria is better defined and communicated. Recommendation 3: EPSRC needs to articulate why postal peer review with right of reply is not being used for distribution of funds in specific cases and to manage the risks that it generates. Recommendation 4: EPSRC should consider how the structure of the postal referees’ form affects engagement from the community, including international referees, in order to maximise assessment of the proposals by the best and most appropriate experts. Recommendation 5: EPSRC should reinvest the funds currently used for the incentive payments (£1M) into science. Recommendation 6: EPSRC should work closely with HEIs to ensure participation in Peer Review by those who benefit from considerable funding.
5
Recommendation 7: EPSRC should consider the status of the College. The panel cannot see justification for its existence when so many referees are not chosen from the College. Recommendation 8: EPSRC needs to reassure the peer community that the choice of referees and panel members is transparent, appropriate with oversight and ownership by the peer community. Recommendation 9: EPSRC should work with the community to identify and capture efficiencies but also to maintain sufficient funds for an effective peer review process to take place. Recommendation 10: EPSRC should look to engender a far greater shared ownership in peer review giving the peer community more input, ownership and oversight into different aspects of the process.
6
3.
MEMBERSHIP
The membership of the panel was jointly agreed by the EPSRC Chair, EPSRC CEO and a sub-group of Council members. The members of the panel were: • • • • •
Prof Dame Julia Goodfellow, University of Kent (Chair); Mr David Eyton FREng, BP; Dr Angelika Kalt, Swiss National Science Foundation; Prof Matt Rosseinsky FRS, University of Liverpool; Prof James Stirling CBE FRS, Imperial College London.
A profile for each member is attached in Annex 1.
7
4.
TERMS OF REFERENCE
The visiting panel was asked to undertake an independent review of EPSRC’s peer review process. The visiting panel was responsible for defining the precise scope and approach of the review. The panel agreed the following Terms of Reference: •
•
•
•
To examine the extent to which peer review was at the heart of EPSRC decision making: how it is used across the range of funding schemes operated; how decisions are made about which scheme proposals are assigned to; how the use and scope of calls for proposals are decided; and any circumstances where funding may be granted without being subject to peer review. To examine how EPSRC seeks peer review advice and from whom: is the college inclusive and representative and how are members selected; how are panel members selected and are they appropriate and representative; how are reviewers assigned to proposals and how appropriate are they; what advice is sought from them; is this process sufficiently transparent and to what extent does it have the confidence of the community. To examine how this advice is used to inform funding decisions: what is the process for this; who makes the funding decisions and how is the budget for a particular panel decided; what factors other than the panel rank order list are taken into account. To examine how the EPSRC process compares to other similar funding agencies in terms of transparency, cost-effectiveness, and the extent of community ownership.
The purpose, scope and operational approach of the review are attached at Annex 2.
8
5.
METHODOLOGY
The panel met on the 11th July 2013 to agree the terms of reference, scope and purpose of the review and methodology. Dr Paul Golby attended this meeting to share the background to the review with the Panel. The panel started with no pre-conceived ideas to the problems but responded to issues presented to it during the consultations. It did not undertake a forensic review of the peer review process. The panel spent the 2nd September 2013 at the EPSRC’s offices in Swindon where they met with members of EPSRC’s executive Leadership Team (ELT), namely, Lesley Thompson, Atti Emecz and Neil Viner, Theme Leads and portfolio managers. The panel met with Prof Dave Delpy, EPSRC CEO, at a later date (26th September 2013). On the 26th September 2013, the panel convened in London where it conducted face to face meetings with: • • •
Dr Mark Goulding, Merck - representative of a Panel Chair Prof Mark Smith, Lancaster University – representative of a nonFramework University Prof Alan Champneys (University of Bristol), Prof Bob Fisher (University of Edinburgh), Prof Tom McLeish (Durham University), Prof Ian Walmsley (University of Oxford) – representative of EPSRC Framework universities.
The panel met on the 25th October to consider the evidence received from the on-line survey (available 9th September to 7th October 2013) and written responses from stakeholder groups. A final meeting was held on the 20th November 2013 to review the main findings of the panel and to share privately these with Dr Paul Golby and Prof Dave Delpy, CEO EPSRC and Dr Lesley Thompson, Director, Science and Engineering. During the course of the review the panel had access to all EPSRC staff, requesting any information or documentation it deemed necessary. A key aim of the review was to draw views from EPSRC’s wider community. Time and resource limitations meant that the panel could not consult with every member of the EPSRC community. It did, however, reach out to a broad group of organisations and key groups who represented a significant proportion of the EPSRC stakeholder community. A full list of those invited to submit their views is given in Annex 4. Those approached were asked to answer a number of questions; a blank version is given in Annex 4. A total of 122 letters were sent out and 30 were returned. An on-line survey was also made available to the community to provide their views. A total of 756 replies were received which equates to 2% of EPSRC’s active community 1 . A copy of the survey is given in Annex 5 and its analysis is provided in Annex 6. The initial analysis of the survey was undertaken by EPSRC staff seconded to this Review and working 1
’Active community’ is defined by EPSRC as people who since 01/07/05 have had either: (1) a live grant (end date in period); (2) a request to review; (3) a submitted application to EPSRC.
9
directly to the Panel. Subsequently a more detailed analysis was undertaken by the panel. All responses were given to the panel and discussed at the meeting held on the 25th October 2013 and subsequent meetings. A final meeting was held on the 20th November 2013 to review the main findings of the panel and to share these privately with Dr Paul Golby and separately with Prof Dave Delpy, CEO EPSRC and Dr Lesley Thompson, Director, Science and Engineering.
10
6.
SUMMARY OF VIEWS PRESENTED TO PANEL
6.1
Summary of Panel Discussions with EPSRC Staff
When the panel visited Swindon and spoke to EPSRC staff about EPSRC’s peer review process they received information about the outcomes of EPSRC’s peer review, the evolution of EPSRC’s peer review process, key relationships involved and an overview of the process. The overview of EPSRC’s peer review process is given in Annex 3. The panel summarised the discussions as follows: Complexity – The panel acknowledged that peer review is a complex process. However, the panel questioned what level of complexity (including number of schemes available) is acceptable before it becomes too difficult or cumbersome for the community to fully engage in the process. Communication – The panel recognised the importance of communications and the efforts made by EPSRC to undertake this. The size and breadth of the community and the variety of funding schemes makes this a complex activity. Communication is a two-way process. Engagement – There is a perception within EPSRC that the community as a whole, and in particular research leaders, is increasingly becoming reluctant to be involved in peer review due to time pressures. EPSRC are concerned about this. Ownership – Who owns and who is involved in EPSRC’s peer review process was discussed. It was acknowledged that peer review cannot operate without the positive engagement of the peer review community but this community also requires a body i.e. EPSRC to act as a custodian and fair arbiter of the process and interpreter of the outcomes to BIS. Tensioning - Peer review is now expected to tension a number of criteria and factors. There was a discussion whether it is a realistic expectation for reviewers to achieve this. Peer review must focus on and drives quality. However, there is an increasing demand for peer review to also consider strategic priorities of EPSRC, for example, national importance and impact. The tensioning of different criteria in peer review by the community happens at both the refereeing stage and at panels. Role of Peer Review within EPSRC - The value placed on successful completion of the process and support of peer review by staff in terms of staff performance was not evident to some of them and they felt that strategic or policy driven activities had higher status. Efficiency - EPSRC’s administration budget is 1.5% of its total budget. It is anticipated that the extra cuts to administration budgets that have to be made by BIS will result in further cuts to EPSRC’s administration budget. There is concern that the quality of EPSRC’s peer review process will be compromised if further cuts are imposed on administration spends.
11
6.2
Response to Invitations to Submit Written Evidence
The panel invited evidence from universities, HEI representative groups, learned societies and strategic partners (industry). Information was requested under four question headings, listed below, and general comments were also invited. A total of 30 responses were received by the deadline, from universities (21), industry (4), and others (5). A copy of the letter sent and full list of those invited to submit their views is given in Annex 4. The vast majority of responses are positive about EPSRC’s peer-review process, and many strongly so. There is clearly a great deal of respect and admiration for the way that EPSRC conducts the research funding application system, with an acknowledgement of the difficulties of managing a system which is fair, robust, transparent and able to accommodate large numbers of applications for funding. The following comment is typical: “The principles and procedures of [EPSRC] peer review are understood well within the academic community, which on the whole has confidence in its operation and decision making.” However, in the above context of generally positive comments most respondents have specific criticisms and suggestions for improvements to which they wished to draw the panel’s attention. Even where the experiences described did result in a funded proposal, there is clear recognition of the difficulties for EPSRC in managing a complex series of priorities and balancing this with competing demands for funding within the research council. It is also notable in the comments received that the work of reviewers and panel members is much appreciated by the peer community. In the following paragraphs, the panel seeks to summarise the responses and to illustrate the generic nature of the comments by quoting from them. Question 1: How robust and fair do you believe the EPSRC Peer Review Process to be? Almost every response concurs with the view that the EPSRC peer review process is robust and fair, within the limitations that could reasonably be expected of any such process. Many aspects of the system are considered to work well, for example, the right to nominate reviewers, reviewer anonymity, the requirements for panel members to moderate reviews rather than re-review proposals (as was claimed to be case for other research councils) and the right of academics to reply to reviews. Comments from industry broadly reflect the views of the universities, although there is perhaps inevitably a slightly lesser sense of engagement and ownership: “Industry is also in general woefully ignorant of the process and will moan, but if they engaged more … they would understand the merits of the process.” Question 2: How open and transparent do you consider the process?
12
The responses are again generally positive, although many respondents would value greater transparency around, for example, the exact mechanism used to select reviewers for grants and the precise role of the National Importance and Impact sections on the review form. Putting a summary of Review Panel meetings in the public domain was also suggested. “ […] a member of EPSRC staff, who selects the reviewers for a grant application, has immense powers to favour or disadvantage a proposal by judicious choice of the reviewers that a proposal is sent to for primary review.” “The major weakness, although it is understood how difficult this is to manage as a funder, is the lack of feedback from the panel meeting itself. Academics are keen to learn how to improve their grants and would greatly appreciate even limited feedback that will give them guidance and drive up quality in the future.” In this context, instances are reported of proposals being graded 6, 6, 6, 5 (or similar) by reviewers and then ending up in the bottom half of the panel list, suggesting that the precise moderating role of the panel needs to be made more transparent. “The view of a number of [our] researchers is that there can be too much ‘re-refereeing’ even though this is explicitly forbidden; the extent to which this happens is dependent on the quality and strength of the chairperson.” There is also concern that ‘outlier’ or ‘rogue’ reports could unduly influence the panel’s judgement. A number of respondents comment on the lack of transparency around EPSRC’s input into peer review decisions, particularly with regard to the Shaping Capability agenda and on where the funding cut-off is placed. They encourage the research council to disseminate to the community detailed information on the ‘contextual briefings’ supplied to the panels, and the steps taken when there is a clash between the excellence of a proposal as judged by peer review and its fit to EPSRC’s strategic objectives. There is quite widespread concern, particularly from industry, that proposals of excellent research quality can be ‘locked out’ of the funding system should they fail for reasons not attributable to research quality. “Novel ideas can be deprioritised due to their risky nature.” Finally, there are a few suggestions that those universities with excellent track records in securing funding enjoyed a privileged position, with a perception that the Peer Review College is predominantly populated by academics from the larger universities: “[Is the process open and transparent?] In theory it is, but if you are not in a partner or managed university … the process [can] feel more distant and mysterious.” Question 3: How cost-effective and efficient do you believe the EPSRC Peer Review Process is? 13
This question elicits more of a mixed response, with some respondents rating the ‘efficiency’ more highly than the ‘effectiveness’. A number of comments relating to these criteria have already been made in answer to earlier questions. Overall the process is felt to keep to tight and reasonable timelines, although it is claimed that with the introduction of an additional outline proposal stage in many calls, the total turnaround time from the submission of a proposal outline to the funding of a project can be up to 18 months. This raises the potential for the ‘new’ ideas proposed to become stale or dated by the time they are funded. A faster evaluation process is considered particularly beneficial where industrial involvement is significant. Many organisations are keen to work with EPSRC to ensure that their researchers contribute more effectively to the peer review process: “[Our university] is committed to ensure that our academic staff contribute to this effort, and we review and act on any information provided to us by EPSRC in the event that it is drawn to our attention that [our] applicants are not contributing to the peer review of other’s applications.” There are also calls for more international and non-academic reviewers to be involved in the process, especially for larger calls involving a wide cross-section of the community, although as one respondent warned: “Often overseas or non-academic reviewers do not understand the appraisal game and so give marks of less than 6 without meaning to kill the activity.” There is general acknowledgement that a balance needs to be struck between efforts driving quality and their cost. “It would be easy to say that spending longer deliberating with more reviews and greater opportunity for PI feedback would be helpful, but there is a point of diminishing returns and our feeling is that the current balance is probably about right” Question 4: How does the EPSRC process compare to other funders? There is almost universal agreement that the quality of the EPSRC peer review process is at least as high as that of other funders, and while for many the difference in quality is particularly marked in comparison with the various EC funding schemes, some respondents commenting that the EC Panels’ focus on a single topic area from a call, rather than having to deal simultaneously with several subject areas. Unfavourable comparisons are also made between the clarity of the ERC’s focus on the quality of the PI and the research project and EPSRC’s inclusion of additional and more ‘nebulous’ Importance and Impact criteria. ERC’s policy of copying all correspondence to the applicant, including feedback of referees’ comments, to the university administrative authority was also commended. Question 5: Do you have any other comments on the process? A number of suggestions for improvement were submitted. For example, it is felt that all research councils should use the same forms and processes to facilitate 14
the submission and review of projects that bridge research council subject areas. The reinstatement of ‘mock panel’ training that was formerly offered to universities is also recommended. It is argued that this helps applicants to understand some of the issues that panels had to contend with, and provided reassurance to the community that the process was robust and fit for purpose. It can also help to improve the internal (i.e. university) support that is provided to applicants, thereby ensuring that the highest quality applications are submitted for peer review. A number of respondents suggest that consideration should be given to the use of ‘double blind’ reviewers, who do not receive information on the applicant, working alongside standard reviewers who receive the full documentation. It is argued that this would help ensure greater accuracy in judging the quality of the proposal, although the additional administrative burden associated with implementing this is acknowledged. Finally, it is suggested that EPSRC consider the appointment of an independent ‘peer review ombudsman’ to whom difficult cases can be referred in the event that an applicant, supported by their University, believes that they have been unfairly treated by the Peer Review process and where EPSRC has been unable to resolve their complaint.
15
6.3 Overview of Results from Survey The survey consisted of nine questions five of them aiming to clarify the participants’ involvement with EPSRC peer review process and four of them asking opinions concerning understanding, fairness, transparency and efficiency of the peer review process (Annex 5). A total of 756 replies were received which equates to 2% of EPSRC’s active community1. 83% of the participants have reviewed a proposal or participated in a panel for the EPSRC in the last 3 years. 79% have applied for EPSRC funding in the last three years. The survey results are thus representative of a community involved in the EPSRC peer review process. They reflect the opinion of non-involved researchers only to a very limited extent. This is very clearly reflected by the overall high scores for the understanding of EPSRC peer review process as shown in Figure 1 below (and Annex 6).
Figure 1: Boxplot of answer scores to survey questions on (Q6) Understanding, (Q7) Fairness, (Q8) Transparency and (Q9) Efficiency of EPSRC peer review process. The bottom and top of the shaded areas correspond to first and third quartiles and the solid line within the shaded area is the median. More often than not, the appreciation of the EPSRC peer review process is positive and there is a general agreement on the process’s openness and transparency, fairness and efficiency and effectiveness as seen from the scores. In order to better understand and visualise the survey data we carried out a multivariate analysis of the data which is presented in Annex 6. The analysis reveals that college and panel members tend to have higher opinions about the fairness of the EPSRC peer review process. In fact, the higher the participation in the EPSRC peer review process, the higher the appreciation tends to be. Interestingly, participants who have not applied to EPSRC funding also have more favourable opinions. On the other hand, applicants left out from the process of peer review tend to have the lowest opinions about the process. 16
However, there are over 200 pages (A4) of comments written by those who filled in the questionnaire. Within the given time frame, the panel could not undertake a detailed analysis all of all these comments. However, these comments were read and some used as evidence within our report. The participants’ answers concerning the three most liked and the three least liked features of EPSRC peer review process (questions 9a and 9b are described in Annex 6) were analysed semi-quantitatively. Also, the numerous suggestions for improvement made by the participants (question 9c) were looked at and visualised by a word cloud, Figure 2.
Figure 2: word cloud representing survey respondents suggested improvements to EPSRC’s peer review process. There is a clear vote for ‘Ability to respond to reviewers’ comments’ (14% of responses) as the most liked element. The second most liked feature (panel operation and process, 9%) figures at the same time on top of the list of least liked feature (10%), followed by ‘quality’ and consistency of reviews (9%). In general, the low frequencies of all other answers suggest a high diversity of opinions. In the suggestions for improvements, the most used words are reviewer, review, proposal and panel, which is in itself not meaningful. In addition, the words 'comments', 'feedback' and 'process' came out. No sound conclusion can be drawn from this preliminary screening, and the panel strongly recommends that EPSRC analyse the individual comments in depth assuming that they are fully anonymised.
17
7. RECOMMENDATIONS Within this section, we have divided each topic into the description of the primary issues, the evidence for these, the Panel’s views and our Recommendations. We use evidence from the oral sessions, the survey and associated comments and the written submissions. We use the words ‘peer community’ through-out but most of the responses came from the academic community. Issue 1:
Peer Review should be at the heart of EPSRC’s mission
Everyone agrees that peer review should be at the heart of EPSRC’s mission and that it is the best way of ensuring the high quality of UK research. The key question is does this happen in practice. Evidence from staff including the Chief Executive made clear that it is at the heart of what they do and that staff feel that they own the process. There is evidence of considerable thought about and evolution of the peer review process since EPSRC was formed. There was some discussion that staff involvement in developing strategy may now be seen as more important than the actual running of the peer review process within EPSRC itself. The ‘peer’ community agree that peer review is at the heart of activity but some felt that in some initiatives it was perhaps being by-passed. For example, Issue 3b below. In the panel’s view there should be more emphasis on peer review having shared ownership between EPSRC staff and the peer community, but it is pleasing that the staff clearly have ownership of running the process. We accept that there does have to be a disinterested party running the assessment to see fair play and that the community cannot do this without EPSRC. Recommendation 1: EPSRC should ensure that their communications and associated actions emphasise that peer review is seen as a shared endeavour. Moreover, these should emphasise the primary role that peer review plays in EPSRC decision making. Within EPSRC, Council and the Executive should ensure that those of its staff who are involved in peer review feel valued.
Issue 2: EPSRC’s peer review process is generally held in high regard The overwhelming view from peers is that the overall process is a good one and many commented on how it was superior to some processes used for example for European Union grants. Features that are almost universally praised are: postal peer 2 reviews managed by EPSRC, with the right of reply from the PI being a strength over international systems; no major changes to matters of fact are imposed by panels. This latter point is on balance favoured over other RCs where panels can take much more drastic action without any right of reply. This is evidenced by oral evidence and written submissions. 2
Postal peer review describes individual written comments received from reviewers
18
“Overall we believe that the EPSRC peer review process is quite good, and while there are minor elements that could be improved, it is definitely a robust and transparent process that succeeds in funding world-leading research.” “Generally considered to be good compared to others where for example BBSRC and NERC are rather opaque, and their panels re-review, hence demeaning the full peer-review process. EPSRC doesn’t do this as a rule, so is much the fairest system.” The panel heard very positive comments about the rigour of the structure of the process and the PIs right to reply to the referees’ comments before the panel was convened. Representative positive comments to survey question: “Which three things do you most like about the process?” are: “(1)It is quite transparent, in comparison to the other UK charities and funding agencies. (2) The applicant has the opportunity to reply to the referees' comments and this reply is taken into account at panel level. (3) A poor referee report can still be overturned at panel level if the panel suspects that the referee had a conflict of interest.” “(1) Applicants' ability to respond to criticism (2) Relatively non bureaucratic (e.g. compared with EU) (3) Panels take great care to be as fair as possible.” “(1) It is a genuine attempt to make informed decisions on what to fund (2) EPSRC try their best to run it 'as advertised' (3) The fixed panels of yesteryear are gone: they did a worse job.” However, the survey shows that many are still worried by the panel’s process and operation, the quality and consistency of reviews, selection of reviewers, fairness, transparency and efficiency. The right of reply is the only feature to get unequivocal positive views. The panel’s view is that the overall process is far from broken and all recommendations should be seen in this light. The panel is not asking for a complete re-think of the peer review process, and the number and detail of the issues raised subsequently should not obscure the overall satisfactory structure of the process used currently.
Issue 3: Many concerns were raised by the community about the execution of the different stages of the peer review process Having said clearly in the last section that the process is far from broken, there are a considerable number of issues raised by a wide section of the peer community about the execution of the specific sub-stages of the process. In general terms, these relate to transparency and effectiveness of the process at an individual application or panel level. The panel has focused on a number of these resonant themes (arising from the written responses and questionnaire); ones that the panel judge to be important. These are addressed below.
19
The panel is clear that these issues are from across the community and need to be addressed by EPSRC. Some may be about perception rather than reality but again EPSRC needs to look at these and not just assume that they come from a small lobby group. The panel makes a number of recommendations that relate to these issues, which are set out in the subsequent points.
Issue 3a: The use of peer review to comment on national importance, and impact Postal Peer Review Stage There is considerable disquiet amongst reviewers about being asked to comment on national importance and impact on the postal referee forms. Peer reviewers do not feel comfortable in these areas and would prefer to focus on the quality of the science. This may also relate to the lack of buy in by some potential referees, as discussed under Issue 4. We recognise that EPSRC has adopted a strategy that requires assessment of national importance and of impact, as well as fit to portfolio, but we feel that the forms filled in by postal referees, with sections of equal length on Quality, Impact, National Importance and Resources and Management, do not give the primary focus on quality of the science and cause most concern to reviewers. Survey comment: “The "impact" and "national importance" stuff puts of (sic) most of the best people from refereeing, and pretty much all foreigners.” The panel noted that equal space is given for these areas as well as a review of the science on the referees form. As no information is given about the relative weighting of these areas, this gives referees the view that these are potentially all equally important. As above, this can lead to reviewer disengagement with the process. Survey comment: “The form that referees have to complete is way too complicated and repetitive at places. At the end, it should all be about good science. I recently reviewed a proposal for a US funding body and was surprised how straightforward the review form was.” Having commented on all parts of the form, a referee is asked for only one score. They are then in a position of having to weight the relevant areas in their own minds. It is understandable that a proliferation of individual scores may lead to the risk of the panel simply adding these up to assess the quality of a proposal rather than considering the tenor of the written comments and associated evidence, but it is currently unclear how reviewers and panels weight the different and multiple criteria that are requested. The risk is that all four sections are equally weighted, simply because they have equal length, which will not give a primary focus on quality of the science.
20
Panel Stage There is also mistrust of how the priority order of the grants is produced. The panel meetings start with grants in descending order based on the referees’ scores. We did not find detailed evidence of how national importance and impact are used. “Strategic criteria of national importance or impact normally influenced a reviewer or panel member when it was missing from a proposal that could easily demonstrate impact or national importance.” “Panels would expect proposals that are more applied to be able to demonstrate relevance of research in an applied context through a more developed impact plan.” Perhaps they are not used in which case one asks why referees are asked about them. If they are used, then it should be made clear how this is to happen. It is clear that the introduction of these different criteria have added considerably to the complexity of the review process and thus to the difficulty of successfully demonstrating transparency to the community. These problems may be a reflection of the difficulty of addressing issues other than quality in peer review processes. We recognise that there are more high quality grants than can be funded. A lot of time goes into producing a rank order in which perhaps 80% are not funded. Moreover, the panel does not know and is not consulted about the ‘spend’ cut off. This seems to be left to EPSRC staff to make this decision post the panel meeting. These issues lead to confusion and a perception of lack of transparency. In the panel’s view postal peer referees should comment and score principally on the quality of the science and they should be asked to comment, if they wish, on the strategic issues as far as is appropriate for the individual proposal. EPSRC should make very clear to the panel how the strategic criteria are to be used through the contextual briefing. This briefing should continue to be made available to the community after the panel. Recommendation 2: EPSRC should think through more clearly how the strategic criteria are incorporated into the processes and documentation. In particular we recommend that the postal peer review form is changed to reflect a primary focus on quality of science and that the panel process in the use of strategic criteria is better defined and communicated.
Issue 3b: Concerns were raised about processes that did not involve postal peer review Given the strong support noted above for the structure by which EPSRC runs postal peer review, it is perhaps not surprising that concerns were expressed both by institutions and individual respondents concerning occasions where full postal peer review with right of reply is not used. 21
“The postal / traditional peer review system is as transparent as it can be, but in situ and EoI review perceived as much less so. So whilst the process is open and transparent, it is unclear around the consistency of approach... The only issue with EPSRC is the increasing use of EoIs, which can be less transparent, meaning that full peer review only happens at the final stages. This is unfair to the majority of applicants, who don’t benefit from full review, and also tends to inflate the ‘success rates’, as only those making it past the EoI stage ‘count’ in the overall success rate published by EPSRC. This is disingenuous practice.” “Against this positive context, some recent, ‘shorter’ assessment processes have been flagged as less open and transparent. Some calls have not included an opportunity to respond to, or see, (sift) panel assessments, for example. Whilst the time pressures are very much understood, this can be perceived as a less than transparent process.” Survey concerns: “Short listing panels that operate without expert peer review and have to evaluate proposals outside their expertise” “Decisions by panels (without expert refereeing) can have huge funding implications” The panel believe that the default position should be peer review for grants of a significant size. Recommendation 3: EPSRC needs to articulate why peer review is not being used for distribution of funds in specific cases and to manage the risks that it generates.
Issue 3c: Other points There are a number of other points concerning the peer review process which led to concern or comment including the following: (i)
Insufficient emphasis on investigator track record in assessing proposals
There is clearly a problem with assessing this for early career researchers, but notwithstanding that the survey reveals concern that the current processes do not give due emphasis to the recent track record of the project team: “Insufficient attention is being paid to the previous research of the applicants in the relevant areas, in particular to their published papers.” “Funding should be based on regularly-reviewed track record.” “The track record should play a major role. Why not to give a weight to it?” “Lack of linkage of awards to credibility” (ii)
Feedback from the panel
22
The opportunity to respond to the postal referees is very much appreciated by the peer community. However, the lack of feedback from the panel has been noted. The lack of this feedback is especially felt when a proposal is not funded or way down the priority listing even though the postal referees scores were very high. (iii)
Panel freedom of action
There were frequent comments about a single inappropriately chosen or nonevidenced technically dismissive or inaccurate review “sinking” a proposal – this is where expert, respected panels will play a key role in maintaining community confidence, and is why panel member and chair selection is a critical part of the process. “The panel has no mechanism by which to completely disregard a review” “Panels must be given more scope to dismiss referee reports where either the referee is poorly chosen or clearly has not done the necessary work to provide a proper review. The rule "thou shall not re-referee" must not override the judgment of genuine peer knowledge.”
Issue 4:
Lack of participation in the peer review process
We heard from both staff and peers that some researchers are not participating appropriately in the peer review process. EPSRC Staff are aware of who responds to their requests in a timely manner whether for postal references or attendance at panels. There are others – may be those at the top of their fields – who, although in receipt of funding, are not participating sufficiently in peer review. The reasons for this are not clear but may relate to time pressures on individual academics. There is no evidence that the incentive payments are helping. There is a view that the quality of the refereeing is not as good as it should be because some potential referees are not participating and thus the staff have had to use others. “As far as the actual process goes, more of an effort needs to be made by EPSRC in identifying the quality of referees and appreciating that having time on your hands and an appetite for submitting reviews by deadlines is not necessarily a qualification.” “The robustness and fairness depends intrinsically on the use of appropriate specialist referees. Since in the College systems referees select their own expertise, in the EPSRC database there can be a potential problem. From our experience it appears that refereeing is often declined by senior experienced members of the community and so in order to ‘get three referees ‘ less suitable people may be used. It remains a challenge to ensure that there is a robust system for ensuring better selection of appropriate referees.”
23
“We are concerned that the robustness of the process for a significant minority of proposals is compromised by the failure to select reviewers with appropriate expertise.” There is a strong view that the current peer review forms do not help. As described above (Issue 3a), we found ‘peers’ do not buy into the use of comments by peer reviews on national importance or impact. They are of the view that it is the quality of the project that is of importance and they do not feel confident in responding outside this area. Moreover, it is felt these other areas would not be understood by international referees. “What are different between EPSRC and other funders are the aspects of the proposal that EPSRC asks reviewers to comment on and that reflects the different criteria used for assessment of proposals. For example, the ERC (and most other international funders) reviewers comment form is very short and asks for assessment of the PI (track record and creativity) and research project (ground breaking nature of the research, potential impact, methodology, high gain/risk balance). In contrast, the EPSRC reviewer form is much longer and requires an unpaid reviewer to devote considerable time and effort to how the proposal meets more nebulous criteria such as Importance and Impact.” “Email top foreign people with an attached proposal and ask them for an expert opinion! They will reply, but if you ask them to grapple with a web system and comment on national politics they will ignore you! You are currently getting reviews only from a small subset of domestic people, which is a huge problem.” In the panel’s view, it is the best qualified researcher that should be used to review proposals and we are clear that those benefiting significantly from grant funding should be most willing to participate. Recommendation 4: EPSRC should consider how the structure of the postal referees’ form affects engagement from the community, including international referees, in order to maximise assessment of the proposals by the best and most appropriate experts. Recommendation 5: EPSRC should reinvest the funds currently used for the incentive payments (£1M) into science. Recommendation 6: EPSRC should work closely with HEIs to ensure participation in Peer Review by those who benefit from considerable funding.
Issue 5: Choice of Referees and Panel members One area of general concern is the choice of postal referees and panel members. This choice lies wholly with EPSRC staff (although one referee is chosen by EPSRC from a list of three submitted by the applicant). We recognise that the ‘portfolio managers’ tried hard to find, and in many cases do find, the most appropriate people. It is assumed that referees and panel members come from the College. 24
“Panels having to compare chalk with cheese and either lacking expertise/experience or not being able to overrule incorrect referee comments” On request, we found that only 56% of reviewers and 76% of panel members came from the College in 2012/13. Some senior academics did not know whether they were still members of the College or not. It is not clear to many in the community, nor to the panel, how the current 4000 College members come to be College members whereas the community may be as large as 37,000. Cuts in funding have led to reduced training for college members. The panel is concerned that a significant number of people refereeing grants or attending panel meetings are not from the College. As they are chosen entirely by EPSRC staff, there is an issue over transparency regarding the criteria for referee selection. We emphasize that staff may well be making the right decisions but there seems to be little oversight of this to further enhance the quality of referee selection. “It might be helpful if a list of all reviewers used by each EPSRC Theme were also made available (maybe on a rolling basis) since it would increase transparency and confidence in the appropriateness of the reviewer choice.” “Reviewers at the core of peer review and I found the on-line training for college members very useful and insightful and expressed an interest in completion rates. The reintroduction of mock panels as they were highly valued by the community would be favourable.” Recommendation 7: EPSRC should consider the status of the College. The panel cannot see justification for its existence when so many referees are not chosen from the College. Recommendation 8: EPSRC needs to reassure the peer community that the choice of referees and panel members is transparent, appropriate with ownership and oversight by the peer community.
Issue 6: Efficiency and effectiveness The panel heard from the Chief executive and others that administrative costs, including those involved in peer review, are now 1.5% of the total budget. Further cuts are possible. This brings into focus whether there are sufficient funds in a process which is at the heart of the EPSRC community. We were also given a description (Annex 3) of the review process. It may be that that EPSRC with its peer community need to look at whether this process can be made more efficient without losing the key ‘peer review’ element. For example: •
A large number of applications are returned to HEIs at the very beginning of the process with administrative questions or questions around justification of resources. If on average 80% are not funded, we question why this is necessary at this early stage. 25
•
A lot is made of every panel producing an ordered list of grants. Again we cannot see the need for a rank order for all grants or at least they could be done in blocks rather than individually as again we point out that majority of grants are not funded.
The panel would have serious concerns if further monies are taken away from the peer review process. Recommendation 9: EPSRC should work with the community to identify and capture efficiencies but also to maintain sufficient funds for an effective peer review process to take place.
Issue 7: Transparency and engendering community ownership The peer community is committed to peer review. They have broad concerns about parts of the process which may be about perception of the role of EPSRC. We have mentioned the choice of referees and panel members. There is also the lack of transparency over the ‘spend’ per panel meeting. The panel think that EPSRC could do far more to encourage shared ownership involving peers far more critically in the process rather than handing over critical decisions to staff who appear to have ultimate control over the allocation of funding to different subject areas and award schemes. For example, a high level group drawn from the peer community could review the choice of panel members and panel chairs and oversee the choice of referees. In the panel’s view how the EPSRC works with the peer review community to tackle issues of transparency is the most important message to come from this Review. In particular we feel that consultation with the community about the issues and recommendations raised in this review would be a good starting point and provide a good opportunity to build ownership and greater confidence in the role of EPSRC itself. Recommendation 10: EPSRC should look to engender a far greater shared ownership in peer review giving the peer community more input, ownership and oversight into different aspects of the process.
General comment In conclusion, the EPSRC should be proud of its focus on peer review in making decisions on the quality of science it supports, and should ensure that the primary role of peer review in the organisation is further strengthened. It is clear from the responses that the peer review process as a whole is not broken but there are wide spread concerns about individual aspects. Our highest priority recommendation (10) is that there should be far more focus on a peer review process shared between staff and the peer community with clear oversight from the broader community. We urge EPSRC to use discussion of this review with the peer community as a starting point for a renewed focus on two-way communication. 26
8.
ANNEX 1 – PANEL MEMBERSHIP AND PROFILES
PROFESSOR DAME JULIA GOODFELLOW
Professor Dame Julia Goodfellow has been Vice-Chancellor of the University of Kent since 2007. She is a board member of UniversitiesUK, and was appointed to both the Science and Technology facilities Council and Prime Minister’s Council for Science and Technology in 2011. Before joining Kent, Julia was Chief Executive of the Biotechnology and Biological Sciences Research Council (BBSRC) which she joined from Birkbeck College where she had been Professor of Biomolecular Science, and Vice-Master. She was the first woman to become Chief Executive of any of the UK Research Councils. MR DAVID EYTON FREng
As Group Head of Technology, David Eyton is accountable for technology strategy and its implementation across BP, conducting research and development in areas of corporate renewal and overseeing the technological capability of the company. Prior to this, David was BP’s Upstream Group Vice President for Technology including responsibility for research and development, technical service work, IT, procurement and supply chain management.
27
David joined BP in 1982 from Cambridge University with an Engineering degree and has held Upstream General Manager roles in Australia, the UK, Trinidad and the Gulf of Mexico. DR ANGELIKA KALT
Angelika Kalt was appointed Deputy Director of the Swiss National Science Foundation (SNSF) in 2008. She is responsible for quality assurance and development of the organisation’s evaluation principles and procedures. She studied geology in Freiburg, Germany, and received her doctorate at the Universities of Freiburg and Münster. After several brief stays in Holland and France, Dr. Kalt became a full professor of petrology at the University of Neuchâtel, Switzerland, in 2000. Her main research topics were modern and ancient chemical and mineralogical processes in the Earth’s mantle as well as mountain-building building processes and paleogeography of different parts of Europe. Besides her broad expertise in petrology, geochemistry and mineralogy, she can draw upon many years of experience in designing and carrying out research projects, fund-raising, journal and proposal reviewing and teaching on all academic levels.
28
PROFESSOR MATTHEW ROSSEINSKY FRS
Matthew Rosseinsky is a materials chemist who has worked at the University of Oxford and at Bell Laboratories, and since 1999, in the Department of Chemistry at the University of Liverpool. He was elected a Fellow of the Royal Society in 2008. He has received several awards, including the inaugural de Gennes Prize of the Royal Society of Chemistry in 2009, the C.N.R. Rao Award of the Chemical Research Society of India in 2010, and the Hughes Medal of the Royal Society in 2011. Since March 2013 he has been a Royal Society Research Professor. He has published over 300 papers, including 11 in Nature and 7 in Science. PROFESSOR JAMES STIRLING CBE FRS
James Stirling is the Provost at Imperial College London, responsible for the College’s core academic mission in research, teaching and translation.James studied Mathematics at Cambridge University, obtaining his PhD in Theoretical Particle Physics in 1979. Between 1986 and 2008 he worked at Durham University, latterly as Pro-Vice-Chancellor for Research. He returned to Cambridge in 2008 to take up the Jacksonian Professorship of Natural Philosophy in the Cavendish Laboratory, and was appointed Head of Department in 2011. In 2013 he moved to Imperial to become the College’s first Provost. His research area is theoretical particle physics, with particular emphasis on the physics of high-energy particle colliders such as the LHC. He has published more than 300 research papers, including some of the most frequently cited papers in the physical sciences. He has been a Fellow of the Royal Society since 1999 and in 2006 he was awarded a CBE for services to science. He is currently a member of 29
the Council of the Science and Technology Facilities Council (STFC) and HEFCE’s Strategic Advisory Committee for Research and Knowledge Exchange. Return to Main Report
30
9.
ANNEX 2 – PURPOSE, SCOPE AND OPERATION
9.1
BACKGROUND
At its meeting in October 2012 EPSRC’s Council discussed the relationship between the executive and its stakeholder community as a result of concerns amongst some stakeholders about decisions that had been taken (e.g. shaping capability, project studentships) in the context of a reduced budget and increased international competition. The concerns that had been raised related variously to the decisions themselves and the process by which decisions were taken. Council concluded that it was important to acknowledge these perceptions and to demonstrate that EPSRC is listening and responding to these. It therefore proposed a number of communication activities to restate and re-emphasise the rationale behind specific strategic decisions and the changes that resulted. Council also agreed to commission an independent and transparent review of how strategic advice is obtained within EPSRC and a separate review looking at the advisory and decision making processes associated with individual research grant funding, namely, the Review of Peer Review. Both reviews will have some key generic features: • • • • 9.2
Be totally independent –membership will be guided by Council and not the Executive; Be free ranging – each panel can choose what to look at or how they conduct the review; Have unlimited access - each panel will have full access to any documentation, data or people that the panel feels appropriate to aid their deliberations; Be transparent - the outcome will be reported to Council and will be made public together with our response. PURPOSE
The EPSRC uses peer review to deliver its Royal Charter on behalf of BIS. The visiting panel will undertake an independent review to look at the nature of the expert advice EPSRC seeks on research proposals, how this advice is solicited and how it is utilised to inform grant funding decisions. 9.3
SCOPE
The visiting panel will be responsible for defining the precise scope of the review. Areas which we would expect to investigate are: • •
•
Nature of advice being sought – with regard to the fact that peer review must appropriately address EPSRC strategic priorities, is the advice sought appropriate to support effective decision making; Structures – to review the adequacy and effectiveness of the peer review structures in place for obtaining expert advice, particularly, the structure and operation of the Peer Review College and arrangements for peer review prioritisation panels; Credibility of Advisory Sources - assurance over the extent to which the individuals providing advice are appropriate and credible sources; 31
•
• • • 9.4
Roles and responsibilities – to review the clarity of roles and responsibilities of those involved in soliciting, and providing, expert peer review advice in EPSRC, and of those using this advice to make funding decisions; Provision and utilisation of advice – to review how effectively and fairly the peer review advice provided is used in informing EPSRC’s funding decisions; Transparency – to review the clarity and transparency of the peer review processes to EPSRC’s wider stakeholder community; Cost-effectiveness of the mechanisms and processes used. MEMBERSHIP
Members will be drawn from senior and highly regarded individuals from academia, business and a peer organisation. At least one member will be drawn from an overseas organisation. Members are not required to represent their own organisation nor to represent their entire sector. The membership will be appointed by the Chair and CEO of EPSRC in conjunction with a sub-group of Council members. 9.5
METHODOLOGY
The visiting panel will define its own methodology and schedule of tasks in keeping with the desired scope of the review. This might include: • • • • •
Review and scrutiny of documentation and data; Interviews with EPSRC staff; Consultation with members of current EPSRC advisory streams and Council; Discussions with members of EPSRC’s key stakeholder groupings Making reference, where appropriate, to contextual information about international analogue organisations.
The Panel will have access to all areas and staff of EPSRC. The Panel will be free to request access to any documentation or data considered appropriate for the review tasks. A designated member of EPSRC will act as Secretary to the Panel for the duration of the review and will be responsible for arranging meetings/interviews for the Panel and ensuring that all necessary documentation and requested information is provided in a timely manner, and that a record is maintained of the panel’s work. 9.6
TIMESCALES
The visiting panel will begin its review in July 2013 and will report to Council meeting scheduled 2nd December. The visiting panel will determine the number of meetings that will be required based on the agreed scope and modus operandi.
32
Issues that the panel would not wish to include in a report should be covered in an oral report from the Panel Chairman to the Chair and CEO. Return to main report
33
10.
ANNEX 3 – EPSRC’S PEER REVIEW PROCESS (return to main report)
34
11.
ANNEX 4 – INVITATIONS TO SUBMIT EVIDENCE
Higher Education Institutes (HEIs) All UK based HEI’s eligible to receive funding HEI Representative Groups & other groups The Campaign for Science and Engineering Science for the Future Campaign 1994 Group Universities UK ARMA Recent Appointees in Materials Science
Council for the Defence of British Universities AIRTO Russell Group Million Plus Recent Appointees in Polymer Science University Alliance
Learned Societies Biochemical Society
British Association of Academic Phoneticians British Computer Society British Pharmacological Society British Science Association
British Society for Research on Ageing Institute of Chemical Engineers
Institute of Food Science & Technology Institution of Structural Engineers
Royal Academy of Engineering Royal Pharmaceutical Society Royal Society of Edinburgh Institute of Physics 3 Strategic Partners (Industry) Baesystems BP Rail Safety and Standards Board NPL Matthey QinetiQ NNL EADS EON BT ARUP Rolls Royce AWE Dyson NDevr
British Association for Applied Linguistics British Biophysical Society British Neuroscience Association British Psychological Society British Society for Proteome Research British Society for Soil Science Institute of Engineering and Technology Institute of Mechanical Engineers Institution of Civil Engineers Institution of Mechanical Engineers
Royal Institution Royal Society
Syngenta CESG Mobile VCE GE Thales group EDF Energy ARM EADS Astrazeneca Unilever GSK Jaguar Land Rover Pfizer Siemens Airbus
Return to main report 3
The Institute of Physics was asked to provide input to the report in December 2013 as they did not receive the original invite letter to submit their views.
35
REVIEW OF EPSRC’S PEER REVIEW PROCESS Dear I am writing to you in my capacity as Chair of an independent Panel which is charged with reviewing the EPSRC peer review process. This review was commissioned by EPSRC’s Chair, Dr Paul Golby, and will report to EPSRC Council by the end of this year. Further information about the background to this review, the membership of the panel, and how we intend to undertake the review can be found here The purpose of this letter is firstly to draw attention to a community survey, available at http://www.survey.bris.ac.uk/epsrc/irpr_survey open until 30th September 2013, and to ask that you encourage members of your society for whom this would be relevant to take the opportunity to feed in their individual views. Secondly I would like to invite you to submit written evidence to this review based on your [group] perception of the EPSRC peer review process. Please feel free to comment on aspect of the process you feel to be appropriate, but note the panel would particularly value your perception of: •
The level of understanding of how this process operates and its usability;
•
How open and transparent the process is;
•
The efficiency and cost effectiveness of this process
•
How the EPSRC process compares to that of other funders
Please submit your responses directly to
[email protected] by 30th September 2013. Finally I can assure you that all responses will be treated in the strictest confidence and any reference made to these in the ensuing report will not be attributed. Yours sincerely
Professor Dame Julia Goodfellow
Return to Chapter 5 Return to Chapter 6.2
36
12.
ANNEX 5 – COPY OF ON-LINE SURVEY
Independent Review of Peer Review: Community Survey Thank you for deciding to complete this survey. Your views will be of great help to the panel in reaching its conclusions. You will note that we have asked respondents to this survey to identify themselves. This is simply so that the panel have the option to be able to follow up on information provided where this might be appropriate. You can however rest assured that all responses will be kept in the strictest confidence and any reference made to these in the ensuing report will not be attributed.
Section 1: Demographics 1. Your Name
2. Your Organisation
3. Have you applied for EPSRC funding in the last three years?
Yes No 4. Do you currently hold an EPSRC grant as either a Principle investigator or a Coinvestigator?
Yes No 5. Are you a member of the current EPSRC Peer Review College?
Yes No 6. Have you reviewed a proposal for EPSRC in the last three years?
37
Yes No 7. Have you served on an EPSRC panel in the last three years?
Yes No
Section 2: Process Approval 8. Please indicate using the 10 point scale (1 low - 10 high) your opinion of the EPSRC peer review process in terms of your confidence in:
please use the drop down list for each row to add you answer a. Your understanding of this process b. The fairness of this process
Select an answ er
c. The openness and transparency of this process d. The efficiency and effectiveness of this process
Select an answ er
9. Please indicate
Please enter your answer here
a. Which three things do you like most about the process?
38
b. Which three things do you like least about the process? c. What could be added to improve the process?
10. Please use this section to provide brief additional comments about any aspect of EPSRC peer review not covered elsewhere in the survey that you wish to bring to the panel's attention (Optional)
Return to main report (Chapter 5) Return to main report (Chapter 6.3)
39
13.
ANNEX 6 – DETAILED ANALYSIS OF SURVEY RESULTS
Description of the Survey Data - Quantitative We consider data collected from an on-line EPSRC survey which was brought forth to evaluate the EPSRC peer review process. The survey was based on a questionnaire which consisted of 9 questions and was answered by 756 participants. The first 5 questions were Yes or No questions concerning the overall involvement of the participants in the process whereas the remaining 4 questions concerned the participants' appreciation of the process and required a mark between 1 and 10. The questions and the summary statistics of the respective given answers are the following: 1. [Applied] Have you applied for EPSRC funding in the last three years? No Yes 161 595 2. [Grant.Holder] Do you currently hold an EPSRC grant as either a Principle investigator or a Co-investigator? No Yes 348 408 3. [In.College] Are you a member of the current EPSRC Peer Review College? No Yes 234 522 4. [College.Active] Have you reviewed a proposal for EPSRC in the last three years? No Yes 138 618 5. [Panel.Active] Have you served on an EPSRC panel in the last three years? No Yes 513 243 6. [Understanding] The understanding of the process. (cf. boxplot) 7. [Fairness] The fairness of this process. (cf. boxplot) 8. [Transparency] Openness and transparency of this process. (cf. boxplot) 9. [Efficiency] Efficiency and effectiveness of this process. (cf. boxplot)
Summary statistics of the answers reveal the following points: •
in the last 3 years, 82% of the participants reviewed a proposal for the EPSRC and 79% applied for EPSRC funding 40
•
most participants indicate very high understanding of the EPSRC peer review process. (82.5% of the participants scored question 6 higher than 7.)
Multivariate analysis
Left: Survey answers shown in the first two components of a principal component analysis of the point scores of questions 6 to 9, with a centre on the scores {5, 5, 5, 5}. The points represent the 756 answers’ scores combination for each participant and the arrows indicate the preferential directions for each question. They define a four dimensional space which is projected in two dimensions of the principal components. The horizontal line corresponds to the first principal component (which captures most of the variance in the data) and the vertical line corresponds to the second principal component (which captures most of the remaining variance in an orthogonal direction relative to the first component). The intersection of the horizontal and the vertical lines accounts for “neutral” (5, 5, 5, 5) answer combinations. This type of plot allows to apprehend both the similarity between participants’ answers individually (by looking at the points) and the correlation between answers as a whole (by looking at the arrows). The figure was produced with the ade4 R package.
A representation of a principal component analysis of the data shows that the vast majority of the participants' answers are projected in the upper quadrants of the plane defined by the two first principal components. This indicates a very high understanding of the EPSRC peer review process by most participants. As it could also be expected, the scores assigned to openness and transparency, fairness and efficiency and effectiveness are highly correlated, i.e. their arrows’s projection (transparency, fairness, and efficiency) is close to each other. As most points spread transversally to the projection of the arrow of understanding, which is also orthogonal to the other arrows, it makes sense to focus the analysis on the point scores of transparency, fairness, and efficiency, only. A closer look at these questions with regard to the answers of questions 1 to 5 shows that the opinion of the fairness of the EPSRC peer review process is the most sensitive issue (see the orientation of the arrows relative to the horizontal axis in the figures below) — its arrow is closest to the horizontal line, the first principal component, which can be interpreted as a summary indicator of the overall appreciation of the process. On this note, appreciation is positive but that it can vary vastly from one participant to another. As it could also be expected, openness and transparency and fairness are more correlated to each other than to efficiency and effectiveness (note the angles between the arrows). The first panel shows that participants who have not applied to EPSRC funding in the last three years have (statistically significant) higher opinions about the EPSRC peer review process. This indicates that, even though it remains generally positive, appreciation decreased for those who have asked for funding. 41
The second panel shows that holding a grant also increases the generally favourably level of satisfaction with the EPSRC peer review process. The third and fourth panels however reveal a higher disparity of opinions between the participants who are members of the EPSRC peer review college or have reviewed in the last three years and those who have not. Interestingly, despite of the high appreciation given by college members, the peer review process is seen as about average for those who do not belong to the EPSRC college. The fifth panel also shows high disparity of opinions between participants who have served in panels in the last three years and those who have not. Remarkably, a few participants who have been selected to serve in panels in the last three years have nevertheless negative views on the EPSRC peer review process. The last panel shows how the appreciation about the peer review process evolves with the degree of involvement of participants, which is given by the sum of affirmative answers to questions 1 to 5. In the main, appreciation raises with deeper involvement in the process. Next page: Two-dimensional projection of a tridimensional space defined by the scores given to the questions 7, 8, and 9 with regard to yes or no answers to the questions 1 to 5. The ellipses regroup answers according to the corresponding yes or no answers for the questions 1 to 5. Again, the arrows refer to increasing scores for questions 7, 8, and 9, on the transparency, fairness, and efficiency of the peer review process. The projection is determined by the horizontal and vertical lines, i.e. the two principal components of a principal component analisys of the answers to the questions 7, 8, and 9. The first five panels refer to each of the first five questions and the last panel corresponds to a grouping of the points according to the sum of affirmative answers to questions 1 to 5. The figure was produced with the ade4 R package.
42
Conclusions More often than not, the appreciation of the EPSRC peer review process is positive and there is a general agreement within the scores on the process’s openness and transparency, fairness and efficiency and effectiveness. College and panel members tend to have higher opinions about the fairness of the EPSRC peer review process. In fact, the higher the participation in the EPSRC peer review process, the higher the appreciation tends to be. Interestingly, participants who have not applied to EPSRC funding also have better opinions. On the other hand, applicants left out from the process of peer review tend to have the lowest opinions about the process. The principal components analysis also highlighted a high scattering of opinions. It would be desirable to further investigate the opinions of specific groups of participants (for example, it would be interesting to focus on the few participants who have low views on the peer review process despite of having served in the 43
panels) and perform a qualitative analysis on the reasons of their poor opinions. Generally, given the great amount of qualitative information available, it would be desirable to take the time to best integrate it in the light of these results. Furthermore, it would be interesting to submit this same survey across different funding agencies to compare the satisfaction of applicants across different countries.
Description of the Survey Data - Free Text The following are the most remarked upon answers from the qualitative (free text) questions asked as part of the survey. These have been collated into order of the frequency they have been mentioned in the responses.
Return to main report (Chapter 5)
Return to main report (Chapter 6.3)
44
Survey Question 9a: Which three things do you like most about the process? Survey Question 9a. - Which three things do you like most about the process? Answer/Aspect frequency percentage of responses Ability to respond to reviewers comments 205 13.96 The panel operation, process and deliberations 132 8.99 The fairness of the process 108 7.35 The electronic submission system (J-eS) 103 7.01 The transparency and openness of the process 95 6.47 Reviews are conducted by experts 81 5.51 The fast speed of the process 63 4.29 There is effective and clear guidance provided for applicants and reviewers 58 3.95 Responsive mode / freedom of application 56 3.81 The fact the peer review is conducted at all 55 3.74 Receiving feedback from reviewers 54 3.67 Expertise and helpful nature of EPSRC staff 49 3.33 Proposals are judged against multiple criteria (Quality, National Importance, 45 3.06 Impact Etc.) The process is efficient 37 2.52 The application forms are concise in nature and science focused 35 2.38 Multiple reviews are obtained for each proposal 33 2.24 Forms provided for justification of reviewer scores are effective 31 2.11 The six point scoring system used by reviewers 30 2.04 Total number of responses received for question 9a = 1468
45
Survey Question 9b: Which three things do you like least about the process? Survey Question 9b. - Which three things to you like least of the process? Answer/Aspect
frequency percentage of responses
The panel process and operation The quality and consistency of reviews The selection of reviewers The perception that one poorly marked review can "kill" a proposal Length of time taken from application to funding decision Pathways to Impact and National Importance criteria Lack of expertise of reviewers Scoring systems/metrics (for reviews and rank ordering at panel) Not enough responsive mode, focus on larger projects and theme areas The application process (content and format of forms, effort required and inability to change/update the application)
170 160 98 80 79 78 72 67 66 66
9.86 9.28 5.68 4.64 4.58 4.52 4.17 3.88 3.83 3.83
The Inability to resubmit applications No feedback to applicants on the panel outcome and discussions Reviewer feedback and comments Reviewer workload and deadlines How interdisciplinary proposals are handled Lack of transparency in the process Review forms (length of, too much box ticking etc.) The process only funds the "old Boy network", large institutions and cliques of researchers
63 57 43 41 38 33 32 30
3.65 3.30 2.49 2.37 2.2 1.91 1.85 1.74
Total number of responses received for question 9b = 1723
46
Survey Question 9c – What could be added to improve the process?
Return to main report
47
14.
ANNEX 7 - EPSRC’S ORGANISATIONAL STRUCTURE and BUDGET ALLOCATION
48
