The Future of Peer Review
Irene Hames, PhD, FSB
@irenehames
Editorial and Publishing Consultant Council Member, COPE (Committee on Publication Ethics) ORCID: http://orcid.org/0000-0002-3806-8786 EASE/ISMTE Conference, Blankenberge, 23 September 2013
Not that long ago … .
Irene Hames, EASE/ISMTE, 23 Sept 2013
2
What is (editorial) peer review? Peer review in scholarly publishing is the process by which research output is subjected to scrutiny and critical assessment by individuals who are experts in those areas. (Hames, 2012, in Academic and Professional Publishing, Chandos Publishing, Eds Campbell, Pentz and Borthwick, p.16)
and …the critical assessment of manuscripts submitted to journals by
experts who are not usually part of the editorial staff (ICMJE, International Committee of Medical Journal Editors, http://www.icmje.org/)
Irene Hames, EASE/ISMTE, 23 Sept 2013
3
Enormous scale ~28,100 active scholarly peer-reviewed journals
publishing ~1.7-1.8 million articles a year
The STM Report: an overview of scientific & scholarly journal publishing, Nov 2012, M Ware & M Mabe http://www.stm-assoc.org/2012_12_11_STM_Report_2012.pdf
Irene Hames, EASE/ISMTE, 23 Sept 2013
4
.
*Good practice and quality in peer review is system and access- and business-model independent*
Irene Hames, EASE/ISMTE, 23 Sept 2013
5
What do people think of peer review?
Editors … value it The research community … does too
Peer review survey: Ware and Monkman, 2007 data; Sense About Science, 2009 data
85% & 82% - peer review greatly helps scientific communication 83% & 84% - without peer review would be no control in scientific communication accuracy and quality of work not peer reviewed cannot be trusted 89% & 91% felt own last accepted paper improved by peer review
Open access survey: Taylor & Francis Group 2012-13 data
authors rated ‘rigorous peer review’ most important service expected when pay to publish OA (above rapid peer review & publication)
Irene Hames, EASE/ISMTE, 23 Sept 2013
6
“Peer review in scholarly publishing, in one form or another, is crucial to the reputation and reliability of scientific research” (Para 277)
Irene Hames, EASE/ISMTE, 23 Sept 2013
7
… but is dissatisfaction
12% (Ware & Monkman) and 9% (SAS) in the two surveys
Only about a third in both surveys think current system of peer review is best that can be achieved
Researchers want to improve peer review, not replace it
Irene Hames, EASE/ISMTE, 23 Sept 2013
8
Criticisms of peer review “Peer review is in crisis”
“Publish all, filter later”
Unreliable and unfair No clear standards, idiosyncratic Open to abuse and bias Stifles innovation Slow, causes delays in publication
Expensive and labour intensive Reviewers overloaded, working ‘for free’ Almost useless at detecting fraud and misconduct Can ‘fail’ in even the best-run systems
[Image, Gideon Burton, Utah, USA (CC BY-SA 2.0)]
Irene Hames, EASE/ISMTE, 23 Sept 2013
9
Critical role of the Editor “…[peer review] works as well as can be expected. The critical feature that makes the system work is the skill and insight of the editor. Astute editors can use the system well, the less able who follow reviewer comments uncritically bring the system into disrepute.” (a respondent, Ware & Monkman, 2008, PRC peer review survey)
“Unfortunately, all too often editors relinquish their responsibilities and treat the peer review process as a vote … the real problem is editors … increasingly, one sees editors who don’t use any judgement at all, but just keep going back to reviewers until there is agreement.” (Dorothy Bishop, Professor of Developmental Neuropsychology, Oxford University, ‘In defence of peer review’, comment 4 Jan 2011, to R Smith (2010) Breast Cancer Research, 12(Suppl 4):S13 )
Irene Hames, EASE/ISMTE, 23 Sept 2013
10
Editors have to act as editors Being an editor is:
not just moving manuscripts automatically through the peer-review process not just ‘counting votes’ not passing on editor responsibilities to reviewers making critical judgements (‘reviewers advise, editors decide’)
Irene Hames, EASE/ISMTE, 23 Sept 2013
11
Some problems due to
Variable quality of peer review
Inconsistency in decision making
Lack of training for new editors
Perceived gaming by journals
Unethical behaviour
Irene Hames, EASE/ISMTE, 23 Sept 2013
12
‘Fake reviewer’ cases
Irene Hames, EASE/ISMTE, 23 Sept 2013
13
Ethics and integrity
Important in research communication/publication whatever the model Lack of knowledge and training COPE
Guidelines and resources
Irene Hames, EASE/ISMTE, 23 Sept 2013
14
.
‘COPE’s new Ethical Guidelines for Peer Reviewers: background, issues, and evolution’, ISMTE, EON May 2013, Vol6, issue4, http://www.ismte.org/Shared_ArticlesCOPEs_new_Ethical_Guidelines_for_Peer_Reviewers_background_issues_and_evolution/ Irene Hames, EASE/ISMTE, 23 Sept 2013
15
.
Innovations in peer review
Irene Hames, EASE/ISMTE, 23 Sept 2013
16
Two functions of peer review separated
Publication based on ‘soundness’ - research methodology, results and reporting - not novelty, interest or potential impact
Evaluation of interest/impact left for post-publication Irene Hames, EASE/ISMTE, 23 Sept 2013
17
Launched December 2006, now a ‘mega’ journal
Published 23,468 articles in 2012 (14,000 in 2011) (~1.4% of world’s scholarly literature)
Used >60,000 reviewers in 2012 (38,400 in 2011) from 154 countries
Impact Factor ~4
‘PLOS ONE clones’ being launched (BMJ Open, Sage Open, Scientific Reports, Biology Open, AIP Advances, SpringerPLus) Irene Hames, EASE/ISMTE, 23 Sept 2013
18
Eliminating ‘wastage’ of reviews
Rejected manuscripts can go from journal to journal, fresh reviews at each
‘Cascading’ submissions and reviews
Within publishers and societies
Between publishers • Neuroscience Peer Review Consortium (http://nprc.incf.org/) • eLife, BMC, PLOS and EMBO – ‘portable peer review’ Irene Hames, EASE/ISMTE, 23 Sept 2013
19
More transparent approaches
Publishing reviews, ms versions and editorial correspondence, reviewers’ names may or may not be revealed BMC series medical journals – ‘pre-publication history’ The EMBO journals – ‘peer review process file’ BMJ Open – ‘peer review history’ eLife – decision letter + author response (have doi’s)
Reviewer interaction: pre-decision at The EMBO Journal (‘cross-peer review’) and eLife Reviewer + author + editor interaction: Frontiers ‘Open’ peer review
Irene Hames, EASE/ISMTE, 23 Sept 2013
20
In 1996 …
Irene Hames, EASE/ISMTE, 23 Sept 2013
21
TPJ anonymous reviewer “That you would consider making a reviewer’s identity known to the submitting authors is alarming in the extreme. How can such a practice not but undermine the peer review process and lower standards? ... I would hold the journal, not the reviewers, responsible for the evaporation of your journal’s credibility, and for an inexcusable erosion of the philosophical framework of modern plant biology.”
Irene Hames, EASE/ISMTE, 23 Sept 2013
22
‘Peer review process goes meta via blogs and Twitter’
‘Ultra-open’ peer review at GigaScience July 2013
Reviewer blogged about a manuscript under peer-review
Shock, followed by, “This is fantastic!!”
“As we are promoting and encouraging more transparent review and the use of pre-print servers, if the authors and reviewers consent then we do allow open discussion of the work prior to publication.”
http://blogs.biomedcentral.com/bmcblog/2013/07/23/ultra-open-peerreview/ Irene Hames, EASE/ISMTE, 23 Sept 2013
23
New initiatives/models
Irene Hames, EASE/ISMTE, 23 Sept 2013
24
… and even newer
Irene Hames, EASE/ISMTE, 23 Sept 2013
25
Peer review doesn’t stop at publication
When real peer review starts?
Post-publication review and evaluation
Increasing opportunities for innovation
Challenges and problems Increasing importance of blogs, twitter and other social media (#arseniclife)
Irene Hames, EASE/ISMTE, 23 Sept 2013
26
.
“Online scientific interaction outside the traditional journal space is becoming more and more important to academic communication”
Mark Hahnel, founder, FigShare (http://figshare.com/)
Irene Hames, EASE/ISMTE, 23 Sept 2013
27
A big challenge - data
Massive amounts being generated
Recognition for producing, making usable by others and curating
Where to put?
Dryad http://datadryad.org/ - international repository of data underlying peer-reviewed articles in basic and applied sciences; can be made securely available for peer review Irene Hames, EASE/ISMTE, 23 Sept 2013
28
A reviewer’s point of view ‘Reviewers peering from under a pile of ‘omics’ data’ J.K. Nicholson, Nature (2006), 440, 992
“The scientific community needs to reassess the way it addresses the peer-review problem, taking into account that referees are only human and are now being asked to do a superhuman task on a near-daily basis.”
Irene Hames, EASE/ISMTE, 23 Sept 2013
29
Will peer review survive?
Mark Ware: ‘far from being in crisis, peer review remains widely supported and diversely innovative’ Ware M (2011) New Review of Information Networking, 16(1): 23-53
Fiona Godlee: (BMJ Editor): ‘At its best I think we would all agree that it does improve the quality of scientific reporting and that it can improve, through the pressure of the journal, the quality of the science itself and how it is performed.’ Godlee F (2011) Evidence given to the UK House of Commons Science and Technology Inquiry into Peer Review, 11 May 2001. Transcript of oral evidence, HC856, Q97. http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
Irene Hames, EASE/ISMTE, 23 Sept 2013
30
2007: Peer review … at its best a very powerful and sophisticated tool … since so much hinges on it, it is essential that it is carried out well and professionally, and that it is viewed with confidence and respect.
In 2013 …
Irene Hames, EASE/ISMTE, 23 Sept 2013
31
But … … what will it look like? … who will be running it? … how will it cope with the increasing volume and range of research output?
Irene Hames, EASE/ISMTE, 23 Sept 2013
32
Thank you!
Dr Irene Hames email:
[email protected] twitter: @irenehames
Irene Hames, EASE/ISMTE, 23 Sept 2013
33
