Wednesday 25 May 2011
Gator in a Tutu
Friday 20 May 2011
More on the UK Inquiry into Peer Review
The UK House of Commons Science & Technology Committee is currently conducting an inquiry into peer review. The second public event of the inquiry was held at Portcullis House, in London, on May 11th.
(NOTE: The third public event will be held next Monday, 23rd May, and will explore Open Access (OA) and post-publication review — details and a link are available at the bottom of this post; some subsequent abstracts and commentary are available here).
UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.
The second public event was split into two sessions.
The first Session was concerned with publication ethics. For this the witnesses were Tracey Brown, Managing Director of Sense About Science, and Dr Liz Wager, Chair of the Committee on Publication Ethics (COPE) and Board Member of the UK Research Integrity Office (UKRIO).
The second session took evidence from publishers. Here the witnesses were Mayur Amin, Senior Vice President, Research and Academic Relations at Elsevier, Dr Philip Campbell, Editor-in-Chief of Nature Publishing Group, Robert Campbell, Senior Publisher at Wiley-Blackwell, Dr Fiona Godlee, Editor-in-Chief at the BMJ Group, and Dr Andrew Sugden, Deputy Editor and International Managing Director of Science.
The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for Ellesmere Port and Neston.
Other politicians to pose the questions extracted from the transcript and listed under the video of the event inserted below were Graham Stringer, Labour MP for Blackley and Broughton, David Morris, Conservative MP for Morecambe and Lunesdale, Roger Williams, Liberal Democrat MP for Brecon and Radnorshire, and Gavin Barwell, Conservative MP for Croydon Central. (A full list of Committee members is available here).
The (uncorrected) transcript of the meeting is available here.
The video of the event can be accessed here.
SOME OF THE ISSUES EXPLORED BY THE COMMITTEE ARE HIGHLIGHTED IN THE QUESTIONS AND ANSWERS LISTED BELOW
Q64 Chair I will quote from the memorandum that we had from UKRIO: "There is a danger that the peer review process can stifle innovation and perpetuate the status quo. Peer reviewers, for example, are more likely to reject a paper or research grant if it challenges their own belief system." Can you elaborate on how big a problem that is for the progression of science and what can we do about it?
Dr Wager: There is some quite nicely crafted evidence that that is true. In general, peer reviewers prefer positive findings. They prefer findings that confirm their own hypotheses and so on. That is just human nature. One of the very important roles of editors, though, is reducing that kind of bias as well as other kinds of bias. One of the things that COPE encourages is to make sure that systems are as objective as possible, to make sure, for example, that journals publish criticism, especially of things that they have published in their own journal, to make sure that they are willing to listen to alternative views and so on. There is a danger of bias towards the status quo. There are other kinds of biases as well, but a well set-up system and a good editor will minimise those biases.
· On the integrity of reviewers and editors …
Q74 Graham Stringer: You have done some research, have you, about the integrity of reviewers and editors in this area?
Dr Wager: I don’t think there has been much research on the integrity of reviewers or editors. Much more research has focused on misconduct by authors. There have been some cases of reviewer misconduct. It is something that COPE picks up now and again. I have done a survey of journal editors to find out how big a problem they thought reviewer misconduct was, and it came pretty low on their list. COPE has produced a flowchart about how to handle allegations and how they should be investigated, because a classic complaint by an author would be, "Someone stole my idea," but that is really pretty uncommon. I don’t think it is a huge problem. Signing up to COPE and getting the complaints procedure working will be one mechanism, we hope, to deal with misconduct by editors.
· On the limits of peer review …
Q81 David Morris: Is the kitemark of peer review really a gold standard that tells the public and policy makers a particular piece of research is reliable?
Tracey Brown: I don’t think it is. For the reasons that Liz [Wager] and I have already outlined, it is a dynamic system that has all the benefits of human judgment, in that it can recognise good ideas. It can sometimes recognise ideas of which even the authors themselves don’t recognise the full implications. It has all the downsides of the system with human judgment, in that it doesn’t always recognise good ideas and sometimes it can be a bit shoddy. It has all those benefits to it. I don’t think it is something that is a stamp of approval beyond which we ask no further questions. It is seen by the scientific community as the basis on which we select those things that are worthy of further attention, but I would emphasise "further attention".
After giving evidence Liz Wager posted a comment on the BMJ blog entitled Are journal editors like used car salesmen?
Q94 Chair: Thank you very much. You will have heard me ask this same question to the first panel. Peer review is regarded as "fundamental to academia and research". What happens if it disappears tomorrow?
Dr Godlee: It is important to distinguish — I am sure others will do this — between pre-publication peer review and peer review generally. Pre-publication peer review is only one aspect of the peer review process, which begins with grant-funding peer review, ethics committees, the pre-publication process, the editing process and then the peer review that goes on after publication. Then there is correction and, in some rare cases, retraction. All of those systems constitute peer review.
If you are talking about the decline or the loss of pre-publication peer review, there are some areas in science and medicine where that would be a problem, as Phil [Campbell] has said, and others where it might be a benefit. The balance between the benefit and harm of peer review is still very poorly experimented with.
· On replacing pre-publication peer review with post-publication review …
Q95 Chair: If we look at the evidence that Richard Smith, the ex-editor of the BMJ, sent us, he suggested moving from "filter, then publish" to "publish everything, then filter." Is there any sense in that approach?
Robert Campbell: He is ignoring the other very important part of peer review, which is improving the article. Especially in some disciplines, that is a lot of what peer review is about. It is not just filtering but going back to the author, making revisions and even doing new experiments. It is only taking one part of peer review.
Mayur Amin: In the Sense About Science study that Tracey Brown mentioned, 91% of the authors said that the peer review process helped to improve their paper. Where everything is published before it gets its first peer review filter, we may end up with a system where it is hard to differentiate between evidence-based conclusions and conclusion-based evidence. We end up in a situation where there is a lot of noise and uncertainty as to whether it is credible or not.
· On the lack of empirical evidence in support of using peer review
· On whether all research is eventually published anyway
Q117 Roger Williams: With so many journals publishing peer-reviewed work, does almost all research get into a peer-reviewed journal at some stage?
Dr Sugden: The evidence is that it does. We heard that said earlier. More than 80% of what passes through our hands will get published somewhere, and mostly somewhere quite good.
What would also be of value would be for the Committee to allow anyone with views on, or experience of, peer review to post comments in response to what is said in the public events directly on the Committee’s web site.
Those giving evidence will include Mark Patterson from the Public Library of Science, Dr Michaela Torkar, Editorial Director at Biomed Central, Professor Ian Walmsley, Pro Vice Chancellor of the University of Oxford, and Professor Teresa Rees CBE, Professor of Social Science and former Pro Vice Chancellor (Research) at Cardiff University.
It is interesting to note that the Committee appears to view Open Access as being synonymous with alternative forms of peer review — OA advocates might deprecate this apparent assumption.
(NOTE: The third public event will be held next Monday, 23rd May, and will explore Open Access (OA) and post-publication review — details and a link are available at the bottom of this post; some subsequent abstracts and commentary are available here).
UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE DETAILS ARE AVAILABLE HERE.
The second public event was split into two sessions.
The first Session was concerned with publication ethics. For this the witnesses were Tracey Brown, Managing Director of Sense About Science, and Dr Liz Wager, Chair of the Committee on Publication Ethics (COPE) and Board Member of the UK Research Integrity Office (UKRIO).
The second session took evidence from publishers. Here the witnesses were Mayur Amin, Senior Vice President, Research and Academic Relations at Elsevier, Dr Philip Campbell, Editor-in-Chief of Nature Publishing Group, Robert Campbell, Senior Publisher at Wiley-Blackwell, Dr Fiona Godlee, Editor-in-Chief at the BMJ Group, and Dr Andrew Sugden, Deputy Editor and International Managing Director of Science.
The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for Ellesmere Port and Neston.
Other politicians to pose the questions extracted from the transcript and listed under the video of the event inserted below were Graham Stringer, Labour MP for Blackley and Broughton, David Morris, Conservative MP for Morecambe and Lunesdale, Roger Williams, Liberal Democrat MP for Brecon and Radnorshire, and Gavin Barwell, Conservative MP for Croydon Central. (A full list of Committee members is available here).
The (uncorrected) transcript of the meeting is available here.
The video of the event can be accessed here.
SOME OF THE ISSUES EXPLORED BY THE COMMITTEE ARE HIGHLIGHTED IN THE QUESTIONS AND ANSWERS LISTED BELOW
#####
== FIRST SESSION ==
· On whether peer review tends to perpetuate the status quo …Q64 Chair I will quote from the memorandum that we had from UKRIO: "There is a danger that the peer review process can stifle innovation and perpetuate the status quo. Peer reviewers, for example, are more likely to reject a paper or research grant if it challenges their own belief system." Can you elaborate on how big a problem that is for the progression of science and what can we do about it?
Dr Wager: There is some quite nicely crafted evidence that that is true. In general, peer reviewers prefer positive findings. They prefer findings that confirm their own hypotheses and so on. That is just human nature. One of the very important roles of editors, though, is reducing that kind of bias as well as other kinds of bias. One of the things that COPE encourages is to make sure that systems are as objective as possible, to make sure, for example, that journals publish criticism, especially of things that they have published in their own journal, to make sure that they are willing to listen to alternative views and so on. There is a danger of bias towards the status quo. There are other kinds of biases as well, but a well set-up system and a good editor will minimise those biases.
· On the integrity of reviewers and editors …
Q74 Graham Stringer: You have done some research, have you, about the integrity of reviewers and editors in this area?
Dr Wager: I don’t think there has been much research on the integrity of reviewers or editors. Much more research has focused on misconduct by authors. There have been some cases of reviewer misconduct. It is something that COPE picks up now and again. I have done a survey of journal editors to find out how big a problem they thought reviewer misconduct was, and it came pretty low on their list. COPE has produced a flowchart about how to handle allegations and how they should be investigated, because a classic complaint by an author would be, "Someone stole my idea," but that is really pretty uncommon. I don’t think it is a huge problem. Signing up to COPE and getting the complaints procedure working will be one mechanism, we hope, to deal with misconduct by editors.
· On the limits of peer review …
Q81 David Morris: Is the kitemark of peer review really a gold standard that tells the public and policy makers a particular piece of research is reliable?
Tracey Brown: I don’t think it is. For the reasons that Liz [Wager] and I have already outlined, it is a dynamic system that has all the benefits of human judgment, in that it can recognise good ideas. It can sometimes recognise ideas of which even the authors themselves don’t recognise the full implications. It has all the downsides of the system with human judgment, in that it doesn’t always recognise good ideas and sometimes it can be a bit shoddy. It has all those benefits to it. I don’t think it is something that is a stamp of approval beyond which we ask no further questions. It is seen by the scientific community as the basis on which we select those things that are worthy of further attention, but I would emphasise "further attention".
After giving evidence Liz Wager posted a comment on the BMJ blog entitled Are journal editors like used car salesmen?
#####
== SECOND SESSION ==
· On the necessity of pre-publication peer review …Q94 Chair: Thank you very much. You will have heard me ask this same question to the first panel. Peer review is regarded as "fundamental to academia and research". What happens if it disappears tomorrow?
Dr Godlee: It is important to distinguish — I am sure others will do this — between pre-publication peer review and peer review generally. Pre-publication peer review is only one aspect of the peer review process, which begins with grant-funding peer review, ethics committees, the pre-publication process, the editing process and then the peer review that goes on after publication. Then there is correction and, in some rare cases, retraction. All of those systems constitute peer review.
If you are talking about the decline or the loss of pre-publication peer review, there are some areas in science and medicine where that would be a problem, as Phil [Campbell] has said, and others where it might be a benefit. The balance between the benefit and harm of peer review is still very poorly experimented with.
· On replacing pre-publication peer review with post-publication review …
Q95 Chair: If we look at the evidence that Richard Smith, the ex-editor of the BMJ, sent us, he suggested moving from "filter, then publish" to "publish everything, then filter." Is there any sense in that approach?
Robert Campbell: He is ignoring the other very important part of peer review, which is improving the article. Especially in some disciplines, that is a lot of what peer review is about. It is not just filtering but going back to the author, making revisions and even doing new experiments. It is only taking one part of peer review.
Mayur Amin: In the Sense About Science study that Tracey Brown mentioned, 91% of the authors said that the peer review process helped to improve their paper. Where everything is published before it gets its first peer review filter, we may end up with a system where it is hard to differentiate between evidence-based conclusions and conclusion-based evidence. We end up in a situation where there is a lot of noise and uncertainty as to whether it is credible or not.
· On the lack of empirical evidence in support of using peer review
Q105 Gavin Barwell: My final question is, particularly, for Dr Godlee. The BMJ Group told us in their submission that "little empirical evidence is available to support the use of editorial peer review". How should a programme of such research be organised, and who would fund it?
Dr Godlee: It has long been felt that a system as important as peer review to most known science is remarkably under-evaluated. There have been studies. There has been an editorially led or research-led approach to this, and some of that funding has come from the NIHR in the UK. We have been very grateful for that. The overall level of evaluation of peer review is very poor — not only journal, editorial peer review, but grant peer review, which is right at the beginning of this process and has an enormous amount of influence on what does and doesn’t get funded. I am sure we should have it. The UK could lead on this. As to where the funding should come from, you could say that it is a combination of the journal publishing world, the grant-giving world, industry, but also public funding. It is a very important part of what we do. We can improve it; there are huge flaws. Lots of good things are going on and there are many new experimental ways of going about things. We need to evaluate these so that different specialty areas can take on different approaches as appropriate. A lot could be done with some decent funding.· On whether all research is eventually published anyway
Q117 Roger Williams: With so many journals publishing peer-reviewed work, does almost all research get into a peer-reviewed journal at some stage?
Dr Sugden: The evidence is that it does. We heard that said earlier. More than 80% of what passes through our hands will get published somewhere, and mostly somewhere quite good.
#####
COMMENT: While it is valuable to see representatives from research organisations, publishers and their representatives, and the great and the good in academia, questioned about the strengths and weaknesses of peer review, why have we seen no common-or-garden researchers and academic editors giving evidence? Perhaps that is to come.What would also be of value would be for the Committee to allow anyone with views on, or experience of, peer review to post comments in response to what is said in the public events directly on the Committee’s web site.
#####
NEXT UP: The third public event of the inquiry will be held next Monday, 23rd May, when the Committee will hear from experts on open access publishing and post-publication review, and from representatives of the research community. Those giving evidence will include Mark Patterson from the Public Library of Science, Dr Michaela Torkar, Editorial Director at Biomed Central, Professor Ian Walmsley, Pro Vice Chancellor of the University of Oxford, and Professor Teresa Rees CBE, Professor of Social Science and former Pro Vice Chancellor (Research) at Cardiff University.
It is interesting to note that the Committee appears to view Open Access as being synonymous with alternative forms of peer review — OA advocates might deprecate this apparent assumption.
Thursday 19 May 2011
UK Inquiry into Peer Review
On 27th January 2011 the UK House of Commons Science & Technology Committee announced that it planned to hold an inquiry into peer review. That inquiry is now underway. Below are some links to the first public event held to take oral evidence. (Click here for details of the second public event).
The terms of reference for the inquiry are the following:
The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:
The terms of reference for the inquiry are the following:
The Committee welcomes submissions on all aspect of the process and among the issues it is likely to examine are the following:
- the strengths and weaknesses of peer review as a quality control mechanism for scientists, publishers and the public;
- measures to strengthen peer review;
- the value and use of peer reviewed science on advancing and testing scientific knowledge;
- the value and use of peer reviewed science in informing public debate;
- the extent to which peer review varies between scientific disciplines and between countries across the world;
- the processes by which reviewers with the requisite skills and knowledge are identified, in particular as the volume of multi-disciplinary research increases;
- the impact of IT and greater use of online resources on the peer review process; and
- possible alternatives to peer review.
The first public event took place on May 4th, with oral evidence being given by a number of experts, including:
- Dr Nicola Gulley, Editorial Director, Institute of Physics Publishing Ltd
- Professor Ronald Laskey CBE FRS FMedSci, Vice-President, Academy of Medical Sciences
- Dr Robert Parker, Interim Chief Executive, Royal Society of Chemistry
- Professor John Pethica FRS, Physical Secretary and Vice-President, Royal Society
The opening question from the chair of the Committee, Andrew Miller, Labour MP for Ellesmere Port and Neston, was:
Q: Peer review is perceived to be "fundamental to scholarly communications". If it disappeared tomorrow, what would the consequences be?
The first reply, from Dr Robert Parker, was:
A: You would have to come up with something else with which to replace it. There isn’t anything very obvious to replace peer review with currently. The danger would be to the scientific record, really. The importance of it is laid out in the evidence that has been submitted with great clarity from most people who have submitted evidence in writing to this review. The value and quality of that scientific record is paramount, and peer review helps to keep that in place.
The written evidence Dr Parker refers to can be read here.
Q: Peer review is perceived to be "fundamental to scholarly communications". If it disappeared tomorrow, what would the consequences be?
The first reply, from Dr Robert Parker, was:
A: You would have to come up with something else with which to replace it. There isn’t anything very obvious to replace peer review with currently. The danger would be to the scientific record, really. The importance of it is laid out in the evidence that has been submitted with great clarity from most people who have submitted evidence in writing to this review. The value and quality of that scientific record is paramount, and peer review helps to keep that in place.
The written evidence Dr Parker refers to can be read here.
Subscribe to:
Posts (Atom)