Dec 3, 2021 09:36 AM
https://physics.aps.org/articles/v14/169
EXCERPTS: After working for about a decade as a journal editor, I’m concerned that peer review is not the safeguard against bad science that we think it is. I’m convinced that peer review could be a lot better and that increased transparency in the process is key to improving it.
Peer review is an obvious improvement over earlier forms of research evaluation [...] But how well does our modern peer review system work? Does it keep bias, misconduct, and errors at bay? Alas, we don’t have enough data to answer these questions. As scientists, we frequently advocate for evidence-based practice, but we don’t always practice what we preach when it comes to keeping our own houses in order.
Despite a vibrant and growing field of research on peer review, focused mainly on medical journals, peer review is still mostly a black box. [...] Both scientists and the public have great confidence in peer review, considering it one of the primary mechanisms by which science vets itself. But seeing peer review from the inside has shaken my confidence.
In my view, there aren’t sufficient guarantees that each paper is thoroughly vetted—often, there is no vetting at all for the most basic qualities you’d expect a quality control system to check, such as computational reproducibility for data-based papers. There is evidence that many errors, even quite obvious ones, make it through peer review.
Moreover, several studies have demonstrated that status bias exerts a significant influence on reviewer evaluations. The impact of other types of bias—based on the gender, geographic location, or ethnic origin of a paper’s authors—still needs to be properly assessed.
Finally, there have been several proven cases of editorial misconduct, whereby editors were found to have manipulated the content of reviews or changed reviewers’ recommendations. I believe that the mechanisms for holding editors accountable in these cases are generally unsatisfactory.
There are thus sufficient reasons to question the validity and limitations of peer review, and it’s clear that we need to gather more evidence. But the opaqueness of peer review—which makes it ripe for error and bias— also makes it hard to study... (MORE - missing details)
EXCERPTS: After working for about a decade as a journal editor, I’m concerned that peer review is not the safeguard against bad science that we think it is. I’m convinced that peer review could be a lot better and that increased transparency in the process is key to improving it.
Peer review is an obvious improvement over earlier forms of research evaluation [...] But how well does our modern peer review system work? Does it keep bias, misconduct, and errors at bay? Alas, we don’t have enough data to answer these questions. As scientists, we frequently advocate for evidence-based practice, but we don’t always practice what we preach when it comes to keeping our own houses in order.
Despite a vibrant and growing field of research on peer review, focused mainly on medical journals, peer review is still mostly a black box. [...] Both scientists and the public have great confidence in peer review, considering it one of the primary mechanisms by which science vets itself. But seeing peer review from the inside has shaken my confidence.
In my view, there aren’t sufficient guarantees that each paper is thoroughly vetted—often, there is no vetting at all for the most basic qualities you’d expect a quality control system to check, such as computational reproducibility for data-based papers. There is evidence that many errors, even quite obvious ones, make it through peer review.
Moreover, several studies have demonstrated that status bias exerts a significant influence on reviewer evaluations. The impact of other types of bias—based on the gender, geographic location, or ethnic origin of a paper’s authors—still needs to be properly assessed.
Finally, there have been several proven cases of editorial misconduct, whereby editors were found to have manipulated the content of reviews or changed reviewers’ recommendations. I believe that the mechanisms for holding editors accountable in these cases are generally unsatisfactory.
There are thus sufficient reasons to question the validity and limitations of peer review, and it’s clear that we need to gather more evidence. But the opaqueness of peer review—which makes it ripe for error and bias— also makes it hard to study... (MORE - missing details)