Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5

The rise and fall of peer review

#1
C C Offline
Why the greatest scientific experiment in history failed, and why that's a great thing
https://experimentalhistory.substack.com...eer-review

EXCERPTS (Adam Mastroianni): For the last 60 years or so, science has been running an experiment on itself. The experimental design wasn’t great; there was no randomization and no control group. Nobody was in charge, exactly, and nobody was really taking consistent measurements. And yet it was the most massive experiment ever run, and it included every scientist on Earth.

[...] Maybe nobody objected because the hypothesis seemed so obviously true: science will be better off if we have someone check every paper and reject the ones that don’t pass muster. They called it “peer review.”

This was a massive change. From antiquity to modernity, scientists wrote letters and circulated monographs, and the main barriers stopping them from communicating their findings were the cost of paper, postage, or a printing press, or on rare occasions, the cost of a visit from the Catholic Church...

[...] That all changed after World War II. Governments poured funding into research, and they convened “peer reviewers” to ensure they weren’t wasting their money on foolish proposals...

[...] Now pretty much every journal uses outside experts to vet papers, and papers that don’t please reviewers get rejected...

[...] This is the grand experiment we’ve been running for six decades. The results are in. It failed. 

[...] Peer review was a huge, expensive intervention. By one estimate, scientists collectively spend 15,000 years reviewing papers every year. It can take months or years for a paper to wind its way through the review system...

[...] Similarly, if peer review improved science, that should be pretty obvious, and we should be pretty upset and embarrassed if it didn’t.

It didn’t. In all sorts of different fields, research productivity has been flat or declining for decades, and peer review doesn’t seem to have changed that trend. New ideas are failing to displace older ones. Many peer-reviewed findings don’t replicate, and most of them may be straight-up false...

[...] Here’s a simple question: does peer review actually do the thing it’s supposed to do? Does it catch bad research and prevent it from being published?

It doesn’t. Scientists have run studies where they deliberately add errors to papers, send them out to reviewers, and simply count how many errors the reviewers catch. Reviewers are pretty awful at this...

[...] In fact, we’ve got knock-down, real-world data that peer review doesn’t work: fraudulent papers get published all the time...

[...] Why don’t reviewers catch basic errors and blatant fraud? One reason is that they almost never look at the data behind the papers they review...

[...] The invention of peer review may have even encouraged bad research...

[...] Here’s another way that we can test whether peer review worked: did it actually earn scientists' trust? 

Scientists often say they take peer review very seriously. But people say lots of things they don’t mean, like “It’s great to e-meet you” and “I’ll never leave you, Adam.” If you look at what scientists actually do, it’s clear they don’t think peer review really matters...

[...] Second: once a paper gets published, we shred the reviews. A few journals publish reviews; most don't. Nobody cares to find out what the reviewers said or how the authors edited their paper in response, which suggests that nobody thinks the reviews actually mattered in the first place...

[...] And third: scientists take unreviewed work seriously without thinking twice...

[...] Peer review doesn’t work and there’s probably no way to fix it. But a little bit of vetting is better than none at all, right?

I say: no way. 

Imagine you discover that the Food and Drug Administration’s method of “inspecting” beef is just sending some guy (“Gary”) around to sniff the beef and say whether it smells okay or not, and the beef that passes the sniff test gets a sticker that says “INSPECTED BY THE FDA.” You’d be pretty angry. Yes, Gary may find a few batches of bad beef, but obviously he’s going to miss most of the dangerous meat. This extremely bad system is worse than nothing because it fools people into thinking they’re safe when they’re not.

That’s what our current system of peer review does, and it’s dangerous...

[...] Why did peer review seem so reasonable in the first place?

I think we had the wrong model of how science works. We treated science like it’s a weak-link problem where progress depends on the quality of our worst work. If you believe in weak-link science, you think it’s very important to stamp out untrue ideas—ideally, prevent them from being published in the first place. You don’t mind if you whack a few good ideas in the process, because it’s so important to bury the bad stuff.

But science is a strong-link problem: progress depends on the quality of our best work. Better ideas don’t always triumph immediately, but they do triumph eventually, because they’re more useful. You can’t land on the moon using Aristotle’s physics, you can’t turn mud into frogs using spontaneous generation, and you can’t build bombs out of phlogiston. Newton’s laws of physics stuck around; his recipe for the Philosopher’s Stone didn’t. We didn’t need a scientific establishment to smother the wrong ideas. We needed it to let new ideas challenge old ones, and time did the rest.

[...] Nobody was in charge of our peer review experiment, which means nobody has the responsibility of saying when it’s over. Seeing no one else, I guess I’ll do it: We’re done, everybody! Champagne all around! Great work, and congratulations. We tried peer review and it didn’t work.

Honesty, I’m so relieved. That system sucked!  [...] I know we all might be a little disappointed we wasted so much time, but there's no shame in a failed experiment... (MORE - missing details)
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Article What’s wrong with peer review? + No, TCM has not been vindicated by science C C 1 82 Nov 13, 2023 03:09 PM
Last Post: Zinjanthropos
  Article Corruption of the academic peer-review process (climate science) C C 4 154 Aug 5, 2023 05:29 PM
Last Post: C C
  Article Partisan science is bad for society + Astrobiology: Rise & fall of a nascent science C C 0 61 Apr 12, 2023 04:38 PM
Last Post: C C
  Article Is peer review doomed? + AI makes sci fraud easier + Non-humans authoring sci papers C C 0 62 Mar 18, 2023 11:35 PM
Last Post: C C
  How scientific is ‘peer-reviewed’ science? C C 0 58 Feb 1, 2023 08:42 PM
Last Post: C C
  AI was author of research paper (controversy) + That viral screed against peer review C C 1 101 Jan 25, 2023 09:07 AM
Last Post: confused2
  Don’t fall for the snake oil claims of ‘structured water’. C C 3 107 Aug 6, 2022 10:31 PM
Last Post: RainbowUnicorn
  Opening the black box of peer review C C 0 63 Dec 3, 2021 09:36 AM
Last Post: C C
  Michael Shermer documents the decline and fall of Scientific American C C 0 78 Nov 19, 2021 07:07 PM
Last Post: C C
  I published a fake paper in a ‘peer-reviewed’ journal C C 1 198 Nov 30, 2020 04:23 AM
Last Post: Zinjanthropos



Users browsing this thread: 1 Guest(s)