Science has been in a “replication crisis” for a decade. Have we learned anything?

C C Offline

INTRO: Much ink has been spilled over the “replication crisis” in the last decade and a half, including here at Vox. Researchers have discovered, over and over, that lots of findings in fields like psychology, sociology, medicine, and economics don’t hold up when other researchers try to replicate them.

This conversation was fueled in part by John Ioannidis’s 2005 article “Why Most Published Research Findings Are False” and by the controversy around a 2011 paper that used then-standard statistical methods to find that people have precognition. But since then, many researchers have explored the replication crisis from different angles. Why are research findings so often unreliable? Is the problem just that we test for “statistical significance” — the likelihood that similarly strong results could have occurred by chance — in a nuance-free way? Is it that null results (that is, when a study finds no detectable effects) are ignored while positive ones make it into journals?

A recent write-up by Alvaro de Menard, a participant in the Defense Advanced Research Project’s Agency’s (DARPA) replication markets project (more on this below), makes the case for a more depressing view: The processes that lead to unreliable research findings are routine, well understood, predictable, and in principle pretty easy to avoid. And yet, he argues, we’re still not improving the quality and rigor of social science research.

While other researchers I spoke with pushed back on parts of Menard’s pessimistic take, they do agree on something: a decade of talking about the replication crisis hasn’t translated into a scientific process that’s much less vulnerable to it. Bad science is still frequently published, including in top journals — and that needs to change... (MORE)
Syne Offline
The social sciences, in particular, are rife with bias. It's like how most psych students seem to get into the subject to cure their own mental ills. IOW, the impetus for entering those fields is motivated reasoning...from the get go. No surprise motivated reasoning persists.

Possibly Related Threads…
Thread Author Replies Views Last Post
  What findings do skeptical psychologists still believe in, after replication crisis? C C 0 150 Jun 3, 2019 05:50 PM
Last Post: C C
  7 'Facts' You Learned at School That Are No Longer True C C 0 166 Oct 1, 2017 04:15 AM
Last Post: C C
  What I learned from predatory publishers: They pose a unique threat to science C C 0 230 Jun 19, 2017 06:56 PM
Last Post: C C
  Ravetz reflects on the science crisis + Doctors we trust, even when they admit bias C C 0 410 Jun 22, 2016 11:09 PM
Last Post: C C

Users browsing this thread: 1 Guest(s)