Mar 19, 2018 05:19 PM
http://www.pnas.org/content/early/2018/03/08/1708272114
EXCERPT: . . . Is there a reproducibility crisis in science? Many seem to believe so. In a recent survey by the journal *Nature*, for example, around 90% of respondents agreed that there is a “slight” or “significant” crisis, and between 40% and 70% agreed that selective reporting, fraud, and pressures to publish “always” or “often” contribute to irreproducible research. Results of this nonrandomized survey may not accurately represent the population of practicing scientists, but they echo beliefs expressed by a rapidly growing scientific literature, which uncritically endorses a new “crisis narrative” about science.
Put simply, this new “science in crisis” narrative postulates that a large and growing proportion of studies published across disciplines are unreliable due to the declining quality and integrity of research and publication practices, largely because of growing pressures to publish and other ills affecting the contemporary scientific profession.
I argue that this crisis narrative is at least partially misguided. Recent evidence from metaresearch studies suggests that issues with research integrity and reproducibility, while certainly important phenomena that need to be addressed, are: (i) not distorting the majority of the literature, in science as a whole as well as within any given discipline; (ii) heterogeneously distributed across subfields in any given area, which suggests that generalizations are unjustified; and (iii) not growing, as the crisis narrative would presuppose. Alternative narratives, therefore, might represent a better fit for empirical data as well as for the reproducibility agenda.
[...] Ultimately, the debate over the existence of a reproducibility crisis should have been closed by recent large-scale assessments of reproducibility. Their results, however, are either reassuring or inconclusive. [...] the very notion of “reproducible research” can be confusing, because its meaning and implications depend on what aspect of research is being examined: the reproducibility of research methods can in principle be expected to be 100%; but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies, for reasons that have nothing to do with questionable research and publication practices. [...] In light of multiple recent studies, there is no evidence that scientific misconduct and QRPs have increased.
[...] To summarize, an expanding metaresearch literature suggests that science—while undoubtedly facing old and new challenges—cannot be said to be undergoing a “reproducibility crisis,” at least not in the sense that it is no longer reliable due to a pervasive and growing problem with findings that are fabricated, falsified, biased, underpowered, selected, and irreproducible. While these problems certainly exist and need to be tackled, evidence does not suggest that they undermine the scientific enterprise as a whole. Science always was and always will be a struggle to produce knowledge for the benefit of all of humanity against the cognitive and moral limitations of individual human beings, including the limitations of scientists themselves.
The new “science is in crisis” narrative is not only empirically unsupported, but also quite obviously counterproductive. Instead of inspiring younger generations to do more and better science, it might foster in them cynicism and indifference. Instead of inviting greater respect for and investment in research, it risks discrediting the value of evidence and feeding antiscientific agendas.
Furthermore, this narrative is not actually new. Complaints about a decline in the quality of research recur throughout the history of science, right from its beginnings. Only two elements of novelty characterize the current “crisis.” The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above. The second element of historical novelty is the rising power of information and communication technologies, which are transforming scientific practices in all fields, just as they are transforming all other aspects of human life. These technologies promise to make research more accurate, powerful, open, democratic, transparent, and self-correcting than ever before. At the same time, this technological revolution creates new expectations and new challenges that metaresearchers are striving to address.
Therefore, contemporary science could be more accurately portrayed as facing “new opportunities and challenges” or even a “revolution”. Efforts to promote transparency and reproducibility would find complete justification in such a narrative of transformation and empowerment, a narrative that is not only more compelling and inspiring than that of a crisis, but also better supported by evidence.
MORE / DETAILS: http://www.pnas.org/content/early/2018/03/08/1708272114
EXCERPT: . . . Is there a reproducibility crisis in science? Many seem to believe so. In a recent survey by the journal *Nature*, for example, around 90% of respondents agreed that there is a “slight” or “significant” crisis, and between 40% and 70% agreed that selective reporting, fraud, and pressures to publish “always” or “often” contribute to irreproducible research. Results of this nonrandomized survey may not accurately represent the population of practicing scientists, but they echo beliefs expressed by a rapidly growing scientific literature, which uncritically endorses a new “crisis narrative” about science.
Put simply, this new “science in crisis” narrative postulates that a large and growing proportion of studies published across disciplines are unreliable due to the declining quality and integrity of research and publication practices, largely because of growing pressures to publish and other ills affecting the contemporary scientific profession.
I argue that this crisis narrative is at least partially misguided. Recent evidence from metaresearch studies suggests that issues with research integrity and reproducibility, while certainly important phenomena that need to be addressed, are: (i) not distorting the majority of the literature, in science as a whole as well as within any given discipline; (ii) heterogeneously distributed across subfields in any given area, which suggests that generalizations are unjustified; and (iii) not growing, as the crisis narrative would presuppose. Alternative narratives, therefore, might represent a better fit for empirical data as well as for the reproducibility agenda.
[...] Ultimately, the debate over the existence of a reproducibility crisis should have been closed by recent large-scale assessments of reproducibility. Their results, however, are either reassuring or inconclusive. [...] the very notion of “reproducible research” can be confusing, because its meaning and implications depend on what aspect of research is being examined: the reproducibility of research methods can in principle be expected to be 100%; but the reproducibility of results and inferences is likely to be lower and to vary across subfields and methodologies, for reasons that have nothing to do with questionable research and publication practices. [...] In light of multiple recent studies, there is no evidence that scientific misconduct and QRPs have increased.
[...] To summarize, an expanding metaresearch literature suggests that science—while undoubtedly facing old and new challenges—cannot be said to be undergoing a “reproducibility crisis,” at least not in the sense that it is no longer reliable due to a pervasive and growing problem with findings that are fabricated, falsified, biased, underpowered, selected, and irreproducible. While these problems certainly exist and need to be tackled, evidence does not suggest that they undermine the scientific enterprise as a whole. Science always was and always will be a struggle to produce knowledge for the benefit of all of humanity against the cognitive and moral limitations of individual human beings, including the limitations of scientists themselves.
The new “science is in crisis” narrative is not only empirically unsupported, but also quite obviously counterproductive. Instead of inspiring younger generations to do more and better science, it might foster in them cynicism and indifference. Instead of inviting greater respect for and investment in research, it risks discrediting the value of evidence and feeding antiscientific agendas.
Furthermore, this narrative is not actually new. Complaints about a decline in the quality of research recur throughout the history of science, right from its beginnings. Only two elements of novelty characterize the current “crisis.” The first is that the validity of these concerns is being assessed scientifically by a global metaresearch program, with results that have been briefly overviewed above. The second element of historical novelty is the rising power of information and communication technologies, which are transforming scientific practices in all fields, just as they are transforming all other aspects of human life. These technologies promise to make research more accurate, powerful, open, democratic, transparent, and self-correcting than ever before. At the same time, this technological revolution creates new expectations and new challenges that metaresearchers are striving to address.
Therefore, contemporary science could be more accurately portrayed as facing “new opportunities and challenges” or even a “revolution”. Efforts to promote transparency and reproducibility would find complete justification in such a narrative of transformation and empowerment, a narrative that is not only more compelling and inspiring than that of a crisis, but also better supported by evidence.
MORE / DETAILS: http://www.pnas.org/content/early/2018/03/08/1708272114