Feb 11, 2020 06:05 AM
(This post was last modified: Feb 11, 2020 06:09 AM by C C.)
Peer Review Is Science’s Wheel of Misfortune
https://www.bloomberg.com/opinion/articl...misfortune
EXCERPT: . . . Modern scientists can only marvel at Einstein’s contempt for peer review. Over the years, this process has become so central to scientific publishing that nowadays even the most distinguished researchers must regularly subject themselves to its trials and tribulations.
We can, however, easily identify with Einstein’s disregard for the opinions of the so-called expert. [...] peer review has problems that run deeper than the quality of any particular reviewer. The process is inconsistent and subjective to the degree that — in the words of Richard Smith, a former editor of the British Medical Journal — it’s “something of a lottery.” Smith wrote that Robbie Fox, a one-time editor of the Lancet, went so far as to question “whether anybody would notice if he were to swap the piles marked ‘publish’ and ‘reject.’” There’s a mountain of evidence that these claims aren’t far from the truth.
[...] the most alarming cases — papers that are blatantly wrong or fraudulent — are rarely caught in the peer review net. ... Publication is just one part of a much larger process in which important papers are identified and then heavily scrutinized by the relevant scientific community. ... The question of how many papers a scientist published, and where, plays a huge role in decisions about hiring, promotion, funding and — in disciplines like computer science — even admission into Ph.D. programs. A scientist’s career may depend on whether a few reviewers choose to accept or reject a single paper.
To receive tenure at a leading economics department, for example, candidates are expected to have published two or three papers in the discipline's most prestigious journals, imaginatively called the "top 5." Three of these journals famously rejected “The Market for Lemons,” a seminal paper that upended economic thinking and won its author a Nobel Prize, with one reviewer complaining, “If this paper was correct, economics would be different.”
It seems paradoxical that scientists — ostensible paragons of evidence-based reasoning — would give such weight to the outcomes of peer review, despite the growing evidence of the system’s limitations. One reason is laziness: nothing’s easier than skimming through a colleague’s list of publications and noting where they appeared. But another may well be that relatively few scientists recognize just how flawed peer review is.
[...] Scientists should also work on solutions to the problems of peer review, as many are already doing. My own contribution to this effort is reported in a recent manuscript coauthored with two former colleagues at Carnegie Mellon University, Ritesh Noothigattu and Nihar Shah... (MORE - details)
RELATED: The intellectual & moral decline in academic research
How Academic Science Gave Its Soul to the Publishing Industry
https://issues.org/how-academic-science-...-industry/
EXCERPT: . . . In addition to shaping the content of science, researcher evaluation policies focused on publication statistics contribute to what is essentially a publication industry oligopoly that undermines user access to the research that is published. Because of entrenched journal hierarchies, codified by journal impact factor scores, a handful of large corporate publishers have secure positions in owning “must publish” outlets for scientists.
[...] Economists may not have terms adequate to describe a market as dysfunctional as the one operating for academic publishing. Universities employ the content providers—academic researchers—who conduct the research that is paid for by governments and other third-party sponsors. Academics then write the articles, are incentivized to publish them in journals owned by a small number of companies, give them to those journals for free, vet the content for others in the process of peer review—for free—and serve on editorial boards organizing the review process—usually for free—thus providing a free labor force for those aspects of scientific publishing that require the greatest expertise and effort. Libraries in turn have little choice but to pay whatever subscription fees the publishers demand to secure access to the resulting content.
Typically, one or a small handful of corporate publishers (e.g., Elsevier, Springer Nature, Taylor and Francis) owns the top journals within a discipline. Those publishers have sufficient market control that subscription to their content is almost mandatory for libraries that strive to serve an active research community. Publishers use this market power to their advantage in negotiations with libraries and other institutional subscribers. [...]
The power and control of the major publishers in the market is self-reinforcing. Details from my institution, Western Washington University, exemplify the dynamic. Our libraries spend 15% of their resource acquisition budget on individual subscription titles, compared with 45% percent on journal subscription package deals and 26% on databases. The major publishing houses know that institutions such as ours depend on large packages to assure the viability of teaching and research, and thus have been able to demand annual price increases of 5%–15% per year, far outpacing inflation. Yet, our library budget has no built-in capacity to deal with even base-level inflation; cuts in subscriptions are inevitable. And it is much easier for the libraries to eliminate independently published individual titles that appeal to single disciplines, regardless of how intensively those resources are used, than to drop an entire package. Independent journals are thus under substantial pressure to join one of the larger publishers if they hope to remain viable. Through time, more and more of the publishing industry is thus assimilated into the portfolios of a very few corporate publishers.
The predictable consequence of these practices is extraordinary profit margins for corporate publishers and database providers in recent years. [...] By building scientific self-governance around publication statistics, we academic researchers have guaranteed the publication industry a supply of government-subsidized content, free labor for assuring quality through peer review, and a virtually guaranteed demand that our host institutions will purchase those products back. These self-imposed standards orient much of science around the pursuit of knowledge deemed interesting to a small set of journals selected for reasons unrelated to any systematic (let alone democratic) consideration of societal importance. And journals are so expensive that subscriptions are stretching already-limited university budgets even in wealthy countries.
How is it that the academic scientific community came not just to accept this situation, but to vigorously defend it as the essence of scientific self-governance? One explanation lies with Vannevar Bush’s “just-so” story about scientific self-governance: in his telling, because science’s benefits are unpredictable but inevitable, university science best serves society when it is left entirely to its own devices. This in turn means that scientific quality can be judged only by scientists, who quite sensibly judge one another based on contributions to the academic literature.
But this arrangement alone did not have to lead to the publishing industry’s current power over science. Two other vignettes dating to the decades following World War II show how the needs and interests of scientists, profit-seeking entrepreneurs in publishing, and the information management needs of libraries converged to create the dysfunctional publishing market today... (MORE - details)
https://www.bloomberg.com/opinion/articl...misfortune
EXCERPT: . . . Modern scientists can only marvel at Einstein’s contempt for peer review. Over the years, this process has become so central to scientific publishing that nowadays even the most distinguished researchers must regularly subject themselves to its trials and tribulations.
We can, however, easily identify with Einstein’s disregard for the opinions of the so-called expert. [...] peer review has problems that run deeper than the quality of any particular reviewer. The process is inconsistent and subjective to the degree that — in the words of Richard Smith, a former editor of the British Medical Journal — it’s “something of a lottery.” Smith wrote that Robbie Fox, a one-time editor of the Lancet, went so far as to question “whether anybody would notice if he were to swap the piles marked ‘publish’ and ‘reject.’” There’s a mountain of evidence that these claims aren’t far from the truth.
[...] the most alarming cases — papers that are blatantly wrong or fraudulent — are rarely caught in the peer review net. ... Publication is just one part of a much larger process in which important papers are identified and then heavily scrutinized by the relevant scientific community. ... The question of how many papers a scientist published, and where, plays a huge role in decisions about hiring, promotion, funding and — in disciplines like computer science — even admission into Ph.D. programs. A scientist’s career may depend on whether a few reviewers choose to accept or reject a single paper.
To receive tenure at a leading economics department, for example, candidates are expected to have published two or three papers in the discipline's most prestigious journals, imaginatively called the "top 5." Three of these journals famously rejected “The Market for Lemons,” a seminal paper that upended economic thinking and won its author a Nobel Prize, with one reviewer complaining, “If this paper was correct, economics would be different.”
It seems paradoxical that scientists — ostensible paragons of evidence-based reasoning — would give such weight to the outcomes of peer review, despite the growing evidence of the system’s limitations. One reason is laziness: nothing’s easier than skimming through a colleague’s list of publications and noting where they appeared. But another may well be that relatively few scientists recognize just how flawed peer review is.
[...] Scientists should also work on solutions to the problems of peer review, as many are already doing. My own contribution to this effort is reported in a recent manuscript coauthored with two former colleagues at Carnegie Mellon University, Ritesh Noothigattu and Nihar Shah... (MORE - details)
RELATED: The intellectual & moral decline in academic research
How Academic Science Gave Its Soul to the Publishing Industry
https://issues.org/how-academic-science-...-industry/
EXCERPT: . . . In addition to shaping the content of science, researcher evaluation policies focused on publication statistics contribute to what is essentially a publication industry oligopoly that undermines user access to the research that is published. Because of entrenched journal hierarchies, codified by journal impact factor scores, a handful of large corporate publishers have secure positions in owning “must publish” outlets for scientists.
[...] Economists may not have terms adequate to describe a market as dysfunctional as the one operating for academic publishing. Universities employ the content providers—academic researchers—who conduct the research that is paid for by governments and other third-party sponsors. Academics then write the articles, are incentivized to publish them in journals owned by a small number of companies, give them to those journals for free, vet the content for others in the process of peer review—for free—and serve on editorial boards organizing the review process—usually for free—thus providing a free labor force for those aspects of scientific publishing that require the greatest expertise and effort. Libraries in turn have little choice but to pay whatever subscription fees the publishers demand to secure access to the resulting content.
Typically, one or a small handful of corporate publishers (e.g., Elsevier, Springer Nature, Taylor and Francis) owns the top journals within a discipline. Those publishers have sufficient market control that subscription to their content is almost mandatory for libraries that strive to serve an active research community. Publishers use this market power to their advantage in negotiations with libraries and other institutional subscribers. [...]
The power and control of the major publishers in the market is self-reinforcing. Details from my institution, Western Washington University, exemplify the dynamic. Our libraries spend 15% of their resource acquisition budget on individual subscription titles, compared with 45% percent on journal subscription package deals and 26% on databases. The major publishing houses know that institutions such as ours depend on large packages to assure the viability of teaching and research, and thus have been able to demand annual price increases of 5%–15% per year, far outpacing inflation. Yet, our library budget has no built-in capacity to deal with even base-level inflation; cuts in subscriptions are inevitable. And it is much easier for the libraries to eliminate independently published individual titles that appeal to single disciplines, regardless of how intensively those resources are used, than to drop an entire package. Independent journals are thus under substantial pressure to join one of the larger publishers if they hope to remain viable. Through time, more and more of the publishing industry is thus assimilated into the portfolios of a very few corporate publishers.
The predictable consequence of these practices is extraordinary profit margins for corporate publishers and database providers in recent years. [...] By building scientific self-governance around publication statistics, we academic researchers have guaranteed the publication industry a supply of government-subsidized content, free labor for assuring quality through peer review, and a virtually guaranteed demand that our host institutions will purchase those products back. These self-imposed standards orient much of science around the pursuit of knowledge deemed interesting to a small set of journals selected for reasons unrelated to any systematic (let alone democratic) consideration of societal importance. And journals are so expensive that subscriptions are stretching already-limited university budgets even in wealthy countries.
How is it that the academic scientific community came not just to accept this situation, but to vigorously defend it as the essence of scientific self-governance? One explanation lies with Vannevar Bush’s “just-so” story about scientific self-governance: in his telling, because science’s benefits are unpredictable but inevitable, university science best serves society when it is left entirely to its own devices. This in turn means that scientific quality can be judged only by scientists, who quite sensibly judge one another based on contributions to the academic literature.
But this arrangement alone did not have to lead to the publishing industry’s current power over science. Two other vignettes dating to the decades following World War II show how the needs and interests of scientists, profit-seeking entrepreneurs in publishing, and the information management needs of libraries converged to create the dysfunctional publishing market today... (MORE - details)