Jun 23, 2021 01:34 AM
(This post was last modified: Jun 23, 2021 06:23 AM by C C.)
https://mindmatters.ai/2021/06/a-vulnera...cientists/
EXCERPTS: In the last two posts [see article links at bottom], we examined how scientific publication has ceased to be a good measure of scientific accomplishment, and how the peer review system is being gamed by unscrupulous publishers and researchers alike. Now, we will continue the discussion on the undermining of scientific publication using two examples: SCIgen and citation counts.
[...] Publication counts and citation indexes are too noisy and too easily manipulated to be reliable. Nor can we evaluate research simply by noting the journals that publish the results. Because there is so much noise in the review process, lots of great papers have been published in lesser journals and many terrible papers have been published in the most respected journals. John P. A. Ioannidis’s paper, “Why Most Published Research Findings Are False,” has been cited nearly 10,000 times despite being published in PLOS Medicine, a good-but-not-great journal. On the other hand, the British Medical Journal, a truly great journal, published a paper making the preposterous claim that Asian-Americans are unusually susceptible to heart attacks on the fourth day of the month because the bad luck associated with the number four is as terrifying as being chased down a dark alley by a savage dog with “flaming jaws and blazing eyes.”
The best solution is to have experts actually read the research done by applicants for a job, promotion, or grant. Simple counts and indexes simply won’t do. Bob Marks reminded me of the old aphorism in academia, “Deans can’t read, but they can count.” A better one is, “Not everything that can be counted counts, and not everything that counts can be counted.” (MORE - details)
You may also wish to read the first two articles in this three-part series:
Publish or Perish — Another Example of Goodhart’s Law. In becoming a target, publication has ceased to be a good measure. Researchers game the system to beat the publish-or-perish culture, which undermines the usefulness of publication and citation counts. (Gary Smith)
Gaming the System: The Flaws in Peer Review. Peer review is well-intentioned, but flawed in many ways. Predatory journals, dishonest researchers, and escalating costs in academic journals reveal the weaknesses in peer review. (Gary Smith)
EXCERPTS: In the last two posts [see article links at bottom], we examined how scientific publication has ceased to be a good measure of scientific accomplishment, and how the peer review system is being gamed by unscrupulous publishers and researchers alike. Now, we will continue the discussion on the undermining of scientific publication using two examples: SCIgen and citation counts.
[...] Publication counts and citation indexes are too noisy and too easily manipulated to be reliable. Nor can we evaluate research simply by noting the journals that publish the results. Because there is so much noise in the review process, lots of great papers have been published in lesser journals and many terrible papers have been published in the most respected journals. John P. A. Ioannidis’s paper, “Why Most Published Research Findings Are False,” has been cited nearly 10,000 times despite being published in PLOS Medicine, a good-but-not-great journal. On the other hand, the British Medical Journal, a truly great journal, published a paper making the preposterous claim that Asian-Americans are unusually susceptible to heart attacks on the fourth day of the month because the bad luck associated with the number four is as terrifying as being chased down a dark alley by a savage dog with “flaming jaws and blazing eyes.”
The best solution is to have experts actually read the research done by applicants for a job, promotion, or grant. Simple counts and indexes simply won’t do. Bob Marks reminded me of the old aphorism in academia, “Deans can’t read, but they can count.” A better one is, “Not everything that can be counted counts, and not everything that counts can be counted.” (MORE - details)
You may also wish to read the first two articles in this three-part series:
Publish or Perish — Another Example of Goodhart’s Law. In becoming a target, publication has ceased to be a good measure. Researchers game the system to beat the publish-or-perish culture, which undermines the usefulness of publication and citation counts. (Gary Smith)
Gaming the System: The Flaws in Peer Review. Peer review is well-intentioned, but flawed in many ways. Predatory journals, dishonest researchers, and escalating costs in academic journals reveal the weaknesses in peer review. (Gary Smith)