Peer Review: The end of an error?

#1
C C Offline
https://www.the-tls.co.uk/articles/publi...er-review/

EXCERPT: [...] Of course, different disciplines have different needs and very different publishing ­cultures. In many subjects, referees typically demand much more substantial changes than they do in mathematics, journals have less ­permissive policies regarding preprints, and authors are not in the habit of making their work available online. Journals in the biomedical sciences, for example, often do not allow authors to post versions of their articles that have been revised in response to comments from referees. These sometimes differ in important ways from the versions originally submitted: for example, it is not uncommon for referees to go as far as to require authors to carry out further experiments. It also seems that many scientists do not regard posting a preprint as any kind of establishment of priority, and they worry about being scooped if they do so. So, I do not want to propose some unified system that would be used by all academics – a mistake made with depressing frequency by university policy-makers – but I do think that we should each question whether the current system, whatever it might be in our particular discipline, is optimal for that discipline. I also think that in at least some disciplines, my own being an example, the answer is no.

Defences of formal peer review tend to focus on three functions it serves. The first is that it is supposed to ensure reliability: if you read something in the peer-reviewed literature, you can have some confidence that it is correct. This confidence may fall short of certainty, but at least you know that experts have looked at the paper and not found it ­obviously flawed.

The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us. Similarly, a vast amount of academic literature is being produced all the time, most of it not deserving of our attention, and the peer-review system saves us time by selecting the most important articles. It also enables us to make quick judgements about the work of other academics: instead of actually reading the work, we can simply look at where it has been published.

The third function is providing feedback. If you submit a serious paper to a serious journal, then whether or not it is accepted, it has at least been read, and if you are lucky you receive valuable advice about how to improve it.

Let us consider these functions in turn. For each one, we should ask whether formal peer review is the best way of performing that function, and, if so, whether the function is valuable enough to justify the effort and expense of the formal peer review system....

MORE: https://www.the-tls.co.uk/articles/publi...er-review/
Reply
#2
Syne Offline
Would peer review have help Tesla get the acclaim he deserved?
Reply
#3
RainbowUnicorn Offline
(Nov 7, 2017 11:24 PM)C C Wrote: https://www.the-tls.co.uk/articles/publi...er-review/

EXCERPT: [...] Of course, different disciplines have different needs and very different publishing ­cultures. In many subjects, referees typically demand much more substantial changes than they do in mathematics, journals have less ­permissive policies regarding preprints, and authors are not in the habit of making their work available online. Journals in the biomedical sciences, for example, often do not allow authors to post versions of their articles that have been revised in response to comments from referees. These sometimes differ in important ways from the versions originally submitted: for example, it is not uncommon for referees to go as far as to require authors to carry out further experiments. It also seems that many scientists do not regard posting a preprint as any kind of establishment of priority, and they worry about being scooped if they do so. So, I do not want to propose some unified system that would be used by all academics – a mistake made with depressing frequency by university policy-makers – but I do think that we should each question whether the current system, whatever it might be in our particular discipline, is optimal for that discipline. I also think that in at least some disciplines, my own being an example, the answer is no.

Defences of formal peer review tend to focus on three functions it serves. The first is that it is supposed to ensure reliability: if you read something in the peer-reviewed literature, you can have some confidence that it is correct. This confidence may fall short of certainty, but at least you know that experts have looked at the paper and not found it ­obviously flawed.

The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us. Similarly, a vast amount of academic literature is being produced all the time, most of it not deserving of our attention, and the peer-review system saves us time by selecting the most important articles. It also enables us to make quick judgements about the work of other academics: instead of actually reading the work, we can simply look at where it has been published.

The third function is providing feedback. If you submit a serious paper to a serious journal, then whether or not it is accepted, it has at least been read, and if you are lucky you receive valuable advice about how to improve it.

Let us consider these functions in turn. For each one, we should ask whether formal peer review is the best way of performing that function, and, if so, whether the function is valuable enough to justify the effort and expense of the formal peer review system....

MORE: https://www.the-tls.co.uk/articles/publi...er-review/

Quote: In many subjects, referees typically demand much more substantial changes than they do in mathematics,

Quote:whether the function is valuable enough to justify the effort and expense
Reply
#4
Yazata Offline
(Nov 7, 2017 11:24 PM)C C Wrote: The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us.

I don't pay much attention to film reviewers.

A big part of that is due to my love of science fiction movies. Film reviewers are often former literature majors or graduates of film school. They have been taught that science fiction is a low-brow genre and largely speaking worthy of an intellectual's contempt. So film reviewers typically favor "knowing" science fiction movies that poke fun at the tropes and conventions of the genre. They typically dislike science fiction movies that treat their material seriously. (In their mind, doing so is just stupid.)

I treat science fiction very seriously, perceiving it as imaginative fiction and film. It imagines counterfactual situations with huge significance and deep implications (first contact, global pandemics, robot revolts, time travel, total surveillance, runaway biotech or whatever) and sci-fi often addresses themes once addressed by religion (superior beings, transhumanism, ultimate transcendence, even salvation).

So unfortunately, reading film reviewers' opinions of science fiction movies often tells me little or nothing about whether I would appreciate the film. We are coming at the genre from diametrically opposed directions.
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  Peter Doshi + Hoax paper + Dog eyes + Aftermath of work built on an error C C 1 144 Jan 17, 2021 01:45 AM
Last Post: Zinjanthropos
  I published a fake paper in a ‘peer-reviewed’ journal C C 1 118 Nov 30, 2020 04:23 AM
Last Post: Zinjanthropos
  Peer review: How is that working out for ya? C C 0 262 Jun 25, 2020 09:35 PM
Last Post: C C
  Peer review is wheel of misfortune + How sci gave its soul to the publishing industry C C 0 165 Feb 11, 2020 06:05 AM
Last Post: C C
  Peer reviewers getting sophomoric, unreliable & subjective + Is social priming bogus? C C 1 174 Dec 14, 2019 07:04 PM
Last Post: Syne
  Will sensitivity-fueled litigation end peer review? + Bad cannabis science C C 7 1,221 Nov 4, 2017 10:44 AM
Last Post: RainbowUnicorn
  Why It Took Social Science Years to Correct a Simple Error About ‘Psychoticism’ C C 3 613 Apr 14, 2017 11:35 PM
Last Post: Syne



Users browsing this thread: 1 Guest(s)