Scivillage.com Casual Discussion Science Forum
Peer Review: The end of an error? - Printable Version

+- Scivillage.com Casual Discussion Science Forum (https://www.scivillage.com)
+-- Forum: Science (https://www.scivillage.com/forum-61.html)
+--- Forum: Junk Science (https://www.scivillage.com/forum-88.html)
+--- Thread: Peer Review: The end of an error? (/thread-4451.html)



Peer Review: The end of an error? - C C - Nov 7, 2017

https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

EXCERPT: [...] Of course, different disciplines have different needs and very different publishing ­cultures. In many subjects, referees typically demand much more substantial changes than they do in mathematics, journals have less ­permissive policies regarding preprints, and authors are not in the habit of making their work available online. Journals in the biomedical sciences, for example, often do not allow authors to post versions of their articles that have been revised in response to comments from referees. These sometimes differ in important ways from the versions originally submitted: for example, it is not uncommon for referees to go as far as to require authors to carry out further experiments. It also seems that many scientists do not regard posting a preprint as any kind of establishment of priority, and they worry about being scooped if they do so. So, I do not want to propose some unified system that would be used by all academics – a mistake made with depressing frequency by university policy-makers – but I do think that we should each question whether the current system, whatever it might be in our particular discipline, is optimal for that discipline. I also think that in at least some disciplines, my own being an example, the answer is no.

Defences of formal peer review tend to focus on three functions it serves. The first is that it is supposed to ensure reliability: if you read something in the peer-reviewed literature, you can have some confidence that it is correct. This confidence may fall short of certainty, but at least you know that experts have looked at the paper and not found it ­obviously flawed.

The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us. Similarly, a vast amount of academic literature is being produced all the time, most of it not deserving of our attention, and the peer-review system saves us time by selecting the most important articles. It also enables us to make quick judgements about the work of other academics: instead of actually reading the work, we can simply look at where it has been published.

The third function is providing feedback. If you submit a serious paper to a serious journal, then whether or not it is accepted, it has at least been read, and if you are lucky you receive valuable advice about how to improve it.

Let us consider these functions in turn. For each one, we should ask whether formal peer review is the best way of performing that function, and, if so, whether the function is valuable enough to justify the effort and expense of the formal peer review system....

MORE: https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/


RE: Peer Review: The end of an error? - Syne - Nov 7, 2017

Would peer review have help Tesla get the acclaim he deserved?


RE: Peer Review: The end of an error? - RainbowUnicorn - Nov 8, 2017

(Nov 7, 2017 11:24 PM)C C Wrote: https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

EXCERPT: [...] Of course, different disciplines have different needs and very different publishing ­cultures. In many subjects, referees typically demand much more substantial changes than they do in mathematics, journals have less ­permissive policies regarding preprints, and authors are not in the habit of making their work available online. Journals in the biomedical sciences, for example, often do not allow authors to post versions of their articles that have been revised in response to comments from referees. These sometimes differ in important ways from the versions originally submitted: for example, it is not uncommon for referees to go as far as to require authors to carry out further experiments. It also seems that many scientists do not regard posting a preprint as any kind of establishment of priority, and they worry about being scooped if they do so. So, I do not want to propose some unified system that would be used by all academics – a mistake made with depressing frequency by university policy-makers – but I do think that we should each question whether the current system, whatever it might be in our particular discipline, is optimal for that discipline. I also think that in at least some disciplines, my own being an example, the answer is no.

Defences of formal peer review tend to focus on three functions it serves. The first is that it is supposed to ensure reliability: if you read something in the peer-reviewed literature, you can have some confidence that it is correct. This confidence may fall short of certainty, but at least you know that experts have looked at the paper and not found it ­obviously flawed.

The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us. Similarly, a vast amount of academic literature is being produced all the time, most of it not deserving of our attention, and the peer-review system saves us time by selecting the most important articles. It also enables us to make quick judgements about the work of other academics: instead of actually reading the work, we can simply look at where it has been published.

The third function is providing feedback. If you submit a serious paper to a serious journal, then whether or not it is accepted, it has at least been read, and if you are lucky you receive valuable advice about how to improve it.

Let us consider these functions in turn. For each one, we should ask whether formal peer review is the best way of performing that function, and, if so, whether the function is valuable enough to justify the effort and expense of the formal peer review system....

MORE: https://www.the-tls.co.uk/articles/public/the-end-of-an-error-peer-review/

Quote: In many subjects, referees typically demand much more substantial changes than they do in mathematics,

Quote:whether the function is valuable enough to justify the effort and expense



RE: Peer Review: The end of an error? - Yazata - Nov 9, 2017

(Nov 7, 2017 11:24 PM)C C Wrote: The second is a bit like the function of film reviews. We do not want to endure a large number of bad films in order to catch the occasional good one, so we leave that to film critics, who save us time by identifying the good ones for us.

I don't pay much attention to film reviewers.

A big part of that is due to my love of science fiction movies. Film reviewers are often former literature majors or graduates of film school. They have been taught that science fiction is a low-brow genre and largely speaking worthy of an intellectual's contempt. So film reviewers typically favor "knowing" science fiction movies that poke fun at the tropes and conventions of the genre. They typically dislike science fiction movies that treat their material seriously. (In their mind, doing so is just stupid.)

I treat science fiction very seriously, perceiving it as imaginative fiction and film. It imagines counterfactual situations with huge significance and deep implications (first contact, global pandemics, robot revolts, time travel, total surveillance, runaway biotech or whatever) and sci-fi often addresses themes once addressed by religion (superior beings, transhumanism, ultimate transcendence, even salvation).

So unfortunately, reading film reviewers' opinions of science fiction movies often tells me little or nothing about whether I would appreciate the film. We are coming at the genre from diametrically opposed directions.