In order to publish in any scientific journal worth publishing in, ones paper must pass “peer review”. This consists of the editors asking one or more experts in the field to review the paper in order to spot errors, recommend improvements, and opine on whether it is of sufficient merit to publish.
Richard Smith, former editor of the British Medical Journal, has written an article in the Times Higher Education arguing that “the process has little if any benefit and lots of flaws”, and that “peer review doesn’t work” because most of what is published in scientific journals “is plain wrong”.
He tells the story, when at the BMJ, of deliberately inserting multiple errors into manuscripts to see how many reviewers found them. Many of the reviewers missed many of the errors.
This raises the question of what peer review is actually for. Smith states that “Peer review is supposed to be the quality assurance system for science”, that aims to reassure readers “that they can trust what they are reading” (added emphasis).
However, no working scientist actually “trusts” scientific papers, all scientists read papers in a sceptical way, knowing full well that much of what they are reading will turn out to be wrong. “The” quality-assurance system for science is not what happens prior to publication, but what happens after publication. That is when the real peer review begins, when the community can read and think about and criticise a paper.
It needs to be that way for two reasons. First, a peer-review system that attempted to publish only “correct” work, or even just work with no errors in method, would be much too expensive and time-consuming. In my field most papers are reviewed by one expert taking a couple of hours to do so. That is grossly insufficient to spot all errors and to certify the paper as error-free, but any process designed to do more would be overly burdensome. Anyhow, it is not primarily the editor’s, nor the reviewer’s responsibility to ensure that the paper is error-free, that responsibility is primarily with the authors.
Second, we want scientific publishing to be relatively open, we want people to be able to promote likely-wrong work, because any process which didn’t allow that would end up censoring too many actually-right ideas. Again, the sorting out of what is right versus wrong should occur mainly after publication, involving all those in the field who wish to get involved.
So what is peer review actually for? Essentially, the peer-review threshold is not “this is right”, but rather “this is worthy of your attention”. If there were no peer review and no standards at all, good-quality work would be lost in a sea of dross. Thus peer review is indeed there to winnow out low-quality work. It ensures that the work is good enough and presented with enough information that it is worthy of consideration by others in the field. If you submit to a flashy journal there is an additional criterion of needing to have sufficiently interesting implications.
But it does not particularly matter if peer review is only partially effective, it is merely cutting down the number of articles that a scientist need be aware of, and it is better for it to err on the side of letting through too many rather than too few. Scientists in the field are then quite capable of forming their own opinions of a paper’s validity and correctness.
Non-scientists such as journalists need to be aware that being “peer-reviewed” is not even intended to be a certificate of correctness, and thus media reports should take a sceptical view of any and all new findings — just as working scientists do — until they become accepted as mainstream. That might involve canvassing opinions from experts and digging deeper into a story, rather than accepting what a press-release says just because the paper has been peer reviewed.
This is particularly important in fields such as medicine where new findings might matter to a lot of people. Any and all new results are suspect until scrutinised widely by others in the field, and possibly until they have been repeated. Cutting-edge science always works that way.
Thus peer review is only one hurdle, and one with a deliberately low threshold. It is only the beginning of the real scrutiny that the work will receive.
Of course nothing said above is a licence to be sloppy or to not bother aiming for the highest standards. Authors, primarily, but also referees and editors, should always be continually striving for top-quality work. But journal peer-review is only one part of that process, rather than being “the” process, and the limitations pointed out by Richard Smith are “features” of the system rather than “bugs”. Given that we accept those limitations, peer review is actually doing its job and doing it reasonably well.