What is the point of scientific peer review?

peer-review-thumbnailIn order to publish in any scientific journal worth publishing in, ones paper must pass “peer review”. This consists of the editors asking one or more experts in the field to review the paper in order to spot errors, recommend improvements, and opine on whether it is of sufficient merit to publish.

Richard Smith, former editor of the British Medical Journal, has written an article in the Times Higher Education arguing that “the process has little if any benefit and lots of flaws”, and that “peer review doesn’t work” because most of what is published in scientific journals “is plain wrong”.

He tells the story, when at the BMJ, of deliberately inserting multiple errors into manuscripts to see how many reviewers found them. Many of the reviewers missed many of the errors.

This raises the question of what peer review is actually for. Smith states that “Peer review is supposed to be the quality assurance system for science”, that aims to reassure readers “that they can trust what they are reading” (added emphasis).

However, no working scientist actually “trusts” scientific papers, all scientists read papers in a sceptical way, knowing full well that much of what they are reading will turn out to be wrong. “The” quality-assurance system for science is not what happens prior to publication, but what happens after publication. That is when the real peer review begins, when the community can read and think about and criticise a paper.

Peer Review protest sign

It needs to be that way for two reasons. First, a peer-review system that attempted to publish only “correct” work, or even just work with no errors in method, would be much too expensive and time-consuming. In my field most papers are reviewed by one expert taking a couple of hours to do so. That is grossly insufficient to spot all errors and to certify the paper as error-free, but any process designed to do more would be overly burdensome. Anyhow, it is not primarily the editor’s, nor the reviewer’s responsibility to ensure that the paper is error-free, that responsibility is primarily with the authors.

Second, we want scientific publishing to be relatively open, we want people to be able to promote likely-wrong work, because any process which didn’t allow that would end up censoring too many actually-right ideas. Again, the sorting out of what is right versus wrong should occur mainly after publication, involving all those in the field who wish to get involved.

So what is peer review actually for? Essentially, the peer-review threshold is not “this is right”, but rather “this is worthy of your attention”. If there were no peer review and no standards at all, good-quality work would be lost in a sea of dross. Thus peer review is indeed there to winnow out low-quality work. It ensures that the work is good enough and presented with enough information that it is worthy of consideration by others in the field. If you submit to a flashy journal there is an additional criterion of needing to have sufficiently interesting implications.

But it does not particularly matter if peer review is only partially effective, it is merely cutting down the number of articles that a scientist need be aware of, and it is better for it to err on the side of letting through too many rather than too few. Scientists in the field are then quite capable of forming their own opinions of a paper’s validity and correctness.

Non-scientists such as journalists need to be aware that being “peer-reviewed” is not even intended to be a certificate of correctness, and thus media reports should take a sceptical view of any and all new findings — just as working scientists do — until they become accepted as mainstream. That might involve canvassing opinions from experts and digging deeper into a story, rather than accepting what a press-release says just because the paper has been peer reviewed.

This is particularly important in fields such as medicine where new findings might matter to a lot of people. Any and all new results are suspect until scrutinised widely by others in the field, and possibly until they have been repeated. Cutting-edge science always works that way.

Thus peer review is only one hurdle, and one with a deliberately low threshold. It is only the beginning of the real scrutiny that the work will receive.

Of course nothing said above is a licence to be sloppy or to not bother aiming for the highest standards. Authors, primarily, but also referees and editors, should always be continually striving for top-quality work. But journal peer-review is only one part of that process, rather than being “the” process, and the limitations pointed out by Richard Smith are “features” of the system rather than “bugs”. Given that we accept those limitations, peer review is actually doing its job and doing it reasonably well.

Advertisements

7 thoughts on “What is the point of scientific peer review?

  1. Mark Wallace

    “Second, we want scientific publishing to be relatively open, we want people to be able to promote likely-wrong work, because any process which didn’t allow that would end up censoring too many actually-right ideas.”

    Working in the humanities, I know little of peer-review in the sciences, but it’s surprising to me that you see it as making the field more open. Academic journals have very specialized criteria which makes it basically impossible for a non-academic to write to their standards. Of course, once you master these criteria, it becomes relatively easy to produce work which is publishable if not substantial, but that takes a very specialized education. Are science journals more “open” in the sense that it is possible for a person outside of the academy to be published?

    Reply
    1. Coel Post author

      I wasn’t quite arguing that peer review made the field more open, more that the current limited peer review makes it more open than a more stringent peer review would be.

      It is indeed possible for outsiders to publish in science journals, though in practice it rarely happens with outsiders as the only authors. More frequently there are papers that are collaborations between, say, professional astronomers and amateur astronomers.

  2. Phillip Helbig

    “peer review doesn’t work” because most of what is published in scientific journals “is plain wrong”

    In my field (cosmology, astrophysics, astronomy), this is certainly not true. Yes, quality of peer review does vary, and occasionally mistakes slip through, but “plain wrong” papers appear very rarely in respected journals.

    I remember once where the referee pointed out a typo in equation 50 or whatever. At least in that case, I am sure he read the whole paper. (Since then, I have often thought of inserting deliberate errors, just to see if the referee catches them, but this is also a two-edged sword.)

    In general, I agree with what you say.

    While outreach is good, the offer of journals and/or professional societies to assist with press releases etc often produces bad results, since some journalist with no command of the field might write up something based on a press release without putting it into context.

    Ideally, journalists will read the respected journals and contact the authors if they want to write about their work. If a journalist can’t read the paper, understand it, and put it into context, he can’t write a useful popular article about it.

    Of course, the problem is often with very popular (newspaper) press, but they should base their stuff on more specialized, semi-popular articles, not on press releases.

    Reply
  3. Roo Bookaroo

    Excellent and lucid article.
    Its point is to dispel the aura of authority that the general public of ordinary people who are not professionals in one field tends to credit to the appellation of “peer-reviewed”.
    It’s the debunking of another distorted public notion that tends to erect scientific findings or conclusions as the equivalent of a dogma for a religious believer. That the label of “peer-reviewed” should deserve the same trustworthiness as “seal of good housekeeping” or “approved by the Church”, “pope-endorsed”, or “the party line”, in other fields.
    Whatever the general public knows in areas where it does not have direct experience or professional expertise is always second-hand or third-hand, or even more remote from the original source of information.
    The real problem is the distortion happening in the transmission of ideas and statements from the source to following links in the chain of transmission.
    There’s little point in trying to directly educate the public, the task is hopeless, as the whole field of knowledge is filtered by the various modes of its transmission. The weak link is in the presentation to the public by the media, journalists, popularizers, TV and radio announcers, since this is at this stage that the skeptical undertone of scientific research is lost, and scientific results are presented as solid facts in easily digestible language or quick sound bites. One characteristic of this style is well expressed in KISS, “keep it simple, stupid”. It is to this category of simplifiers and disseminators that Coel’s denunciation of the mystique of “peer-review” has most relevance and usefulness.

    In the field where I have devoted enough time to become aware of the ambiguities and uncertainties of the issues, the Origins of Christianity, I am always amused to see the Columbia University PhD Richard Carrier, unable to find any academic position and guaranteed source of income, going around the US, peddling his books and articles by constantly emphasizing his “peer-reviewed” label to impress the ignorant American public, as a modern form of circus barking.

    Reply
  4. YF

    I largely agree. One might add that the key to separating gold from garbage post-peer review is independent replication of findings.

    Reply
  5. Philosopher Eric

    Hi Coel,

    Yes I now see that peer review both isn’t and shouldn’t be an error catching exercise, but rather an up/down vote from a few respected scientists regarding whether or not a given paper seems somewhat promising. This is a way to filter out an avalanche of papers that are *perhaps* less promising, not a way to provide much true validation. While acceptance will grow as further verification comes in, the risk of invalidation remains for even the most successful of ideas. Theory must always remain just that.

    Furthermore we might consider peer review for respected journals, but regarding the far less well founded realm of philosophy. The question is, can an objective assessment be made, when there is so little which has become established in the field? I have a friend who went through six months of this last year, and even if successful I think he would have said that the experience seemed overly confrontational and arbitrary. But then why would we not expect this from a field which contains so many opposing positions?

    Nevertheless I’m not here to throw stones at philosophers, for at least these people are ambitious enough to question our most vital uncertainties. Until progress becomes made regarding such uncertainties, it’s the circle of science that shall remain incomplete.

    Reply
  6. Alex SL

    I agree mostly with what you wrote, but it really depends on the area. In my field two to three reviewers may examine a manuscript, and that means that usually nothing seriously flawed will get into the ‘good’ journals. The process is obviously far from perfect: sometimes a few bad papers make it into really prestigious journals; very often perfectly decent papers get rejected because somebody got out up with the wrong foot or doesn’t like the author’s face – but they will ultimately get published in a different but equivalent journal; and all too often, perfectly good papers get rejected by a fancy journal because they are supposedly not of sufficient ‘general interest’, which usually means that you aren’t working on vertebrates.

    Still, it works as a sorting mechanism that ensures I will find very little nonsense in the important journals in my field. I shudder to think how it would be if the voices who clamour for post-publication review only got their wish. We would have to start judging papers by who wrote them, and that would be a big hurdle for competent newcomers.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s