Tag Archives: Peer review

Is peer review biased against women?

In astrophysics, time-allocation committees allocate the time available on major telescopes and satellites. Facilities such as the Hubble Space Telescope are often over-subscribed by factors of 4 to 10, and writing proposals that win time in peer review is crucial to people’s careers.

I’ve recently had my first experience of serving on a “dual-anonymous” time-allocation committee, in which all the proposals are reviewed without knowing the proposers’ names.

The trend to dual-anonymous review started with an analysis of 18 years of proposals to use Hubble. The crucial data showed that, over more than a decade, the success rate for proposals with a woman as principal investigator was lower (at 19%) than for proposals led by men (at 23%). That 20% difference in success rate then disappeared when proposals were reviewed without knowing the proposer’s names.

Since then, the study has been widely quoted as an example of either conscious or unconscious bias against women, and many time-allocation committees have now moved to dual-anonymous reviewing.

Call me a heretic, but I suspect that the findings actually show a different kind of bias: a bias towards well-known names. We all have such a bias. It’s why brand names are so crucial to marketing; it’s why advertisers spend billions on promoting the names of their products.

If (as is true) well-known, late-career “names” in the field tend to be dominated by men, whereas (as is also true) early-career researchers tend to be more balanced in gender, then a bias towards well-known names would itself account for the success rate of male-led proposals being 20% higher, even if there were no bias against women per se (see footnote).

This suggestion is actually supported by the Hubble study: “[Stefanie] Johnson and her graduate student, Jessica Kirk, found no evidence of gender bias in the preliminary grading that determined which proposals made it to the discussion stage. It was only in the in-person discussions that bias reared its head, and Johnson and Kirk noted a potential reason for it: Much of the in-person discussion on a given proposal focused on the track record of the applicant and colleagues, rather than on the science he or she was proposing to do”.

It is obvious that such a process would suffer from well-known-name bias. One can then ask whether such a bias is a good thing, after all, the established names have presumably acquired that status by repeatedly demonstrating that they can do good science. On the other hand, science is often driven by fresh ideas from the youthful. This could be argued either way, but a process that keeps the “names” on their toes by forcing them to compete on a level playing field is no bad thing.

Why, then, is the gender balance very different between early-career researchers and late-career names? One obvious explanation might be sexist bias against women, particularly in past decades when now-senior names started out. I suspect, however, that the biggest reason is that academia has a career structure which leads to capable early-career researchers being less likely to pursue senior status if they are women.

Establishing a successful academic career is pretty much a rat-race, involving ongoing competition for grants and resources against 10-to-1 competition, often through a series of fixed-term, post-doc contracts with little job security, often moving between institutions to chase opportunities, and all the while ensuring that one produces a stream of good-quality publications. Notably, that is all happening at the same stage in life when many will want to start and support a young family. The structure of academia forces people to prioritise one or the other, and it’s no surprise that there might be a systematic difference between men and women in making that choice.

To summarise: is the move to dual-anonymous peer review a good thing? Probably yes. Does non-anonymous peer review suffer from a sexist bias? Likely not (though it is likely biased towards established names). Does the structure of academia make it easy for researchers with young families? Definitely not. In particular, the necessity of moving to institutions that match ones research speciality, or simply to find an available job, often requires a spouse willing to prioritise their partner’s career over their own.

Footnote: This is an example of Simpson’s paradox. Suppose that a quarter of the proposals are submitted by “senior names”, which are 65:35 male-to-female, and which have a 20% chance of success. The rest of the proposals have a 50:50 gender balance, and a 15% chance of success. A quick calculation shows that this reproduces the same 20% advantage in proposals submitted by men, even if there is no actual sexist bias in the evaluations.

What is the point of scientific peer review?

peer-review-thumbnailIn order to publish in any scientific journal worth publishing in, ones paper must pass “peer review”. This consists of the editors asking one or more experts in the field to review the paper in order to spot errors, recommend improvements, and opine on whether it is of sufficient merit to publish.

Richard Smith, former editor of the British Medical Journal, has written an article in the Times Higher Education arguing that “the process has little if any benefit and lots of flaws”, and that “peer review doesn’t work” because most of what is published in scientific journals “is plain wrong”.

He tells the story, when at the BMJ, of deliberately inserting multiple errors into manuscripts to see how many reviewers found them. Many of the reviewers missed many of the errors.

This raises the question of what peer review is actually for. Smith states that “Peer review is supposed to be the quality assurance system for science”, that aims to reassure readers “that they can trust what they are reading” (added emphasis).

However, no working scientist actually “trusts” scientific papers, all scientists read papers in a sceptical way, knowing full well that much of what they are reading will turn out to be wrong. “The” quality-assurance system for science is not what happens prior to publication, but what happens after publication. That is when the real peer review begins, when the community can read and think about and criticise a paper. Continue reading