Tag Archives: Simpson’s paradox

Is peer review biased against women?

In astrophysics, time-allocation committees allocate the time available on major telescopes and satellites. Facilities such as the Hubble Space Telescope are often over-subscribed by factors of 4 to 10, and writing proposals that win time in peer review is crucial to people’s careers.

I’ve recently had my first experience of serving on a “dual-anonymous” time-allocation committee, in which all the proposals are reviewed without knowing the proposers’ names.

The trend to dual-anonymous review started with an analysis of 18 years of proposals to use Hubble. The crucial data showed that, over more than a decade, the success rate for proposals with a woman as principal investigator was lower (at 19%) than for proposals led by men (at 23%). That 20% difference in success rate then disappeared when proposals were reviewed without knowing the proposer’s names.

Since then, the study has been widely quoted as an example of either conscious or unconscious bias against women, and many time-allocation committees have now moved to dual-anonymous reviewing.

Call me a heretic, but I suspect that the findings actually show a different kind of bias: a bias towards well-known names. We all have such a bias. It’s why brand names are so crucial to marketing; it’s why advertisers spend billions on promoting the names of their products.

If (as is true) well-known, late-career “names” in the field tend to be dominated by men, whereas (as is also true) early-career researchers tend to be more balanced in gender, then a bias towards well-known names would itself account for the success rate of male-led proposals being 20% higher, even if there were no bias against women per se (see footnote).

This suggestion is actually supported by the Hubble study: “[Stefanie] Johnson and her graduate student, Jessica Kirk, found no evidence of gender bias in the preliminary grading that determined which proposals made it to the discussion stage. It was only in the in-person discussions that bias reared its head, and Johnson and Kirk noted a potential reason for it: Much of the in-person discussion on a given proposal focused on the track record of the applicant and colleagues, rather than on the science he or she was proposing to do”.

It is obvious that such a process would suffer from well-known-name bias. One can then ask whether such a bias is a good thing, after all, the established names have presumably acquired that status by repeatedly demonstrating that they can do good science. On the other hand, science is often driven by fresh ideas from the youthful. This could be argued either way, but a process that keeps the “names” on their toes by forcing them to compete on a level playing field is no bad thing.

Why, then, is the gender balance very different between early-career researchers and late-career names? One obvious explanation might be sexist bias against women, particularly in past decades when now-senior names started out. I suspect, however, that the biggest reason is that academia has a career structure which leads to capable early-career researchers being less likely to pursue senior status if they are women.

Establishing a successful academic career is pretty much a rat-race, involving ongoing competition for grants and resources against 10-to-1 competition, often through a series of fixed-term, post-doc contracts with little job security, often moving between institutions to chase opportunities, and all the while ensuring that one produces a stream of good-quality publications. Notably, that is all happening at the same stage in life when many will want to start and support a young family. The structure of academia forces people to prioritise one or the other, and it’s no surprise that there might be a systematic difference between men and women in making that choice.

To summarise: is the move to dual-anonymous peer review a good thing? Probably yes. Does non-anonymous peer review suffer from a sexist bias? Likely not (though it is likely biased towards established names). Does the structure of academia make it easy for researchers with young families? Definitely not. In particular, the necessity of moving to institutions that match ones research speciality, or simply to find an available job, often requires a spouse willing to prioritise their partner’s career over their own.

Footnote: This is an example of Simpson’s paradox. Suppose that a quarter of the proposals are submitted by “senior names”, which are 65:35 male-to-female, and which have a 20% chance of success. The rest of the proposals have a 50:50 gender balance, and a 15% chance of success. A quick calculation shows that this reproduces the same 20% advantage in proposals submitted by men, even if there is no actual sexist bias in the evaluations.