On false positives and false negatives and peer review

I have written previously about peer review. I wrote:

This leads me to the hypothesis that the primary purpose of academic Peer Review is not to review papers and give feedback to authors. It is instead to induce authors to submit work of high quality because they believe someone will read it.

Journals want to ensure good (e.g. novel and important) papers are accepted and bad (e.g. wrong or trivial) papers are rejected. In addition to the evaluative goal, peer review may also have a developmental goal, making papers better, as any paper can be improved. It seems reasonable enough as a goal, it has costs that are unnecessarily high.

There are two sources of errors that can occur, analogous to Type I and type II errors in statistics (which is which depends on what you take as the null hypothesis, rejection or acceptance):

Error 1: Bad papers are accepted. … This is a false positive.

Error 2: Good papers are rejected. … This is a false negative.

There has been a great deal of ink spilled about the acceptance of bad papers, and the retraction of wrong papers.  Obviously we would prefer not to accept bad papers as a community, as it is embarrassing, may mislead researchers and the general public.

However, we spend so much time poring over papers (the amount of time academics spend reviewing other academics’ work would surprise an outsider) to ensure bad papers are rejected that we inevitably cast our net wide enough to reject good papers. And so we almost never accept good papers on the first round.

Any rejected paper can always be resubmitted and a second (third, fourth, fifth) journal can get an opportunity to review it. This costs time. But more than that it costs a significant amount of mental effort. When the paper was originally submitted, it was immediately after the research was completed. The ideas were fresh in the mind. Authors were somewhat enthusiastic about the topic. By submitting the paper, the authors have mentally closed this project and opened the next one. But then 3 or 6 or 9 or 12 months later (or in one sad case of mine 8 years!) the reviews come back. And the reviewers want some change; the reviewers always see some way the paper can be improved. And no doubt in a perfect world with infinite time in a day, we would agree not only that this is an improvement, but that it is worth doing.

But instead, we are apathetic or antagonistic or busy with other things, as what was closed has now been needlessly reopened for what is in reality a very minor improvement most of the time to make the reviewer feel that his or her fingerprints have affected the outcome of the paper.

Some of my coauthors are also faculty members, and should have motivation to revise and resubmit, which may be a few hours to a days worth of work in many cases, and is a far faster way to get a paper accepted than starting from scratch. But the mental burden and pains reopened are that great for work from 1 or 3 or 5 or 10 years ago. I have more understanding for coauthors who are in industry, where the rewards from peer reviewed publication are another line on the CV and maybe an attaboy (attagirl) and a beer from colleagues, but not existential in the way tenure is.

But instead of revising the paper, it sits.

I currently have about 10 papers in this state (almost enough to move someone from Assistant Professor to Associate Professor at many US universities), ignoring papers that have been fully abandoned, and excluding papers that I have some confidence or hope will actually be revised and resubmitted soon. My coauthors have not yet made the revisions necessary,  (nor did I, but they were the lead authors and it was really their work), and so it was not done in a timely way and thus the original reviews effectively expired; and we have not sent it elsewhere. There are always reasons, with which I have empathy, coauthors have young families, new jobs, or are otherwise busy. In the end it is a question of priorities, and the personal benefits to publication for non-academics is not especially great, the benefits accrue to science and society at large. The positive spillovers cannot be captured.

And this is after I encourage, cajole, nag, and flog students and former students to revise and resubmit. And I suspect I am more systematic about this than most people. The amount of knowledge buried on people’s hard drives because of the peer review ‘revise and resubmit’ system is a huge loss to humanity and scientific progress.