Harry Frankfurt wrote a book “On Bullshit“, which Wikipedia summarizes as saying “bullshit is speech intended to persuade without regard for truth.” I think the problem is deeper than that. There is work generated for the sake of saying that work was done.

Peer Review

Consider peer review. I recently received a review from a paper I co-authored in a good journal. The reviews were positive except Reviewer #2 said the word X was not the right word. X is of course exactly the right word, but in order to get accepted we had to change the paper to make Reviewer #2 happy. [I refuse to accept the charitable view that these were Reviewer #2’s genuine beliefs, it is truly nonsense.]

We complied. We wasted our time to increase the utility of anonymous Reviewer #2 in order to satisfy the editor. Reviewer #2’s ego is boosted, by having enforced compliance, and therefore increasing his relative status at the expense of ours, but since he is anonymous, only he knows. Reviewer #2 could have just said “Accept”, but that would be too easy, he felt he had to say something to prove he reviewed the paper. (R2 could be female, but he feels male.)

Now Reviewer #2 is not operating in a vacuum. Undoubtedly some unreasonable reviewer of one of his papers made him go through what he felt were ridiculous contortions, and this rolls downhill. To salvage ego, the abused child becomes an abuser, creating a new generation of abused spouses and children.

What we have here is a cycle of peer review violence, where as more and more research is produced due to increased productivity of academics (in part due to the rise of information technologies, but mostly the publish or perish culture driven by university ranking systems driven by the desire to attract international students driven by revenue), more review requests are generated, more reviewers get more annoyed at the requests, and more hoops laid out before us.

This Reviewer #2 was actually not so bad. Many others are unhappy if you don’t regurgitate all scientific knowledge up until the present day, and lay out all prospective policy outcomes going forward. This attitude has led to an explosion of paper lengths.

Reviewers would be much more polite were reviews not anonymous. This raises other problems, that junior people would be afraid to confront senior people on cases which were not bullshit. Unless everything is open, an open (non-blind) review policy from a single journal cannot  extract fair reviews. Retaliation, on say grant reviews, or promotion reviews, which remain anonymous, is a risk.

Without any peer review, Gresham’s Law of Journal Articles: Bad knowledge drives out good, would surely apply. Peer review of some form is a Good Housekeeping Seal of Approval for scientific articles. But that said, what is really required here? If you, a journal, trust me enough to review (or edit) other people’s work, why do you not trust me enough to publish my own? There are a few arguments in favor of review:

• First, we have Linus’s Law: “with many eyes all bugs are shallow”, and so some editorial review will improve quality and find problems. Good authors want good editors.
• Second, the anticipation of peer review improves quality as you know a paper will have to get through review to be published

However in contrast, if we have a peer review system where nothing passes the first round (regardless of how good), but many papers go into the revise and resubmit limbo, authors will, in fact, submit lower quality work, and wait for the reviewers to make their recommendations, and spend their scarce time trying to satisfy those reviewers instead of themselves.  In short, we have constructed a system where peer review lowers the initial quality of submission. We have become so afraid of publishing false positives (a wrong paper), we create many false negatives (decline competent papers). History can judge false positives retrospectively just fine, we don’t actually need to spend so many resources to do this prospectively.

I have talked previously about how peer review also costs society knowledge, as the inevitable delay in the Revise and Resubmit round, and the cost of going back to closed projects.  Instead of rewarding academics by which journals they published in, reward them for how important their work is. This is either known, because history rewards them with citations, or arguable that the future will recognize them because colleagues believe in them now.

Instead of over-reliance on peer-review, we should view it as a filter to ensure wrong or poorly written papers are not published, not a filter to ensure only perfect papers are published. We should have a system that rewards the creation of small (or large) academic building blocks, and lets scientists and engineers and even economists file their work respectably as they develop it in the length appropriate, and not feel the need to expand their work to develop a whole new theory of civilization with every research output.

History can be the evaluator – it is attention which is now scarce, not the number of pages in a journal. Compliance with systems built for another age needs to be tossed with those systems.

Promotion

Seeking letters for promotion cases aims to ensure that an outsider (someone not at your university) says you do good work, because for some reason, the university cannot trust people at its own university to make such a judgment. My promotion cases required 10 or 11 letters from other academics at other universities saying that my work was good enough to warrant promotion, and I have written numerous anonymous letters. I have not retaliated (nor had the opportunity to ‘retaliate’ against a youngster I was offended by, or their senior allies) by trying to undermine a promotion case, but I certainly can see how some senior people might if they were offended by a junior faculty somewhere, or by those junior’s senior colleagues, like a peer review.

I understand that such letters help assure that promotion is warranted, but imagine Apple computer asking Microsoft, Google, and Facebook to write letters in support of promotion of their own software engineers. That’s absurd.  The evidence of my research is in my publications, and other people’s citations of those publications, not in whether someone else says my work is good. My colleagues should be able to judge that. The evidence of my teaching is in whether my students learned (and retained) anything, not in end-of-semester surveys.

But if I as a junior faculty know that I have to get 10 senior people to write letters for me, I will spend effort to curry favour by doing things like reviewing papers when assigned by editors, and serve on committees, and so on. In short, I will comply in advance so that the favour will be returned. Letter writing enforces compliance on the part of junior faculty structurally.

Let universities take on the burden themselves of deciding whom they should promote, rather than offloading this to the community. If they don’t feel comfortable assessing their own staff, maybe that’s a field they shouldn’t be in.

Conferences

Sometimes compliance-enforcement takes an even more ridiculous turn. I recently had a conference paper at an Australasian conference accepted on its substance but declined because of some mysterious MS Word formatting problem that I refused to spend even more time to rectify after 2 previous revision attempts. Despite using the organisers templates, they still decided the paper somehow didn’t meet the correct format, and so was rejected. Obviously it’s their loss, they’ll miss me, and our research,  (and my student who would have also presented something else), and the revenue we would have paid to attend the conference. If the papers were to be published in a book, I might understand why this matters, but that in fact was not the case, it was simply for electronic distribution, and the aesthetic judgment of the organiser, which is lacking (obviously, as it was a pretty ugly MS Word template to begin with).

Dissertations

Now I understand Tyler Cowan’s quote (can’t find the original, but essentially)

 “The most important thing I learned in my PhD was to get the margins on the pages right.”

Back in the day, an older woman in the registrar’s office would go through your thesis or dissertation with a ruler and measure to make sure the margins on each page were just so. And if not, the dissertation would be turned back, and you got to reformat it. This was the University’s final lesson in compliance.

But really, why does marginal perfectionism matter? We did it because the system required it. Fortunately, this particular requirement has disappeared, but why did the system require it? One imagines so that reproductions of the dissertation on a smaller sheet of paper would not lose important information. There may have once been a good reason, but margin size enforcement was promulgated as a rule that lasted long past the original need.

Establishing Degrees

At a major university with which I have an affiliation, I am working on establishing a new degree program. This is a relatively cost-free enterprise the university, the units of study are almost entirely already offered. However to get the program established, we have to have an Expression of Interest, vetted by three faculties involved, including 2 committees in my faculty, as well as two committees at the university level. Then we have a proposal, whose form is 57 pages. And then we need to go through all the same committees. I am told the 57 page form is designed to dissuade people who aren’t serious. But for those who are, that and all the meeting for something so technically simple to implement is pointless.

One of the faculty committees has about 50 members, all of whose job, apparently is to ratify what the other committees said and supervise one or two full-time staff members.

Let a thousand degrees flourish, and if they don’t succeed, they can be cancelled.

Accreditation

Just as universities accredit students (who undoubtedly think exams and homeworks and projects are a nuisance), degree programs often go through accreditation themselves to show that the curriculum they require students engage in comports with what the industry associations who control ABET think is important (or was important, as this is an exceptionally conservative process designed to stifle innovation.) The last time I went through this (fortunately I did not have to lead it in my Department), each required course produced a notebook with sample poor, average, and good work from students for each assignment, as well as printouts of the assignments and other miscellany. It is pretty clear the review panel did not actually review the contents of each notebook. They may have sampled them.  The  wall of notebooks was there to demonstrate compliance. Each assignment was cross-referenced against objectives and qualities students were supposed to accomplish by successfully completing that assignment. While this sounds good in principle, it is basically a database exercise, labelling things as satisfying objectives rather than changing things to meet objectives.

Let universities produce students whose value is they graduated from a university that taught what it thought important, and if that aligns with market demand, all the better. It is not as if students don’t also have to take and pass exams to be a Professional this or that, and an education that helped would be appropriately recognized, or universities don’t have well-established and largely self-fulfilling reputations.

Discussion

Academics do nothing if not evaluate each other’s work. The amount of time writing letters of recommendation, evaluations of promotion cases, reviews of proposals and each other’s programs, and conducting peer reviews  of articles is surprisingly, and in my view unnecessarily, high. It is academia generating work for academics who ought to be in the primary business of creating and transmitting knowledge, not evaluating knowledge creation and transmission. It is, in economic terms, a deadweight loss. If all this evaluation improved the quality of knowledge production or transmission sufficiently, it might not be, but there is no evidence I see such is the case. We adopt the forms because those before us adopted the forms.