Well, someone has finally come out and said what we are all thinking. A professor, Marc Kirschner, has an editorial in the tabloid ‘Science’ criticizing the NIH’s emphasis on pre-judging “impact” and translational “significance” of proposed studies.

The editorial is frank, with some of my favorite quotes being:

Thus, under the guise of an objective assessment of impact, such requirements [of judging impact of proposed and current research] invite exaggerated claims of the importance of the predictable outcomes—which are unlikely to be the most important ones. This is both misleading and dangerous.

In today’s competitive job and grant market, these demands create a strong inducement for sloppy science.

And they should reemphasize humility, banishing the words “impact” and “significance” and seeing them for what they really are: ways of asserting bias without being forced to defend it.

I’m not sure, however, if speaking our minds, even when done by senior scientists, is going to change anything. The problem is not really of attitude, as Kirshner suggests. I think it is simply of funds, as in not enough of compared to the number of scientists vying for them.

When ever demand exceeds supply the supplier can do any arbitrary thing they want. In many jobs, like a coder position at Google or a tenure track research position, where there are many more applicants than positions we would think quality would be selected for, that the cream would rise to the top.

What really happens, however, is that since imperfect humans are judging other imperfect humans in qualities they hope are predictive of future success, we pick arbitrary criteria. We think we are being clever and competitive, when we are actually asking people to jump through hoops just for the heck of it. It would be better to just have a lottery.

I think that until we have more grants or fewer scientists we will continue to apply stupid criteria simply because we need to filter. When one thing is as good (or bad) as the other, its just chance. Like Kirshner, I agree we should not use “impact” and “significance” to judge suitability of grant proposals.

I have a more radical suggestion: use a lottery.

My former adviser (John Maunsell) has said to me that he believes that once you go below the 20 percent line in grant scoring its all noise. The bad ones have been taken out and now all the good ones are in the mix and no one can tell which ones will pan out and be significant.

We should be honest, filter the badly designed experiments out, and then do a lottery on the proposals that contain technically sound science.

(On the topic of grant scoring, have you noticed how NIH scores have tended towards bimodal? It’s like reviewers only use two numbers 1 and 5. Almost as if they know that with the tight paylines its a lottery and so in order for their favorite grants to win they really need to push the distributions apart)