The National Association of Scholars Weighs in on ‘The Irreproducibility Crisis’

STOP PRESS: Since first writing this post I’ve discovered that all may not be as it seems–especially to me at the other end of the Earth. As ever, we need to be vigilant for any dark forces wishing to use the replication crisis as an excuse to discredit science. Science has, overwhelmingly, been highly successful and effective, even if the knowledge it has provided has not always been used for the benefit of humanity. Open Science is the positive and creative response of scientists to the discovery that some findings in some fields can’t be replicated. Science can only become even more successful and effective as OS practices become more widespread. Developing OS practices further and, above all, working to see them adopted more widely–these should be the aims of everyone, and all organisations, concerned to enhance scientific research and its use to advance human well-being. To the extent that the NAS report does this, and is used to do this, it deserves our support.

My original post, slightly revised:
I’ve just received notice of the launch of a report titled The Irreproducibility Crisis of Modern Science: Causes, Consequences, and the Road to Reform. The public launch will be in Washington, D.C., on April 17.

The report is by the (U.S.) National Association of Scholars, which is described in Wikipedia as “an American non-profit politically conservative advocacy group, with a particular interest in education”.

I was sent a two-page summary of the report; this summary is here. [Sorry, I’ve now (6 April) taken down the summary, at the request of the NAS–they wish it to remain embargoed until the launch on 17 April. I’ll re-post it, and/or the whole report, after the launch.]

Judging by that brief summary, the full report, to be released on April 17, *seems* like a reasonable report that recognizes some of the problems that Open Science seeks to address. However, there is a lot to be concerned about as well: the report will apparently be launched at an event chaired by a noted climate-change denier… so there is some strong concern here that the science reforms being advocated may be cover for an overall anti-science program.

It’s hard to tell….but here are brief paraphrases of a few of the report’s 40 recommendations:

● Statistical Standards. Researchers should define statistical significance as p < .01 rather than as p < .05.

● Research Practices. Researchers should pre-register their research protocols.

● Schools. High school and colleges should teach basic statistics, including discussion of uncertainty.

In my view, the first is weak and largely irrelevant, while the other two are fine. Further recommendations emphasise the importance of requiring evidence from reproducible research to guide U.S. government policies and research practice.

Geoff