Open Science: A Detailed Telling of the Story So Far

Concern about publication bias apparently goes back at least as far at Robert Boyle in the 17^th century. That’s Barker Bausell’s starting point for his highly detailed account of the replication crisis and rise of Open Science. Bausell, the author of a number of books, brings a broad perspective, across many disciplines beyond only psychology, to the story he tells so engagingly. Frequent mentions of health and medical research reflect one aspect of his expertise.

The book is The Problem With Science: The Reproducibility Crisis and What to Do About It. You can read the first couple of dozen pages here, or go here and click at left for a brief summary of each chapter.

Chapter 2 is appropriately critical in discussing NHST and p values, and false positives. Recommendations focus on using larger and higher-power studies, ESs, and α values much smaller than .05. I, of course, wish for recommendations against using p values at all, and in favour of estimation—for single studies and as the basis for meta-analysis. That’s surely at the core of Open Science practice.

Chapter 3 discusses a monumental list of 23 (!) QRPs—and some of those actually comprise more than one practice.

All through there’s much that will be familiar to many readers of this blog. We come across HARKing, forking paths, and Bem, Bargh, and Cuddy. But we also hear about cold fusion, and other goodies (?) from chemistry, physics, and more. I love the whole-of-science perspective. Of course, there’s lots of discussion of replication. Also of publishing practices, and the terrible contingencies that researchers have traditionally faced.

For any particular problem there’s discussion of the full sequence: the problem’s origins, investigations of how serious and widespread it is, what’s needed to solve it, then finally how we’re going so far with implementing the necessary changes. This sequence reminds me of Replicability, Robustness, and Reproducibility in Psychological Science, by Brian Nosek et al., which is a detailed discussion of exactly what the title states and developments over the last decade or so. This is in press for The Annual Review of Psychology and the final version is available here as a preprint. By comparison, Bausell ranges over many disciplines.

Education

Bausell’s brief final chapter looks forward. I like his emphasis on the incorporation of Open Science practices into education—I would say that, wouldn’t I, with ITNS2 designed to be a resource for just that. He takes an optimistic position, judging that, given sufficient support by researchers, positive developments in research practices are likely to flourish, even if success is not (yet) assured. Good to see; I dearly hope he’s correct.

While discussing the challenge of overcoming ingrained beliefs that ‘statistically significant’ is equivalent to ‘notable’, Bausell writes “the best educational option is probably to keep the sanctity of obtaining p-values at all costs from being learned in the first place” (p. 264). I couldn’t agree more! Again, I’d go further and advocate starting with estimation, then mentioning p values, if at all, only later.

What’s missing?

While Bausell generally gives us an upbeat and forward-looking discussion, I confess that I miss the sense of excitement that I’ve sometimes felt, for example at a couple of APS Conventions, roughly around 2014-2015. At those, some talks and symposia sparked animated discussions and a real sense that right here we are trying to figure out how science could be done better.

I also have in mind the youth of many advocates of better practices. I’ve had young researchers approach me after talks, sometimes in tears. They know what’s needed, but how, they ask, should they cope with a professor who says “reanalyse—your job is to find statistical significance in your data”? Despite such pressures, the Society for the Improvement of Psychological Science (SIPS) has always been energised by young folks. One-third of its board positions are reserved for graduate students and postdocs. I continue to be inspired and excited by young researchers who are insisting that science is done better, and who are active in helping figure out just how.

My conclusion is that Bausell gives us a valuable, wide-ranging and well-informed history, progress report, and agenda for the Open Science project.

Enjoy,

Geoff

Bausell, R. B. (2021). The Problem With Science: The Reproducibility Crisis and What to Do About It. OUP. ISBN-13: 9780197536537 https://doi.org/10.1093/oso/9780197536537.001.0001