Another disappointing replication result, but with as happy an ending as can be…

A few years back, a paper in Science caught the eye of one of my students, Clinton Sanchez.  Clinton brought me the paper in a state of excitement–it showed 4 different experiments in which very subtle nudges meant to foster analytic/logical thinking produced large decreases in religious belief.  Clinton, who had completed one of his statistics term projects on the predictors of religious belief, immediately wanted to explore this issue.  And so we set off on a replication adventure….

What a journey it was.  We contacted the lead author on the original paper, Will Gervais, to obtain original materials and feedback on the replication attempt (he was extremely gracious and helpful).  We settled on replicating one specific study in which images of The Thinker were used to prime analytic thinking, resulting in a large decrease in self-reported belief in G-d.  We recruited 2 other institutions to complete the replication on different campuses.  We developed the whole study in Qualtrics, added a positive control to help check our work, obtained IRB approval, pre-registered the design, and then collected data from over 900 participants (some at Dominican, some via Mechanical Turk, some at our 2 partner institutions: Concordia University Chicago and the College of DuPage).

Our results: little to no effect of the manipulation on religious belief.  Crap.  (The paper is here; all the data and materials are on the OSF here).

Then a second stage of the journey began, towards publication.  We submitted the paper as a 300-word comment to Science.  It was rejected without review (it was “not given a high priority rating during the initial screening process”).  We tried Psych Science (though this didn’t feel like a good fit) and Social Psychology (this seemed like exactly the right journal).  Both returned the paper without review.  So we turned to PLOS One, which was much more receptive.  This still took a good long while, though, as PLOS One rightly has the original authors conduct a secondary review of accepted replication manuscripts.  The original authors were kind enough to take the time to do this and never held us up.

So, at long-last, the paper is now out: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172636

In the time it took from the beginning of the project to publication, the world really shifted around us.  First, Greg Francis and colleagues published a meta-analysis of psychology papers published in Science which showed that most of the papers there, and this one in particular, have improbably wonderful results (2014).  Then a paper by Deppe and colleagues (2015) came out showing that the pilot data in the original paper validating the manipulation was way too good to be true: in fact, exposure to The Thinker does nothing to change analytic thinking.  In fact, all the methods in the original paper have now fallen apart.  For example, one of the studies had used the verbal disfluency effect (a tough-to-read font), but the notion that this influences analytic thinking has crashed and burned (Meyer et al., 2015).

Ok – so this is pretty depressing.  This was an effect we were excited to explore, deconstruct, and learn more about.  It was published in a top journal with multiple conceptual replications and seemingly rock solid validation of the experimental manipulations.  And yet, with hindsight, it is pretty clear that things simply could not have been as Gervais & Norenzayan found them.  It takes alot to influence analytic thinking–subtle manipulations like looking at the the Thinker should not have produced the enormous changes reported in the supplemental materials.  Similarly, many of the predictors of religious belief are already well characterized–there should not have been the space for a subtle change in analytic thinking to produce such large changes in religious belief.  As scientists, we need to listen to the data…but we need to plug our ears when it whispers “jump off a cliff”.

What’s the happy ending to this?  Well, it is in two parts.  First, the lead author of the original article, Will Gervais, posted a really thoughtful and searching blog post responding to the failed replication.  You can read it here; it is well-worth it (fair warning: some profanity at the end; I didn’t mind).  It is an exemplary response that shows how much the field has grown and changed.  Will uses the phrase “methodological awakening” which is perfect.  In fact, I propose we stop using the phrase “replication crisis”.  Lack of reproducibility is symptom of bad methods; the main thing going on here is an sea change in how we use and apply statistics towards scientific inference.

Another part of the happy ending is that the new replication was picked up in the press.  Science writer Dalmeet Singh wrote up a really thorough and accurate piece for NY Magazine (you can read it here).  Dalmeet explained that he sees covering negative results to be part of his mission in science journalism… Amen!

References:

Deppe, K. D., Gonzalez, F. J., Neiman, J. L., Jacobs, C., Pahlke, J., & Smith, K. B. (2015). Reflective liberals and intuitive conservatives : A look at the Cognitive Reflection Test and ideology. Judgment and Decision Making, 10(4), 314–331.

Francis, G., Tanzman, J., & Matthews, W. J. (2014). Excess success for psychology articles in the journal science. PloS One, 9(12), e114255. https://doi.org/10.1371/journal.pone.0114255

Meyer, A., Frederick, S., Burnham, T. C., Guevara Pinto, J. D., Boyer, T. W., Ball, L. J., … Schuldt, J. P. (2015). Disfluent fonts don’t help people solve math problems. Journal of Experimental Psychology: General, 144(2), e16–e30. https://doi.org/10.1037/xge0000049

Sanchez, C., Sundermeier, B., Gray, K., & Calin-Jageman, R. J. (2017). Direct replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious belief. PLOS ONE, 12(2), e0172636. https://doi.org/10.1371/journal.pone.0172636

About

I'm a teacher, researcher, and gadfly of neuroscience. My research interests are in the neural basis of learning and memory, the history of neuroscience, computational neuroscience, bibliometrics, and the philosophy of science. I teach courses in neuroscience, statistics, research methods, learning and memory, and happiness. In my spare time I'm usually tinkering with computers, writing programs, or playing ice hockey.

Leave a Reply

Your email address will not be published. Required fields are marked *

*