Adventures in Replication: Introduction

Over the past 5 years or so, I (Bob) have been a bit replication crazy–I’ve conducted about 10 direct replication projects in collaboration with undergraduate students at Dominican.  I became obsessed in part because I wanted to know for myself if the alarm bells being raised during the ‘replication crisis’ were really valid–could it really be that many of the important scientific findings celebrated in the press are unreliable?   Unfortunately, my conclusion from personal experience is yes.  The vast majority of the replication projects I’ve been involved in have been unable to reproduce the very strong patterns of results reported in the original manuscripts.  My students and I have found that power doesn’t really improve motor skill (Cusack, Vezenkova, Gottschalk, & Calin-Jageman, 2015) , superstition doesn’t really improve golf skills (Calin-Jageman & Caldwell, 2014) , being exposed to organic food doesn’t really make you a jerk (Moery & Calin-Jageman, 2016) , holding your face in the form of a smile doesn’t really make you think cartoons are funnier  (Wagenmakers et al., 2016) , engaging in analytic thinking doesn’t really decrease your religious belief (Sanchez, Sundermeier, Gray, & Calin-Jageman, 2017) , and seeing red doesn’t make a romantic partner massively more attractive (Lehmann & Calin-Jageman, 2017).  In each case it’s not just that we found results that are ‘not significant’–it’s that we found effect sizes close to 0 with CIs that generally exclude all but the very weakest and most scientifically intractable of effects.  It’s not just that we failed to find strong effects once–it’s that in each case we did a series of studies with varying conditions/populations but found consistently next-to-no effect. And it’s not a matter of competence, because early on we developed the approach of adding in positive controls, and we’ve had no problem obtaining reliable effects of well-known psychological phenomena with the same participants and conditions.  So, the balance of the evidence I’ve published so far suggests that the replication crisis is very, very real.  And that’s just what we’ve managed to publish at this point–there are still 3 more papers on the way, at least.

I can feel my replication fever now starting to break.  I’ve learned a lot not just about replication but about science in general.  I’ve picked up new methods and view points that I’m finding useful in my own research.  And I’ve convinced myself of the sad and depressing truth that much of what has recently passed as the very best of empirical research is almost complete nonsense.  Blech.  I’m not sure if I have much more to learn from additional replications–though I have found them wonderful vehicles for student growth and development, so I imagine I’ll continue to supervise replication projects for the foreseeable future.

As the fever fades, I want to collect my thoughts and reflect on the process of replication.  So I’m starting a series of blog posts on what I’ve learned from being a replicator.  This is the inaugural post–just kind of a place holder to say: Stay tuned!

References

Calin-Jageman, R. J., & Caldwell, T. L. (2014). Replication of the Superstition and Performance Study by Damisch, Stoberock, and Mussweiler (2010). Social Psychology, 45(3), 239–245. https://doi.org/10.1027/1864-9335/a000190
Cusack, M., Vezenkova, N., Gottschalk, C., & Calin-Jageman, R. J. (2015). Direct and Conceptual Replications of Burgmer & Englich (2012): Power May Have Little to No Effect on Motor Performance. PLOS ONE, 10(11), e0140806. https://doi.org/10.1371/journal.pone.0140806
Lehmann, G. K., & Calin-Jageman, R. J. (2017). Is Red Really Romantic? Social Psychology, 48(3), 174–183. https://doi.org/10.1027/1864-9335/a000296
Moery, E., & Calin-Jageman, R. J. (2016). Direct and Conceptual Replications of Eskine (2013). Social Psychological and Personality Science, 7(4), 312–319. https://doi.org/10.1177/1948550616639649
Sanchez, C., Sundermeier, B., Gray, K., & Calin-Jageman, R. J. (2017). Direct replication of Gervais & Norenzayan (2012): No evidence that analytic thinking decreases religious belief. PLOS ONE, 12(2), e0172636. https://doi.org/10.1371/journal.pone.0172636
Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., … Zwaan, R. A. (2016). Registered Replication Report. Perspectives on Psychological Science, 11(6), 917–928. https://doi.org/10.1177/1745691616674458
About

I'm a teacher, researcher, and gadfly of neuroscience. My research interests are in the neural basis of learning and memory, the history of neuroscience, computational neuroscience, bibliometrics, and the philosophy of science. I teach courses in neuroscience, statistics, research methods, learning and memory, and happiness. In my spare time I'm usually tinkering with computers, writing programs, or playing ice hockey.

Posted in Replication

Leave a Reply

Your email address will not be published. Required fields are marked *

*