Not replicable, but citable

What happens to the reputation of a paper when the results reported cannot be replicated?

Here’s a graph of citations/year from two studies–an original and a replication study that found little to no effect.  It’s just one example, but it doesn’t seem like the replication study has had much impact on citations to the original article.  There is a brief fall off (which is actually normal as a paper gets older), but there has lately been a rebound (which is rare; most papers don’t have much longevity in terms of citation history).  Most interesting, you can see that the majority of those citing the original research are not citing the replication study, so most aren’t even raising questions about the results.

I suppose this isn’t surprising, given that even retracted papers earn citations, and it is estimated that retraction only cuts citation yields by about half (Grieneisen & Zhang, 2012) (plus a bunch of other references).  It would be interesting and useful to look more systematically at this question as it relates to replications (and to see if a successful replication helps).

The details:  The original study is by Damisch et al.  (Damisch, Stoberock, & Mussweiler, 2010).  It was published in Psych Science and suggested that belief in luck can produce substantial improvements in motor skill (e.g. having a lucky golf ball increases golf performance).  The replication paper is Calin-Jageman & Caldwell (2014) , a study I helped complete (Calin-Jageman & Caldwell, 2014).  We conducted 2 direct and well-powered replications of one of the key studies in the original article (the lucky golf ball study).  The replications indicated little to no effect of superstition on motor skill.  There’s no such thing as Truth with a capital T in science, but I don’t think there’s a lot of ambiguity on this one.  Not only are the replications convincing, but there is good reason to believe that placebo effects have boundary conditions: believing you will do well at golf shouldn’t magically make you substantially better.  .