Replication: ‘Psychological Science’ does the right thing

I have been enjoying Bob’s series of posts about replication. (Go to our home page and scroll down to see links and a few lines of text about each of the 5 posts, with title starting ‘Adventures in Replication’.) Actually, “enjoying” isn’t quite the right word. “Appreciating”, perhaps, and “sharing the frustration”?

Bob tells of the replicator’s frustrations when a journal just doesn’t want to know, or a referee or editor just doesn’t seem to get it. Science relies on replication to decide what to really trust?? Not always in psychology it seems.

Anyway, now for some good news. Psychological Science, which The Association for Psychological Science describes as its ‘flagship’ journal, recently announced a new category of article that it would publish: Preregistered Direct Replications (PDRs).

Editor-in-chief, Steve Lindsay, writes that “PDR articles report high-quality, preregistered, direct replications of studies published in Psychological Science.”

He also writes that “One of the motivations for adding PDRs to Psychological Science is the belief that a journal is responsible for the works it publishes (as per Sanjay Srivastava’s,
2012, “Pottery Barn rule” blog post). (The pottery barn rule says that ‘You break it, then you own it and you pay for it’. We don’t have pottery barns in Australia, not by that name anyway, but I can guess.)

Preregistration is required for PDRs, and submission for review when the study is at the proposal stage is strongly encouraged, although not (yet) required. Of course, review at that early stage can lead to improvements to the proposal, and acceptance of the proposal gives assurance of publication, subject only to careful implementation of the preregistered plan, and full disclosure.

The advent of PDRs is just the latest step taken by Psychological Science, which has been a pioneer in the encouragement of Open Science. Recall the introduction by previous editor-in-chief, Eric Eich, of new disclosure requirements and OS badges, and the encouragement of use of the new statistics rather than NHST–supported by publication of my tutorial article.

So, consider encouraging your groups of students seeking worthwhile research projects to scour Psychological Science looking for interesting findings that deserve replication. Or consider doing a PDR yourself. You may find something surprising, but even if you don’t you can make a valuable contribution to science.

And here’s a challenge: Can you sniff out a published effect that is reported as statistically significant, or even highly statistically significant, but for which a larger replication finds evidence that the effect is tiny or maybe zero? Bob may be the reigning world champion at doing that: He and his students are, I believe, currently running at about 9 out their 10 replication projects yielding zero or tiny estimates of effects.

Geoff

P.S. The editorial announcing PDRs was released online on 9 August 2017. Steve tells me that NO submissions (either proposals or manuscripts) for PDRs have been received since then–as at 23 October 2017. You could rush to be the first to submit?