Adventures in Replication: Your replication appears to be somewhat underpowered

Many journals now proclaim their openness to replication research.  Behind the scenes, though, replication manuscripts are often met with impossible demands and/or insane double-standards.

Here’s an example from an editor at a prominent social psychology journal:

the studies appear to be somewhat underpowered. This is (as reviewer 1 notes) because you estimated sample sizes from a power analysis based on the (very likely overestimated) d from the B & E study.”

Here’s why this makes my head explode:

• The original study had 36 participants (18/group).
• We conducted 5 replications that encompassed almost 700 participants!  (Two studies had only 2x the original sample size, but all the the others had > 2.5x)
• The reviewers (and the editors) suggest that our power analysis should have assumed the original study was very wrong!

It’s been 2 years but I still sometimes wake up in the middle of the night thinking about this.  I sit bolt upright and bellow out incredulous ripostes into the darkness: “So now you care about sample size!!” or “YOUR UNDERSTANDING OF POWER IS SOMEWHAT UNDERPOWERED!”.  My wife, G-d bless her, has become used to this and has not sought to have me committed (that I know of).

My occasional night rage is mellow compared to how I felt when the review first came back.  I probably didn’t handle it perfectly.  I bashed out a very snarky reply and hit send.  Here are the highlights:

It is fortunate that computers were designed to process data without regard for veracity or your email would have broken the internet….I’ve published enough to know that this review is an outlier on the bananas scale

I guess I got what I was looking for, because the editor did send back a very irritated response:

I see no need for any further correspondence about this decision.  If you feel that you must reply, please refrain from inflammatory, ad hominem comments such as the formulation in your email that indirectly questions my veracity, or the characterization of the letter as “bananas.”  Such comments have absolutely no place in professional discourse.

This tells me that people confused about power are also confused about what an ad hominem attack is.

I’m sure things have changed for the better… right?  We did, at least, find a home for our somewhat underpowered replication results at PLOS ONE (Cusack, Vezenkova, Gottschalk, & Calin-Jageman, 2015).