Significance Roulette 2

In my post of a couple of days ago I gave the link to Significance Roulette 1, a video that explains how to generate the roulette wheel for a ‘typical experiment’, by which I meant an independent groups experiment, N = 32 in each group, with half a standard deviation difference between the population means. In other words, the population effect size is assumed to be 0.5, traditionally considered a medium-sized effect.

However, assuming a known population effect size is unrealistic: If we knew that, we’d hardly have reason to run the experiment! Can we do better? The second video, Significance Roulette 2 explains how we can. Give me just the p value from an initial experiment, and I will generate the roulette wheel that represents the probability distribution of replication p, meaning the p value we would get if we made a single replication of the original experiment–just the same but with new samples.

The remarkable thing is that we don’t need to know the sample sizes, the power, or the population effect size for the initial experiment. All we need is the p value from the initial experiment, and assurance that the replication is just the same as the original, but with new samples.

The shortcut to the Significance Roulette 2 video is: http://tiny.cc/SigRoulette2

The article in which I explain the calculations is here.

Once again I conclude that we should never trust a p value, we should not use p values at all, and that there are much better ways.

Enjoy…

Geoff