At the kind invitation of Neil Bailey I recently gave a talk at Deakin Unversity. Here’s the video:
You can download here the above video, my PowerPoint slides, and a hand-out summarising the slides.
My talk was a brief take on the 20+ years it has taken me to discover that statistical significance testing is a disaster, that it’s extremely hard to get researchers to change, and that now there are now much better alternatives.
My Argument
- For approaching a century numerous distinguished scholars, including philosophers of science, statisticians, and psychologists, have published cogent critiques of p values, significance testing, and how researchers across science use these.
- Even so, a large proportion of researchers, teachers, journals, and granting bodies use null hypothesis significance testing (NHST)–often based on p < .05 or p < .01–as the standard for concluding whether or not an effect exists, whether or not a result is large or important. Despite this logic being wrong-headed in so many ways!
- Devotion to NHST and p <.05 resembles an addiction–the researcher’s heroin. Rational argument is not sufficient to shake the addiction. Could a dramatic demonstration, perhaps persuading via the gut rather than the brain, shake this addiction?
- A striking but little-known feature of p values is that they are highly unreliable–their sampling variability is astonishingly large. Replicate a study, exactly the same but with a new random sample, and expect to obtain replication p that can take just about any value between 0 and 1!
- Jerry Lai in his lovely PhD studies took three converging approaches to find that a large proportion of published researchers in psychology, medicine, and statistics severely under-estimate the amount of variability in the p value with replication.
- My first demonstration of p value variability was the dance of the p values, the first video of which dates from 2009. Search YouTube for ‘dance of the p values’ to find several videos. You can also play with the dances in esci web.
- My second approach arose from study of the probability distribution of replication p, the p value obtained in a replication. My highly-cited 2008 article has details.
- I define the p interval as the 80% prediction interval for replication p. It’s astonishingly long! For example, if an initial study obtains p = .05, the p interval is (.0002, .65), meaning an 80% chance of p within that interval and fully a 10% chance it falls below .0002 and 10% above .65. After p = .01 the interval is (.00001, .41). After initial p = .001 (*** highly statistically significant) there is fully a 1 in 6 chance a replication does not even achieve p < .05! After p = .20 ns there is a 1 in 3 chance a replication finds p < .05!
- I take the probability distribution of replication p following initial p = .01 and divide the area under the curve, which represents probability, into 38 equal areas. I label each area with the p value in the centre of the interval. I have 38 p values, a few large and many small and very small, which accurately represent that probability distribution.
- I scatter those 38 p values randomly around the 38 bins of a roulette wheel. Simply click to spin, wait for a few moments, and see the replication p you might have obtained from a replication. Much faster and cheaper than the hassle of raising a grant, recruiting participants, hassling with ethics approval… and collecting data!
- At Significance Roulette in esci web you can click between initial p of .05 and .01. With initial p = .01, replication p values tend a little smaller, of course. But the striking thing is how widely spread the p values are! The distributions are pictured to left of the wheel. For example, switch from .05 to .01 and note a slightly smaller number of deep blue, deep trombone sound, despairing figures for p > .1 ns and slightly more bright red, triumphant trumpet blast, elated figures for p < .001 ***. Click SPIN, and note your quickened heart beat, sweaty palms, and that you are holding your breath–will you be despairing or elated–and in only a few seconds you’ll know!
You can visit the p-value casino here. A picture is below.
Enjoy, and weep!
Geoff
I have a question about interpreting p values which I haven’t seen addressed in the literature.
If the p value is small enough the null hypothesis is rejected. But, since we’re dealing with a conditional probability — p(Data|H0) — isn’t it equally acceptable to reject the data?
Having numbers in a table isn’t sufficient to not reject the data. The sample data must also (among other things) be representative of the population under consideration. Representativeness can easily be compromised. SPSS, for example, identifies ‘outliers’ in the sample which encourages users to remove them from analysis. But although these values are outliers in the sample they may not be outliers in the population sampled. The ‘trimmed’ sample then isn’t representative of the population, leading to invalid inferences. Isn’t rejecting the data then a valid logical inference from the small p value?
Thanks Geoff,
Makes sense to reject the data for small p, but what would that mean?
Leaving a Bayesian approach aside and taking a correct High Church, capital-F, Frequentist stance we can say only: “Assuming our statistical model of independent sampling and a normally distributed population, we can reject the null at our chosen alpha.” Or, if we reject the data: “Assuming….., our data allows us to…”
IRL we want to draw conclusions beyond that model. That’s when we become informal Bayesians and talk about the chance that our effect is non-zero, or matters to us in some way. That’s how virtually all researchers, and statisticians also once they get off their high horse, think and talk.
Of course I advocate thinking instead in terms of estimation and confidence intervals. Formally, assumptions as above, we can say “our CI is one from a notional infinite series of such intervals, alpha % of which capture the unknown population mean”. As useful as… choose your rude metaphor.
I’ve argued in lots of places that it’s reasonable in most realistic situations to think in terms of strength of evidence and, in more detail, that points near the centre of the CI are more likely bets for the true size of effect, points progressively out to the ends of the CI, and beyond, are successively less good bets.
There’s lots on all this stuff in Chapters 2-5 of UTNS, my first book, 2012.
Your second, longer, para: Yes indeed, tho I’m not sure that rejecting the data helps. The data gives us all the info we have about the world (assuming no Bayesian priors); our job is to glean what we can from that data set, preferably first in a pre-registered way, them maybe an exploratory way.
May all your confidence intervals be short 🙂
Geoff
The logic of NHST–rejecting H0 when the p value is sufficiently small is inexorable, even though it is not required logically; one might as well reject the data.
Maybe–just maybe–if this were widely understood and appreciated the tendency to ‘refine’ the data (often by eliminating apparent outliers) to get a ‘significant’ result might be less common. My main issue, I think, is that the concentration on the usual NHST protocol doesn’t encourage consideration of the data.
Thanks again Geoff,
I agree entirely! (How boring… :-()
Geoff