Statisticians see the light–Hooray!

“Scientific Method for the Twenty-First Century: A World Beyond p<.05." I keep thinking about that tagline, which as we've mentioned a couple of times comes from the recent ASA Symposium on Statistical Inference–see the logo again, below. Bob’s wonderful talk explained that estimation and meta-analysis (aka the new statistics) provide a highly attractive and immediately available way forward, helping to forge the better science of the post-p<.05 research world we are (or should be) entering. Great! I'm thinking also that the tagline tells us something important about how statisticians are thinking about statistical inference. First, in May 2016, the ASA put out their official statement about the p value, and highly critical it was. Then they ran the seminar mentioned above, with the aim of discussing how statistical inference should be done post-p<.05. My own experience over many years is that many statisticians have been rather dismissive, even patronising, about the argument that estimation should replace p values. Some of them say things very close to:

“If only people (meaning psychologists and other applied researchers) had been taught properly (unstated: by real statisticians) and therefore understood NHST and p values properly, you wouldn’t be having all these problems. Besides, p values and confidence intervals are pretty much equivalent, so there is little point in demanding the change.”

On the second point, yes, for sure. In Chapter 6 of ITNS we explain easy ways to translate in your mind’s eye between a p value (plus the point estimate) and a CI. You can indeed translate back and forth. But knowing the CI is way more informative than knowing the p value. It’s a prime example of the way information is presented (CI rather than p) having a massive effect on what message a reader receives, and how the reader thinks about the result. Statistically equivalent–to some extent yes, but far from equivalent in terms of making interpretable, justifiable, and informative inferences. The change is definitely worth making!

So I take that tagline, when chosen as a public statement by the statisticians (well, ASA anyway) to signal a highly important shift in thinking. Statisticians seem to be saying “Yes, there is a problem after all. We all need to move beyond p<.05 and do better inference." To which I say "Yes indeed!" and "Three cheers!" The next question is--did that seminar lead to useful conclusions about what researchers should now do? Will ASA make a follow-up statement about best practice for statistical inference? I'll write more about those questions sometime soon. Geoff