I (Geoff) recently came across an article published in 2014 with the title Six Persistent Research Misconceptions. All six are important, but it’s no. 6 that would be most familiar to anyone reading ITNS:
Misconception 6. Significance testing is useful and important for the interpretation of data.
Some quotes from the article:
“Significance testing has led to far more misunderstanding and misinterpretation than clarity in interpreting study results. … The unfortunate consequence of the focus on statistical significance testing has been to foster a dichotomous view of relationships that are better assessed in quantitative terms. …Every day there are important, regrettable and avoidable misinterpretations of data that results from the confusing fog of statistical significance testing. Most of these errors could be avoided if the focus were shifted from statistical testing to estimation.”
That may seem pretty familiar if you are working with ITNS, but it’s a message that needs stating in every discipline that uses NHST. The article is, in fact, from medicine, and the author is Kenneth J. Rothman, who’s a distinguished epidemiologist based in Boston. Ken is also statistical reform royalty, having been writing persuasively about how to do things better since the 1970s. Here’s a brief note on 3 important things he has done:
CIs in medicine
Ken published numerous articles advocating confidence intervals (CIs) and the avoidance of p values and NHST. He and others eventually persuaded ICMJE (the International Committee of Medical Journal Editors), in the mid-1980s, to include in their recommendations for reporting research:
“When possible, quantify findings and present them with appropriate indicators of measurement error or uncertainty (such as confidence intervals). Avoid relying solely on statistical hypothesis testing, such as P values, which fail to convey important information about effect size and precision of estimates.” (That’s the current wording, which you can see under the subhead ‘Statistics’ here. It has scarcely changed for 30 years.)
As a consequence, the great majority of empirical articles in medicine report CIs, even if they still report NHST as well and largely use NHST to guide conclusions.
From NHST to CIs in AJPH
In the mid- and late-1980s, Rothman was an assistant editor at the American Journal of Public Health. He insisted that p values be removed from any manuscript he reviewed, before he would accept it for publication. Many saw this as dramatic over-reach, but it worked. I and colleagues investigated the statistical techniques published in AJPH and found that sole reliance on p values dropped from 63% to 5%, and CI reporting rose from 10% to 54% over the period Rothman was assistant editor. Our article is here.
Epidemiology: Ten years (almost) without p values
In 1990, Rothman founded the journal Epidemiology. He continued as editor for a decade. He announced that his journal would not publish NHST or p values. CIs were expected as the main way to report results. We found that he was very largely successful, with hardly any articles over the 10 years mentioning NHST or p values, and 86% reported CIs.
So far, so good, but here comes the ‘but’: In only about 10% of articles (in AJPH or Epidemiology) that reported CIs, were the lengths of the CIs used as the basis for interpretation and discussion!
- Good science is perfectly possible without p values and NHST–witness the ten years during which Epidemiology flourished.
- Editorial policy and insistence can change publishing practices, but more may be needed to achieve all that is desirable.
- Medicine largely made a big step forward in the 1980s by making CI reporting routine, but even now it usually bases interpretation on p values rather than CIs. There’s still work to be done!
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004). Editors can lead researchers to confidence intervals, but can’t make them think: Statistical reform lessons from medicine. Psychological Science, 15, 119-126.
Rothman KJ. Six persistent research misconceptions. J Gen Intern Med. 2014 Jul; 29(7):1060-4. PMID: 24452418 DOI: 10.1007/s11606-013-2755-z