Science Spin – Sackler Colloquim on Reproducibility Field Report 3

The conference on reproducibility I (Bob) attended in early March was so invigorating I figured I would spread these posts out.  Here’s the next installment.

Another good talk on the first day was from Isabelle Boutron, an MD PhD at Oxford who has extensively researched scientific spin: making claims about research results which go beyond what is clearly indicated in the data.  According to Boutron: “Publication bias is showing only the tip of the iceburg.  Spin is trying to make the iceburg look beautiful.”

Boutron reviewed evidence that over half of all applied medical papers have at least some “spin”–some attempt to claim stronger findings than clearly demonstrated in the data.  Strategies documented include:

• Interpretation of non-significance as indicating comparability
• Mining for significance in subgroups, focus on secondary outcomes, etc.
• Approached significance phrasing.
• Overly forceful language relative to the finding “demonstrated safety” for merely non-siginficant difference in safety.

Does spin matter?  Yes–clinicians who examined an abstract with spin found the results more promising than those who examined the abstract without spin (but with same data).  Moreover, spin from the scientist is propagated and amplified if the study makes its way into the popular press.  Although there is a strong contribution from the press in distorting research results, more distortions seem to occur when the paper and/or original press release already has some spin.

One interesting tidbit–does press coverage matter to a researcher’s career?  Probably.  Boutron mentioned a study by Phillips (1991, NEJM, I think) which looked at citations to articles that were either a) covered in the NYT, b) not covered but otherwise similar, or c) selected to be covered but not due to a strike at the NYT (what a cool control!).  The study concluded that being written up in the NYT boosts citations by 73%.  Fascinating!

Overall, Boutron provided a convincing case that scientists are overhyping their research in a way that is likely to have negative consequences.  Although I expected no positive solutions to this, Boutron highlighted one possible way forward: HealthNewsReview.org, a website which de-spins health-science news.  Here’s a post just about this cool resource: https://thenewstatistics.com/itns/2017/03/20/the-unspin-cycle-for-science-news/