Category: Uncategorized

A chilling picture

In ITNS you may notice dot points like:

  • Focus on effect sizes
  • Be mindful of variability
  • Find a revealing picture

Here’s an illustration of the potency of those 3 bullet points.

It’s from a recent article in The Conversation. It plots the world’s total area of sea ice, in millions of square kilometres, for each year since 1978. The total sea ice is, of course, the sum of Artic and Antarctic sea ice. There’s clearly an annual pattern, with variation from year to year. But then a dramatic change since August 2016. The article discusses a range of contributions–why sea ice at both ends of the earth has decreased in recent months. It also discusses how little we really understand about the underlying causes of such enormous changes from past patterns.

Indeed, a chilling picture.

Geoff

Video too slow? Then speed it up!

Video lectures can sometimes drag, maybe even statistics videos? Did you know that YouTube lets you adjust the playback speed? Simply go to ‘Settings’, the little cogwheel lower right, choose ‘speed’, and select from the options. Often, speeding up to 1.25 or 1.5 can be good. Of course, you can pause, rewind, or tweak the speed up or down at any moment. Better still, pause, write down the main messages in your own words, then watch some more.

You might even like to fiddle with the speed of the ITNS videos, which are all on YouTube, as described in this post.

Happy (fast?) watching.

Geoff

This is how things get better…

This is a time of especially intense self-examination in science, a time when the “replication crisis” is rapidly changing our ideas of how to conduct quality research.  As our best practices have changed, we look at prior research in a new and often quite critical light.  Certainly, when established results don’t replicate or are called out for what we now know are problematic research practices, this can feel intensely critical and dispiriting for the original authors.

When criticism comes our way, it never feels great.  But we have to make a decision of where to go from there: if we will simply be defensive and dismissive, or if we will find ways to grapple with criticism and find ways to respond.

On that note, here is short essay that can serve as a model for how to respond to criticism. It comes from Daniel Lakens, who has emerged as real leader in the efforts to improve science, and it tells how he came into this role by grappling seriously with intense criticism of one of his most cited papers.  It’s an incredible story, and it shows the way forward out of the replication crisis: by working hard to do even better.

Check out the post here: http://daniellakens.blogspot.com/2016/09/why-scientific-criticism-sometimes.html

 

Making pre-registration work

Here is link to the COMPARE project, an incredible, though depressing, project designed to check if clinical trials are following their preregistered analysis plans when published.

Pre-registration is a great technique to help draw a bright line between planned and exploratory analyses.  It’s not that there is anything wrong with exploration, but it is essential when presenting results that findings made through exploration are clearly presented this way.  It is an amazing thing to test a drug and find it has exactly the predicted effect on a specific dependent variable.  It is interesting but much less amazing to find that out of forty variables assessed the drug had an interesting effect on two of them.  The really crucial thing, though, is not to have one type of analysis masquerading as another: not to dress up a finding fished out of the sea as exactly what one was expecting.

Pre-registration makes your analysis plan publicly verifiable.  But that means the public must actually pay attention.  The Compare project does just that.  The researchers examined 6 weeks worth of clinical trials published in top medical journals.  For each trial they compared the published article with the preregistered analysis plan.  They flagged places where planned analyses had been silently dropped and where new analyses had been introduced without flagging them as exploratory.  The results were sobering: out of 67 trials checked, only 9 were perfectly reported; all the others added or omitted analyses without flagging these changes.  Overall, over 300 new exploratory analyses were reported without being flagged as exploratory and over 3oo planned analyses were silently dropped from the manuscripts.  That’s terrible!  Even worse, when the researchers pointed out these issues to the journals, many of them had negative reactions that indicated a lack of commitment or understanding about the importance of drawing a bright line between exploratory and planned analyses.  Ugh!

Click the link to see the results in detail.  In particular, check out the blog section which tracks the response to the research including email correspondence with the journal editors.  It is really worth reading.  The take away, it simple: we need to work harder at ensuring preregistration is used and at COMPARIng manuscripts with preregistration documents to ensure researchers are reporting results as they should.

 

Criminal justice and the perils of regression analysis

Regression analysis is incredible–it can literally help us predict the future based on patterns observed in the past. There are many pitfalls, however, to using regression. First, the correlations observed in the past may not apply to new cases or new contexts.  Second, even when the correlation holds, regression does best at predicting what the *typical* or *average* outcome will be–it usually does not do a great job at predicting individual outcomes. You can see this in ESCI: compare the CI for the predicted mean (the range that is plausible for the *typical* outcome) to the prediction interval (PI) for a specific prediction (the range that is plausible for an *individual* outcome). You’ll notice that the PI is much longer. Moreover, as the relationship gets stronger, the CI gets much shorted, but the PI is barely affected–that’s because the PI is influenced not only by the strength of the relationship but also by the natural variation in the Y variable.

In such abstract terms it may be difficult to get all worked up about the nuances of regression analysis.  Here’s a rather remarkable piece of science reporting, however, that examines the use of regression analysis to help make individualized predictions–in this case to provide ‘risk scores’ for those in the criminal justice system. These risk scores are being used to help influence many individual decisions–about bail levels, sentencing levels, and parole.   As the article makes clear, however, the predictions are often wildly inaccurate and potentially racially biased. It’s well worth reading as a case study in the complexities of applying regression analysis in the real world, and as an example of why statistical savvy is increasingly essential for good citizenship.

 

The article is “Risk scores attached to defendants unreliable, racially biased”. It appeared in the Milwaukee-Wisonsin Journal Sentinal on 5/30/206 and was written by Julia Angwin, Jeff Larson, Surya Mattu, Lauren Kirchner (of ProPublica). The full link is here:http://www.jsonline.com/news/crime/risk-scores-attached-to-defendants-unreliable-racially-biased-b99732973z1-381306991.html