Blog

Good research in the service of more effective activism

Research can be vitally important, helping shape how we see the world and the policies we enact.  Here is an example of an applied research lab aiming to use applied research to figure out the most effective ways to lobby for animal rights.  You don’t have to agree with their politics to appreciate what an incisive and thoughtful lab this is.  Click through the blog to read posts on the perils of significance testing, the importance of adequate sample sizes, and excellent research reports that are a model of Open Science practices.

 

http://www.humaneleaguelabs.org/blog/

 

Get involved: registered replication project on “professor priming”

Registered replication reports are a new initiative sponsored by the APS where labs around the world collaborate to conduct a large-scale, precise replication of an important finding in psychology.  The really cool thing about these projects is that they are perfect for undergraduate researchers.  A lead lab develops the protocol and training materials that will ensure each lab replicates the desired protocol precisely.  Interested labs apply to take part and commit to collecting a sample of a specific size within a certain time frame.

Here is a call to participate in an RRP that will examine “professor priming” (we’re not going to explain what that is, click the link to find out).  Hurry, the deadline for applying to participate is September 9.

“Professor Priming” Focus of APS Registered Replication Report Project

Making pre-registration work

Here is link to the COMPARE project, an incredible, though depressing, project designed to check if clinical trials are following their preregistered analysis plans when published.

Pre-registration is a great technique to help draw a bright line between planned and exploratory analyses.  It’s not that there is anything wrong with exploration, but it is essential when presenting results that findings made through exploration are clearly presented this way.  It is an amazing thing to test a drug and find it has exactly the predicted effect on a specific dependent variable.  It is interesting but much less amazing to find that out of forty variables assessed the drug had an interesting effect on two of them.  The really crucial thing, though, is not to have one type of analysis masquerading as another: not to dress up a finding fished out of the sea as exactly what one was expecting.

Pre-registration makes your analysis plan publicly verifiable.  But that means the public must actually pay attention.  The Compare project does just that.  The researchers examined 6 weeks worth of clinical trials published in top medical journals.  For each trial they compared the published article with the preregistered analysis plan.  They flagged places where planned analyses had been silently dropped and where new analyses had been introduced without flagging them as exploratory.  The results were sobering: out of 67 trials checked, only 9 were perfectly reported; all the others added or omitted analyses without flagging these changes.  Overall, over 300 new exploratory analyses were reported without being flagged as exploratory and over 3oo planned analyses were silently dropped from the manuscripts.  That’s terrible!  Even worse, when the researchers pointed out these issues to the journals, many of them had negative reactions that indicated a lack of commitment or understanding about the importance of drawing a bright line between exploratory and planned analyses.  Ugh!

Click the link to see the results in detail.  In particular, check out the blog section which tracks the response to the research including email correspondence with the journal editors.  It is really worth reading.  The take away, it simple: we need to work harder at ensuring preregistration is used and at COMPARIng manuscripts with preregistration documents to ensure researchers are reporting results as they should.

 

Draw your own data tool

Sometimes it is nice to be able to make up a set of data to explore.  Here is a cool tool that makes it easy to craft your own data set: just draw the data and you will instantly see how each new data point affects the mean, standard deviation, and Pearson’s r.  You can even download the data you have created.  A nice tool to explore, it could be especially handy for instructors who need to make quiz and test items. The page was developed by stats guru Robert Grant.

http://robertgrantstats.co.uk/drawmydata.html

 

Criminal justice and the perils of regression analysis

Regression analysis is incredible–it can literally help us predict the future based on patterns observed in the past. There are many pitfalls, however, to using regression. First, the correlations observed in the past may not apply to new cases or new contexts.  Second, even when the correlation holds, regression does best at predicting what the *typical* or *average* outcome will be–it usually does not do a great job at predicting individual outcomes. You can see this in ESCI: compare the CI for the predicted mean (the range that is plausible for the *typical* outcome) to the prediction interval (PI) for a specific prediction (the range that is plausible for an *individual* outcome). You’ll notice that the PI is much longer. Moreover, as the relationship gets stronger, the CI gets much shorted, but the PI is barely affected–that’s because the PI is influenced not only by the strength of the relationship but also by the natural variation in the Y variable.

In such abstract terms it may be difficult to get all worked up about the nuances of regression analysis.  Here’s a rather remarkable piece of science reporting, however, that examines the use of regression analysis to help make individualized predictions–in this case to provide ‘risk scores’ for those in the criminal justice system. These risk scores are being used to help influence many individual decisions–about bail levels, sentencing levels, and parole.   As the article makes clear, however, the predictions are often wildly inaccurate and potentially racially biased. It’s well worth reading as a case study in the complexities of applying regression analysis in the real world, and as an example of why statistical savvy is increasingly essential for good citizenship.

 

The article is “Risk scores attached to defendants unreliable, racially biased”. It appeared in the Milwaukee-Wisonsin Journal Sentinal on 5/30/206 and was written by Julia Angwin, Jeff Larson, Surya Mattu, Lauren Kirchner (of ProPublica). The full link is here:http://www.jsonline.com/news/crime/risk-scores-attached-to-defendants-unreliable-racially-biased-b99732973z1-381306991.html