Applied research | Introduction to the New Statistics

Make it stick–the Cliff’s Notes version

Learning statistics can be a challenge. Using effective study habits can make that challenge more manageable. Unfortunately, many students ignore or are unaware of the best study habits and instead adopt approaches that are less than optimal.

In the new textbook, Geoff and I try to help point students in the right direction, offering a brief summary of the invaluable Make it Stick–a superbly well-researched examination of the best study habits by Brown, Roediger, & McDaniel (2014).

Today, APS has published a shorter, peer-reviewed version of Make it Stick in the Journal Perspectives in Psychological Science (Putnam, Sungkhasettee & Roediger, 2016). The article was commissioned by editor Robert J. Sternberg “so that instructors would be able to give students who wanted to improve something that is short, easy to read, and to the point.”

Click here to get this gem of an article. [[An earlier version of this post claimed that the article was not open-access. But I was wrong! The article is open-access. WINNING!]]

Mission mostly accomplished–the new article indeed distills the essential aspects of Make it Stick and provides a great handbook for students looking to get the most out of their studies. The one problem? The article is pay walled. SAD.

If you’re library has a subscription to Perspectives in Psychological Science, then click here to get this gem of an article. Otherwise, you could get a used copy of Make it Stick for 1/2 the price APS would charge you to buy this short article (used copies of Make it Stick are currently selling for about $16 USD on Amazon; APS charges $35 to purchase a pdf of a single article!) . Perhaps if enough people emailed APS to let them know this is an article that should be open access…

MoE, the Margin of Error: What the New York Times says

The title of this NYT article is a good summary: “When You Hear the Margin of Error Is Plus or Minus 3 Percent, Think 7 Instead”. The NYT piece is based on this article by famous statistician (imagine that!) Andrew Gelman and colleagues.

But first, a few words about Chapter 1 in ITNS. A fictional poll reports 53% support for a proposition, with margin of error (MoE) of 2%. So the 95% confidence interval is [51, 55]. This CI tells us about the sampling variability, according to our statistical model and N for the poll.

OK, but later in the chapter we discuss how, in real life, there’s even greater uncertainty–extra reasons why the poll result may differ from the true level of support in the population of all likely voters. Perhaps the poll sample doesn’t represent that population well? What about the people who couldn’t be reached by the pollster, or who refused to answer the question? Was the poll question loaded in any way? Unfortunately, the confidence interval can’t tell us about all that.

Gelman and colleagues investigated more than 4,000 U.S. polls for more than 600 presidential, senatorial, and state governor elections since 1998. All the polls were conducted in the final three weeks of the election campaigns. They compared the polls with the election results to estimate total error, which they found to be considerably larger than sampling variability as indicated by the pollster’s stated MoE.

So, yes, in real life polls do typically have uncertainty beyond sampling variability. Furthermore, the Gelman team could estimate that this extra uncertainty is often considerable. Their conclusion was the NYT title: On average, the polls reported MoE of around 3%, but the researchers found total uncertainty was more like a 7% error.

So now we can go back and read Chapter 1 again, with confidence that there’s evidence to back up our discussion, and even to give us a rough idea about the typical amount of extra uncertainty, at least for recent U.S. political polls. Take-home message: There’s often more error than we appreciate. Alas!

Good research in the service of more effective activism

Research can be vitally important, helping shape how we see the world and the policies we enact. Here is an example of an applied research lab aiming to use applied research to figure out the most effective ways to lobby for animal rights. You don’t have to agree with their politics to appreciate what an incisive and thoughtful lab this is. Click through the blog to read posts on the perils of significance testing, the importance of adequate sample sizes, and excellent research reports that are a model of Open Science practices.

http://www.humaneleaguelabs.org/blog/