MoE, the Margin of Error: What the New York Times says

The title of this NYT article is a good summary: “When You Hear the Margin of Error Is Plus or Minus 3 Percent, Think 7 Instead”. The NYT piece is based on this article by famous statistician (imagine that!) Andrew Gelman and colleagues.

But first, a few words about Chapter 1 in ITNS. A fictional poll reports 53% support for a proposition, with margin of error (MoE) of 2%. So the 95% confidence interval is [51, 55]. This CI tells us about the sampling variability, according to our statistical model and N for the poll.

OK, but later in the chapter we discuss how, in real life, there’s even greater uncertainty–extra reasons why the poll result may differ from the true level of support in the population of all likely voters. Perhaps the poll sample doesn’t represent that population well? What about the people who couldn’t be reached by the pollster, or who refused to answer the question? Was the poll question loaded in any way? Unfortunately, the confidence interval can’t tell us about all that.

Gelman and colleagues investigated more than 4,000 U.S. polls for more than 600 presidential, senatorial, and state governor elections since 1998. All the polls were conducted in the final three weeks of the election campaigns. They compared the polls with the election results to estimate total error, which they found to be considerably larger than sampling variability as indicated by the pollster’s stated MoE.

So, yes, in real life polls do typically have uncertainty beyond sampling variability. Furthermore, the Gelman team could estimate that this extra uncertainty is often considerable. Their conclusion was the NYT title: On average, the polls reported MoE of around 3%, but the researchers found total uncertainty was more like a 7% error.

So now we can go back and read Chapter 1 again, with confidence that there’s evidence to back up our discussion, and even to give us a rough idea about the typical amount of extra uncertainty, at least for recent U.S. political polls. Take-home message: There’s often more error than we appreciate. Alas!