Twelve years ago, I created this blog to keep a public record of predictors' track records as polling was entering its Golden Age. Nate Silver and other poll aggregators predicted the 2012 election brilliantly, and I kept a record of it. Now I return with less confidence in polls from 2024 than from any previous year in my life.
Polling as we once knew it, with genuinely representative samples from calling random people, no longer exists because they don’t pick up the phone. What we now call polls are essentially models tested on a little bit of data. This approach resulted in unprecedented errors in the 2024 Republican primary, which should lower our confidence that poll averages will predict the upcoming election.
Steven Shepard in Politico summarizes the most severe polling failures: “In Michigan, Trump’s margin over Haley was 15.3 points smaller in the actual results than in the final polling average. Pollsters also had double-digit whiffs in Massachusetts (a 14.3-point difference), Tennessee (11.3 points) and Virginia (20.8 points).” These races weren’t heavily polled around the voting, which may explain the errors.
There is no similar excuse in New Hampshire, which was heavily polled right before its election. The poll aggregators’ average had Trump winning by 19.2%, and he actually won by 11.1%. South Carolina was also heavily polled, and the aggregators had Trump winning by 26% where he actually won by 20.3%. (For NH and SC, the aggregator average is 538, 270towin, and RCP averaged together.)
People didn’t complain because the polls said Trump would win and he won. But missing the margin by 6-8% in a well-polled race is severe. Moreover, all these errors were overestimates of Trump, and underestimates of Haley.
The polls underestimated Trump in 2016 and 2020, and got things about right in 2022. Why would they suddenly overestimate Trump in the 2024 primaries? And what does this mean for the general election?
The fundamental problem is that response rates to phone polls have been declining for years. This means that phone polling mainly reaches the ever-shrinking sliver of the US population that picks up when called by an unknown number. If that sliver of the population is representative of the whole, polling will still work. But if it becomes unrepresentative, polling will present all opinion as being like the unrepresentative and skewed sample. If 1% of the US population consists of chaotic people who are always willing to pick up the phone and say extreme things to a pollster that they won't follow up on – maybe they’re playing pranks or high on intense drugs or having a mental health problem – a poll with a response rate of 5% will be one-fifth chaos.
There are other ways of polling than the phone, but none get a representative sample. For example, ActiVote polls people who downloaded its app, but self-selection problems there are severe. Pollsters now must put a lot of weight on how they antecedently model the composition of the 2024 electorate rather than what they learn about it from the data they acquire. Such models can be tweaked to get whatever result the pollster likes.
In 2020, phone polls were still doing okay because Trump and Haley voters were still on the same side. Trump voters are more likely to pick up the phone, while Haley voters are less likely. But their different tendencies cancelled out, and both voted for Republicans. Then January 6 happened, and divided the Republican Party into factions that couldn’t be reconciled. In 2022 each faction could be optimistic about winning control of the party in 2024, so they still mostly voted together.
The exception was when seriously Trumpy candidates turned off the sorts of people who became Haley voters. This happened in Pennsylvania, were Trump favorites ran for both Senator and Governor. The Senate race had the infamous Dr. Oz against John Fetterman, while the Governor’s race had Doug Mastriano against Josh Shapiro. Polls estimated that Shapiro would win by 11.4%; he actually won by 14.8%. Polls estimated a 0.4% victory for Oz; Fetterman won by 4.9% despite being incapacitated by a stroke.
All of this makes sense if Haley voters are less likely to respond to pollsters. What would that mean for 2024?
What data we have about Haley voters suggests that they’re especially likely to support Harris. This makes demographic and ideological sense. This voter bloc swung the 2022 PA Senate race, and kept Trump’s margins down in this year’s primaries. It’s because of silent Haley voters adding something like 1-2% to Harris’ margins in crucial swing states that I think she is more likely than not to win the Presidential election.
No comments:
Post a Comment