Wednesday, November 6, 2024
The failure of the One True Pollster
Tuesday, November 5, 2024
One True Pollster: Harris by 5%, 309-229
Sunday, November 3, 2024
Ann Selzer, the One True Pollster
I’ve made a general case for poll skepticism, and I should explain why this poll has some chance of being unusually reliable. Selzer has been overseeing polling at the Des Moines Register since 1987, focusing on how to get an accurate read of Iowa. She applies decades of experience and well-tested methods to a simpler task than other pollsters have, working only with a small and relatively homogeneous state.
Selzer’s track record is unparalleled, including Iowa caucuses and general elections. Four years ago, her final pre-election poll shocked complacent Democrats by showing Trump up 7. Several other pollsters were finding a Biden lead, and of the last 37 polls before the election, no one else found such a large Trump lead. Trump actually won the state by 8.
She did pretty well in the 2024 Iowa caucuses, predicting a 48-16-20-8 Trump-DeSantis-Haley-Vivek popular breakdown when the outcome was 51-21-19-8. Perhaps her polls are especially sensitive to Haley voters. In the primary she underestimated Trump’s margin over Haley by 4, while the NH average overestimated by 6, the SC average by 8, and other state averages missed by double digits. (Her last Iowa poll beats ALL the other state AVERAGES!) This supports my view that other polls are missing Haley voters.
A herd of polls disagrees with her. The herd does not predict an epic Harris victory, perhaps fearing the costs of being wrong. Rather than follow the herd, I will follow her. She predicts the election using evidence rather than fear. To me, she is the one true pollster.
Her modest Trump underestimation may suggest treating her results as underestimating him again. So perhaps we should accordingly trim 4 points off the +3 margin so Trump wins Iowa by 1. But if his margin is anywhere near that small in Iowa, he’s doing terribly in similar regions. Harris easily holds PA-MI-WI-Omaha for 270+ electoral votes and victory.
Saturday, November 2, 2024
As polls crumble into models, are Haley voters slipping through the cracks?
Twelve years ago, I created this blog to keep a public record of predictors' track records as polling was entering its Golden Age. Nate Silver and other poll aggregators predicted the 2012 election brilliantly, and I kept a record of it. Now I return with less confidence in polls from 2024 than from any previous year in my life.
Polling as we once knew it, with genuinely representative samples from calling random people, no longer exists because they don’t pick up the phone. What we now call polls are essentially models tested on a little bit of data. This approach resulted in unprecedented errors in the 2024 Republican primary, which should lower our confidence that poll averages will predict the upcoming election.
Steven Shepard in Politico summarizes the most severe polling failures: “In Michigan, Trump’s margin over Haley was 15.3 points smaller in the actual results than in the final polling average. Pollsters also had double-digit whiffs in Massachusetts (a 14.3-point difference), Tennessee (11.3 points) and Virginia (20.8 points).” These races weren’t heavily polled around the voting, which may explain the errors.
There is no similar excuse in New Hampshire, which was heavily polled right before its election. The poll aggregators’ average had Trump winning by 19.2%, and he actually won by 11.1%. South Carolina was also heavily polled, and the aggregators had Trump winning by 26% where he actually won by 20.3%. (For NH and SC, the aggregator average is 538, 270towin, and RCP averaged together.)
People didn’t complain because the polls said Trump would win and he won. But missing the margin by 6-8% in a well-polled race is severe. Moreover, all these errors were overestimates of Trump, and underestimates of Haley.
The polls underestimated Trump in 2016 and 2020, and got things about right in 2022. Why would they suddenly overestimate Trump in the 2024 primaries? And what does this mean for the general election?
The fundamental problem is that response rates to phone polls have been declining for years. This means that phone polling mainly reaches the ever-shrinking sliver of the US population that picks up when called by an unknown number. If that sliver of the population is representative of the whole, polling will still work. But if it becomes unrepresentative, polling will present all opinion as being like the unrepresentative and skewed sample. If 1% of the US population consists of chaotic people who are always willing to pick up the phone and say extreme things to a pollster that they won't follow up on – maybe they’re playing pranks or high on intense drugs or having a mental health problem – a poll with a response rate of 5% will be one-fifth chaos.
There are other ways of polling than the phone, but none get a representative sample. For example, ActiVote polls people who downloaded its app, but self-selection problems there are severe. Pollsters now must put a lot of weight on how they antecedently model the composition of the 2024 electorate rather than what they learn about it from the data they acquire. Such models can be tweaked to get whatever result the pollster likes.
In 2020, phone polls were still doing okay because Trump and Haley voters were still on the same side. Trump voters are more likely to pick up the phone, while Haley voters are less likely. But their different tendencies cancelled out, and both voted for Republicans. Then January 6 happened, and divided the Republican Party into factions that couldn’t be reconciled. In 2022 each faction could be optimistic about winning control of the party in 2024, so they still mostly voted together.
The exception was when seriously Trumpy candidates turned off the sorts of people who became Haley voters. This happened in Pennsylvania, were Trump favorites ran for both Senator and Governor. The Senate race had the infamous Dr. Oz against John Fetterman, while the Governor’s race had Doug Mastriano against Josh Shapiro. Polls estimated that Shapiro would win by 11.4%; he actually won by 14.8%. Polls estimated a 0.4% victory for Oz; Fetterman won by 4.9% despite being incapacitated by a stroke.
All of this makes sense if Haley voters are less likely to respond to pollsters. What would that mean for 2024?
What data we have about Haley voters suggests that they’re especially likely to support Harris. This makes demographic and ideological sense. This voter bloc swung the 2022 PA Senate race, and kept Trump’s margins down in this year’s primaries. It’s because of silent Haley voters adding something like 1-2% to Harris’ margins in crucial swing states that I think she is more likely than not to win the Presidential election.
Tuesday, November 6, 2012
2012 Presidential Prediction Rankings
# | Predictor | Total | Obama / Romney picks | Pop=3.9% | State Fail | Grade |
1 | Markos Moulitsas and Daily Kos Elections | 332-206 | NV PA MN WI IA NH OH CO VA FL | 3.5% | 0 | A+ |
2 | Nate Silver, New York Times | 332-206 | NV PA MN WI IA NH OH CO VA FL | 2.5% | 0 | A |
3 | Simon Jackman, Huffington Post | 332-206 | NV PA MN WI IA NH OH CO VA FL | 1.7% | 0 | A |
4~ | Josh Putnam, Davidson College | 332-206 | NV PA MN WI IA NH OH CO VA FL | 0 | A | |
4~ | Drew Linzer, Emory University | 332-206 | NV PA MN WI IA NH OH CO VA FL | 0 | A | |
6 | Sam Wang, Princeton University | 303-235 | NV PA MN WI IA NH OH CO VA FL | 2.34% | 0.9 | A- |
7 | Jamelle Bouie, American Prospect | 303-235 | NV PA MN WI IA NH OH CO VA FL | 2.2% | 0.9 | A- |
8~ | TPM Polltracker | 303-235 | NV PA MN WI IA NH OH CO VA FL | 0.7% | 0.9 | A- |
8~ | RealClearPolitics | 303-235 | NV PA MN WI IA NH OH CO VA FL | 0.7% | 0.9 | A- |
10 | Intrade Prediction Market | 303-235 | NV PA MN WI IA NH OH CO VA FL | 0.9 | A- | |
11~ | Ezra Klein, Washington Post | 290-248 | NV PA MN WI IA NH OH CO VA FL | 3.9 | B | |
11~ | Larry Sabato, University of Virginia | 290-248 | NV PA MN WI IA NH OH CO VA FL | 3.9 | B | |
13 | Cokie Roberts, ABS NEWS | 294-234 | NV PA MN WI IA NH OH CO VA FL | 5.6 | B | |
14 | Dean Chambers, Unskewed Polls | 275-263 | NV PA MN WI IA NH OH CO VA FL | 1.79% | 10.5 | C+ |
15 | Erik Erickson, Redstate | 285-253 | NV PA MN WI IA NH OH CO VA FL | 17.2 | C | |
16 | SE Cupp, MSNBC | 270-268 | NV PA MN WI IA NH OH CO VA FL | 22.0 | C- | |
17 | Karl Rove, Bush advisor (popular) | 285-253 | NV PA MN WI IA NH OH CO VA FL | 3% | 23.9 | D+ |
18 | Ben Shapiro, National Review | 311-227 | NV PA MN WI IA NH OH CO VA FL | 28.0 | D | |
19 | Ben Domenech, The Transom | 278-260 | NV PA MN WI IA NH OH CO VA FL | R | 26.7+ME2 | D |
20 | Christian Schneider, Milwaukee Journal-Sentinel | 291-247 | NV PA MN WI IA NH OH CO VA FL | 30.5 | D- | |
21 | James Pethokoukis, AEI | 301-227 | NV PA MN WI IA NH OH CO VA FL | 2% | 30.5 | D- |
22 | Michael Barone, Washington Examiner | 315-223 | NV PA MN WI IA NH OH CO VA FL | 33.8 | F | |
23 | George Will, Washington Post | 321-217 | NV PA MN WI IA NH OH CO VA FL | 35.7 | F | |
24 | Steve Forbes, Forbes Magazine | 321-217 | NV PA MN WI IA NH OH CO VA FL | 40.5 | F | |
25 | Dick Morris, Fox News | 325-213 | NV PA MN WI IA NH OH CO VA FL | 41.5 | F |
Here are the ranking methodology and criteria for inclusion. I'm only putting professional pundits and big things like Intrade on the chart. But I should additionally mention that my co-blogger Nicholas Beaudrot got every state right.
The rankings are based on how well people predicted individual states, with the penalty for missing a state being the percentage margin of victory. This penalizes people less for getting close states wrong, and more for losing big. The values of the various states, via Huffington Post, are MN=7.7 WI=6.7 NV=6.6 NH=5.8 IA=5.6 PA=5.2 CO=4.7 VA=3.0 OH=1.9 FL=0.9.
I use accuracy of popular vote prediction to break ties. I'm working with a popular vote margin of 3.9%. Since many people didn't try to predict the popular vote, this is somewhat artificial. So I've added the letter-grade component on the right, which doesn't take popular vote into account, except in the case of the Daily Kos folks who get an A+ for getting the popular vote closest to right.
For now, I'll outsource commentary on the success of Nate and the other poll aggregators to xkcd:
Prediction-Gathering Post
I'm intending this site to be more a ranking of the punditocracy than a fantasy football league for all of us, so what I'm asking for are links to professional pundits. (But if you'd like to have your election predictions recorded for posterity in this thread, feel free to put them there!) I see that Brad Plumer's prediction-compiling post has Nate Silver, Sam Wang, Drew Linzer, Michael Barone, Ezra Klein, Larry Sabato, Josh Putnam, Jay Cost, Philip Klein, Ross Douthat, Jamelle Bouie, George Will, Ben Domenech, Markos Moulitsas, Karl Rove, James Pethokoukis, Dick Morris, Jim Cramer, and Dean Chambers in it. There's also a National Review prediction thread. Where else should I look? I'll try to use the last predictions people made before the first ballots of November 6 were cast.
As I suggested in the methodology post, I need people who have come up with an electoral map and preferably a popular vote margin as well. They should also be reasonably famous political prognosticators who are cited in print or televised media. A Facebook friend suggested that I rank P'Lod the All-Seeing, an alien quoted in the Weekly World News. But unfortunately, P'Lod didn't specify an electoral map.
Monday, November 5, 2012
Welcome To VoteSeeing!
In comments below, people have suggested methods that would give more weight to Florida than Alaska, since it's worth more votes. I think this is a very reasonable suggestion, and I've been chatting with friends about it. But in the end, I've stuck with the percentage system. To see why, look at the results from 2008. Do we want to penalize someone more for saying Obama would win Alaska, or that McCain would win Florida? I think Obama in Alaska is the sillier pick. The %Fail system delivers this result, while summing vote counts says that Obama in Alaska is better, since you were off by fewer voters. Nevertheless, once raw vote counts stabilize, I may add in those numbers as well, as it's also a nice metric. And in the end, I doubt this will make a huge amount of difference.
There might be a lot of more sophisticated ways of doing this, if people were making more complex predictions. If people were picking an amount by which candidates would win each state, we could take the sum of the error (or maybe the sum of the square of the error). But usually all people give you is a map and a popular vote percentage, so I'll be working with that.
Maybe I'll do things with Senate predictions later on if people are interested. But for now, just the presidential election. This is going to be interesting!