The failure of the One True Pollster

Barring something really surprising, Donald Trump has won the election. I'm dismayed by the outcome and sorry for giving my friends false hope. There will be time to analyze political implications of a Trump victory later. Since I've been posting about polling and election predictions lately, I'll stick to that now.

I bet everything on Ann Selzer's Harris +3 result in Iowa being highly informative about national trends. This completely failed. It's understandable that she didn't predict the strong Hispanic swing towards Trump, since Iowa has a fairly small Hispanic population. 

But I expected Selzer to at least be accurate about people in Iowa. With almost 95% reporting, Harris is down 14 in Iowa. That Selzer would miss by 17 in Iowa is the biggest surprise of the election to me, as she had done so well in previous elections. I was using her Iowa result to predict the rest of the country, and it was in the wrong direction about Iowa! Maybe she'll have some kind of explanation. I have no idea how this happened.

Perhaps the most important thing to understand that I don't understand how low-response-rate poll aggregation works. Somehow, it seems to work pretty well. Maybe the types of modeling errors I forecast this year will come to pass sometime in the future, but it's so far so good for the aggregators.

That people would vote for Donald Trump doesn't surprise me that much – I've seen it before. But that people would vote solidly against Trump's endorsees Herschel Walker in GA and Dr. Oz in PA in 2022, and then vote for Trump himself over Harris in 2024 is a bit unexpected. I suppose people are more into the boss than the minions? There also seems to be a fair amount of ticket-splitting where people voted for Trump and a Democratic Senate or Governor candidate. I thought they'd do it the other way around, but this is what we're seeing. 

Somehow, we may take the House. How we might be doing that while losing everywhere else is among the many things I don't understand here – obviously there were some redistricting-related court cases that helped, but the bad news elsewhere seems overwhelming enough that winning a chamber is very surprising. That will be important for policy and worth watching.

Overall, though, I'm just astonished. Betting on Selzer is the sort of thing I usually do and it usually works – finding something deep in the weeds of politics that has an excellent track record and relying on it. But there's no guarantee that what worked before will work again, and here it spectacularly failed.

One True Pollster: Harris by 5%, 309-229

I predict a Kamala Harris victory, with a 309-229 Electoral College majority and a 52-47 popular vote majority. I arrive at this prediction by using the One True Pollster model, outlined below.

Respect for representative-sample phone polling drives the One True Pollster model. Political analysts' respect for polls comes from the ability of phone polling to generate a representative sample, which supports good predictions. Such polls flourished and could be usefully aggregated in 2012 when I started this blog and celebrated the rise of poll aggregators.

Such polls are at this point almost gone, and I return to chart the demise of the era in which this blog rose. In the modern era, response rates have collapsed, and it's really hard to get a representative sample. Poll aggregators have lost their value, clogged by things that aren't really polls, the worst of which are simply used to distort public opinion. Catastrophic failures of polling little noticed by the media demonstrate this. 

Here I lay out the three principles of the One True Pollster model. In doing so, I'll explain how I arrived at the map above. The model isn't some kind of neural net with opaque parameters. It uses polling data in line with three principles. I'll outline them below, and in doing so, explain the map. 

1. Ann Selzer is the One True Pollster. Her commitment to true representative-sample polling with minimal additional assumptions sets her apart methodologically. It explains her unparalleled record of surprising, informative, and extremely accurate results. 

This doesn't mean she's infallible – the One True Pollster is after all a pollster. She slightly underestimated Trump in the primary and we calibrate accordingly. But she's the only pollster who I'm sure is running a real poll these days. 

From a recent interview with The Bulwark:

As I read "I keep my dirty fingers off my data", my hand moved towards my heart, like she was singing the National Anthem. As she says about sticking to her rigorous methodology in a Bulwark interview: "I’m prepared that one day it will not work and I’ll blow up into tiny little pieces and be scattered across the city of Des Moines." Holy Saraswati, Goddess of Knowledge! 

Selzer found Harris 47, Trump 44 in Iowa. Since she underestimated Trump's margin over Haley by 4 points, we'll subtract 4 and expect Trump +1 in Iowa. If Harris is keeping it that close in Iowa, she's doing so well among the sorts of people who vote there that she's winning MI, WI, PA, and Omaha for 270 electoral votes and victory. We'll try to maintain conservative assumptions apart from "Harris is winning much bigger than expected with Iowa-type people." 

The map has Harris winning all other swing states but AZ, where rightward shifts among Latino men favor Republicans. In NC, Mark Robinson's disastrous 'Black Nazi' candidacy for Governor sensitizes voters to Trump's awfulness. I've seen some early voting data suggesting strong turnout among women and Zoomers in Georgia; they carry our hopes. NV is the hardest to predict because confounding factors have devoured the utility of the early voting data, but Jon Ralston has a perfect record since 2010 and he says Harris by 0.3, so I'll just run with that. I'm guessing that's a Harris +5% map? 

2. The NYT/Siena poll is informative too, but its predictive power is severely limited. It has enough resources to detect something, but if Harris is doing spectacularly well, NYT/Siena won't be able to tell you that. 

Here Nate Cohn explains why, and admits that he and most of his professional community have stopped running real polls. This is the kind of thing that we should have heard months ago from some responsible steward of American political polling. But of course we hear it now after we've been consuming their product frantically for months, and cautiously herded polls make the truth unavoidable: 

With all its resources, NYT/Siena is actually doing polls, which is why they aren't herding like so many other pollsters. But they're shying away from publishing any especially strong Harris result. When the guy running the polls tells you that, you have to adjust their polls accordingly. If reality is Harris +7 in PA, they won't show it to you. I'm guessing it's because of a baked-in assumption of their turnout model, but maybe it's somewhere else in the system. 

There is a systematic perversity behind Cohn's admission. When the polls favor Democrats but they lose, Democrats blame the polls. When the polls favor Republicans but they lose, Republicans blame shenanigans in the real elections. (They may also do this when polls don't favor Republicans.) This has the perverse effect of making pollsters like election deniers better than people who hold pollsters accountable for their failures. 

3. Other polls must be ignored, due to their systematic and severe errors during 2022 Trump-endorsed Senate campaigns and especially the 2024 Republican primary. Low response rates have destroyed the representative-sample polling we once knew and loved. All these polls systematically failed to find the sorts of Republicans who don't like Trump-endorsed Senate candidates and vote for Nikki Haley. This is why the polls wildly overestimated Trump's margin of error in primary after primary. Selzer did better in Iowa than any polling average in any other state. 

The surprise of this election will be delivered by silent Haley voters. Selzer's methods alone detect them – longtime Republicans who prefer stability and order, and disdain both lefty-activist and MAGA chaos. Many have business or military backgrounds, or are married to people who do. Pollsters used to focus mainly on these voters because they were the most reliable voters, many regarding voting as a patriotic duty. 

Haley voters don't like Trump, but many were scared into sticking with him for a while. They voted against Hillary in 2016 because of 25 years of Republican propaganda against her. BLM may have scared them into staying Republican in 2020. 

But then January 6 happened, and shocked these law-and-order loving people. They were on Liz Cheney's side, and were shocked again by her demotion and defeat. Some still voted for Republicans until 2022 in hopes that they could retake the party from Trump, but they wouldn't come out for Trump-endorsed Senate candidates like Dr. Oz and Herschel Walker, who generally underperformed their poll numbers while other Republicans didn't. Now the Haley voters are fully giving up on the party, and many are becoming Harris voters.

Most pollsters used to model Republicans as Haley voters. After 2016 and 2020, they model Republicans as Trump voters. These days, Haley voters may be less likely to pick up the phone than Trump voters. I don't know how it was in 2012. It may have been the opposite back then, when a ringing phone seemed more of an obligation. Selzer alone finds Haley voters, because of her random-dialing methods without frightened modeling assumptions. 

Or maybe other true pollsters are running real polls like Selzer. I apologize to them and hope they will soon be revealed to me. But I don't have much confidence that I can find them. I'm sure data can be extracted from some other polls in aggregate, perhaps with appropriate adjustments like I'm making with NYT/Siena. But I have no way of aggregating that data without including lots of fake polls. I despair of finding another needle in that haystack full of snakes.

Poll aggregators are worthless. Especially after 2022 when herding together for safety worked fairly well, lots of polls are incentivized to herd again. Now they're herding around the safest and least committal results – swing states even, Harris up a percent or two nationally. 

Aggregators are also clogged with highly rated fake polls from companies like AtlasIntel that 'predicted well' in previous years. I don't trust new firms' past predictions anymore. Scammers running highly manipulated online polls can follow the herd to a safe result one year, and sell out to some billionaire who wants to manipulate media coverage the next year. They could also profit by running pump-and-dumps on prediction markets. Someone could even run five totally fake 'polling firms', predict a range of results, and end up certain of having a 'highly rated' polling firm at the end of it. Goodhart's Law – “when a measure becomes a target, it ceases to be a good measure” – has ruined aggregators' pollster ratings. 

AtlasIntel is especially dubious as it's pumping out 'interactive polls' at very high speed near the election, using a 'proprietary' and unexplained methodology. Interactive polls should be ignored entirely, as they involve opt-in methods of sampling that attract chaotic respondents. Pew did an opt-in survey and asked people if they were licensed to operate a nuclear submarine. 

If you believe interactive polls, say hello to the young nuclear submarine pilots all around you.

People today are calling anything a 'poll' and throwing it into easily manipulated aggregators. One light shines amidst this darkness. She is Ann Selzer, the One True Pollster. 

Ann Selzer, the One True Pollster

Yesterday Ann Selzer released a Harris 47, Trump 44 result in Iowa. It has to be too optimistic, but if the score is even close to that in Iowa, Harris is winning the election.

I’ve made a general case for poll skepticism, and I should explain why this poll has some chance of being unusually reliable. Selzer has been overseeing polling at the Des Moines Register since 1987, focusing on how to get an accurate read of Iowa. She applies decades of experience and well-tested methods to a simpler task than other pollsters have, working only with a small and relatively homogeneous state. 

Selzer’s track record is unparalleled, including Iowa caucuses and general elections. Four years ago, her final pre-election poll shocked complacent Democrats by showing Trump up 7. Several other pollsters were finding a Biden lead, and of the last 37 polls before the election, no one else found such a large Trump lead. Trump actually won the state by 8. 

She did pretty well in the 2024 Iowa caucuses, predicting a 48-16-20-8 Trump-DeSantis-Haley-Vivek popular breakdown when the outcome was 51-21-19-8. Perhaps her polls are especially sensitive to Haley voters. In the primary she underestimated Trump’s margin over Haley by 4, while the NH average overestimated by 6, the SC average by 8, and other state averages missed by double digits. (Her last Iowa poll beats ALL the other state AVERAGES!) This supports my view that other polls are missing Haley voters.

A herd of polls disagrees with her. The herd does not predict an epic Harris victory, perhaps fearing the costs of being wrong. Rather than follow the herd, I will follow her. She predicts the election using evidence rather than fear. To me, she is the one true pollster. 

Her modest Trump underestimation may suggest treating her results as underestimating him again. So perhaps we should accordingly trim 4 points off the +3 margin so Trump wins Iowa by 1. But if his margin is anywhere near that small in Iowa, he’s doing terribly in similar regions. Harris easily holds PA-MI-WI-Omaha for 270+ electoral votes and victory.

As polls crumble into models, are Haley voters slipping through the cracks?

Twelve years ago, I created this blog to keep a public record of predictors' track records as polling was entering its Golden Age. Nate Silver and other poll aggregators predicted the 2012 election brilliantly, and I kept a record of it. Now I return with less confidence in polls from 2024 than from any previous year in my life. 

Polling as we once knew it, with genuinely representative samples from calling random people, no longer exists because they don’t pick up the phone. What we now call polls are essentially models tested on a little bit of data. This approach resulted in unprecedented errors in the 2024 Republican primary, which should lower our confidence that poll averages will predict the upcoming election.  

Steven Shepard in Politico summarizes the most severe polling failures: “In Michigan, Trump’s margin over Haley was 15.3 points smaller in the actual results than in the final polling average. Pollsters also had double-digit whiffs in Massachusetts (a 14.3-point difference), Tennessee (11.3 points) and Virginia (20.8 points).” These races weren’t heavily polled around the voting, which may explain the errors. 

There is no similar excuse in New Hampshire, which was heavily polled right before its election. The poll aggregators’ average had Trump winning by 19.2%, and he actually won by 11.1%. South Carolina was also heavily polled, and the aggregators had Trump winning by 26% where he actually won by 20.3%. (For NH and SC, the aggregator average is 538, 270towin, and RCP averaged together.) 

People didn’t complain because the polls said Trump would win and he won. But missing the margin by 6-8% in a well-polled race is severe. Moreover, all these errors were overestimates of Trump, and underestimates of Haley. 

The polls underestimated Trump in 2016 and 2020, and got things about right in 2022. Why would they suddenly overestimate Trump in the 2024 primaries? And what does this mean for the general election?

The fundamental problem is that response rates to phone polls have been declining for years. This means that phone polling mainly reaches the ever-shrinking sliver of the US population that picks up when called by an unknown number. If that sliver of the population is representative of the whole, polling will still work. But if it becomes unrepresentative, polling will present all opinion as being like the unrepresentative and skewed sample. If 1% of the US population consists of chaotic people who are always willing to pick up the phone and say extreme things to a pollster that they won't follow up on – maybe they’re playing pranks or high on intense drugs or having a mental health problem – a poll with a response rate of 5% will be one-fifth chaos. 

There are other ways of polling than the phone, but none get a representative sample. For example, ActiVote polls people who downloaded its app, but self-selection problems there are severe. Pollsters now must put a lot of weight on how they antecedently model the composition of the 2024 electorate rather than what they learn about it from the data they acquire. Such models can be tweaked to get whatever result the pollster likes. 

In 2020, phone polls were still doing okay because Trump and Haley voters were still on the same side. Trump voters are more likely to pick up the phone, while Haley voters are less likely. But their different tendencies cancelled out, and both voted for Republicans. Then January 6 happened, and divided the Republican Party into factions that couldn’t be reconciled. In 2022 each faction could be optimistic about winning control of the party in 2024, so they still mostly voted together. 

The exception was when seriously Trumpy candidates turned off the sorts of people who became Haley voters. This happened in Pennsylvania, were Trump favorites ran for both Senator and Governor. The Senate race had the infamous Dr. Oz against John Fetterman, while the Governor’s race had Doug Mastriano against Josh Shapiro. Polls estimated that Shapiro would win by 11.4%; he actually won by 14.8%. Polls estimated a 0.4% victory for Oz; Fetterman won by 4.9% despite being incapacitated by a stroke. 

All of this makes sense if Haley voters are less likely to respond to pollsters. What would that mean for 2024? 

What data we have about Haley voters suggests that they’re especially likely to support Harris. This makes demographic and ideological sense. This voter bloc swung the 2022 PA Senate race, and kept Trump’s margins down in this year’s primaries. It’s because of silent Haley voters adding something like 1-2% to Harris’ margins in crucial swing states that I think she is more likely than not to win the Presidential election.

2012 Presidential Prediction Rankings

This chart ranks how well 25 major forecasters predicted the outcome of the 2012 presidential election. Obama won the election 332-206, with a 3.9% popular vote margin.

#	Predictor	Total	Obama / Romney picks	Pop=3.9%	State Fail	Grade
1	Markos Moulitsas and Daily Kos Elections	332-206	NV PA MN WI IA NH OH CO VA FL	3.5%	0	A+
2	Nate Silver, New York Times	332-206	NV PA MN WI IA NH OH CO VA FL	2.5%	0	A
3	Simon Jackman, Huffington Post	332-206	NV PA MN WI IA NH OH CO VA FL	1.7%	0	A
4~	Josh Putnam, Davidson College	332-206	NV PA MN WI IA NH OH CO VA FL		0	A
4~	Drew Linzer, Emory University	332-206	NV PA MN WI IA NH OH CO VA FL		0	A
6	Sam Wang, Princeton University	303-235	NV PA MN WI IA NH OH CO VA FL	2.34%	0.9	A-
7	Jamelle Bouie, American Prospect	303-235	NV PA MN WI IA NH OH CO VA FL	2.2%	0.9	A-
8~	TPM Polltracker	303-235	NV PA MN WI IA NH OH CO VA FL	0.7%	0.9	A-
8~	RealClearPolitics	303-235	NV PA MN WI IA NH OH CO VA FL	0.7%	0.9	A-
10	Intrade Prediction Market	303-235	NV PA MN WI IA NH OH CO VA FL		0.9	A-
11~	Ezra Klein, Washington Post	290-248	NV PA MN WI IA NH OH CO VA FL		3.9	B
11~	Larry Sabato, University of Virginia	290-248	NV PA MN WI IA NH OH CO VA FL		3.9	B
13	Cokie Roberts, ABS NEWS	294-234	NV PA MN WI IA NH OH CO VA FL		5.6	B
14	Dean Chambers, Unskewed Polls	275-263	NV PA MN WI IA NH OH CO VA FL	1.79%	10.5	C+
15	Erik Erickson, Redstate	285-253	NV PA MN WI IA NH OH CO VA FL		17.2	C
16	SE Cupp, MSNBC	270-268	NV PA MN WI IA NH OH CO VA FL		22.0	C-
17	Karl Rove, Bush advisor (popular)	285-253	NV PA MN WI IA NH OH CO VA FL	3%	23.9	D+
18	Ben Shapiro, National Review	311-227	NV PA MN WI IA NH OH CO VA FL		28.0	D
19	Ben Domenech, The Transom	278-260	NV PA MN WI IA NH OH CO VA FL	R	26.7+ME2	D
20	Christian Schneider, Milwaukee Journal-Sentinel	291-247	NV PA MN WI IA NH OH CO VA FL		30.5	D-
21	James Pethokoukis, AEI	301-227	NV PA MN WI IA NH OH CO VA FL	2%	30.5	D-
22	Michael Barone, Washington Examiner	315-223	NV PA MN WI IA NH OH CO VA FL		33.8	F
23	George Will, Washington Post	321-217	NV PA MN WI IA NH OH CO VA FL		35.7	F
24	Steve Forbes, Forbes Magazine	321-217	NV PA MN WI IA NH OH CO VA FL		40.5	F
25	Dick Morris, Fox News	325-213	NV PA MN WI IA NH OH CO VA FL		41.5	F

Here are the ranking methodology and criteria for inclusion. I'm only putting professional pundits and big things like Intrade on the chart. But I should additionally mention that my co-blogger Nicholas Beaudrot got every state right.

The rankings are based on how well people predicted individual states, with the penalty for missing a state being the percentage margin of victory. This penalizes people less for getting close states wrong, and more for losing big. The values of the various states, via Huffington Post, are MN=7.7 WI=6.7 NV=6.6 NH=5.8 IA=5.6 PA=5.2 CO=4.7 VA=3.0 OH=1.9 FL=0.9.

I use accuracy of popular vote prediction to break ties. I'm working with a popular vote margin of 3.9%. Since many people didn't try to predict the popular vote, this is somewhat artificial. So I've added the letter-grade component on the right, which doesn't take popular vote into account, except in the case of the Daily Kos folks who get an A+ for getting the popular vote closest to right.

For now, I'll outsource commentary on the success of Nate and the other poll aggregators to xkcd:

Prediction-Gathering Post

Happy Election Day, everyone! Now that things are up and running, I'm putting up a post to gather people's election predictions.

I'm intending this site to be more a ranking of the punditocracy than a fantasy football league for all of us, so what I'm asking for are links to professional pundits. (But if you'd like to have your election predictions recorded for posterity in this thread, feel free to put them there!) I see that Brad Plumer's prediction-compiling post has Nate Silver, Sam Wang, Drew Linzer, Michael Barone, Ezra Klein, Larry Sabato, Josh Putnam, Jay Cost, Philip Klein, Ross Douthat, Jamelle Bouie, George Will, Ben Domenech, Markos Moulitsas, Karl Rove, James Pethokoukis, Dick Morris, Jim Cramer, and Dean Chambers in it. There's also a National Review prediction thread. Where else should I look? I'll try to use the last predictions people made before the first ballots of November 6 were cast.

As I suggested in the methodology post, I need people who have come up with an electoral map and preferably a popular vote margin as well. They should also be reasonably famous political prognosticators who are cited in print or televised media. A Facebook friend suggested that I rank P'Lod the All-Seeing, an alien quoted in the Weekly World News. But unfortunately, P'Lod didn't specify an electoral map.

Welcome To VoteSeeing!

We're in the last days of the 2012 presidential election, and we're about to see a lot of predictions. I've started VoteSeeing to help us keep track of the predictions and see which ones were more accurate than others.

I've been talking with a few smart mathy friends about this, and my plan is to rank people's presidential election predictions by the two following criteria:

1. Summed State %Fail: I'll consider all the states where people didn't pick the winner, and add up the winner's percentage margin of victory in each of them. Lowest score is best here, with 0 meaning that you didn't miss a single state.

2. Popular %Fail: The percentage by which people missed the popular vote. When people get the same score on 1, probably because they picked an identical electoral map, I'll use this to distinguish between them. Again, lowest score is best, with 0 meaning you hit it exactly.

If someone doesn't pick a winner in a state, I'll just count half the margin of victory against them. That way, correct prediction is better than saying nothing, which in turn is better than error. If somebody doesn't say anything about the popular vote, I'll use a default value of the popular vote being tied (but don't worry, I'll mark this clearly, because it's totally artificial). 

Why this method? And in particular, why do 1 this way? Consider the following example: Donkey picks Obama to win Florida and Alaska, while Elephant picks Romney to win both states. And suppose Obama wins Florida by 0.1%, while Romney wins Alaska by 30%. I'd consider Elephant to be the better predictor here. Florida was really tight and could've gone either way. Elephant made a reasonable guess that was almost right. Donkey, meanwhile, was way off on Alaska in a way that just looks silly. Donkey, however, got closer to the total electoral vote, since Florida is worth a lot more.  So if you count based on that, you end up with the result that Donkey is the better predictor. And if you just count based on how many states people missed, you get them being equal. The method I've suggested rates Elephant higher in this case, which I think is the right answer.

In comments below, people have suggested methods that would give more weight to Florida than Alaska, since it's worth more votes. I think this is a very reasonable suggestion, and I've been chatting with friends about it. But in the end, I've stuck with the percentage system.  To see why, look at the results from 2008. Do we want to penalize someone more for saying Obama would win Alaska, or that McCain would win Florida? I think Obama in Alaska is the sillier pick. The %Fail system delivers this result, while summing vote counts says that Obama in Alaska is better, since you were off by fewer voters. Nevertheless, once raw vote counts stabilize, I may add in those numbers as well, as it's also a nice metric. And in the end, I doubt this will make a huge amount of difference.

There might be a lot of more sophisticated ways of doing this, if people were making more complex predictions. If people were picking an amount by which candidates would win each state, we could take the sum of the error (or maybe the sum of the square of the error). But usually all people give you is a map and a popular vote percentage, so I'll be working with that.

Maybe I'll do things with Senate predictions later on if people are interested. But for now, just the presidential election. This is going to be interesting!