Tuesday, November 6, 2012

2012 Presidential Prediction Rankings

This chart ranks how well 25 major predictors foresaw the outcome of the 2012 presidential election.

#Predictor Outcome Obama / Romney picksPop=3.9% StateFail Grade
1 Markos Moulitsas and Daily Kos Elections 332-206 NV PA MN WI IA NH OH CO VA FL3.5% 0 A+
2 Nate Silver, New York Times 332-206 NV PA MN WI IA NH OH CO VA FL 2.5% 0 A
3 Simon Jackman, Huffington Post 332-206NV PA MN WI IA NH OH CO VA FL1.7%0A
4~ Josh Putnam, Davidson College 332-206 NV PA MN WI IA NH OH CO VA FL 0 A
4~ Drew Linzer, Emory University 332-206 NV PA MN WI IA NH OH CO VA FL 0 A
6 Sam Wang, Princeton University 303-235 NV PA MN WI IA NH OH CO VA FL 2.34% 0.9 A-
7 Jamelle Bouie, American Prospect 303-235NV PA MN WI IA NH OH CO VA FL 2.2% 0.9 A-
8~TPM Polltracker 303-235NV PA MN WI IA NH OH CO VA FL 0.7% 0.9 A-
8~RealClearPolitics 303-235NV PA MN WI IA NH OH CO VA FL 0.7% 0.9 A-
10 Intrade Prediction Market 303-235 NV PA MN WI IA NH OH CO VA FL
0.9 A-
11~ Ezra Klein, Washington Post 290-248 NV PA MN WI IA NH OH CO VA FL 3.9 B
11~ Larry Sabato, University of Virginia 290-248 NV PA MN WI IA NH OH CO VA FL 3.9 B
13 Cokie Roberts, ABS NEWS 294-234 NV PA MN WI IA NH OH CO VA FL 5.6 B
14 Dean Chambers, Unskewed Polls 275-263 NV PA MN WI IA NH OH CO VA FL1.79% 10.5 C+
15 Erik Erickson, Redstate 285-253 NV PA MN WI IA NH OH CO VA FL 17.2 C
16 SE Cupp, MSNBC 270-268 NV PA MN WI IA NH OH CO VA FL 22.0C-
17 Karl Rove, Bush advisor (popular) 285-253 NV PA MN WI IA NH OH CO VA FL3% 23.9 D+
18Ben Shapiro, National Review 311-227 NV PA MN WI IA NH OH CO VA FL
28.0 D
19 Ben Domenech, The Transom 278-260 NV PA MN WI IA NH OH CO VA FLR 26.7+ME2 D
20 Christian Schneider, Milwaukee Journal-Sentinel 291-247 NV PA MN WI IA NH OH CO VA FL
30.5 D-
21 James Pethokoukis, AEI 301-227 NV PA MN WI IA NH OH CO VA FL2% 30.5 D-
22 Michael Barone, Washington Examiner 315-223NV PA MN WI IA NH OH CO VA FL 33.8 F
23 George Will, Washington Post 321-217NV PA MN WI IA NH OH CO VA FL 35.7 F
24Steve Forbes, Forbes Magazine 321-217NV PA MN WI IA NH OH CO VA FL 40.5 F
25 Dick Morris, Fox News 325-213 NV PA MN WI IA NH OH CO VA FL 41.5 F

Here are the ranking methodology and criteria for inclusion. I'm only putting professional pundits and big things like Intrade on the chart. But I should separately mention that my co-blogger Nicholas Beaudrot got every state right.

The rankings are based on how well people predicted individual states, with the penalty for missing a state being the percentage margin of victory. This penalizes people less for getting close states wrong, and more for losing big. The values of the various states, via Huffington Post, are MN=7.7 WI=6.7 NV=6.6 NH=5.8 IA=5.6 PA=5.2 CO=4.7 VA=3.0 OH=1.9 FL=0.9.

I use accuracy of popular vote prediction to break ties. I'm working with a popular vote margin of 3.9%. Since many people didn't try to predict the popular vote, this is somewhat artificial. So I've added the letter-grade component on the right, which doesn't take popular vote into account, except in the case of the Daily Kos folks who get an A+ for getting the popular vote closest to right.

For now, I'll outsource commentary on the success of Nate and the other poll aggregators to xkcd:

Prediction-Gathering Post

Happy Election Day, everyone! Now that things are up and running, I'm putting up a post to gather people's election predictions.

I'm intending this site to be more a ranking of the punditocracy than a fantasy football league for all of us, so what I'm asking for are links to professional pundits. (But if you'd like to have your election predictions recorded for posterity in this thread, feel free to put them there!) I see that Brad Plumer's prediction-compiling post has Nate Silver, Sam Wang, Drew Linzer, Michael Barone, Ezra Klein, Larry Sabato, Josh Putnam, Jay Cost, Philip Klein, Ross Douthat, Jamelle Bouie, George Will, Ben Domenech, Markos Moulitsas, Karl Rove, James Pethokoukis, Dick Morris, Jim Cramer, and Dean Chambers in it. There's also a National Review prediction thread. Where else should I look? I'll try to use the last predictions people made before the first ballots of November 6 were cast.

As I suggested in the methodology post, I need people who have come up with an electoral map and preferably a popular vote margin as well. They should also be reasonably famous political prognosticators who are cited in print or televised media. A Facebook friend suggested that I rank P'Lod the All-Seeing, an alien quoted in the Weekly World News, but unfortunately P'Lod didn't specify an electoral map.

Monday, November 5, 2012

Welcome To VoteSeeing!

We're in the last days of the 2012 presidential election, and we're about to see a lot of predictions. I've started VoteSeeing to help us keep track of the predictions and see which ones were more accurate than others.

[I've modified this post a bit, and I intend to extend and tweak it a little before elections get going, in response to people's comments and suggestions.]

I've been talking with a few smart mathy friends about this, and my plan is to rank people's presidential election predictions by the two following criteria:

1. Summed State %Fail: I'll consider all the states where people didn't pick the winner, and add up the winner's percentage margin of victory in each of them. Lowest score is best here, with 0 meaning that you didn't miss a single state.

2. Popular %Fail: The percentage by which people missed the popular vote. When people get the same score on 1, probably because they picked an identical electoral map, I'll use this to distinguish between them. Again, lowest score is best, with 0 meaning you hit it exactly.

If someone doesn't pick a winner in a state, I'll just count half the margin of victory against them. That way, correct prediction is better than saying nothing, which in turn is better than error. If somebody doesn't say anything about the popular vote, I'll use a default value of the popular vote being tied (but don't worry, I'll mark this clearly, because it's totally artificial). 

Why this method? And in particular, why do 1 this way? Consider the following example: Donkey picks Obama to win Florida and Alaska, while Elephant picks Romney to win both states. And suppose Obama wins Florida by 0.1%, while Romney wins Alaska by 30%. I'd consider Elephant to be the better predictor here. Florida was really tight and could've gone either way. Elephant made a reasonable guess that was almost right. Donkey, meanwhile, was way off on Alaska in a way that just looks silly. Donkey, however, got closer to the total electoral vote, since Florida is worth a lot more.  So if you count based on that, you end up with the result that Donkey is the better predictor. And if you just count based on how many states people missed, you get them being equal. The method I've suggested rates Elephant higher in this case, which I think is the right answer.

In comments below, people have suggested methods that would give more weight to Florida than Alaska, since it's worth more votes. I think this is a very reasonable suggestion, and I've been chatting with friends about it. But in the end, I've stuck with the percentage system.  To see why, look at the results from 2008. Do we want to penalize someone more for saying Obama would win Alaska, or that McCain would win Florida? I think Obama in Alaska is the sillier pick. The %Fail system delivers this result, while summing vote counts says that Obama in Alaska is better, since you were off by fewer voters. Nevertheless, once raw vote counts stabilize, I may add in those numbers as well, as it's also a nice metric. And in the end, I doubt this will make a huge amount of difference.

There might be a lot of more sophisticated ways of doing this, if people were making more complex predictions. If people were picking an amount by which candidates would win each state, we could take the sum of the error (or maybe the sum of the square of the error). But usually all people give you is a map and a popular vote percentage, so I'll be working with that.

Maybe I'll do things with Senate predictions later on if people are interested. But for now, just the presidential election. This is going to be interesting!