Monday, November 5, 2012

Welcome To VoteSeeing!

We're in the last days of the 2012 presidential election, and we're about to see a lot of predictions. I've started VoteSeeing to help us keep track of the predictions and see which ones were more accurate than others.

[I've modified this post a bit, and I intend to extend and tweak it a little before elections get going, in response to people's comments and suggestions.]

I've been talking with a few smart mathy friends about this, and my plan is to rank people's presidential election predictions by the two following criteria:

1. Summed State %Fail: I'll consider all the states where people didn't pick the winner, and add up the winner's percentage margin of victory in each of them. Lowest score is best here, with 0 meaning that you didn't miss a single state.

2. Popular %Fail: The percentage by which people missed the popular vote. When people get the same score on 1, probably because they picked an identical electoral map, I'll use this to distinguish between them. Again, lowest score is best, with 0 meaning you hit it exactly.

If someone doesn't pick a winner in a state, I'll just count half the margin of victory against them. That way, correct prediction is better than saying nothing, which in turn is better than error. If somebody doesn't say anything about the popular vote, I'll use a default value of the popular vote being tied (but don't worry, I'll mark this clearly, because it's totally artificial). 

Why this method? And in particular, why do 1 this way? Consider the following example: Donkey picks Obama to win Florida and Alaska, while Elephant picks Romney to win both states. And suppose Obama wins Florida by 0.1%, while Romney wins Alaska by 30%. I'd consider Elephant to be the better predictor here. Florida was really tight and could've gone either way. Elephant made a reasonable guess that was almost right. Donkey, meanwhile, was way off on Alaska in a way that just looks silly. Donkey, however, got closer to the total electoral vote, since Florida is worth a lot more.  So if you count based on that, you end up with the result that Donkey is the better predictor. And if you just count based on how many states people missed, you get them being equal. The method I've suggested rates Elephant higher in this case, which I think is the right answer.

In comments below, people have suggested methods that would give more weight to Florida than Alaska, since it's worth more votes. I think this is a very reasonable suggestion, and I've been chatting with friends about it. But in the end, I've stuck with the percentage system.  To see why, look at the results from 2008. Do we want to penalize someone more for saying Obama would win Alaska, or that McCain would win Florida? I think Obama in Alaska is the sillier pick. The %Fail system delivers this result, while summing vote counts says that Obama in Alaska is better, since you were off by fewer voters. Nevertheless, once raw vote counts stabilize, I may add in those numbers as well, as it's also a nice metric. And in the end, I doubt this will make a huge amount of difference.

There might be a lot of more sophisticated ways of doing this, if people were making more complex predictions. If people were picking an amount by which candidates would win each state, we could take the sum of the error (or maybe the sum of the square of the error). But usually all people give you is a map and a popular vote percentage, so I'll be working with that.

Maybe I'll do things with Senate predictions later on if people are interested. But for now, just the presidential election. This is going to be interesting!

5 comments:

  1. Shouldn't you weight the % failure by the electoral college votes of the state? It seems like mis-calling Ohio should count more than miscalling Nevada?

    ReplyDelete
  2. Shouldn't you weight the % failure by the electoral college votes of the state? It seems like mis-calling Ohio should count more than miscalling Nevada?

    ReplyDelete
  3. Obama wins all the swing states except FL and NC. (I think that's Obama 303, Romney 235.) And wins the popular vote by 2.7%.

    ReplyDelete
  4. Agreed with Cy, it needs to be weighted by the electoral college votes. Missing Florida by 1% is a much bigger fail than missing Alaska by 1% (sticking with your example and assuming Alaska was suddenly in play via some "November Suprise" that affected Alaskans... Romney comes out in favor of an Igloo tax or something...)

    ReplyDelete
  5. Thanks for raising the issue, Cy and Unknown! I've gone back and forth on this, and I've decided to go with the original way I had up there. I've updated the post to explain why.

    ReplyDelete