America’s Predictable Elections
I liked reading 538.com during the election season as much as anyone, and still think it’s an insightful site with a lot of interesting things to say. That said, I’ve been kind of surprised by the post-election surge of praise for Nate Silver. You would think, based on some of the commentary, that his innovative and complicated formula wound up giving us some incredible insight we couldn’t have gotten from anywhere else. In fact, though the 538 algorithm performed pretty well, the core reason it performed well was that crude polling averages are very accurate in US Presidential elections:
Assuming that Obama will win by 7.1%, the Pollster.com regression line of all national polls was only off by only 0.5%. The same can be said for the simple mean of national polls performed by Real Clear Politics. The lesson here is that when there are a high number of public polls, election forecasters are not very useful. Even a schmo like me can just conduct a simple mean of all the polls, and come pretty close to the final result.
The high stakes in US Presidential elections make people feel a lot of anxiety about outcomes. This makes people very interested in the subject of election forecasting. And they would like their level of interest to be matched by the forecasting process itself being interesting. But in fact the large number of public polls on something like a presidential election makes the outcomes quite easy to forecast based on crude measures. What’s more, even absent polling, Presidential election outcomes seem to be pretty predictable based on nothing more than macroeconomic variables. But even if you don’t believe in the fundamentals, the reality is that “the polls,” in the aggregate, are very accurate and there’s not much more to it. For other things, like primary elections in states that have rarely held competitive primaries (and this is where 538 initially made a name for itself), or in little-polled House races, there’s lots of room for other methods. But presidential elections are easy.
88 Responses to “America’s Predictable Elections”
Register
RSSLogin
Topic Cloud
ArchivesBlog RollAbout Matt YglesiasContact Me
Donate
December 2nd, 2008 at 3:54 pm
Did everyone get it right in 2000? I seem to remember that the final pre-election polls had a slight lead for W., but no one that I knew of predicted what eventually happened with any degree of certainty. Even Rove was so (over) confident that he had Bush out in California pre-election. And of course the macro-economic fundamentals that year were in Gore’s favor, although Monica et al. and the weariness of the Clinton era scandals probably hurt him.
December 2nd, 2008 at 4:01 pm
i would pay $20 to see matt yglesias and nate silver get in a fist fight.
December 2nd, 2008 at 4:03 pm
I have a foolproof way of predicting elections and events: pick the opposite of Matt’s prediction.
December 2nd, 2008 at 4:04 pm
Meow
December 2nd, 2008 at 4:07 pm
I thought it was 538’s electoral map prediction that was really impressive.
December 2nd, 2008 at 4:07 pm
Hidden subtext here: Baseball-o-phobe Matt hatin’ on sabermetric pioneer Nate Silver.
December 2nd, 2008 at 4:09 pm
I agree that some pundits gushed a little too much over Nate Silver, but the gushed over Bush and Sarah Palin even more.
What I found compelling about 538.com was the attempt to bat down a bunch of made up explanations for why people voted the way that they did. There was more of that in the primary. There was also a lot of reality checks (for all sides) and information on what to expect in polling during and after the conventions.
The more sites we have like this the better. Somehow the unchecked media was allowed to paint Gore as a liar. Part of the media story line is driven by reading too much into polls, single polls, not reading too much into an average of polls on the top line outcome. (Remember how Obama couldn’t win the votes of older white working class voters. Yet there was never a survey where they asked anyone are you a white working class voter, there was never even an attempt to define how you end up in that demographic and what percentage of the pie they make up.)
Yes Nate got gushed over…but by a class of folks who are paid to either gush or attack or ignore.
December 2nd, 2008 at 4:10 pm
538.com gave more than just the popular vote tally though; it also provided an estimate of win percentage (which isn’t easily computed from the national popular vote polling), an electoral college prediction, an electoral college histogram based on the election simulations, and a state-by-state breakdown of polling, summarized in a shades-of-blue-and-red electoral map. Nate and Sean also added a lot of value with their data-based commentary (often of a far higher quality than you would find on op-eds, blogs or TV) and election travelogues.
December 2nd, 2008 at 4:14 pm
someBrad: Yeah, but websites with more naive methods like RealClearPolitics (which just uses a straight average of recent polls deemed worthy) were basically just as impressive. However, although I might be misremembering things, I think that FiveThirtyEight was able to reach its conclusions somewhat earlier than poll-only methods, which is nice. The ability to predict the election two months in advance, although inherently less reliable than predicting it two days in advance, is something worth aspiring to in of itself.
December 2nd, 2008 at 4:15 pm
What makes Nate Silver worth gushing over is not the accuracy of his model, but his commitment to illuminating the process. Most pollsters are reticent, or even evasive, about their methodology. Unless you’re a statistician or well-studied in polling, the whole thing seems like something pulled out of a black box.
Silver explained in detail how the whole mess worked and gave detailed rationales for weighting polls. You could disagree with his assessments, but at least you always knew his basis for making them. He lifted the veil, owned up to not only his biases but also his mistakes, and wrote about a fairly dry subject in an engaging way. His accuracy was less important than his transparency.
December 2nd, 2008 at 4:16 pm
Matt’s (basically) right here. 538’s state-by-state popular vote predictions (including the national vote as a “state”) did not perform, on average, any differently than Pollster’s or RCP’s.
I’ve looked at this fairly closely actually. All three methods did quite well (by this metric; predicting electoral outcomes was easier in this particular election using almost any method since that’s such a coarse measurement).
That’s not really a criticism of 538 or nate silver; it’s just to say that all that extra stuff silver was doing didn’t actually provide any systematic improvement in accuracy (as measured by popular vote margin predictions) over Pollster or RCP. (Again, one can cherry pick states that each of the three sites did well on; overall though, they all did basically the same.)
But it certainly is fun and provides some interesting and different looks at the polling data out there.
December 2nd, 2008 at 4:17 pm
I would make a distinction: by the time of election day, 538, the individual pollsters, pollster.com, and real clear politics had largely converged, and the election played out as predicted for the most part.
But before that, there were some dynamics: the impact of the Celebrity ad, then the conventions, the v-p picks, the emergence of the financial mess. In those cases, Nate was picking up trends more quickly than the other pollsters, because he could update electoral college projections without waiting for each particular state to be polled. That put him a week ahead of everyone else in identifying trends.
It was a useful model.
December 2nd, 2008 at 4:19 pm
As far as I can tell, Silver deserves credit for two things that no one else saw: (1) that Indiana would be extremely close in the primaries, (2) putting some quantitative estimate on the size and timing of the convention bounces. It should be noted that Silver predicted (incorrectly) that South Dakota would go for Obama.
I think silver’s information could be incredibly useful at the CD level; it would be interesting to see how the correlation to dem performance down the ticket is when compare to his various factors (chiefly the “American” ethnicity).
December 2nd, 2008 at 4:21 pm
Matt, I like you and this site a lot. But Nate Silver’s very hard technical work this election was a lot more helpful than your musings, to be blunt.
December 2nd, 2008 at 4:23 pm
Fivethirtyeight did a couple of things that Pollster, RCP, and the rest didn’t feel comfortable doing. First, it stepped right up and said that the Bradley Effect wasn’t a factor while the polling organizations demurred. Nate Silver, along with many political scientists, was way ahead of that one while mainstream pundits were saying that Obama needed to be up by 7 in the polls just to win by 2 points. And he said it forecefully and with a confidence not often seen in the wishwashy world of electoral projections.
Second, Silver provided an unprecedented push for accountability from polling firms, often embarrassing old stand-bys into submission. In a very real way, he raised the veil of the way public opinion is gauged in the US. I’m sure Silver would look askance at my hyperbole, but I don’t think it’s unreasonable to suggest that that man is responsible for freeing up more information about the American electorate than any one person in recent history.
December 2nd, 2008 at 4:23 pm
What the last half dozen commentators said: Nate provided a play by play on how his statistics work; he did not just displayed them. More to the point- national polls are not how the presidential election are decided, so one must look if the states are following the trends- there have been several elections (4+?) where the popular vote winner lost.
December 2nd, 2008 at 4:23 pm
I think the appeal of Silver has less to do with the innovations of his number crunching than it does with the idea that Silver is *the* authority on polling data. Early in the primaries there were lots of surprises and not only were the certainties of pundits called into question, but the validity of polling itself was in doubt. Silver doesn’t represent a single polling house, but is rather an expert on polls. He analyzes and assesses the data, and as such he’s been the go-to-guy to talk about polls. Plus, most polling houses maintain a strict impartiality (at least any pollster that expects to be taken seriously), while Nate Silver not only makes judgments about the significance of data but has his own opinions about politics. That means he’s a more interesting guest for a television show than most other experts in his field.
December 2nd, 2008 at 4:25 pm
Nicholas Beaudrot:It should be noted that Silver predicted (incorrectly) that South Dakota would go for Obama.
Not true — he predicted an 8.7% win for McCain.
December 2nd, 2008 at 4:25 pm
This http://election.princeton.edu/ website has some good analysis of different forecasting methods from someone who uderstands statistical analysis much better than I do.
December 2nd, 2008 at 4:26 pm
Matt:
If you were a baseball fan, instead of a fan of that other sport no right-thinking people care about, you would already be aware of the problems with Silver. He originally made a name for himself with a player projection method called PECOTA. Thing is, PECOTA was only marginally more accurate than a “generic” projection termed Marcel (’cause a monkey could do it…). By coming up with a fairly complex model, and frequently making ad hoc changes to it, Silver focuses attention on sophistication rather than accuracy, despite the fact that simpler models perform just as well or better. Silver has always been better at marketing himself and his “models” than really contributing.
Sam Wang (http://election.princeton.edu/) used a simpler model to predict the results more accurately (only slightly, because as you say, elections are easy) and also made many useful criticisms of Silver’s metholodogy along the way.
December 2nd, 2008 at 4:27 pm
It should be noted that Silver predicted (incorrectly) that South Dakota would go for Obama.
Just to clarify, you mean in the primary, right?
I’ve linked this before, but here’s some basically correct (I haven’t rechecked the state tallies for a while) data comparing 538, pollster, and electoral-vote in their predictions of the margin of victory in each state. I remain shocked, and a little bothered, that Nate hasn’t published something similar.
December 2nd, 2008 at 4:30 pm
Tom Veil,
I believe Nicholas Beaudrot was referring to the South Dakota primary, not the general election.
December 2nd, 2008 at 4:30 pm
Silver doesn’t represent a single polling house, but is rather an expert on polls.
He’s someone good at math and very good at creating mathematical models of real world events, I don’t know what expertise on polls he had prior to starting 538, but I’m pretty sure it’s near-zero.
December 2nd, 2008 at 4:34 pm
538 gave some fun and insightful commentary, both on the statistics as well as the politics. The site was clean, well presented and professional – I loved it during the race. But I agree with MY, it wasn’t anything groundbreaking or game-changing, and Silver kind of butchered the Minnesota recount. I’ll be back there next election season though.
December 2nd, 2008 at 4:41 pm
I would actually argue that Nate Silver’s model underperformed vs. many simpler models. You could say that everybody converged to the point where Missouri, Indiana, and North Carolina were toss-ups and then it was few a coin flips… but the whole point of of Mr. Silver’s “demographic modeling” deal was to reduce the uncertainty of such things, and instead he added to it.
That said, in 2008, toward the end there we were getting over 20 polls a day… there was really no reason to suspect that Nate Silver’s hunches were going to prove more powerful than that kind of data depth just aggregated. Where his models might prove more useful are races where there isn’t a whole lot of depth and polling is sparse.
I also have to give him credit for popularizing a serious analysis of the polls and making it sexy and relevant… and I certainly enjoy his commentary. For a serious mathematical treatment of the issue however, I went to the aforementioned Princeton Election Consortium.
December 2nd, 2008 at 4:56 pm
I’ve enjoyed 538 immensely, and like above commenters, really appreciated his discussion of the methodology of polling.
And I noticed that many polls seem to have a subtext, or at least a target audience. Zogby, in particular, seemed invested in providing good news to Republicans…
The results were seemingly tailored to the client’s wishes, it seems sometimes. And Silver made this less sustainable.
December 2nd, 2008 at 4:59 pm
Where Silver was very impressive, I thought, was during the primaries, which, it’s worth noting, were largely not based on aggregating polls, but on ignoring them and looking at the demographics in each state.
I won’t speak beyond that.
December 2nd, 2008 at 5:01 pm
Nate Silver puts his information in a form most people can read. He’s a great editor who has dense informative graphics. His formula may be a common variation of other formulas but when you aggregate everything he’s doing it’s a better product.
Nate Silver is The Weather Channel when it first came out. Weather geeks love them some weather. The raw data was available to the local weather broadcasts and cable news. They just never figured out how to present the information.
And don’t underestimate the quality of the non-polling posts on 538.com. I knew it was over for McCain when 538 started going to various McCain offices across the country and they were empty. 538’s photography and other posts showed how little enthusiasm there was for McCain across the country.
December 2nd, 2008 at 5:03 pm
Zack,
I’m pretty sure that PECOTA outperforms pretty much every other projection system. It’s only a little bit better than Marcel, but that’s because it’s really hard to do much better than Marcel. PECOTA manages to wring a bit more predictive value out of the data, which is more than you can say for most systems.
December 2nd, 2008 at 5:04 pm
Information to be read before using makkale.blogcu.com Blog but you see to go my home Hertha Berlin galatasaray maçının golleri, Hertha Berlin Galatasaray uefa kupası maçının özeti http://makkale.blogcu.com/cimbom-un-hertha-11-i-makkale-blogcu-com-canli-hertha-berlin-galatasaray-macinin-golleri-hertha-berlin-galatasaray-uefa-kupasi-macinin-ozeti_30285761.html
December 2nd, 2008 at 5:06 pm
Good points from most commenters, but one thing missed about Silver’s awesomeness is that he wasn’t just spectacularly right at the end- he was remarkably accurate throughout the election season, including the brief period after the RNC, when democrats were running scared. Your hypothetical schmo averaging together polls would have been completely off at that stage.
@J.W.Hamner, how does predicting the popular vote within .1% underperform anything? How does (accurately) assigning a probability to outcomes add uncertainty to anything? What are you talking about exactly?
December 2nd, 2008 at 5:21 pm
LaurencePassmore,
Nate’s results tracked the polls then entire time. Just like everyone else, his model showed McCain slightly ahead after the RNC.
At any given time, his model’s poll aggregate and projection, essentially agreed with Pollster’s at the time. For the most part, when his model deviated from this, it turned out to be wrong.
You are confusing Nate’s tone with his model’s actual predictions.
December 2nd, 2008 at 5:23 pm
One reason that Nate did a good job is that he called states BEFORE the polls even opened.
What is the point of waiting until Utah closed to call it for McCain? Or California for Obama?
I knew there were 77 votes on the west coast where McCain didn’t spend a dime and where the polls were heavily in Obama’s favor. It would have taken a miracle for McCain to win one of those states without wining virtually every battleground state. (Think about it. Can you come up with a reasonable case for McCain to win Oregon and not win states like New Mexico and Colorado? )
So, I KNEW that as soon as Obama won Ohio and had over 193 electoral votes that Obama had won.
Nate had the guts to say it was OVER when it was over. The networks all pretended that McCain had a chance. All he had to do was win states where the polls had him behind by double digits and McCain had never campaigned in.
I’ll make a prediction now. The Democrat will win DC in 2012. Did I call the race too early?????
December 2nd, 2008 at 5:30 pm
Your hypothetical schmo averaging together polls would have been completely off at that stage.
At the bottom of Obama’s post RNC polling collapse, on 9/16, Nate’s model was projecting 250.3 Electoral votes for Obama, and 249.7 on 9/17. Electoral-vote.com, which non-hypothetically just averages together polls, was projecting 247 electoral votes for Obama on those dates. Per your comment, a three vote difference is the difference between remarkably accurate and completely off.
It is, by the way, absurd that 538 does not archive each day’s prediction from the model and that I only know those two numbers because I wrote them down at the time.
December 2nd, 2008 at 5:45 pm
Neil Wilson at #33: The networks had a gentleman’s agreement not to call any state before the polls closed, and thus could not officially call the election before 11:00 pm, while Nate had no such restrictions. But watching MSNBC, at least, there was no doubt that Obama had the win after Ohio — Olbermann said something like “check my addition, but if Obama wins these states that he is virtually certain to win, he has over 270, right?”
December 2nd, 2008 at 5:45 pm
LaurencePassmore:
I was referring to 538’s final EV projection, which I believe was 346-347. Princeton Election Consortium, and other “hobbyist” sites with a little more mathematical rigor, were at or around 364.
Like I said, any one of us could have gone to Pollster on election eve and quickly determined that there were three toss-up states and the rest was pretty much writ in stone. That means there were 3*2=6 possible EV outcomes (not including Nebraska funkiness obviously) unless something crazy happened on the tails… so it wasn’t super hard to guess right (especially with Indiana and Missouri being worth the same amount of EV’s)… yet Nate Silver’s very very complicate model, with all sorts of parameters to tune, hit it on the low end. I would call that “under performance”.
December 2nd, 2008 at 5:46 pm
To be fair, the two dates I picked are, I’m fairly certain (but not sure, see my complaint immediately above), the lowest numbers the model ever projected for Obama, and the model moved away from those low numbers four or five days faster than Electoral-vote did.
December 2nd, 2008 at 5:47 pm
You’re claiming that one thing that Nate did was not that impressive because a lot of people also did it and it’s not that hard to do.
Even if that’s true, he did so much more. He predicted the Senate races, and the only one he had as being close was . . . dunh du duh . . . Minnesota.
He called the states almost entirely accurately. His closest state? Missouri? He was far more confident in Nev, NC, and Colo. than anyone else.
But the most important and valuable thing he did was teach me about polling. He did pull the curtain back. I feel like a polling expert and now have the confidence to completely ignore the daily changes in the horse race.
Nate rocked and his book(s) will sell way more copies than HITS. Sorry. Americans love people who can take complex things and make us feel like we get them.
December 2nd, 2008 at 5:56 pm
MikeF, how did Silver butcher the Minnesota recount?
December 2nd, 2008 at 6:06 pm
@J.W. Hamner: Three toss-ups implies 8 different outcomes, not six (it’s 2*2*2, not 2*3). Also, it’s not enough to say that one could have predicted the outcome on the eve of the election- the site was up (and largely spot on) for a long period of time.
@David Shor: I don’t know how to get historical predictions from 538, and perhaps you have a point about tone vs. prediction.
That said, I think you’re missing something when you say Nate’s model “tracked” Pollster.com. Of course it did- both 538 and Pollster are ultimately poll aggregators relying on the same underlying data, so you wouldn’t expect them to be wildly different. Nate’s model distinguishes itself in how it weights and interprets the various polls. These additions are less useful when the pollsters do a good job (e.g. during the november election) because here even a simple averaging will do; they are immensely useful when the polls are widely off, as during the primaries.
December 2nd, 2008 at 6:15 pm
All you need to know about 538 is that Petey thinks Nate Silver is brilliant. Nuff said.
More seriously, there’s no evidence that all the bells and whistles of Silver’s model actually added any value to predicting the election results. How did he compare to unsophisticated polling aggregators? Answer: he did basically the same, they all predicted the election more or less correctly.
Fanboys will counter that Silver was “way ahead of the trend” but as pointed out above this is inaccurate: like everyone else, Silver was projecting a McCain victory in mid-September. When the polls turned against Obama, the model’s bells and whistles weren’t smart enough to figure out “this will change in a few weeks.”
So you have a complex, elaborate model that doesn’t do any better than simpler, more transparent models. What’s the point? The point is that the fancy model gives distant observers the impression that brilliant and important work is going on behind the scenes. And that makes us all feel better, right?
December 2nd, 2008 at 6:17 pm
At the bottom of Obama’s post RNC polling collapse, on 9/16, Nate’s model was projecting 250.3 Electoral votes for Obama, and 249.7 on 9/17.
Right, but it seems unfair to note that Nate was also saying that the polling around the conventions was going to be unpredictive because of convention bounces, and that he had a model for how the overlapping bounces would eventually fade, which IIRC was pretty accurate.
December 2nd, 2008 at 6:19 pm
DaveMB:
I know the networks have an agreement not to jump the gun.
However, it is absurd to lie to the audience.
Bob Sheifer(sp?) said something very good that night. he said something like “it is going to be extremely difficult for McCain. He will have to win some states where he didn’t campaign. I just don’t see it happening.” Sorry for the terrible paraphase.
If people like Nate Silver can use common sense and the networks are not allowed to use common sense then the networks are in trouble.
December 2nd, 2008 at 6:21 pm
They pretty much called it very early on CNN as well. For show ,after OH was called, the commentator gave McCain every state left except for CA, WA, OR and HI and he was still short of 270. It was pretty much over after Obama won PA with a double digit lead, but McCain still had a “chance” with OH. However, hardly anybody could see PA going for Obama so heavily and OH + every other battleground state going for MCCain.
December 2nd, 2008 at 6:35 pm
The primary was the key. In the primary, Nate did something truly novel- using real voting results from other primaries and demographic regressions, he largely outperformed the polls. Because he had real results to work with he didn’t need to worry about calibrating turnout models, non-random samples, etc. In effect the super Tuesday states were a 20 million person exit poll for future states, just needing demographic reweighting.
In the general, what he was doing wasn’t as novel. But lacking states running their elections before anyone else, I’m not sure what you wanted from him.
Also, in terms of simple/complicated- normally a few variables do provide most of the explanatory power. You’re never going to see proof of a better model one way or the other in a single draw. Nate himself offered a distribution, and 1 in 100 draws McCain would have won, for goodness sake. Way too many people, either here or on TV, put the emphasis on the result that occurred, rather than the distribution.
December 2nd, 2008 at 6:38 pm
Perhaps that is because we don’t get the benefit of running the election 10000 times, so there’s absolutely no way to check if McCain had a 1% chance of winning or a 4% chance of winning.
December 2nd, 2008 at 6:41 pm
Or in other words, the idea that Silver’s model is better than simpler models because he had assigned *some* probability to the event of McCain winning is nonsense. In this election, the distribution was basically useless, except to give people the impression of sophistication.
I didn’t follow the primaries so it’s possible he did very good work there.
December 2nd, 2008 at 6:46 pm
He predicted the Senate races, and the only one he had as being close was . . . dunh du duh . . . Minnesota.
Right, his predictions were badly off on how close Oregon and Alaska would be.
December 2nd, 2008 at 6:52 pm
Neil,
The rules force them to talk in code like that, they all made it very clear as soon as Ohio was called that is was over, they just couldn’t say it explicitly.
On MSNBC especially they were obvious about it.
December 2nd, 2008 at 6:56 pm
Right, but it seems unfair to note that Nate was also saying that the polling around the conventions was going to be unpredictive because of convention bounces, and that he had a model for how the overlapping bounces would eventually fade, which IIRC was pretty accurate.
Matt, I thought he was saying that he was changing the sensitivity of the model to new results so that it wouldn’t be as badly affected by convention bounces, not that this was a reason to distrust the model.
the site was up (and largely spot on) for a long period of time.
This is not correct for most definitions of spot on and long periods of time. It was badly off six weeks out (almost everyone else was too), due to the lack of archives and the fact that I wasn’t writing down the prediction every day (just in the period during McCain’s RNC bounce) I don’t know when the model got close to being spot on, but I don’t think it was more than a week or two in advance.
December 2nd, 2008 at 6:58 pm
Washerdreyer-
He predicted OR 56.5 to 41.8. Actual: 57 to 41.
He predicted AK 42.2 to 56.4. Actual: 60 to 38.
I don’t either of those can be considered “badly off.” This is especially true considering the fact that there wasn’t much polling in Alaska.
December 2nd, 2008 at 6:59 pm
Er, AK was 38 to 60.
December 2nd, 2008 at 7:03 pm
Ray: I’m sure washerdryer was referring to the Senate races, which were very close. And yes, 538 gave the Dems a 90% chance of winning each seat.
December 2nd, 2008 at 7:09 pm
David:
Silver’s EV distribution was… ugly… to say the least, and he wasn’t the only one who constructed one. The decision to Monte Carlo it, fantasy baseball style, made for a lot of the messiness… but a lot of that mess also appears to come from his assumptions.
December 2nd, 2008 at 7:23 pm
Barbar is correct, Ray. Why would I quote what you wrote about the Senate races if I were talking about Presidential election? And now that you mention it:
He was far more confident in Nev, NC, and Colo. than anyone else.
There are two problems with this. One, the model, on the final day, gave North Carolina to Obama 59% of the time. That’s not very confident. Two, I don’t know who counts as “anyone else,” and it is relevant at what point in time the predictions were made, but tons of people projected Obama wins in all those states. Here’s a large collection of predictions, and of course all of pollster, RealClearPolitics, and electoral-vote were predicting wins in those states.
December 2nd, 2008 at 7:46 pm
There were several important things that 538 provided.
1. A realisting look at Obama v. Hillary chances in a race against McCain. It took the whole “Big States” myth to task.
2. A great plot of the expected historical convention bounce, and Nate’s prediction of what would happen with the conventions were back to back. His graph was dead on! Which I think contradict the whole Sarah Palin frenzy. The bounce was predicted by Nate exactly prior to Palin ever being chosen. Maybe McCain wouldn’t have had the traditional bounce without her, but never the less, it was nothing more than traditional.
3. A good comentary on which polls were worth taking seriously, instead of getting in a frenzy about outliers.
December 2nd, 2008 at 7:49 pm
What drew me to the 538 site was not just the predictions and daily numbers but the commentary, which gave me a lot of insight into election polling and regression analysis as tools. I think it is an intelligent site, and not so predictable as many, in part because of the contrast between Nate and Sean as partners. The campaign site visits gave us Sean’s experience of the two campaigns, another strong contrast. They were partial (and competent) without ever letting ideology take control (like Obama?).
December 2nd, 2008 at 10:33 pm
@eriks,
There was a point in the recount – I don’t remember the exact date – when Silver used a certain tool (binomial distribution?) to predict, based on the partial recount results, that Franken would probably win. A professional statistician (forget who, sorry!) jumped immediately on it saying that the methodology was not optimal and Coleman really had the edge. I was skeptical, but the results seem to be backing him up.
December 2nd, 2008 at 10:48 pm
@eriks again,
Here’s the 538 projection I was talking about:
http://www.fivethirtyeight.com/2008/11/projection-franken-to-win-recount-by-27.html#comments
I still don’t remember where I saw the contrarian stats model, but this is a pretty good contemporary response:
http://election.princeton.edu/2008/11/24/statistical-malpractice/
The response probably holds equally true to the contrarian model I mentioned but still can’t find. Point is, it was not an insightful prediction, but it was presented in a headline-grabbing manner (I think it was on the front page of HuffPo).
December 2nd, 2008 at 11:26 pm
how did Silver butcher the Minnesota recount?
By my count he has now changed his projection three times.
December 3rd, 2008 at 12:51 am
I’ll take Nate’s analysis anytime. Matt, you’re a fun read if not always angling for things that make your life easier (I want a restaurant on MY street! Let’s sell the forest service lands “out there” and build electric buses that won’t wake ME up!), but you need a proofreader and a trip outside your mid-Atlantic cocoon.
December 3rd, 2008 at 9:30 am
I also don’t mind the changing predictions, but thanks for the links, MikeF.
December 3rd, 2008 at 10:55 am
@21: The first tab of that document looks OK, but the other tabs have serious problems. Look at Ohio for instance – all three poll aggregators are within a percent of the actual result on the first tab, but click on “States by projection error” and you see them all off by huge margins.
Furthermore, all of 538’s biggest errors are underestimating the size of Obama’s margin in states where he was already winning crushing victories – this has no effect on the outcome. I think the pattern is the same for the other accumulators.
Overall, what I take away from that chart is that 538 and the other aggregators understate huge margins when they are present. Since they are all working from the same raw data the obvious hypothesis is that pollsters won’t release a poll that shows DC going 93-6, even though it actually did. Very unbalanced states are also underpolled because everyone knows who will win them.
If and when Nate does a postmortem on the accuracy of different pollsters and aggregators, I wonder if he’ll look at this (apparent) tendency to call states closer than they actually are.
December 3rd, 2008 at 12:29 pm
I agree with Harriet-J. The road trip aspect of 538 gave a comparison of Obama and McCain’s ground game and was a real indication of why McCain couldn’t catch up: No GOTV.
I think 538 was less sterile than other polling sites, and showed a lot of the hard work and humanity of campaigns and voters. It really should be said that Brett Marty took some of the most compelling photographs on the trail.
Some people went to polling sites to scan. I went to 538 to read.
December 3rd, 2008 at 3:02 pm
Yeah, I actually like 538 quite a bit and I still do. (Notice that everybody’s talking about the site in the past tense?)
I just think the proclamations about what a genius Nate Silver is are a bit overblown.
December 3rd, 2008 at 3:26 pm
“I was referring to 538’s final EV projection, which I believe was 346-347. Princeton Election Consortium, and other “hobbyist” sites with a little more mathematical rigor, were at or around 364.”
JW Hammer,
Actually, both Matt’s EV projection and the Princton Election Consortium’s EV projection were the same. They both called 49 out of 50 states correctly, they both called Indiana incorrectly, and they both failed to predict Obama’s win in Nebraska’s 2nd Congressional district. When you add up the electoral votes of the states (and the District of Columbia) they predicted for Obama, both were predicting an electoral college win of 353. Since they both failed to call Indiana and Nebraska’s 2nd Congressional District, they were both off by 12 EVs.
So while there was a signficant difference between them in the average or expected number of EVs predicted for Obama, in terms of the states they actually called for Obama, both were equally correct. This is the most significant metric to judge them by, since the actual Electoral College vote totals are not detemined by mathematical averages of aggregate polling results. They are determined by the combination of states that you win in. If the combination gets you to 270 or above, you win.
December 3rd, 2008 at 4:21 pm
@21: The first tab of that document looks OK, but the other tabs have serious problems. Look at Ohio for instance – all three poll aggregators are within a percent of the actual result on the first tab, but click on “States by projection error” and you see them all off by huge margins.
Thanks, fixed and data updated. New numbers decreased how much 538 was off on Alaska Presidential vote.
If and when Nate does a postmortem on the accuracy of different pollsters and aggregators, I wonder if he’ll look at this (apparent) tendency to call states closer than they actually are.
If Nate did this I would immediately have almost no complaints about the site. It is possible that he’s waiting for the electoral count act safe harbor deadline of 12/12 (everyone remember their Bush v. Gore litigation?) to pass so that he has final numbers for each state before doing so.
December 3rd, 2008 at 8:33 pm
I don’t have a dog in this fight, really, but would like to point out that there’s a whole lot of hindsight bias going on in the comment section. At the time of the election, pretty much everyone I knew, Republicans and Democrats alike, were saying stuff like, “Oh, you can’t trust polls, they’re all over the place.” Very, very few people would have thought that a “poll of polls” would be able predict the popular vote within 0.1%. But now, thanks to the polling aggregators and, especially, Nate Silver– who may not be a “genius,” but is probably a little better than everyone else and a lot better at marketing himself and his techniques — we won’t be hearing a whole lot of that next time. That may not be groundbreaking, and if I were Matt Y., I’d probably be jealous of all the attention/cable TV appearances, too, but that’s a pretty significant contribution to the process.
December 3rd, 2008 at 9:07 pm
What I most admire about Silver’s approach is the way he tries to bypass the cloud of rhetoric spewed by pundits and bloggers and actually try, by statistical analysis, to analyze real data. I love the fact that much of what he says is based on analysis of data, not statements by bloviators.
This, rather than the fact that he “got it right” on the vote, is what makes him exceptional.
December 4th, 2008 at 1:09 pm
Matt’s missing the point:
Before the election, most people didn’t realize that John Zogby was crap, that ARG was even more crap, and that IBD/TIPP rigs their samples.
Before the election, few people knew Scott Rasmussen, and now he’s (rightly) regarded as top of the heap.
Before the election, networks reported their own polls as gospel.
Before the election, RCP’s poll averages were regarded as the gold standard, even though they were biasing their samples by including and excluding polls at a whim.
December 9th, 2008 at 12:24 am
this might have already been mention in previos comments, but anyway: the reason 538 and nate in particular became so popular was not that he was able to calculate the average of all the polls (for that, we had pollster.com).
rather, it was about the fact that there were millions of neurotic obama supporters (i was one of them) who obsessed about any sign of “tightening” polls, or whatever the paranoia-of-the-week was about. and nate was always, always, there to explain why we, in fact, did not have a reason to worry.
Oh, how I remember totally freaking out about that post RNC-convention ABC poll from early September (poll nostalgia, that’s a first?) showing McCain/Palin up by 10 percent. But, yup, Mr. Silver was there, just a couple of hours later, to explain why we needn’t worry, why the poll was an outlier.
He was a comforting, soothing, reasonable, and very very reliable dork in an ocean of loud, stupid talking heads on TV.
December 9th, 2008 at 12:26 am
“He was a comforting, soothing, reasonable, and very very reliable dork in an ocean of loud, stupid talking heads on TV.”
come to think of it, this also applies to matt yglesias.
and barack obama.
also chuck todd.
January 19th, 2009 at 11:35 pm
laptop battery
laptop batteries
March 11th, 2009 at 4:22 am
Excellent site. It was pleasant to me.
March 14th, 2009 at 5:03 am
Excellent site. It was pleasant to me.
xanax
March 22nd, 2009 at 5:59 am
tramadol
Very interesting site. Hope it will always be alive!
April 2nd, 2009 at 5:06 am
Great site. Good info
buy cheap viagra
April 9th, 2009 at 5:06 am
It is the coolest site,keep so! viagra