Controversy: Criticism of presidential forecasts
In a series of posts in 2011 and 2012, FiveThirtyEight criticized the forecasting methods that relied on macro-economic modeling of the electoral outcomes. According to Silver, models based primarily on the macro-level performance of the economy (such as unemployment, inflation, and the performance of the stock market), presidential approval ratings (when an incumbent is running for re-election), and the ideological positioning of the (potential) opposing candidates were, according to Silver, useful for making forecasts of the election outcome well in advance of election day, though not very precise ones.
An article stating such a position published exactly a year before the 2012 election day was attacked in an online article in Bloomberg News by Ron Klain, the former chief-of-staff to Vice President Biden and a political advisor to Barack Obama.
For many years, a group of political scientists, mathematicians and scholars have argued that a handful of factors determine of presidential elections, irrespective of the campaigns.
Most famous among those thinkers is Allan Lichtman, whose "13 Keys to the White House" model (which looks at factors such as incumbency, the outcome of the previous midterm election and per capita economic growth) has forecast the popular-vote winner in each of the last seven elections.
More recently, the brilliant data analyst Nate Silver has employed a three-factor model (presidential approval rating, economic outlook and opponent's ideology) to forecast the 2012 outcome under a variety of scenarios. At least implicitly, he, too, is suggesting that the campaign itself is irrelevant to the result of the election.
One immediate reason to be skeptical of the models' forecasting prowess is that they point in opposite directions: Lichtman has interpreted his keys to forecast that President Barack Obama will be re-elected in 2012, while Silver rates Obama's chances at less than 50 percent.
As described elsewhere in this article, Silver's actual model for predicting the outcomes of the 2012 election is more elaborate than the three-factor model that he used for making his long-term forecast, which set out a variety of scenarios whose electoral outcomes depended in part on who the Republican Party nominee would be. Silver, too, criticized models that rely only on long-term or underlying macroeconomic and macropolitical factors – which Klain refers to as extrinsic factors – to make predictions of the outcome, including Lichtman's model and that of other political scientists and economists who did not look at conditions that were more proximate to the election date, especially as reflected in the results of opinion surveys. However, Klain argued that intrinsic factors are critical to the outcome of elections. In short, campaigns matter, and campaign spending matters.
In a response, Silver began by stating,
Unfortunately, Mr. Klain's article attributes to me a number of views that I am ambivalent about or actively disagree with, so it deserves a fairly long reply. I will also use this opportunity to respond to some criticisms that I have been receiving from political scientists. The irony is that I agree with Mr. Klain more than he realizes.
But let's start with Mr. Klain's central question: how much difference does campaign strategy make in determining the outcome of presidential elections?
Do all the ads, speeches, mailings, debates, online activity and rallies really change minds? Or is the outcome of the election the product of underlying fundamentals that are scarcely affected by such efforts?
This is obviously something of a false juxtaposition. It is extremely unlikely that campaigns don't matter at all. Now and then, you'll see a political scientist come fairly close to expressing this viewpoint, but that is certainly not the majority opinion within the discipline. The question, instead, is how much campaigns matter, and that is a difficult question to answer.
I strongly agree with Mr. Klain that political scientists as a group badly overestimate how accurately they can forecast elections from economic variables alone. I have written up lengthy critiques of several of these models in the past, which suffer from fundamental problems regardless of which variables they choose.
One of the things it took me a long time to learn about forecasting is that there's a difference between fitting data to past results and actually making a prediction. A regression model built from historical data is really just a description of statistical relationships that existed in the past. The forecaster hopes or assumes that the relationships will also apply in the future, but there is often a significant deterioration in performance..... Presidential forecasting models that rely on economic data are likely to be especially susceptible to these problems. Most of them are fit to data from a small sample of 10 to 20 past elections but have a choice of literally hundreds of defensible economic or political variables to sort through.
A more tangible question is how well economic statistics alone can really predict elections. I have written previously that a good assumption is that they can explain perhaps 50 percent of the results. But based on some further research that I will soon publish, I suspect that estimate was too high, and that the answer is more like 30 or 40 percent when the models are applied to make real, out-of-sample forecasts. Economic variables that perform better than that over a small subset of elections tend to revert to the mean or even perform quite poorly over larger samples.
So say that 60 percent of elections cannot be explained by economic variables. Should all of the remaining credit go to campaigns?
No, of course not. First, the fact that widely available published economic statistics cannot explain more than about 40 percent of election results does not mean that the actual living and breathing economy cannot....
After further discussing how and why campaigns and candidates do make a difference, Silver concluded:
I apologize if some of this seems prickly. I lived through the Moneyball wars in baseball and then saw how much progress the sport made once everyone learned how much they had in common.
Baseball games, however, are played 162 times per year, so the learning process is accelerated. But presidential elections are held only once every 4 years, and we make the same mistakes over and over again. The outcome of the election isn't especially predictable right now, but here are four predictions you can take to the bank:
1. Next year, the strategists of the winning campaigns will be praised as brilliant.
2. Next year, the strategists of the losing campaign will be blamed for a long series of mistakes.
3. Next year, some of the political science models will hit the outcome right on the nose.
4. Next year, some of the political science models will miss wildly in one direction or another.
Silver's response was followed by another one from Klain: "Respectfully, Silver Is Still Wrong," as well as by comments from others on Silver's article and the debate with Klain.
^ Ron Klain, "Why Data Wonks Are Wrong About Presidential Elections," Bloomberg, November 14, 2011.
^ Ron Klain, "Respectfully, Silver Is Still Wrong," Bloomberg, November 17, 2011.
^ Micah Cohen, "Reads and Reactions," The New York Times," November 19, 2011; John Sides, "Underemphasized Points about the Economy and Elections," The Monkey Cage, November 18, 2011; Alan I. Abramowitz, "Why Barack Obama Has a Good Chance of Winning a Second Term: And why Nate Silver may have underestimated his chances," Sabato's Crystal Ball, November 10, 2011