Silver explains a variety of reason why an inexact realtionship exists:
The problem is that whatever signal there is gets filtered through an awful lot of noise. Consider:
The unemployment rate itself is subject to fairly significant measurement error.
Voters will interpret the unemployment rate in different ways, and assign the president varying amounts of credit or blame for it.
The unemployment rate is but one of a number of salient economic indicators.
Economic performance is but one of the ways that voters evaluate a president.
Voters’ evaluation of a president is important, but they also consider the strength of a president’s opponents, including third-party alternatives in some elections.
If you could hold each of these other factors constant, you could come to a more confident conclusion about how much each tick in the unemployment rate affects Mr. Obama’s re-election odds. But the real world is not set up with these sorts of experiments in mind, and since presidential elections are infrequent, the likelihood that truly comparable cases will exist in the historical data is relatively low.
http://fivethirtyeight.blogs.nytimes.com/2011/06/02/on-the-maddeningly-inexact-relationship-between-unemployment-and-re-election/