Another spring, another publication of the World Economic Outlook by the International Monetary Fund (IMF). Twice a year, the IMF evaluates the state of the world economy and presents forecasts of future economic growth, leading to news headlines such as this one and this one. The nice thing about specific predictions for the future is that, once that future has arrived, you can have another hard look at those predictions and learn how much, or little, sense they made.
The following post is an attempt at exactly that: to take a hard look at the IMF forecasts of the world's GDP growth. Why? Because churning out these forecasts is serious business that is taken very seriously when economic policies are set, so it’s fair to ask how accurately these forecasts have historically reflected real global GDP growth. This post is also an exercise in digesting a set of numbers that says something about the world and trying to squeeze out as much information as possible.
Accuracy
Let’s start at the beginning and have a look at the most basic graph possible for this excercise: a timeline comparing the real world economy growth with the IMF predictions made 5 years earlier. Ignoring the murkiness of the standard deviation, I’m choosing to use the (median) absolute deviation, or simply the difference between forecast and real growth in absolute values (percentage points in this case).
Timeline of real GDP growth and forecasted GDP growth
Making forecasts on the state of the world economy 5 years from now is obviously ridiculous when you judge it on accuracy at the scale of individual years: not a single forecast on the growth of the world economy made between 1990 and 2014 actually turned out to match reality.
Of the 40 forecasts made between 1990 and 2014, 18 underestimated the actual growth while 22 overestimated it. This may seem a pretty even over/under split, but the averages with which the forecasts were off show something else. On average the overestimations were way more profound than the underestimations:
Difference real GDP growth and forecasted GDP growth in percentage points
When the IMF was overly optimistic of the direction of the world economy it was way more off the actual numbers than when it was overly pessimistic. When the forecast growth percentage turned out to be higher than the actual number, it was, on average, 1.65 percent point off; when it turned out to be lower than the actual number it was off on average by 0.63 percent point.
This points to the most glaring problem with the forecasts: they tend to be overly bullish and have completely missed every crisis the world economy suffered the past decades. For 1998 and 2001, when growth took hard hits, and for 2009, when the world economy actually shrunk, the predictions 5 years before had been as rosy and steady as ever. “Sure!” you’ll say, “how could anyone have known beforehand?” Well, exactly.
Variance
The suspicious steadiness of the forecasts can be shown in numbers, by calculating its variance. Variance is a measure to show how far a set of numbers is spread out from their mean: the higher the variance, the greater the spread.
Plotting the results for our sets of numbers shows real life GDP growth has a variance of 1.97. The 5-year forecast doesn't compare with a variance of 0.17; nor do the 4-year, 3-year or 2-year forecasts. The forecast from 1 year before is the first to break the monotony with a variance of 0.38, but only the same-year forecast numbers have a variance that approaches reality with 1.82.
Variance for real and forecasted GDP growth
Trendlines
So, let’s forget about the accuracy of a single forecast for a given year for a moment and let’s focus on the trend lines: how do the general trends of the forecasts compare to the real growth trend?
Linear trendlines of real and forecasted GDP growth
All of the forecasts’ trends are upwards, showing higher and higher growth, and so is the actual trend, so at least that’s sort of accurate!
But the forecast trends are still way off: while the 5-year out forecast trend line hovers between 4% and 5% of GDP growth, the trend line of the real GDP growth stayed between 3% and 4%. When 18 years worth of your predictions turn out to be consistently off by 25% from the real deal, it's hard to claim success in accuracy. The trend lines do tend to get closer to the real growth trend when they predict less far into the future. Note how the real growth trend line even intersects with the trend line of the 1-year out forecasts.
Interestingly, the trend lines clearly show how each of the forecasts overestimate the world’s GDP growth, except for the same-year forecasts: this is the only forecast that is actually more pessimistic than the actual growth numbers turn out to justify. So, to summarise that last point, the general IMF feeling is “The near future is gonna be better, it's gonna be awesome, it's gonna be great, it's gonna be fantastic, it's gonna be grea-- Nope, this year's gonna suck!”
Correlations
At least two sorts of correlations came to mind that the dataset could be checked for:
First of all, I've checked the correlations between each of the forecasts with the real growth numbers. Scatterplots can do an excellent job at showing the correlation (or lack thereof) between the forecasts and real growth, “in all its messy glory”. As the two scatterplots (below) show, the same year predictions tend to correlate pretty well with the real growth -- you could call 1999 and 2010 outliers but overall you see a nice linear line tracking from bottom-left to top-right.
What they also show is that the predictions from 5 years out are all over the place. There is a cluster of forecasts that correlate somewhat OK with real GDP growth but the majority of forecasts are grouped way off to the middle-right or middle-left. On top of that, there were the 5-year-out forecasts for 2009: those were off so far they're almost literally off the charts.
Correlation real GDP growth and forecasts from the same year
Correlation real GDP growth and forecasts from 5 years earlier
The second type of correlation to be checked is between a forecasts’ offset from real growth and the number of years it looks ahead -- a hypothesis that seems fair to check is "Forecasts perform worse when they predict farther into the future." This seems to be confirmed by the data, in general, but there are some surprising exceptions.
The results show that for 31 of the 40 forecasts there was a positive correlation (sometimes only mild but often very strong) between how much the forecasts were off from the real figures and the number of years that they looked ahead.
The other 9 forecasts however showed some level of negative correlation, meaning the predictions made 5 years earlier turned out to be more accurate than the same-year predictions. Four years in particular stand out with a strong negative correlation: 2000, 2003, 2005 and 2010. Given the small size of this sample and the always-present likelihood of seeing random noise for something meaningful, this probably says nothing. (But, note to self: find out if there was anything different about these years!)
Correlation "forecast further out" with "forecast accuracy"
Tiny Case Study
The last 30 odd years have seen at least 3 major crises that resulted either in a decline in economic growth (in 1998 and 2001) or in shrinking of the world's GDP (in 2009). It is worth zooming in on these 3 events and see how the forecasts performed. How did they anticipate these crises? And how did they respond during the aftermath and the recovery?
Comparing the 5-year out forecasts here is a waste of time as they failed to show any awareness of these real world 'anomalies'. So we'll take a look at the forecasts made in the same year as the real GDP growth numbers refer to. Looking at the years leading up to each of the declines, there is a consistent underestimation apparent in the forecasts. All same-year forecasts made in 1997/98, 2000/01 and 2008 were higher than real GDP growth. At the bottom of the real GDP growth drop however, the forecasts for 1998 and 2009 overestimated how deep the fall would be. And for all 3 events, the forecasts lag behind the real economic recovery. So, the tendency is to underestimate both the economic decline about to occur, and the recovery that follows it.
Timeline 2006-2010
Timeline 1996-2002
Take Aways
Predicting the future in precise numbers is really difficult – shocker! Predicting the future five years from now, in precise numbers, and interpret those numbers to have some real predictive value seems…absurd (don’t do that!).
The predictive accuracy of the IMF’s GDP growth forecasts does tend to be better when made less far into the future. The forecasts made in the same year are far more accurate than those made 5 years earlier. Also, the overall trendlines of the forecasts do show a similar direction as the real GDP growth trend.
Economic predictions about the world’s GDP are overly optimistic way more often than overly pessimistic (except for predictions made in the same year; those are actually more gloomy than they should). This is a particularly dangerous bias seeing that the world’s GDP steadily climbs upwards but suffers regular major drops. We have proven to be blind to an impending drop in the world economy and are apparently left standing with a stupid celebratory smile on our faces when a crisis first hits. Then, after being caught off guard, the knee-jerk response is to overestimate the free fall and we tend to remain a step or two behind in correctly predicting the recovery.
Finally, it is worth noting that an economy is a feedback machine, that absorbs and responds to any information, including predictions about its performance. This means there are (probably many) feedback loops at work and the lines between “reality” and “prediction” should be considered pretty murky.