Edited By Ulrich Fritsche, Roman Köster and Laetitia Lenel
Few areas in economics are as controversial as economic forecasting. While the field has sparked great hopes for the prediction of economic trends and events throughout the 20th and 21st centuries, economic forecasts have often proved inaccurate or unreliable, thus provoking severe criticism in times of unpredicted crisis. Despite these failures, economic forecasting has not lost its importance. Futures Past considers the history and present state of economic forecasting, giving a fascinating account of the changing practices involved, their origins, records, and their implications. By bringing together economists, historians, and sociologists, this volume offers fresh perspectives on the place of forecasting in modern industrial societies, thereby making a broader claim for greater interdisciplinary cooperation in the history of economics.
Continuities and Discontinuities in Economic Forecasting: Tara M. Sinclair
Tara M. Sinclair (George Washington University)
“The cost of a recession is so great that a forecaster should never miss one.”
Herman O. Stekler (11/4/1932 - 9/4/2018)2
Abstract: Throughout the history of macroeconomic forecasting, several major themes have remained surprisingly consistent. The failure to forecast economic downturns ahead of time is perhaps the most significant of these. Forecasting approaches have changed, but forecasts for recessions have not improved. What can we learn from past evaluations of macroeconomic forecasts? Is it possible to predict major economic shocks or is it a fool’s errand? This chapter discusses how forecasting techniques have evolved over time and yet the record on forecasting recessions remains dismal. There are several competing hypotheses for why forecasters fail to foresee recessions, but little evidence any of them are going to be addressed before the next recession occurs. This suggests planners and policymakers should expect to be surprised by the arrival of downturns and develop ways to be prepared for recessions without having clear warning of their coming.
Keywords: Forecast evaluation, recessions←31 | 32→
One of the key tenets of economic theory is that decisions are forward-looking. Economic forecasts are used in all sorts of planning and in particular are relevant for policymakers who are charged with preventing recessions. Unfortunately the record is not good for forecasting recessions. Ahir and Loungani (2014) and An, Jalles, and Loungani (2018) document that forecasters have a poor record of predicting recessions across countries and for both private and official sectors. Does this poor record reveal that forecasting recessions is a fool’s errand? Or are improvements possible? This chapter provides an overview of the historical record of economic forecasting focused on recessions, considers some common explanations of this poor record, and discusses the implications of this record for future planning and policymaking.
2. The Historical Record of Economic Forecasting
Economic forecasts before and through the Great Depression tended to focus on qualitative predictions: would the economy get better or worse? This qualitative nature made them difficult to evaluate until textual analysis approaches were introduced by Goldforb, Stekler, and David (2005) and Mathy and Stekler (2017). According to Hardy and Cox (1927), three common forecasting methods were used: (1) a “cross cut” approach of judgemental comparison and weighting of positive and negative news, (2) modeling the economy as following a regular rhythm, and (3) forecasting by analogy, comparing current events to past events to predict future outcomes. These forecasting approaches led to what Goldfarb, Stekler, and David (2005) called “egregious errors” where forecasters in 1930 predicted 1931 would show a recovery in the U.S. Instead the economy contracted for two more years.
Many new forecasting techniques have been introduced since the Great Depression. They have been predominantly quantitative and have focused on continuous rather than binary or directional forecasts. Two broad camps have evolved over time: (1) theory-based and (2) data-driven. The theory-based approaches started with large-scale macroeconomic models which have since been replaced by Dynamic Stochastic General Equilibrium (DSGE) models. The appeal of theory-based models is that ←32 | 33→they provide structure and stories to explain the patterns in the forecasts. They can also be used to analyze the impact of different proposed policies on the forecasts. In terms of forecast quality, however, the theory-based models typically cannot out-perform simple benchmarks such as autoregressive models (Chauvet and Potter 2013).
Data-driven approaches have focused on mostly time series econometric models such as autoregressive integrated moving average (ARIMA), vector autoregressive (VAR), and factor models of various kinds. New techniques are being developed now using “Big Data,” machine learning, and artificial intelligence. Data-driven approaches, however, cannot consistently beat judgemental forecasts, particularly the average forecast from forecast surveys (Ang, Bekaert, and Wei 2007).
With all the advances in forecasting techniques, it would be reasonable to expect that forecasts would have improved over time. Unfortunately there is little evidence that there has been substantial improvement, particularly if focused on predicting recessions. The Global Financial Crisis of 2007–2008 and the associated Great Recession took economic forecasters by surprise. Culbertson and Sinclair (2014) document how both private sector forecasters and policymakers completely failed to predict the Great Recession in the U.S. And this is not just a U.S. story. In a response to a question from Her Majesty the Queen of England about why everyone missed the Global Financial Crisis, Besley and Hennessy wrote: “the exact form that it would take and the timing of its onset and ferocity were foreseen by nobody” (2009, page 8). Ahir and Loungani (2014) found that around the world, none of the 62 recessions in 2008–2009 was predicted by September of the previous year by the consensus of professional forecasters.
For an example of how economic forecasts perform around recessions, Fig. 1 presents a graph of U.S. real Gross Domestic Product (GDP) growth and the median of the four quarter ahead forecasts for US real GDP growth from the Philadelphia Fed’s Survey of Professional Forecasters (SPF). Median forecasts from surveys, particularly from the SPF, tend to out-perform other forecasting methods (see Ang, Bekaert, and Wei 2007; similar results are true for Europe using the European Central Bank SPF, see Genre et al. 2013). Fig. 1 shows that the forecasts perform fairly well outside of recessions, but there is little to no anticipation of a downturn a year in advance of recessions.
Fig. 1: U.S. Real GDP Growth and 4 Quarter Ahead SPF Forecasts
Perhaps a year ahead is asking too much of forecasters, but it is a relevant horizon for planning and policymaking. Even if we look at forecasts just one quarter ahead, forecasters miss the arrival of the downturn in the next quarter, although once in a recession they do adjust their forecasts downward. They consistently miss the turning point and the depth of recessions, however, even at this short horizon, as can be seen in Fig. 2.
Fig. 2: U.S. Real GDP Growth and 1 Quarter Ahead SPF Forecasts
One interpretation of these figures is that forecasters focus on predicting normal times and ignore recessions, at least until the recession has arrived. In their study of 19 advanced economies, Dovern and Jannsen (2017) provide evidence that forecasters produce forecasts that are unbiased conditional on being in an expansion and therefore neglect recessions in their models and forecasts. Fildes and Stekler (2002) similarly conclude that forecasters are better when economic conditions are relatively stable. This might reflect the standard training for economists to fill in the status quo when other information is not available. Forecasting recessions may therefore still be out of reach for our existing models and knowledge. There are however, various potential reasons why forecasters consistently miss recessions, described in the next section.←34 | 35→
3. Why Do Forecasters Miss Recessions
A number of different explanations have been put forward as to why forecasters consistently miss recessions. Some suggest we need better models or better/more timely data sources. Others suggest that falsely predicting a recession when one does not occur is much worse than missing a recession entirely, which explains why forecasters are conservative in forecasting recessions. Still others suggest that by their very nature recessions are inherently unpredictable.
In a sense these explanations range from optimistic to completely pessimistic. The solution in the case of poor models, methods, or data is to invest further in these directions. The new methods and data sources coming from the Big Data revolution may help us to forecast future recessions. Historical experience, however, tempers this optimism since there have been substantial improvements in these directions to date without noticeable improvement in forecasting recessions.
Despite the under-prediction of recessions, it is still a common joke that forecasters over-predict recessions, which suggests economists are very sensitive to over-prediction. For example, Paul Samuelson said in ←35 | 36→1966 that the stock market predicted nine of the past five recessions.3 This might mean that forecasters could predict recessions, but they do not have the right incentives to do so. If it is an issue of forecaster reputation, where predicting a recession when one does not occur is more costly than missing one entirely, then we might see forecasters only slowly respond to new information, particularly around downturns. This might look like forecasters are smoothing their predictions over time (Nordhaus 1987). But recent research suggests forecasters sometimes over-respond to new information, not always smoothing (e.g. Azeredo da Silveira and Woodford 2019; Bordalo et al. 2018; Messina, Sinclair, and Stekler 2015; Dovern and Weisser 2011). Similarly we might expect forecasters to herd, i.e. to produce forecasts similar to their peers, to protect their reputation. Rülke, Silgoner, and Wörz (2016), however, find evidence of anti-herding across an international set of business cycle forecasters, particularly in times of increased uncertainty. These findings suggest that even if we could find a way to change forecaster incentives around predicting recessions, that may not improve their record on forecasting recessions.
Thus we are left with the most dismal explanation, that recessions may be caused by purely random shocks, which by their nature are impossible to forecast (e.g. Drautzburg 2019). Consistent with this explanation, we see forecasters adjust their models after a downturn so that they would have better predicted the past, but do no better at predicting the future. This was particularly obvious after the Global Financial Crisis where forecasters added financial and housing sectors into their models so they would have been able to forecast the Great Recession with those models. Only time will tell if these improvements help predict the next recession.
To give a sense of the challenge facing forecasters, Fig. 3 provides an example using 3-month decline in the industrial production index. This ←36 | 37→was a leading indicator originally proposed in the 1950s (Alexander and Stekler 1959; Stekler 1972).4 Every US recession identified by the NBER5 is signaled by this indicator in some way, but there are both false signals of recession and false indications of expansions. Perhaps the most disconcerting is that we miss the start of many recessions, not by much, typically just one to two months, but it suggests that we cannot breathe easy even when this indicator is in positive territory. Indicators are often maligned for falsely predicting recessions, but we might be willing to take some false signals if we consistently had an accurate prediction of the timing of recessions. Unfortunately no model, forecaster, or indicator has yet achieved that standard.
Fig. 3: Industrial Production Index and Recession Signal
4. Nowcasting Recessions
The record for identifying a recession once it’s occurring (nowcasting) is much better than predicting one even one quarter ahead. There is evidence that policymakers such as the Federal Reserve are able to identify recessions once they are in progress (Sinclair, Joutz, Stekler 2010). Giusto and Piger (2017) have shown that several approaches identify recessions in real time. These approaches provide faster identification of recessions than waiting for the NBER business cycle dating committee to provide official classification of the turning point, but for monetary policy with its long and variable lags, knowing a recession is occurring only in real time may be too late. There are however, other policies that might work in a world where we can only nowcast recessions.
For example, recently there has been much attention directed to the “Sahm rule” based on Claudia Sahm’s proposal (Sahm 2019) to use a 3-month moving average of the unemployment rate as a trigger for automatic stimulus payments. Sahm argues that an increase of 0.50 percentage points or more, relative to the unemployment rate’s low in the prior 12 months (in order to allow for changes in the natural rate of unemployment), has historically only occurred during or closely after recessions in the US. Thus this rule does not predict recessions, but it is a simple and useful one to trigger automatic fiscal stimulus. This sort of policy approach can quickly react to a recession as it is occuring to offset some of its impact even if we cannot predict recessions in advance.
The failure of forecasters to predict past recessions does not necessarily imply we will never be able to forecast recessions. It is possible that with further development of techniques and insights into the structure of the economy along with new and more timely data sources our forecasts will improve. But it is important for the public and policymakers to understand the current state of forecasting and not rely on predictions to prepare for downturns.
Despite advances in forecasting techniques, computational power, as well as data quality and quantity, forecasters continue to systematically miss recessions. Harding and Pagan (2016) advise that we should know ←38 | 39→the limits of forecasting and focus research instead on better understanding the business cycle rather than trying to predict it We may need to accept that nowcasting recessions is the best we can do and build policy plans with that information in mind. As we continue to develop new models and methods, deepen our understanding of the structure of the economy, and build the quantity and timeliness of data sources, we need to continue to heed Stekler’s (2007) advice for evaluating forecasters and remember that forecasters are responding to their own set of incentives that affect their judgement as well as the models they choose.
Ahir, Hites, and Prakash Loungani. 2014. “ ‘There Will Be Growth in the Spring’: How Well Do Economists Predict Turning Points?” VoxEU.Org. April 14. https://voxeu.org/article/predicting-economic-turning-points (last access: 8/5/2019).
Alexander, Sidney S., and Herman O. Stekler. 1959. “Forecasting Industrial Production--Leading Series versus Autoregression.” Journal of Political Economy 67 (4): 402–9. doi:10.1086/258200.
An, Zidong, João Tovar Jalles, and Prakash Loungani. 2018. “How Well Do Economists Forecast Recessions?” International Finance 21 (2): 100–21. doi:10.1111/infi.12130.
Ang, Andrew, Geert Bekaert, and Min Wei. 2007. “Do Macro Variables, Asset Markets, or Surveys Forecast Inflation Better?” Journal of Monetary Economics 54 (4): 1163–212. doi:10.1016/j.jmoneco.2006.04.006.
Azeredo da Silveira, Rava, and Michael Woodford. 2019. “Noisy Memory and Over-Reaction to News.” AEA Papers and Proceedings 109 (May): 557–61. doi:10.1257/pandp.20191049.
Besley, Tim, and Peter Hennessy. 2009. “The Global Financial Crisis – Why Didn’t Anybody Notice?” British Academy Review 14: 8–10.
Bordalo, Pedro, Nicola Gennaioli, Yueran Ma, and Andrei Shleifer. 2018. “Over-Reaction in Macroeconomic Expectations.” Working Paper 24932. National Bureau of Economic Research. doi:10.3386/w24932.
Chauvet, Marcelle, and Simon Potter. 2013. “Forecasting Output.” In Handbook of Economic Forecasting, Vol. 2, edited by Graham ←39 | 40→Elliott and Allan Timmermann, 141–94. Elsevier. doi:10.1016/B978-0-444-53683-9.00003-7.
Culbertson, Daniel S., and Tara M. Sinclair. 2014. “The Failure of Forecasts in the Great Recession.” Challenge 57 (6): 34–45. doi:10.2753/0577-5132570603.
Dovern, Jonas, and Johannes Weisser. 2011. “Accuracy, Unbiasedness and Efficiency of Professional Macroeconomic Forecasts: An Empirical Comparison for the G7.” International Journal of Forecasting 27 (2): 452–65. doi:10.1016/j.ijforecast.2010.05.016.
Dovern, Jonas, and Nils Jannsen. 2017. “Systematic Errors in Growth Expectations over the Business Cycle.” International Journal of Forecasting 33 (4): 760–69. doi:10.1016/j.ijforecast.2017.03.003.
Drautzburg, Thorsten. 2019. “Why Are Recessions So Hard to Predict? Random Shocks and Business Cycles.” Economic Insights 4 (1): 1–8.
Fildes, Robert, and Herman Stekler. 2002. “The State of Macroeconomic Forecasting.” Journal of Macroeconomics 24 (4): 435–68. doi:10.1016/S0164-0704(02)00055-1.
Genre, Véronique, Geoff Kenny, Aidan Meyler, and Allan Timmermann. 2013. “Combining Expert Forecasts: Can Anything Beat the Simple Average?” International Journal of Forecasting 29 (1): 108–21. doi:10.1016/j.ijforecast.2012.06.004.
Giusto, Andrea, and Jeremy Piger. 2017. “Identifying Business Cycle Turning Points in Real Time with Vector Quantization.” International Journal of Forecasting 33 (1): 174–84. doi:10.1016/j.ijforecast.2016.04.006.
Goldfarb, Robert S., Herman O. Stekler, and Joel David. 2005. “Methodological Issues in Forecasting: Insights from the Egregious Business Forecast Errors of Late 1930.” Journal of Economic Methodology 12 (4): 517–42. doi:10.1080/13501780500343524.
Harding, Don, and Adrian Pagan. 2016. The Econometric Analysis of Recurrent Events in Macroeconomics and Finance. Princeton University Press. https://muse.jhu.edu/chapter/1984459 (last access: 8/5/2019).
Hardy, Charles O., and Garfield V. Cox. 1927. Forecasting Business Conditions. New York: MacMillan.
Joutz, Frederick L. 2010. “Interview with Herman O. Stekler.” International Journal of Forecasting, Special Section: European ←40 | 41→Election Forecasting 26 (1): 195–203. doi:10.1016/j.ijforecast.2009.12.001.
Joutz, Fred, Prakash Loungani, and Tara Sinclair. 2015. “Honoring Herman: A Special Section for Stekler.” International Journal of Forecasting 31 (1): 93–95. doi:10.1016/j.ijforecast.2014.11.001.
Mathy, Gabriel, and Herman Stekler. 2017. “Expectations and Forecasting during the Great Depression: Real-Time Evidence from the Business Press.” Journal of Macroeconomics 53 (September): 1–15. doi:10.1016/j.jmacro.2017.05.006.
Messina, Jeffrey D., Tara M. Sinclair, and Herman Stekler. 2015. “What Can We Learn from Revisions to the Greenbook Forecasts?” Journal of Macroeconomics 45 (September): 54–62. doi:10.1016/j.jmacro.2015.04.002.
Nordhaus, William D. 1987. “Forecasting Efficiency: Concepts and Applications.” The Review of Economics and Statistics 69 (4): 667–74. doi:10.2307/1935962.
Rülke, Jan-Christoph, Maria Silgoner, and Julia Wörz. 2016. “Herding Behavior of Business Cycle Forecasters.” International Journal of Forecasting 32 (1): 23–33. doi:10.1016/j.ijforecast.2015.02.004.
Sahm, Claudia. 2019. “Direct Stimulus Payments to Individuals.” Brookings. May 16. https://www.brookings.edu/research/direct-stimulus-payments-to-individuals/ (last access: 8/5/2019).
Sinclair, Tara M., Fred Joutz, and Herman O. Stekler. 2010. “Can the Fed Predict the State of the Economy?” Economics Letters 108 (1): 28–32. doi:10.1016/j.econlet.2010.04.010.
Stark, Tom, and Dean Croushore. 2002. “Forecasting with a Real-Time Data Set for Macroeconomists.” Journal of Macroeconomics 24 (4): 507–31. doi:10.1016/S0164-0704(02)00062-9.
Stekler, Herman O. 1972. “An Analysis of Turning Point Forecasts.” American Economic Review 62 (4): 724–29.
Stekler, Herman O. 2007. “The Future of Macroeconomic Forecasting: Understanding the Forecasting Process.” International Journal of Forecasting 23 (2): 237–48. doi:10.1016/j.ijforecast.2007.01.002.←41 | 42→
1 The author thanks Jacob Jones for excellent research assistance; Fred Joutz, Prakash Loungani, James Morley, Adrian Pagan, and Nigel Ray, for insightful discussions; and participants in the Futures Past: Economic Forecasting in the 20th and 21st Century Conference at the University of Hamburg and the Second Macroeconomic Modelling Workshop at the University of Tasmania for helpful comments. This chapter is dedicated to my colleague, co-author, mentor, and friend, Herman Stekler, whose passion for forecast evaluation was so incredibly contagious. He left a lasting imprint on the profession and is greatly missed.
3 Samuelson, Paul (September 19, 1966), “Science and Stocks,” Newsweek, p. 92. Herman Stekler proudly claimed to have “predicted n + x of the last n recessions” (recorded by Joutz, 2010, in an interview of Stekler for the International Journal of Forecasting), but he saw this as in contrast to the profession that typically missed recessions completely. If policymakers were predicting and preventing some recessions then we would see a poor forecasting record coming from predicting more recessions than occur, but unfortunately the record is too few recessions forecasted by policymakers rather than too many.
4 Board of Governors of the Federal Reserve System (US), Industrial Production Index [INDPRO], retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/INDPRO, June 23, 2019. Note that because this series is revised, it is important to evaluate it in real time (Stark and Croushore 2002). This simple example, however, uses the latest available data.