Edited By Ulrich Fritsche, Roman Köster and Laetitia Lenel
Few areas in economics are as controversial as economic forecasting. While the field has sparked great hopes for the prediction of economic trends and events throughout the 20th and 21st centuries, economic forecasts have often proved inaccurate or unreliable, thus provoking severe criticism in times of unpredicted crisis. Despite these failures, economic forecasting has not lost its importance. Futures Past considers the history and present state of economic forecasting, giving a fascinating account of the changing practices involved, their origins, records, and their implications. By bringing together economists, historians, and sociologists, this volume offers fresh perspectives on the place of forecasting in modern industrial societies, thereby making a broader claim for greater interdisciplinary cooperation in the history of economics.
Never Change a Losing Horse?: On Adaptations in German Forecasting after the Great Financial Crisis: Jörg Döpke, Ulrich Fritsche, and Gabi Waldhof
Jörg Döpke (Hochschule Merseburg), Ulrich Fritsche
(Universität Hamburg), and Gabi Waldhof (Martin-Luther-
Abstract: Using data from a recent survey among German professional macroeconomic forecasters, we analyze whether their (self-reported) behavior and attitude towards new methods has changed since the Great Recession. We find that several forecasters claim to use new methods and some known methods more frequently. By contrast, forecasters do not report to have changed their loss function after the Great Recession. Although linking forecasters´ attitudes towards a change in methods to socio-demographic variables (age, gender, nature of the institution) did not yield precise estimates, point estimates still suggest that the openness towards new methods is negatively related to medium-run forecasting success. Forecasters with good medium-term track record seem to be more reluctant to change technology whereas forecasters with a bad record seem to be more open for new methods.
Keywords: Forecast error evaluation, questionnaire, survey, business cycle forecasts, professional forecaster
The Great Financial Crisis (GFC) is related to a huge macroeconomic forecast error of the Great Recession (GR) which tremendously impaired the reputation of the entire macroeconomic forecasting guild and the economics profession as a whole (see for example Besly and Hennessy 2009; Nienhaus 2009; Gaffney 2011; Bezemer 2010). Consequently, intensive debates have emerged, e.g. over probate macroeconomic modelling strategies (Aghion et al. 2002 Krugman 2018; Reis 2018; Stiglitz (2018); Vines and Wills 2018), as well as about updating stylized business ←191 | 192→cycle facts following the GFC (see S. Ng and Wright 2013, for a survey). On the academic side, there is a ongoing discussion about the necessity to re-formulate macroeconomic models after the GFC. Positions range from fundamental criticism of the road macroeconomics has taken since Sargent and Lucas (1979) (see Romer 2016, for a long and detailed discussion) to moderate arguments in Vines and Wills (2018) or Reis (2018) to a defence of established models Christiano, Eichenbaum, and Trabandt (2018).1 The idea that the academic discourse is influenced by real-world development is supported by Lüdering and Winker (2016), who undertook a text-mining analysis of the Jahrbücher für Nationalökonomie und Statistik, a German economic scholarly journal.
We contribute to these debates by asking, whether the behavior of professional macroeconomic forecasters and the discussion within the economic profession has changed. Furthermore, we investigate what determines forecasters’ openness to apply new methods or models. Although at first glance it may seem unlikely that there is a direct relationship between a single but huge macroeconomic forecast error and the behavior and the toolkit used by macroeconomic forecasters, the academic debate draws a very distinct picture.
As Friedman (1953) argued, the ultimate goal of economic models in a non-normative, i.e. positive way is to deliver “predictions” of phenomena unknown in advance. In this methodological tradition based on Hempel (1942), there is often an implicit parallelism between conditional (i.e. model-based with a priori assumptions) forecast and the scope of explanation of a certain model/ method/ paradigm. The implicit parallelism can be found e.g. in the arguments of Sargent and Lucas (1979).
Box 1: Quotes Highlighting the Role of Forecast Errors in the Evolution of Macroeconomic Theory
“Thus, the logical structure of a scientific prediction is the same as that of a scientific explanation.”
(Hempel 1994 , p. 38)
“Judging econometric models by their forecasting success seems such a natural procedure that it might occasion surprise to question its usefulness.”
“[The Keynesian models, UF, JD, GW] predictions were wildly incorrect and that the doctrine on which they were fundamentally awed are now simple matters of fact, involving no novelties of economic theory.”
(Sargent and Lucas 1979)
“Nonetheless, I remember the 1970s quite well, and stagnation did indeed play a role in the rise of New Classical macroeconomics, albeit in a subtler way than the caricature that it proved Keynes wrong, or something like that. What mattered instead was the fact that stagnation had in effect been predicted by Friedman and Phelps; and the way they made that prediction was by taking a step in the direction of microfoundations.”
“The stagnation of the 1970s, when both inflation and unemployment rose, is one of the greatest successes of out-of-sample forecasting by a macroeconomist.”
(Mankiw and Reis 2018)
“This analysis shows that a valid model can forecast badly, and a poor model can forecast successfully.”
(Castle and Hendry 2011)
Box 1 shows some quotes of influential scholars representing different viewpoints on the relation between forecast errors and macroeconomic thinking.
The research question of this chapter can be stated as follows: Did the experience of the GFC and the GR lead to changes in the way macroeconomic forecasts are produced in Germany? This question could be addressed in several ways: For example, Frenkel, Lis, and Rülke (2011) analyse the expectation formation process of professional forecasters before and after the recession. The authors argue that certain important relations of applied macroeconomics, namely an Okun relation, a Phillips curve, and a Taylor rule have not changed in the eyes of professional forecasters. Döpke, Fritsche, and Waldhof (2019) evaluate forecasting ←193 | 194→errors before and after the recession and conclude that forecast accuracy has not changed, but that there are some signs of a change in the loss functions of the forecasters before and after the financial crisis. Pain et al. find that the OECD has learned from the crisis, since they subsequently re-thought their forecasting practice and now pay more attention to global economic or financial developments. Heilemann and Schnorr-Bäcker (2017) provide an in-depth post-mortem analysis of the failed forecasts of the downturn after the financial crisis in Germany and remain more sceptical on the possibility of learning effects from the GR. Forecasters, they argue, had low priors about the probability of a recession in the first place. Drechsel and Scheufele (2012) take a different perspective and argue that, based on leading indicators, forecasters had little chances to correctly predict the recession. While the combination of forecasts provides same gains of accuracy, the forecasts made in the dawning of the recession came pretty close to the best indicator based forecasts. Thus, there are only small incentives to look for better indicators. Based on a structural model of the U.S. economy, Fair (2012) finds that a large share of forecasts uncertainty is based on fluctuations of asset prices, which – as the author argues – are almost unpredictable. Therefore, one might see the financial crisis and the subsequent recession as a “perfect storm” that teaches not much lessons to improve forecasts in the future.
Common to all approaches is an indirect way of assessing the behavior of the forecasters: observed forecasts, forecast errors, and information over the available information at the forecasting date are used. The literature, however, also suggests a more direct way to collect relevant information: ask the forecasters. Surveys among professional forecasters have frequently been used to analyze a range of problems. To name just a few: D. Batchelor and Dua (1990a) and R. Batchelor and Dua (1990b) analyze how divergent theories and models are across different forecasting institutions and do not find a strong impact of theoretical positions and forecasting techniques on the accuracy of the forecasts. In a similar vein, Ashiya (2006) cannot find a respective connection based on Japanese data. The European Central Bank (2009) and European Central Bank (2014) has conducted special surveys among participants of the regular “Survey of Professional Forecasters.” The results confirm a great importance of judgemental forecasting as opposed to model-based forecasting ←194 | 195→(Fildes and Stekler 2002; Lawrence et al. 2006). Furthermore, they find a very low “relative weight” of use of modern macroeconomic (i.e. DSGE) models, which contrasts to the high academic reputation of these models (see, e.g., Wieland and Wolters 2013). Stark (2013) presents results based on a special survey among the U.S. “Survey of Professional Forecasters.” According to his results, forecasters use a combination of methods with a high degree of judgemental methods.
In the following, we are going to argue that astonishingly little has changed after 2008. We find only weak evidence that the (self-reported) behavior of forecasters has changed substantially since the GR. While some forecasters claim that they use new methods or known methods more frequently, they remain silent on the question of what methods are meant by these statements. The vast majority of forecasters reject the idea of a changed loss function. Linking the attitude of the forecasters towards a change in methods or theories to demographic information on the forecasters (age, gender, nature of the institution) yield only highly imprecise estimates and rests heavily on the assumption of a linear probability model.
In the course of the paper, we discuss several hypotheses and lines of arguments why this might be the case. Briefly:
• First, in line with the hypothesis in Mankiw (2006) we can interpret macroeconomic forecasting as a task of macroeconomists in their role as “engineers.” Colander (2017) takes up this notion and extends the methodology of engineering to a general role model for economists. This might explain the de-coupling of academic and theory-related discussions from the “business-as-usual” in macroeconomic forecasting.
• Second, the populations in the academic “camp” seem to differ from the people living in the forecaster’s “camp” (Geiger, Sauter, and Schmid 2009). The differences mostly refer to the “schools of thought” people sympathize with and to the models people have in mind when asked about how the economy functions.
• Third, in contrast to the period mentioned in Sargent and Lucas (1979), macroeconomic forecasters were not (scientifically and personally) discredited in the scientific community – which in turn is related to the first argument – but in the general public. This provoked several ←195 | 196→“self-evaluation” and “re-assessment” studies but no pressing need for a fundamental change of methodology.
The chapter is organized as follows: Section 2 describes the survey we have conducted among German forecasters. In Section 3 we present the empirical analysis with respect to whether the forecasting process has changed since the GFC and the Great Recession. Section 4 will provide arguments and evidence for the de-coupling hypothesis and Section 5 concludes.
2. The Survey
Included in our statistical population are forecasters that meet the following criteria: The institution the forecaster is working at is based in Germany or provides forecasts for the German economy. These forecasts are quantitative, i.e. includes at least a prediction for real Gross Domestic Product (GDP) growth. Additionally, we only included macroeconomic forecasters, i.e. we exclude institutions that provide forecasts for individual sectors, branches, or regions only from our sample. The institution forecasts on a regular basis. We refer to short-run, i.e. mostly to one-year-ahead or at best two-year-ahead forecasts. We include only forecasts that are – at least in part – offered as a public good. Some institutions provide a detailed explanation of the forecasts only for their customers, but are counted in public rankings with their “headline” numbers of, say, real GDP growth. Our net-based search strategy, however, will miss firms that provide their forecasts exclusively for their customers, although we are not aware of such a firm. In contrast to previous studies, our basic statistical unit are not the forecasting institutions, but the individual forecasters. We refer to currently active forecasters.2
Relying on publicly available information, we have identified 266 persons that match the above-mentioned criteria. We have taken into account institutions that have been listed in the ranking of Fricke (2016) and the ←196 | 197→regular reports of Consensus Forecast ™ (2016). We contacted 266 persons. The overall response rate was 34 % with respect to the invited persons and 67 % with respect to the invited institutions, which is relatively high compared to other online surveys.
Tab. 1 provides some demographic information regarding the respondents. The median years of experience as a forecaster is of particular interest for our topic, since it makes clear that roughly half of the forecasters have no personal professional experience that includes the time span leading to the GR. Therefore, we have specifically asked for the changes within the institution, the respondent might be aware of.3
Tab. 1: Some Demographic Information. In Brackets: 25 % and 75 % Quartiles. Source: Döpke, Fritsche, and Waldhof (2019)
|Median age of respondent||43||49 [37; 52.5]|
|Median years experience as a forecaster||50||10 [5; 18]|
|Share of female forecasters||54||13 %|
|Academic degree or position||56||Diplom: 9|
Master of Science: 4
Professor: 3 Other: 1
|Field of studies||57||Economics: 53|
Mathematics: 1 Others: 2
|Group of institutions||81||Public institutes: 18|
Private institutes: 12
Policy related institutions: 19 Private firms: 31
3. Empirical Results
3.1. Responses to Pre-Formulated Statements
Fig. 1 shows the responses to a question exploring changes in the forecasting process that might have happened following the recent financial crisis and the subsequent GR. The most popular answer is that the ←197 | 198→institutions now use “new” methods in forecasting. However, we offered the possibility to answer a free-text question to provide more information which new models have been used. In this question, it was possible to add “additional” methods that have been used for forecasting in the institution as answer categories. Hence, the “new” method mentioned should shine up in the answers for that question, but this was generally not the case.
While there is some evidence for the use of new methods or the more frequent use of already known methods, or “other” changes in the forecasting process, the overall impression is that there have been little changes in response to the crisis and the subsequent forecast errors. In particular, it seems intuitively reasonable that forecasters might have changed their loss function and, e.g., try to forecast more cautiously. We find, however, no hint to a changed loss function in either direction. In addition, only a minority of the forecasters report that their institution takes dissenting opinions more seriously into account than before. Again, this statement seems to be plausible in advance.
Interestingly, the statement favoring new methods in forecasting is much more popular than the idea of new theoretical models. On the one hand, this is in line with previous findings that applied macroeconomic forecasting prefers data-driven to theory-driven forecasts. For example, Meyler and Rubene. report – based on the Survey of Professional Forecasters conducted by the European Central Bank – that more than 35 % of the respondents refer to judgemental forecasting in short-term forecasting GDP growth. Additional by about 45 % refer to time series methods. These numbers contrast sharply to the share of less than 20 % who use traditional large-scale econometric models, and markedly less than 5 % who use DSGE models. Similarly, Stark (2013) documents – albeit on a small sample – for data from the U.S. Survey of Professional Forecasters that models with subjective adjustment and “No Model – Experience and Intuition” alone dominate mathematical and computer models. On the other hand, it might be a hint that more recent theoretical models are not very popular among forecasters.
Question: In the aftermath of the Financial Crisis 2008/09 economic forecasts have been criticized (again). This leads to the possibility that your institution may have changed its forecasting process. Which statements apply to your institution?
Fig. 1: Consequences of the Great Recession. Source: Own Survey and Calculation
3.2. Answers to Free Questions
Respondents also had the opportunity to answer to a free-text question and provide information about what has changed in the forecasting process due to the Financial Crisis. One person reported that the respective institution has undertaken an overhaul of existing and estimation of new econometric models, in particular new indicators, and now uses methods of model averaging. Another respondent argues that the institution looks more strongly on measures of uncertainty that rely on market prices. Moreover, they consider more strongly the balance sheets of both firms and households, because they feel that balance sheet adjustments have weakened growth after the crisis. Finally, according to the results, this institution finds that bubbles have become more important. A third forecaster argues that his or her institution takes now into account a broader diversity of forecasting methods and models as well as forecast combination. A forecaster from an institution that relies on own surveys among ←199 | 200→firms to build their forecasts reports on changes in the methodology of the surveys, in particular an adjustment of the own survey technique (shorter survey period, faster publication).
More generally, one forecaster observes for his/her institution that they are more aware of inaccuracy, think in broader bandwidths. Furthermore, they place greater emphasis on risk scenarios. One forecaster also points to “systematic forecast error evaluation” as a consequence of the failed forecast of the crisis. It also seems that at least some forecasters feel that their business has become harder: For example, one person points to his/her impression that the literature regarding forecasts has become more complex and requires in-depth studies.
3.3. Evidence from Probability Models
Tab. 2 shows the results of some estimations of determinants of the probability that a certain forecaster responds positively to a certain item. We chose the two answers that led to the most pronounced average support by the respondents:
(i) Agreement to the statement “We use new methods to forecast.” (“New method” for short),
(ii) Agreement to the statement “We use already known methods more frequently than before.” (“More often”).
All answers are recoded to a numerically scale such as “5” denotes full agreement, whereas “1” represents no agreement at all. Thus, we check, whether a forecaster is more open to changes after the GFC.
Fig. 2 shows, for example, the agreement to the statement “Our institution uses new methods” by age group. Forecasters with an age under 35 are rare and have mostly no opinion on whether their institution has changed their forecasting methods. For the two other groups of older forecasters, there is no noticeable difference. As possible determinants for openness to change, we consider the following variables:
Fig. 2: Agreement to the Use of New Methods by Age Group.
Source: Own Survey and Calculation
• Forecaster Age: An older age might make him or her less open for models that are more recent. However, increased age might have different implications for the usage of models. Although forecasters might be unwilling to learn new methods, at the same time their perceived ←200 | 201→pressure to conform to standards in the field might decrease with experience and confidence. Lamont (2002) argues that forecasters become more established with age and therefore have incentives to stick to the consensus, since they face an increased risk of a reputation loss. Therefore, his model predicts that forecasts of older forecaster are less accurate. As the relation of age and the probability of using new methods might be non-linear, we take also into account squared age.
• Type of Institution: We include a dummy variable that assumes the value 1, if the institution of the forecasters is a private one, 0 otherwise.
• Gender: We test with the help of a dummy variable, whether the gender of the forecaster is important for the probability of interest.
• Previous Success of the Institution: We also suspect a possible influence of the previous success of the forecasting institution. Hence, we include the rank of the forecasting institution according to the ranking ←201 | 202→of Fricke (2016). A lower value indicates a more successful forecasting institution.4
• School of Thought and Openness to Change: Finally, we tried to link the attitude towards changes to the theoretical position of a forecaster. It has been argued (see, for example, Farrell and Quiggin 2017) that certain macroeconomic events had been challenges for different macroeconomic schools of thought, in particular, the Great Recession has been a problem for neoclassical models. Therefore, we test for a difference in the attitude towards new models by using a dummy variable that equals 1, if a forecaster leans more to a neoclassical position, 0 otherwise.5
We estimate the relation both by means of the linear probability model (Angrist and Pischke 2008, 101 ff.; Wooldridge 2014, 209 ff.) and by means of a multinomial Logit model.6The results do not provide much support to the view that forecasters’ attributes may explain a lot of the variation of their openness to change. Of course, our sample is very small and, consequently, parameters can be estimated very imprecisely only. Furthermore, the results of columns 1 and 3 of Tab. 2 heavily depend on the assumption of a linear probability model. Note that most results have to be interpreted with care due to relatively high standard errors. Given these caveats, it is still worthwhile to have a brief look at the results:
• Forecaster Age: According to the point estimates, the probability to use new methods increases with experience, but up to a certain point only. After this point in time, forecasters tend to stick with their already known methods.←202 | 203→
• Type of Institution: Working in a private institution makes it more likely to use new methods, but less likely to use known methods more frequently.
Tab. 2: Probability Models for the Likelihood to Change the Forecasting Process. Robust Standard Errors in Parentheses. Source: Own Survey and Calculation
• Previous Success of the Institution: The impact of success, i.e. a good rank in the ranking of the institutions is the largest single effect and also (at least at the 10 % level) statistically significant in all models: the lower the rank of the forecaster (which corresponds to a relatively bad prediction record) the higher the probability that a respondent reports the use of new methods.←203 | 204→
• Neoclassical Positions and Openness to Change: Tending to a neoclassical position goes hand in hand with a higher probability of using new methods and a lower probability of using existing methods more often.
4. Decoupling of Academia and Macroeconomic Forecasting Camp
As pointed out above, we consider the possibility that academic work on forecasting and practical forecasting processes do not overlap by very much. We are, of course, not the first, who see important gaps between different spheres of forecasting. For example, Tichy (1976) diagnosed what he labelled the “Great Dichotomy” between business cycle theory, business cycle empirics, and the receptive policy. He argues that, after the initial large interest in business cycle theory related to the Great Depression the 1960s and 1970s had been a phase with substantial progress in empirical methods and forecasting, but little interest in related theory. Later, he argues in retrospect (Tichy 2013), the picture changed: empirical business cycle research stagnated while interest in business cycle theory increased again.
In a similar vein, we have argued that active forecasters and academic economists, who work on related questions, are likely a rather distinct groups of persons. To give a first impression on how distinct these groups are, we compare our underlying sample – the forecasters we have invited to our survey – with a subgroup of the Verein für Socialpolitik – the German Economic Association.
In particular, we referred to all members that have singled out either the JEL code E3:”Prices, Business Fluctuations, and Cycles” or the JEL code C53: “Forecasting and Prediction Methods, Simulation Methods” as their main scientific area. The overlap of these two groups was very small: while our sample includes 252 persons, and the subgroup of the academic association consist of 109 persons, we have found only 5 persons that are members of both groups. More generally, only 25 subjects from our invited sample have been found in the database of the members of the Verein für Socialpolitik – by about 10 %.
Fig. 3: Comparison of Survey Answers of Döpke, Fritsche, and Waldhof (2019) and Fricke (2017).
Notes: For the present survey the bars represent the share of respondents that see a particular school of thought as either as “extremely important,” “very important,” or “relatively important” personally. For the present survey “Neoclassical economics” is “Neoclassical economics” and “New classical economics,” “Keynesianism” is the average of “Keynesianism (neoclassical synthesis)” and “New Keynesianism,” and “Public choice etc.” represents “Theory of political business cycles.” “Socialism/Marxism” has been no answer option in the present survey (but possible as a free-text answer). Note that the answer options in to Fricke (2017) survey have been exclusively, whereas in Döpke, Fritsche, and Waldhof (2019) respondents have been allowed to choose more than one answer. Source: own survey and calculation; Fricke (2017)
Additional to institutional differences, survey results point to substantially different views on particular economic schools of thought. To illustrate this point, we compare the results of Schneider, Frey, and Humbert (2007), who surveyed the members of the Verein für Socialpolitik with our ←204 | 205→more recent survey. As can be seen from Fig. 3, the popularity of certain schools of thought is markedly different.
The possible decoupling might also in part reflect a changed importance of macroeconomic forecasting for academic research. For example, in 1962, the German Economic Association (Verein für Socialpolitik) devoted a complete annual conference to the problem of forecasting Giersch and Borchardt (1962). Thus, we wonder whether forecasting plays a similar prominent role in broad academic cycles today. Evidence in favor of the decoupling hypothesis might be seen in the frequency of papers in general-interest scholarly journals that are devoted to study related topics. ←205 | 206→Taking the German Economic Review as an example, we searched the IDEAS-database for papers that have either “forecast*” or “predict*” or “macro*” in title or abstract. From this list we picked the papers arguably directly relevant for macroeconomic forecasting. We end up with 7 papers out of 461 (by about 1.5 %) listed for that journal in IDEAS recently. In a similar vein, we use the ECONBIZ database of the Leibniz Information Centre for Economics, Kiel to check for the frequency of papers that deal with questions related to macroeconomic forecasting.
Fig. 4 reveals that the scientific interest in business cycle forecasting in Germany indeed reacts to the financial crisis, but temporarily only. Of course, this is a first illustration of this point only. We therefore test this hypothesis with the help of a broader dataset: the texts of the papers presented at the annual meeting of the Verein für Socialpolitik.
Fig. 4: Hits for Search “Konjunkturprognose* + fehler*”1) in ECONBIZ Database per Year.
Source: Own Calculation Based on https://www.econbiz.de/Search/Results?type=AllFields 1) “Business cycle forecast*+ error*”, * represents truncation and “+” a logical “and”.
We furthermore used the abstracts of the papers presented at the open sessions of the annual meetings of the “Verein für Socialpolitik” to fit a Latent Dirichlet Allocation (LDA) topic model on the abstracts (Blei, A. Y. Ng, and Jordan 2003; Blei 2012). Unfortunately, data are only available from 2010 to 2018 in the ECONBIZ database. LDA stands for “Latent Dirichlet Allocation” and describes a mixed membership model where all word with certain probabilities belong to topics and all topics with certain probabilities appear in documents. The Dirichlet distribution in fact is the conjugate prior to the multinomial distribution in Bayesian statistics. The LDA model – despite its analytical non-tractability – can be estimated using variants of Gibbs sampling or variational inference (Heinrich 2005; Hornik and Grün 2011) conditional the number of topics is fixed a priori. Several criteria were proposed in the literature to deal with that issue. We used the criteria proposed by Arun et al. (2010), Cao et al. (2009), Deveaud, SanJuan, and Bellot (2014) and Griffiths and Steyvers (2004) to determine the number of topics. We fixed the number of topics to 15. Inspection of the most probable words ←207 | 208→in the topics revealed that two of the identified topics are possibly strongly related to the GFC and GR: topic 1 (risk, banking, finance) and topic 7 (macroeconomics, monetary policy)
Fig. 5: Share of Forecasting Papers Presented at the Annual Meeting of the German Economic Association. Source: Own Calculation
Forecasting papers in a narrow sense are papers that have at least on of the following JEL-classifications: E32 (Business Fluctuations, Cycles), E37(Forecasting and Simulation: Models and Applications) or C53 (Forecasting and Prediction Methods, Simulation Methods). Forecasting papers in a broader sense also include papers with the JEL-classifications E30 (Prices, Business Fluctuations, and Cycles, General), E31 (Price Level, Inflation, Deflation).
Fig. 6: Topic 1: Wordcloud Figure. Source: Own Calculation. LDA topic model fitted with number of topics k = 15
Fig. 7: Topic 7: Wordcloud Figure. Source: Own Calculation. LDA topic model fitted with number of topics k = 15
The distribution of estimated topics over time reveals a very stable structure but no obvious trend for the GFC- or GR-related topics 1 and 7 to become more prominent (see Fig. 8).
Fig. 8: Abstracts of Papers Presented at Annual Meetings of “Verein für Socialpolitik”: Distribution of Estimated Topics over Time 2011 to 2018
This data provides insights into the German discourse. Additionally, we searched international data, or data for the Anglosaxon research community. We searched the American Economic Review articles indexed in the JSTOR database and again focused on papers that have either “forecast*” ←208 | 209→ ←209 | 210→or “predict*” or “macro*” in title or abstract. We applied the same search string to the full IDEAS database (data are scaled relative to all IDEAS-indexed economic papers per year).
Fig. 9: AER and IDEAS Key Word Search Results. Source: Own Calculation. Count of JSTOR-indexed AER papers that have either “forecast*” or “predict*” or “macro*” in title or abstract. The same search was done for the full IDEAS database (data are scaled relative to all IDEAS-indexed papers per year).
The figures reveal that in the academic discourse outside Germany, the topics “macroeconomics” and “forecasts” seem to be on the rise. This is in contrast to the analysis for Germany, where the interest rose only shortly after the GFC/ GR and points to another (second) decoupling problem: the particularity or oddity of the German economic discourse with respect to macroeconomic issues. Beck and Kotz (2017) and Winkler (2018) recently analyzed the discourse. Given that there are different perspectives on the issue that the German economic policy debate with respect to monetary policy and fiscal austerity is an “oddity,” two observations remain: first, there is a very strong argumentation in favor of general principles instead of a pragmatic use of policy instruments. The latter is popular among a certain subgroup of German economists. Second, Germany as a net creditor country was never strongly under pressure to adjust or use harsh austerity.
To sum up: On an international level we can observe that the importance of research devoted to macroeconomics and forecasting issues increased over the last couple of years with the German academic scene as an exception.←210 | 211→
Glandon et al. (2018) also report that macroeconomics has responded to the crisis at least to a certain extend. While, on the one hand, they find that the share of theoretical papers has increased at the expense of empirically orientated contributions, the role of financial intermediation in the papers, on the other hand, has increased (again) after the crisis. Still, the gap to the forecasting practitioners has probably increased, since the share of general equilibrium models has become much larger in their sample, while partial equilibrium models, which are more relevant for forecasting, have become less important over time.
Relying on data from a recent survey among professional macroeconomic forecasters in Germany, we conclude that changes in the forecasting process are at best modest. While forecasters claim to use new methods and some known methods more frequently, they remain unspecific which methods they mean by these statements. Forecasters do not report to have changed their loss function after the recession. Linking the attitude of forecasters towards a change in methods or theories to demographic information on the forecasters (age, gender, nature of the institution) yield only highly imprecise estimates. A longer-term bad forecasting record, which exceeds the single event of the GFC, is the single most influential factor explaining the openness for new methods. However, we can infer some tendencies from the open question part: Some forecasters seem to combine forecasts from different methods more often, and re-evaluated their existing toolbox after the GR. Furthermore, the sources of forecast errors are evaluated more often in some cases and measuring macroeconomic uncertainty seems to be more important for the forecast.
This is in contrast to the debate on macroeconomics on the international level but not necessarily on the German level. The astonishingly little reaction of the forecasting community on the macroeconomic turbulences after 2008 in our interpretation might be due to a mixture of facts:
Macroeconomic forecasting is to a very large extent “engineering” and only loosely related to theoretical debates about the “right” theory or school of thought. Consider, for example, the German “oddity” of ←211 | 212→neglecting the need for active macroeconomic policy or opposing the idea of active policy at all, which is common to a relevant subgroup of German economists and parts of the public discourse. This point of view might have decreased the pressure on forecasters to use more modern theoretical models (at least for the German sub-population we analyzed).
These conclusions are obviously based on a small database only, which future research will have to extend. One possible way to do so might be seen in an analysis of citations. Based on such an analysis Fourcade, Ollion, and Algan (2015) show that economics shows a larger degree of insularity as compared to other parts of the social sciences. If our decoupling hypothesis is true, a similar picture should be visible for the forecasters camp. Another way to gain further insights might be a closer look at personal relations and biographies of forecasters. As they are academically trained, it would be interesting to look at their origins in this respect and to check whether they come from certain universities and/or other institutions. This link may well establish social networks (Grimm, Kapeller, and Pühringer 2017), which in turn may explain how certain positions evolve during time.
Aghion, Phillipe, Roman Frydman, Joseph E. Stiglitz, and Michael Woodford, eds. 2002. Knowledge, Information, and Expectations in Modern Macroeconomics: In Honor of Edmund S. Phelps. Princeton, NJ: Princeton University Press.
Angrist, Joshua D., and Jörn-Steffen Pischke. 2008. Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton, NJ: Princeton University Press.
Arun, R., V. Suresh, C. V. Madhavan, and M. N. Murthy. 2010. “On finding the natural number of topics with latent dirichlet allocation: Some observations.” In Advances in Knowledge Discovery and Data Mining, edited by M. J. Zaki, J. X. Yu, B. Ravindran, and V. Pudi, 391–402, Berlin: Springer.
Ashiya, Masahiro. 2006. “Forecast accuracy and product differentiation of Japanese institutional forecasters.” International Journal of Forecasting 22 (2): 395–401.←212 | 213→
Batchelor, Roy., and Pami Dua. 1990a. “All forecasters are equal.” Journal of Business and Economic Statistics 8: 143–44.
Batchelor, Roy, and Pami Dua. 1990b. “Forecaster ideology, forecasting technique, and the accuracy of economic forecasts.” International Journal of Forecasting 6 (1): 3–10.
Beck, Thorsten, and Hans-Helmut Kotz, eds. 2017. Ordoliberalism: A German Oddity? A VoxEU.org eBook. London: CEPR Press. https://voxeu.org/system/files/epublication/Ordoliberalism_Nov22.pdf (last access: 8/13/2019).
Besly, Tim, and Peter Hennessy. 2009. Letter to the Queen. https://www.euroresidentes.com/empresa_empresas/carta-reina.pdf (last access: 8/13/2019).
Bezemer, Dirk J. 2010. “No one saw this coming: Understanding financial crisis through accounting models.” Accounting, Organizations and Society 35 (7): 676–88.
Blei, David M. 2012. “Probabilistic topic models.” Commun. ACM 55 (4): 77–84.
Blei, David M., Andrew Y. Ng, and Michael I. Jordan. 2003. “Latent dirichlet allocation.” Journal of Machine Learning research 3 (Jan): 993–1022.
Cao, Juan, Tian Xia, Jintao Li, Yongdong Zhang, and Sheng Tang. 2009. “A density-based method for adaptive LDA model selection.” Neurocomputing 72 (7–9): 1775–81.
Castle, Jennifer, and David F. Hendry. 2011. “On not evaluating economic models by forecast outcomes.” Istanbul Business Research 40 (1): 1–14.
Christiano, Lawrence J., Martin S. Eichenbaum, and Mathias Trabandt. 2018. “On DSGE models.” The Journal of Economic Perspectives 32 (3): 113–40.
Colander, Dave. 2017. “Economists should stop doing it with models (and start doing it with heuristics).” Eastern Economic Journal 43 (4): 729–33.
Consensus Forecast ™. 2016. “G7 and Western Europe.”.
Deveaud, Romain, Eric SanJuan, and Patrice Bellot. 2014. “Accurate and effective latent concept modeling for ad hoc information retrieval.” Document numérique 17 (1): 61–84.←213 | 214→
Döpke, Jörg, Ulrich Fritsche, and Gabi Waldhof. 2019. “Theories, techniques and the formation of German business cycle forecasts.” Jahrbücher für Nationalökonomie und Statistik 239 (2): 203–41.
Drechsel, Katja, and Rolf Scheufele. 2012. “The performance of short-term forecasts of the German economy before and during the 2008/2009 Recession.” International Journal of Forecasting 28 (2): 428–45.
European Central Bank. 2009. “Results of a special questionnaire for participants in the ECB Survey of Professional Forecasters (SPF).” Monthly Bulletin April: 1–16.
European Central Bank. 2014. “Results of the second special questionnaire for participants in the ECB Survey of Professional Forecasters.” Monthly Bulletin January: 1–28.
Fair, Ray C. 2012. “Analyzing macroeconomic forecastability.” Journal of Forecasting 31 (2): 99–108.
Farrell, Henry, and John Quiggin. 2017. “Consensus, dissensus, and economic ideas: Economic crisis and the rise and fall of Keynesianism.” International Studies Quarterly 61 (2): 269–83.
Fildes, Robert, and Herman Stekler. 2002. “The state of macroeconomic forecasting.” Journal of Macroeconomics 24 (4): 435–68.
Fourcade, Marion, Etienne Ollion, and Yann Algan. 2015. “The superiority of economists.” Journal of Economic Perspectives 29 (1): 89–119.
Frenkel, Michael, Eliza M. Lis, and Jan-Christoph Rülke. 2011. “Has the economic crisis of 2007–2009 changed the expectation formation process in the Euro area?” Economic Modelling 28 (4): 1808–14.
Fricke, Thomas. 2016. Prognostiker des Jahres 2016 – die Langzeitauswertung. https://neuewirtschaftswunder.de/ (last access: 8/13/2019).
Fricke, Thomas. 2017. “Altes Einheitsdenken oder neue Vielfalt? Eine systematische Auswertung der großen Umfragen unter Deutschlands Wirtschaftswissenschaftler_innen.” FGW-Studien 3: 2017. https://www.fgw-nrw.de/fileadmin/user_upload/NOED-Studie-03-Fricke-A1-Web-komplett.pdf (last access: 8/13/2019).
Friedman, Milton. 1953. Essays in Positive Economics. Chicago, IL: University of Chicago Press.←214 | 215→
Gaffney, Mason. 2011. “An award for calling the crash.” Econ Journal Watch 8 (2): 185–92. https://econjwatch.org/articles/an-award-for-calling-the-crash (last access: 8/13/2019).
Geiger, Felix, Oliver Sauter, and Kai D. Schmid. 2009. “The camp view of inflation forecasts.” Hohenheimer Dikussionsbeiträge 320/2009.
Giersch, Herbert, and Knut Borchardt, eds. 1962. Diagnose Und Prognose Als Wirtschaftswissenschaftliche Methodenprobleme: Verhandlungen Auf Der Arbeitstagung Des Vereins Für Socialpolitik, Gesellschaft Für Wirtschafts- Und Sozialwissenschaften in Garmisch-Partenkirchen 1961. Arbeitstagungen zur Erörterung der Aufgaben und Methoden der Wirtschaftswissenschaft in unserer Zeit 1,1961. Berlin: Duncker & Humblot.
Glandon, Philip, Kenneth N. Kuttner, Sandeep Mazumder, and Caleb Stroup. 2018. “Macroeconomic research, present and past.” Unpublished manuscript, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3225417 (last access: 8/13/2019).
Griffiths, Thomas L., and Mark Steyvers. 2004. “Finding scientific topics.” Proceedings of the National academy of Sciences 101 (suppl 1): 5228–35.
Grimm, Christian, Jakob Kapeller, and Stephan Pühringer. 2017. “Zum Profil der deutschsprachigen Volkswirtschaftslehre: Paradigmatische Ausrichtung und politische Orientierung deutschsprachiger Ökonom_innen.” ICAE working paper series. Unpublished manuscript, http://hdl.handle.net/10419/171445 (last access: 8/13/2019).
Heilemann, Ullrich, and Susanne Schnorr-Bäcker. 2017. “Could the start of the German recession 2008-2009 have been foreseen? Evidence from Real-Time Data.” Jahrbücher für Nationalökonomie und Statistik 237 (1): 29–62.
Heinrich, Gregor. 2005. “Parameter estimation for text analysis.” Technical report Universität Leipzig. Unpublished manuscript, http://www.arbylon.net/publications/text-est.pdf (last access: 8/13/2019).
Hempel, Carl G. 1994 . “The function of general laws in history.” In Readings in the Philosophy of Social Science, edited by Michael Martin and Lee C. McIntyre, 43–53. Cambridge, MA: MIT Press.
Hendry, David. F. 1986. “The role of prediction in evaluating econometric models.” Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 407 (1832): 25–34.←215 | 216→
Hippel, Paul von. 2015. “Linear vs. Logistic probability models: Which is better, and when?” https://statisticalhorizons.com/linear-vs-logistic (last access: 8/13/2019).
Hornik, Kurt, and Bettina Grün. 2011. “topicmodels: An R package for fitting topic models.” Journal of Statistical Software 40 (13): 1–30.
Krugman, Paul R. 2014. “Stagflation and the fall of macroeconomics.” https://krugman.blogs.nytimes.com/2014/06/28/stagflation-and-the-fall-of-macroeconomics/ (last access: 8/13/2019).
Krugman, Paul R. 2018. “Good enough for government work? Macroeconomics since the crisis.” Oxford Review of Economic Policy 34 (1/2): 156–68.
Lamont, Owen A. 2002. “Macroeconomic forecasts and microeconomic forecasters.” Journal of Economic Behavior & Organization 48 (3): 265–80.
Lawrence, Michael, Paul Goodwin, O’ Conno, Onkal, Marcusr, and Dilek Onkal. 2006. “Judgmental forecasting: A review of progress over the last 25 years.” International Journal of Forecasting 22 (3): 493–518.
Lüdering, Jochen, and Peter Winker. 2016. “Forward or backward looking? The economic discourse and the observed reality.” Jahrbücher für Nationalökonomie und Statistik 236 (4): 483–515.
Mankiw, N. Gregory. 2006. “The macroeconomist as scientist and engineer.” The Journal of Economic Perspectives 20 (4): 29–46.
Mankiw, N. Gregory, and Ricardo Reis. 2018. “Friedman’s Presidential address in the evolution of macroeconomic thought.” The Journal of Economic Perspectives 32 (1): 81–96.
Meyler, Aidan, and Ieva Rubene. “Results of a special questionnaire for participants in the ECB Survey of Professional Forecasters (SPF).” Unpublished manuscript, https://www.ecb.europa.eu/stats/prices/indic/forecast/shared/files/quest_summary.pdf (last access: 8/13/2019).
Ng, Serena, and Jonathan H. Wright. 2013. “Facts and challenges from the great recession for forecasting and macroeconomic modeling.” Journal of Economic Literature 51 (4): 1120–54.
Nienhaus, Lisa. 2009. Die Blindgänger: Warum die Ökonomen auch künftige Krisen nicht erkennen werden: Frankfurt; New York: Campus Verlag.←216 | 217→
Pain, Nigel, Christine Lewis, Thai-Thanh Dang, Yosuke Jin, and Pete Richardson. 2014.“OECD forecasts during and after the financial crisis.” OECD Economics Department Working Papers 1107.
Reis, Ricardo. 2018. “Is something Rreally wrong with macroeconomics?” Oxford Review of Economic Policy 34 (1/2): 132–55.
Romer, Paul. 2016. “The trouble with macroeconomics.” The American Economist 20: 1–20.
Sargent, Thomas J., and Robert E. Lucas. 1979. “After Keynesian macroeconomics.” Federal Reserve Bank of Minneapolis Quarterly Review 3 (2): 295–319.
Schaffler, Mark. 2012. Probit Better than LPM? http://www.mostlyharmlesseconometrics.com/2012/07/probit-better-than-lpm/ (last access: 8/13/2019).
Schneider, Friedrich, Bruno Frey, and Silke Humbert. 2007. “Was denken deutsche Ökonomen? Eine empirische Auswertung einer Internetbefragung unter den Mitgliedern des Vereins für Socialpolitik im Sommer 2006.” Perspektiven der Wirtschaftspolitik 8 (4): 359–77.
Stark, Tom. 2013. “SPF panelists forecasting methods: A note on the aggregate results of a November 2009 special survey.” Federal Reserve Bank of Philadelphia. https://www.philadelphiafed.org/-/media/research-and-data/real-time-center/survey-of-professional-forecasters/spf-special-survey-on-forecast-methods.pdf?la=en (last access: 8/13/2019).
Stiglitz, Joseph E. 2018. “Where modern macroeconomics went wrong.” Oxford Review of Economic Policy 34 (1/2): 70–106.
Tichy, Gunther. 1976. Konjunkturschwankungen: Theorie, Messung, Prognose. Heidelberger Taschenbücher 174. Berlin: Springer.
Tichy, Gunther J. 2013. Konjunkturschwankungen: Theorie, Messung, Prognose: Berlin: Springer-Verlag.
Vines, David, and Samuel Wills. 2018. “The rebuilding macroeconomic theory project: An analytical assessment.” Oxford Review of Economic Policy 34 (1/2): 1–42.
Wieland, Volker, and Maik Wolters. 2013. “Forecasting and policy making.” In Handbook of Economic Forecasting, Vol 8, edited by ←217 | 218→Graham Elliott, Clive Granger, and Alan B. Timmermann, 239–325. Amsterdam: Elsevier.
Winkler, Adalbert. 2018. “Zehn Jahre nach dem Konkurs von Lehman Brothers: Ordnungspolitische Irrtümer in der Bewegung der EZB-Geldpolitik seit der globalen Finanzkrise.” Perspektiven der Wirtschaftspolitik 19 (2): 141–62.
Wooldridge, Jeffrey M. 2014. Introduction to Econometrics: Europe, Middle East and Africa Edition: Andover, Hampshire: Cengage Learning.
1 It is beyond the scope of the paper to document all discussions in detail but a look at the grants and projects of the “Institute for New Economic Thinking” reveals that there is an ongoing and lively debate about the re-formulation of traditional macroeconomic modelling after the GFC.
2 We have asked retired forecasters and individuals, who are still active as economist, but not as a forecaster for comments on a pre-test version of the questionnaire.
4 Note that the ranking does not exclusively rely on the accuracy of the growth rate forecast. Rather, the ranking takes also into account whether the directional change of some central components of final demand (consumption, exports, investment) have correctly been anticipated. For details please refer to https://neuewirtschaftswunder.de/