NY Fed EVP: "Models Only Get You So Far" (now with more Superforecasting)

From the Federal Reserve Bank of New York:

February 22, 2019 
Models Only Get You So Far

Simon Potter, Executive Vice President

Remarks for the Federal Reserve Banks of Atlanta and New York's First Annual Joint Research Day on Quantitative Tools for Monitoring Macroeconomic and Financial Conditions, Federal Reserve Bank of New York, New York City As prepared for delivery

 

It is a pleasure to deliver the lunchtime address during the Atlanta and New York Fed's first annual research day on quantitative tools for monitoring macroeconomic and financial conditions.¹ My remarks will focus on some insights from the book Superforecasting by Philip Tetlock and Dan Gardner, with respect to how we interpret, use, and evaluate the results from our quantitative tools.² If you have read the book, subtitled "The Art and Science of Prediction," you will probably be aware that it is much less mathematical than the discussions we have been having today, but at the same time is more complex in how it articulates the role of forecasting in good decision-making. Before I continue, I should note that these remarks reflect my own personal views and not necessarily those of the New York Fed or the Federal Reserve System.

The insights in Superforecasting grew out of a forecasting tournament sponsored by the Intelligence Advanced Research Projects Activity (IARPA). In the wake of the controversy surrounding the intelligence community's assessment of the existence of weapons of mass destruction in Iraq, IARPA set out to enhance the accuracy, precision, and timeliness of intelligence forecasts.³ The Federal Reserve and the broader economics and financial community have, of course, had to face similarly humbling shortcomings in our failure to forecast the Great Recession, despite the numerous signals that were available to forecasters and policymakers prior to 2008. On this issue, it is useful to recall a quote from an unusual source on economic forecasting, the Queen of the United Kingdom, who in November 2008 asked, "Why did nobody notice it?" We should not treat this as a rhetorical question. What are some of the underlying reasons individuals and organizations fail to predict? What should we change about our mindsets and practices to improve the chances that we "notice it" next time, whenever that may be?

At the New York Fed, we've made investments in response to these critical questions. We've created a team that is raising awareness of the challenges that make it so difficult to "notice," innovating on how we approach analysis and decision-making, and making these approaches an essential part of what it means to work at the Bank. This is much in the spirit of what IARPA sought to do by sponsoring their tournament.

To compete in the IARPA tournament, Tetlock recruited participants online from outside the intelligence community. A small number of these participants quickly separated themselves from the pack and consistently maintained their exceptional performance. Tetlock dubbed these individuals "superforecasters" and subsequently focused his team on aggregating the forecasts of these standout performers—a strategy that draws from the "wisdom of the crowd." As you well know, this strategy relies on some degree of independence in the information, approaches, and insights producing the forecasts. We often think of independence in a very statistical sense, but Superforecasting conveys a useful framing of independence as how one assembles diverse teams and how team dynamics can be established to maintain this diversity.⁴ This approach won the IARPA tournament by a substantial margin with the use of one subtle but important tweak. Tetlock extremized the aggregated prediction probabilities—so, for example, a probability of 70 percent became 85 percent....MORE