The Perilous Task of Forecasting
To get business right, finance chiefs need to be good forecasters. Yet research has shown that amateurs can actually be better than experts at predicting the future.
Jason Zweig, The Wall Street Journal’s investing columnist, sat down with Philip Tetlock, a professor at the University of Pennsylvania’s Wharton School and the co-author of “Superforecasting: The Art and Science of Prediction,” to explore why that is and what companies can learn from it.
Edited excerpts follow.
Vague verbiage MR. ZWEIG: Why is it so hard for experts to make forecasts about things in their own domain of expertise?
MR. TETLOCK: One reason is that experts sometimes know too much. I was talking once to John McLaughlin, former director of the CIA, about the end of the Cold War period, and he was remarking that the analysts who were slowest to recognize that East Germany was disintegrating were the people who had been on the case for 20 years.
It was the newbies coming in who got it pretty quickly. And there’s a lot of psychological evidence that attests to the power of preconceptions to grip us and make it hard for us to be timely belief updaters. So sometimes knowledge is actually an impediment. Another big factor is that there is a large amount of uncertainty in the world. So no matter how smart you are, it isn’t going to give you a lot of traction.
MR. ZWEIG: Because luck and randomness are such powerful forces?
MR. TETLOCK: Yeah. They aren’t totally dominant, but they’re powerful forces. And forecasters who don’t take them into account do so at their peril.
MR. ZWEIG: After the invasion of Iraq, the U.S. intelligence community did a lot of soul searching. And part of that was the project that you got involved in and that you write about in your book. Tell us what you were asked to do.
MR. TETLOCK: The U.S. intelligence community ran a competition. Five academic teams from major universities around the country competed to generate realistic probability estimates on events that the U.S. intelligence community cares about. And we were being compared or benchmarked against the predictions of professional intelligence analysts with access to classified information.
That’s a remarkable thing to do.
If I’m an intelligence analyst, I’m much safer saying, “Look, I think there’s a distinct possibility the Iranians may cheat on the nuclear deal.” Let’s say I think the true probability is 5% in the next year. If they do cheat and I’m on record with the 5% probability estimate, I’m in trouble. If I say “distinct possibility,” I’m covered no matter what. Vague verbiage gives you political safety.
The downside is that vague verbiage makes it impossible to learn to make more granular and well-calibrated probability estimates. So there’s this deep tension inside organizations between the desire for political safety and doing what needs to be done to extract as much predictive juice as you can out of your experts.
MR. ZWEIG: We know from your work that expert forecasts are nowhere near as reliable as they should be or as the experts think they are. Are the forecasts of CFOs or corporate forecasts in general more accurate?
HT: Value Investing World
MR. TETLOCK: You should expect forecasters to do better to the degree they’re working in a world where they get quick, clear feedback on their forecasts. “Distinct possibility” doesn’t count. You have to be making numerical probability estimates repeatedly over time on a wide range of outcomes. If you do that, you can learn to become one of the better-calibrated professionals....MORE
Previously on Shoulda seen it coming:
Credit Suisse's Mauboussin: "Sharpening Your Forecasting Skills"
How Automation and Big Data Affect Forecasting
Forecasting: The Intelligence Advanced Research Projects Activity (IARPA) Has An "Office for Anticipating Surprise"
"How To Win At Forecasting" (Philip Tetlock and the Intelligence Advanced Research Projects Agency)
"U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events"
Forecasting: "So You Think You're Smarter Than A CIA Agent"
"Superforecasting: The Art and Science of Prediction"
Edge Magazine's Master Class In Forecasting With Phillip Tetlock
Daniel Kahnman's Favorite Paper: "On the Psychology of Prediction"
"Pseudo-Mathematics and Financial Charlatanism...."
“What should one do: predict specifics, or forecast broad trends that necessarily miss specifics?”
"Thinking Clearly About Forecasting"
How to Predict a Nation's GDP per Capita at r=.97 Using "Economic Freedom and average citizenry IQ -- plus slight tweaks from trading block membership and oil"
And many more, use the 'search blog box if interested.
