This class isn't the most interesting but we want the continuity of having links to the full set so off we go.
And yes, Danny is that Danny.
From Edge:
A Short Course in Superforecasting, Class I
Forecasting Tournaments: What We Discover When We Start Scoring Accuracy
Philip Tetlock: Welcome. There really should be two people on stage here. Barb has been, not only my life partner, she's my research collaborator on this. Without her there wouldn't have been any IARPA forecasting tournaments with the intelligence community. It's hard to say where her work ends and my work begins in the IARPA project. My rule of thumb is that Barb does the deep scientific work and I do the public relations. That's the division of labor. I should also thank John and his crew for all their help: Katinka, Max, Nina, and Russell.
I wanted to say a few words about Danny Kahneman and his role in all this. I'm not sure he remembers all the different things he's done over the last twenty-five, thirty years, to facilitate all of this. In the beginning, I was doing very small-scale forecasting tournaments. I was at Berkeley and remember in the late 1980s—I don't know if Danny remembers—he went to Chicago to the McArthur Foundation to argue that McArthur should be supporting forecasting tournaments. They did eventually come up with something, not nearly as much as I wanted, but they came up with something. That was very helpful.HT: Value Investing World
Conversations with Danny through the years have been very helpful in thinking about this. We had lunch at a Chinese restaurant—I think it was called Yen Ching—in Berkeley, in the late 1980s. He had offhandedly tossed off this remark that he thought the average political expert would be hard-pressed to beat, in a forecasting tournament, an attentive reader of the New York Times. And that's been a benchmark that some of my political science colleagues have been struggling to beat ever since.
He also has been very helpful in trying to persuade the intelligence community to take the lessons of the forecasting tournaments seriously, so he did Amtrak down to D.C with me last year to talk to the National Intelligence Council and the Director of National Intelligence about the value of explicit probability scoring, learning from feedback. These might seem like truisms to scientists, but they're still quite radical, even revolutionary ideas in much of the intelligence community. I'm not implying that he agrees with everything I'm going to say. We certainly have some differences of opinion and I'm probably more of an optimist than he is on the degree to which people will ultimately embrace these technologies, but he has been very helpful, so thank you.
I'm going to start this conversation off with a couple of stories that provide a nice point of entry into the 130 slides that you have in front of you in this notebook, and I'll also offer the reassurance that I have no intention of frog-marching you through all 130 slides. As issues come up, we might refer to particular things in here, but we're not going to go through these slides one-by-one.
It would be useful if we went around the table because I'm having a very hard time seeing people beyond about the middle.
John Brockman: Just state your name and what you do in thirty seconds or less.
Robert Axelrod: Political scientist from Michigan. I do game theory international national security.
Danny Hillis: I'm an engineer
Danny Kahneman: I'm a psychologist.
Anne Treisman: I'm a psychologist.
Salar Kamangar: On leave the last year, but my last role was leading YouTube for Google, which I did for about five years.
Wael Ghonim: I used to work for Google as well. I'm a political activist and now working for a startup called Parlio.
Ludwig Siegele: I'm a journalist. I cover technology for The Economist.
Andrian Kreye: I am a journalist and I run the Feuilleton, the arts and essay section of Süddeutsche Zeitung in Germany.
Brian Christian: I'm a writer, and I write about mostly the intersection of computer science and philosophy.
Katinka Matson: Co-Founder of Edge.
Max Brockman: I'm here with Edge.
Dean Kamen: I'm an Edge groupie. When I'm not doing that, I make stuff.
D.A. Wallach: I'm a musician and investor.
Rodney Brooks: I'm a reformed academic and I make robots.
Rory Sutherland: I work for Ogilvy & Mather in London, the advertising agency, and also write on technology for The Spectator.
Peter Lee: I work at a place called Microsoft Research.
Jennifer Jacquet: Environmental Social Sciences, NYU.
John Brockman: Editor of Edge.
Stewart Brand: The Long Now Foundation and Revive & Restore for de-extinction.
Margaret Levi: Director, Center for Advanced Study in Behavioral Sciences at Stanford—CASBS, as it's affectionately known—and a political scientist.
Barb Mellers: Psychologist at UPenn.
Tetlock: I said I would start off with a couple of stories, and that's exactly what I'll do. The first story is about President Obama and the search for Osama bin Laden. In 2010, early 2011, evidence was gradually accumulating about the location of Osama bin Laden. The intelligence community was growing more confident they knew where he was, and as the confidence grew they started reporting higher up in the hierarchy. They reported to Leon Panetta, then Director of the Central Intelligence Agency, and eventually directly to President Obama. They made presentations to both of these men, and in these presentations they made probability judgments about the likelihood that Osama was indeed residing in that mysterious walled-up compound in the Pakistani town of Abbottabad. There is a version of this in the movie, Zero Dark Thirty, which we have checked out—Dan did a great job checking out with the people who were actually there—so don't believe the movie.
The movie is not too far off in this particular sense. They went around the table asking the advisors and the intelligence officers what they thought the likelihood was of Osama being there. The probabilities ranged from about 35 percent to 95 percent with a median of about .75, so 75 percent. President Obama looked a little bit frustrated and he said, "Seems like it's a coin flip, 50-50." And that was that for the purposes of that meeting....MUCH MORE
Previously:
Dec. 2012
"How To Win At Forecasting" (Philip Tetlock and the Intelligence Advanced Research Projects Agency)
We've linked to Edge a few times. The Observer called it "The World's Smartest Website" but sometimes they're a bit too precious for my taste. This isn't one of those times.
"IARPA: It's like DARPA but for spies!"
----------------
"IARPA's mission [is] to invest in high-risk/high-payoff research
programs that have the potential to provide the United States with an
overwhelming intelligence advantage over future adversaries."
– FBI National Press Release, 2009
Sept. 2013
Daniel Kahnman's Favorite Paper: "On the Psychology of Prediction"
June 2014
Elite Forecasters and The Best Way to Predict the Future
Sept. 2014
"U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events"
"Pseudo-Mathematics and Financial Charlatanism...."
“What should one do: predict specifics, or forecast broad trends that necessarily miss specifics?”
"Thinking Clearly About Forecasting"
How to Predict a Nation's GDP per Capita at r=.97 Using "Economic Freedom and average citizenry IQ -- plus slight tweaks from trading block membership and oil"
And many more.
