Who knew?
Got a concept for cutting-edge spy tech? Jason Matheny, who was named director of the Intelligence Advanced Research Projects Activity (IARPA) in August, wants your great ideas. The agency, established in 2006, invests in high-risk, high-payoff research to solve problems faced by the U.S. intelligence community. Partly due to Matheny’s work, the agency is tapping resources outside of government, including crowdsourcing ideas from the general public.
Matheny joined IARPA in 2009 after a career in both academia—Oxford University, Princeton University, and the Johns Hopkins Applied Physics Laboratory—and the startup world. He previously headed IARPA’s comically named Office for Anticipating Surprise—which develops new forecasting capabilities—and served as program manager of Aggregative Contingent Estimation (ACE), an outfit that crowdsourced forecasts from more than 20,000 people on various geopolitical issues. He also worked at the agency’s Office of Incisive Analysis, which analyzes data sets.
IEEE Spectrum contributing editor Tam Harbert talked with Matheny about the agency’s recent work and his goals for the future.
Jason Matheny on… IEEE Spectrum: Because IARPA is associated with the intelligence community, people assume its research is secret. Yet you seem pretty open in collaborating with industry and academia, and even crowdsourcing.
Jason Matheny: One of the features that distinguishes IARPA is its degree of openness and external engagement. We know that a lot of the problems cannot be solved internally. Unlike the secretive Q in the James Bond movies, who is always developing gadgets for 007 deep inside a lab, IARPA identifies Qs who are working outside—in academia and industry—on important technologies. We find these Qs by putting out calls for research proposals and by staging research tournaments.
Most of IARPA’s work is unclassified, and we make sure that work gets published. We also make the data sets produced by our research publicly available. These data sets may be annotated speech files for speech recognition, for example, or event data sets of political instability. They can be quite useful to other researchers who are working on related problems.
IEEE Spectrum: You’ve been at IARPA since its founding. What have been IARPA’s biggest accomplishments so far?
Jason Matheny: First is getting an organization into place that funds high-risk, high-reward research. It’s a nontrivial challenge in government to create such an organization and to sustain it.
Specific successes include our research on human judgment, including theAggregative Contingent Estimation program, which ran from June 2010 until June 2015. [According to IARPA’s website, the goal of ACE was to develop advanced techniques that combine the judgments of many analysts in ways that would enhance the accuracy, precision, and timeliness of intelligence forecasts.] This was the world’s largest forecasting experiment. It involved more than 20,000 people collecting over 2 million judgments that were crowdsourced on hundreds of geopolitical topics. It asked thousands of participants to forecast who would win a political election, for example, or which countries would go to war. We kept score of whose forecasts were right, whose were wrong, what distinguished the good forecasters from the not-so-good forecasters, and discovered ways of combining the judgments from individuals to create better forecasts than any single individual.
ACE provided a template for how to do a range of research at IARPA. Specifically, it inspired several other IARPA forecasting tournaments, including a program to forecast cyberattacks, a program to forecast disease outbreaks and political instability, a program to forecast military mobilization and terrorism, and a program to forecast insider threats....MUCH MOREWe have quite a few posts on this stuff: "IARPA: It's like DARPA but for spies!"
"How To Win At Forecasting" (Philip Tetlock and the Intelligence Advanced Research Projects Agency)
"U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events"
Forecasting: "So You Think You're Smarter Than A CIA Agent"
And a whole bunch of links in last week's Credit Suisse's Mauboussin: "Sharpening Your Forecasting Skills"
