Tuesday, August 1, 2017

DARPA Wants a B.S. Detector

Yesterday the FT's David Keohane commended to our attention an article at BuzzFeed on the reproducibility crisis in science and the soft sciences, a topic we've visited a few times* and to which we pay homage via Retraction Watch on the blogroll at right.

I didn't post on it because someone had the bright idea that I could take a couple of his other Further Reading links and make a post out of them (okay, it was me) and we try to not visit the same well every day.
So now, faced with this latest story from Wired I have a trilemma which can only be solved by risking mass confusion for our busy readers. First up:

From Wired:
Adam Russell, an anthropologist and program manager at the Department of Defense’s mad-science division Darpa, laughs at the suggestion that he is trying to build a real, live, bullshit detector. But he doesn’t really seem to think it’s funny. The quite serious call for proposals Russell just sent out on Darpa stationery asks people—anyone! Even you!—for ways to determine what findings from the social and behavioral sciences are actually, you know, true. Or in his construction: “credible.”
Even for Darpa, that’s a big ask. The DoD has plenty of good reasons to want to know what social science to believe. But plenty more is at stake here. Darpa’s asking for a system that can solve one of the most urgent philosophical problems of our time: How do you know what’s true when science, the news, and social media all struggle with errors, advertising, propaganda, and lies?

Take a scientific claim. Do some kind of operation on it. Determine whether the claim is right enough to act on. So ... a bullshit detector?

“I wouldn’t characterize it that way, and I think it’s important not to,” Russell says. He doesn’t want to contribute to cynicism that lets people think if scientists admit uncertainty, that means they can’t be trusted. “I have a deep faith that there is real science. It’s not that we know nothing about the world.” Science is still the best way of knowing stuff. Darpa just wants to know what stuff science is really sure about, and how it knows it. And how it knows it knows it.

You can imagine why Darpa and the DoD might want to shore up the social sciences. They want to understand how collective identity works, or why some groups (and nations) are stable and some fall apart. The military would like to get a better handle on how humans team up with machines before the machines get smarter and more get deployed. How does radicalization work, especially online? Why do people cooperate sometimes and compete at others? All these questions have two things in common: They are super-important to national security, and no one knows the answer....MORE
*Among our posts on replication and reproducibility:

Serious stuff.

I've mentioned the fact that if you can't can't replicate what you're doing, what you're doing isn't science. It might be metaphysics, it might be pseudoscience, it might be religion, it might be any number of things but it isn't science.

Reproducibility and falsifiability, along with predictive power are pretty much the definition of science.
(here's a quick explanation of the difference between reproducibility and replicability in science) 

What Mendeleev did in describing the properties of as-yet undiscovered elements was science. So-called post-normal science is not, it's policymaking gussied up with sciency sounding words.

See Feynman's 1974 Caltech commencement address. Cargo Cult Science for a really smart guy's take on the issue.... 

And a few others. Meanwhile, the post combining two of the Alphaville links, colonialism in India and the Irish border, remains unwritten but I guarantee, it would have been good.
We do have "Too Good To FactCheck: "Glaciers, Gender, and Science"".