"It's not so much the losing but the not knowing why I lost that gets me."Well, it is the losing but he had a point.
From MIT's Technology Review:
New Research Aims to Solve the Problem of AI Bias in “Black Box” Algorithms
As we automate more and more decisions, being able to understand how an AI thinks is increasingly important.
From picking stocks to examining x-rays, artificial intelligence is increasingly being used to make decisions that were formerly up to humans. But AI is only as good as the data it’s trained on, and in many cases we end up baking our all-too-human biases into algorithms that have the potential to make a huge impact on people’s lives.Previously:
In a new paper published on the arXiv, researchers say they may have figured out a way to mitigate the problem for algorithms that are difficult for outsiders to examine—so-called “black box” systems.
A particularly troubling area for bias to show up is in risk assessment modeling, which can decide, for example, a person’s chances of being granted bail or approved for a loan. It is typically illegal to consider factors like race in such cases, but algorithms can learn to recognize and exploit the fact that a person’s education level or home address may correlate with other demographic information, which can effectively imbue them with racial and other biases.
What makes this problem even trickier is many of the AIs used to make those choices are black boxes—either they’re too complicated to easily understand, or they’re proprietary algorithms that companies refuse to explain. Researchers have been working on tools to get a look at what’s going on under the hood, but the issue is widespread and growing (see “Biased Algorithms Are Everywhere, and No One Seems to Care”).
In the paper, Sarah Tan (who worked at Microsoft at the time) and colleagues tried their method on two black-box risk assessment models: one about loan risks and default rates from the peer-to-peer company LendingClub, and one from Northpointe, a company that provides algorithm-based services to courts around the country, predicting recidivism risk for defendants.
The researchers used a two-pronged approach to shed light on how these potentially biased algorithms work. First, they created a model that mimics the black-box algorithm being examined and comes up with a risk score based on an initial set of data, just as LendingClub and Northpointe’s would. Then they built a second model that they trained on real-world outcomes, using it to determine which variables from the initial data set were important in final outcomes.
In the case of LendingClub, the researchers analyzed data on a number of matured loans from 2007 to 2011. LendingClub’s database contained numerous different fields, but the researchers found that the company’s lending model probably ignored both the applicant’s annual income and the purpose of the loan. Income might make sense to ignore, since it’s self-reported and can be faked. But the purpose of the loan is highly correlated with risk—loans for small businesses are much riskier than those used to pay for weddings, for example. So LendingClub appeared to be ignoring an important variable.
Northpointe, meanwhile, says its COMPAS algorithm does not include race as a variable when making recommendations on sentencing. However, in an investigation by ProPublica, journalists collected racial information on defendants who were sentenced with help from COMPAS and found evidence of racial bias. In their mimic model, the researchers used the data gathered by ProPublica as well as information on the defendants’ age, sex, charge degree, number of prior convictions, and length of any previous prison stay. The method agreed with ProPublica’s findings, suggesting that COMPAS was likely biased for some age and racial groups....MORE
Sept. 28
Let Me Be Clear: I Have No Inside Information On Who Will Win The Man-Booker Prize Next Month (hedge funds, AI and simultaneous discovery)
Over the years we've mentioned one of the oddest phenomena in science, the simultaneous discovery or invention of something or other, the discovery/invention of the calculus by Newton and Leibniz is one famous example (although both may actually have themselves been preceded) but there are dozens if not hundreds of cases. Here's a related phenomena.
On Saturday September 23, 6:28 AM PDT we posted "Cracking Open the Black Box of Deep Learning" with this introduction:
One of the spookiest features of black box artificial intelligence is that, when it is working correctly, the AI is making connections and casting probabilities that are difficult-to-impossible for human beings to intuit.Today Bloomberg View's Matt Levine commends to our attention a story about one of the world's biggest hedge funds and prize-putter-upper of what's probably the most prestigious honor in literature, short of the Nobel, the Man Booker Award.
Try explaining that to your outside investors.
You start to sound, to their ears anyway, like a loony who is saying "Etaoin shrdlu, give me your money, gizzlefab, blythfornik, trust me."
See also the famous Gary Larson cartoons on how various animals hear and comprehend:...
On Tuesday September 26, 2017, 11:00 PM CDT Bloomberg posted:
The Massive Hedge Fund Betting on AI
The second paragraph of the story:
...Man Group, which has about $96 billion under management, typically takes its most promising ideas from testing to trading real money within weeks. In the fast-moving world of modern finance, an edge today can be gone tomorrow. The catch here was that, even as the new software produced encouraging returns in simulations, the engineers couldn’t explain why the AI was executing the trades it was making. The creation was such a black box that even its creators didn’t fully understand how it worked. That gave Ellis pause. He’s not an engineer and wasn’t intimately involved in the technology’s creation, but he instinctively knew that one explanation—“I can’t tell you why …”—would never fly with big clients looking for answers when Man inevitably lost some of their money...Now that is just, to reuse the phrase, spooky. Do read both the Bloomberg Markets and the Bloomberg View pieces but I'll note right now it's only with Levine you get:
"I imagine a leather-clad dominatrix standing over the computer, ready to administer punishment as necessary."The Man Booker Award winner will be announced October 17th.
I have no foreknowledge of the decision.
Sept. 23
Cracking Open the Black Box of Deep Learning
July 2
Fooling The Machine: The Byzantine Science of Deceiving Artificial Intelligence
And previous non-AI black boxes:
July 2015
Quants and Black Box Trading: Why They All “Blow-Up”
Oct 2013
Black Box Investing Versus Common Sense Quant
