From Logic Magazine:
Monkeywrenching the Machine
Silicon Valley’s surveillance-based business model relies heavily on machine learning. But with the right techniques, we can resist the enclosure of our lives for profit and disrupt the disruptors.
This piece is also available in audio from our friends at Curio.io.
Machine learning is the practice of training algorithms to classify and predict in order to support decision-making. In recent years, it has skyrocketed in popularity and ubiquity. It's no stretch to say that most services we use now incorporate machine learning in one way or another. In its pervasiveness, machine learning is becoming infrastructural. And, like all infrastructure, once it matures it will become invisible.
Before that happens, we should develop a way to disrupt it.
A relatively nascent field called "adversarial machine learning" provides a starting point. Described as the intersection of cybersecurity and machine learning, this field studies how these algorithms can be systematically fooled, with or without knowledge of the algorithm itself—an ideal approach, since the specifics of many algorithms are trade secrets. And given the fact that the machine learning regime effectively makes all of us its workers—most of our online activity is in fact labor towards the improvement of these systems—we as individual users have an opportunity to inflict major sabotage.
Consider an early and now-ubiquitous application of machine learning: the everyday spam filter. The job of the spam filter is to categorize an email as either "spam"—junk—or "ham"—non-spam. The simplest case of adversarial machine learning in this context is constructing an email that is spam—a pitch for a pharmaceutical product, for example—but in such a way that the spam filter misclassifies it as ham, thus letting it through to the recipient.
There are a variety of strategies you might employ to accomplish this. A relatively simple one is swapping out the name of "Viagra" for something more obscure to a machine but equally readable to a human: "Vi@gr@", for example.
Today, most spam filters are resistant to this basic obfuscation attack. But we could consider more sophisticated approaches, such as writing a longer, professional-looking email that hints at the product without ever explicitly mentioning it. The hint may be strikingly obvious to a human, but incomprehensible to a spam filter.
A spam filter is less insidious than many other applications of machine learning, of course. But we can generalize from this example to develop techniques for disrupting other applications more worthy of sabotage.
Poisoning the Well
Most machine learning models are constructed according to the following general procedure:
Many websites collect training data with embedded code that tracks what you do on the internet. This information is supposed to identify your preferences, habits, and other facets of your online and offline activity. The effectiveness of this data collection relies on the assumption that browsing habits are an honest portrayal of an individual.
- Collect training data.
- Run a machine learning algorithm, such as a neural network, over the training data to learn from it.
- Integrate the model into your service.
A simple act of sabotage is to violate this assumption by generating "noise" while browsing. You can do this by opening random links, so that it's unclear which are the "true" sites you've visited—a process automated by Dan Schultz's Internet Noise project, available at makeinternetnoise.com. Because your data is not only used to make assumptions about you, but about other users with similar browsing patterns, you end up interfering with the algorithm's conclusions about an entire group of people.
Of course, the effectiveness of this tactic, like all others described here, increases when more people are using it. As the CIA's Simple Sabotage Field Manual explains, "Acts of simple sabotage, multiplied by thousands of citizens, can be an effective weapon...[wasting] materials, manpower, and time. Occurring on a wide scale, simple sabotage will be a constant and tangible drag on...the enemy."
Attacks of this sort—where we corrupt the training data of these systems—are known as "poisoning" attacks.
The Pathological and the Perturbed
The other category of adversarial machine learning attacks are known as "evasion.” This strategy targets systems that have already been trained. Rather than trying to corrupt training data, it tries to generate pathological inputs that confuse the model, causing it to generate incorrect results.
The spam filter attack, where you trick an algorithm into seeing spam as ham, is an example of evasion. Another is "Hyperface," a collaboration between Hyphen Labs and Adam Harvey, a specially designed scarf engineered to fool facial recognition systems by exploiting the heuristics these systems use to identify faces. Similarly, in a recent study, researchers developed a pair of glasses that consistently cause a state-of-the-art facial recognition system to misclassify faces it would otherwise identify with absolute certainty....MORE
