Netflix believes, algorithmically at least, that I am the kind of person who likes to watch “Dark TV Shows Featuring a Strong Female Lead.” This picksome genre is never one that I that seek out intentionally, and I’m not sure it even represents my viewing habits. (Maybe I fell asleep watching The Killing one night?) It is an image of me that Netflix compiled from personal data it gathers, and, like a portrait taken slantwise and at a distance, much finer detail is missing. As it happens, television sometimes puts me to sleep; other times I stream a movie as I work on my laptop, and by the time I’ve finished typing and look back, the credits are rolling. Either way, the idea offered of me after my data has been mined is curiously off-base.
More than a decade ago, Netflix ushered in a cultural conversation about big data and algorithms with stunts like the Netflix Prize—an open competition to improve user rating predictions—and its eventual use of subscriber data to produce and cast the show House of Cards. Now, with Cambridge Analytica and driverless cars in the headlines, the artless future that some technology critics forecasted back then—movies cast by algorithms!—sounds quaint in comparison. For the time being, the stakes are low (rifle through streaming titles to find something good to watch), and the service declares the way it categorizes me—as a fan of the “Strong Female Lead”—rather than clandestinely populating the interface with lady detective shows. To be sure, there is plenty to criticize about its micro-targeting practices, but now that “surveillance capitalism” has eclipsed “big data” as the tech media buzzphrase of choice, at least its subscriber-based business model suggests the company has little incentive to partner with data brokers like Acxiom and Experian, to determine whether mine is a BoJack Horseman household or more apt to stream 13 Reasons Why.
Netflix is an accessible example of the gap between an algorithmically generated consumer profile and the untidy bundle of our lived experiences and preferences. The reality of living a digital life is that we’re routinely confronted with similarly less than spot-on categories: Facebook ads for products you would never buy, iPhoto tagging your house as a person’s face, false positives, false negatives, and all the outliers that might be marked as red dots on prediction models. Mix-ups like these might be laughable or bothersome; the octopus of interlinked corporate and state surveillance apparatuses has inevitable blind spots, after all. Still, I wonder if these blunders are better than the alternative: perfect, all-knowing, firing-on-all-cylinders systems of user tracking and categorization. Perhaps these mistakes are default countermeasures: Can we, as users, take shelter in the gaps of inefficacy and misclassification? Is a failed category to the benefit of the user—is it privacy, by accident?
Surveillance is “Orwellian when accurate, Kafkaesque when inaccurate,” Privacy International’s Frederike Kaltheuner told me. These systems are probabilistic, and “by definition, get things wrong sometimes,“ Kaltheuner elaborated. “There is no 100 percent. Definitely not when it comes to subjective things.” As a target of surveillance and data collection, whether you are a Winston Smith or Josef K is a matter of spectrum and a dual-condition: depending on the tool, you’re either tilting one way or both, not in the least because even data recorded with precision can get gummed up in automated clusters and categories. In other words, even when the tech works, the data gathered can be opaque and prone to misinterpretation.
Companies generally don’t flaunt their imperfection—especially those with Orwellian services under contract—but nearly every internet user has a story about being inaccurately tagged or categorized in an absurd and irrelevant way. Kaltheuner told me she once received an advertisement from the UK government “encouraging me not to join ISIS,” after she watched hijab videos on YouTube. The ad was bigoted, and its execution was bumbling; still, to focus on the wide net cast is to sidestep the pressing issue: the UK government has no business judging a user’s YouTube history. Ethical debates about artificial intelligence tend to focus on the “micro level,” Kaltheuner said. When “sometimes the broader question is, do we want to use this in the first place?”
Mask OffThis is precisely the question taken up by software developer Nabil Hassein in “Against Black Inclusion in Facial Recognition,” an essay he wrote last year for the blog Decolonized Tech. Making a case both strategic and political, Hassein argues that technology under police control never benefits black communities and voluntary participation in these systems will backfire. Facial recognition commonly fails to detect black faces, in an example of what Hassein calls “technological bias.” Rather than working to resolve this bias, Hassein writes, we should “demand instead that police be forbidden to use such unreliable surveillance technologies.”
Hassein’s essay is in part a response to Joy Buolamwini’s influential work as founder of the Algorithmic Justice League. Buolamwini, who is also a researcher at MIT Media Lab, is concerned with the glaring racial bias expressed in computer vision training data. The open source facial recognition corpus largely comprises white faces, so the computation in practice interprets aspects of whiteness as a “face.” In a TED Talk about her project, Buolamwini, a black woman, demonstrates the consequences of this bias in real time. It is alarming to watch as the digital triangles of facial recognition software begin to scan and register her countenance on the screen only after she puts on a white mask. For his part, Hassein empathized with Buolamwini in his response, adding that “modern technology has rendered literal Frantz Fanon’s metaphor of ‘Black Skin, White Masks.’” Still, he disagrees with the broader political objective. “I have no reason to support the development or deployment of technology which makes it easier for the state to recognize and surveil members of my community. Just the opposite: by refusing to don white masks, we may be able to gain some temporary advantages by partially obscuring ourselves from the eyes of the white supremacist state.”...MUCH MORE