Saturday, August 6, 2022

Data Centers and "The people of the cloud"

As we saw in August 1's "If You Thought Market Leadership Was Concentrated In 1999...." the earnings of the big three of tech, Amazon, Google and Microsoft to a great extent are what are driving the market. And not simply those companies but the 'cloud' divisions within those companies, meaning the growth of the cloud is pretty much the driver of stock market valuations and that the risks of failure at the large data centers have become an economy-wide threat.

From Aeon, July 28:

Hot, strenuous and unsung. There is nothing soft and fluffy about the caretaking work that enables our digital lives

The ‘cloud’ is not an intangible monolith. It’s a messy, swelling tangle of data centres, fibre optic cables, cellular towers and networked devices that spans the globe. From the tropical megalopolis of Singapore to the remote Atacama Desert, or the glacial extremes of Antarctica, the material infrastructure of the cloud is becoming ubiquitous and expanding as more users come online and the digital divide closes. Much has been written about the ecological impact of the cloud’s ongoing expansion: its titanic electricity requirements, the staggering water footprint required to cool its equipment, the metric tonnes of electronic waste it proliferates, and the noise pollution emitted by the diesel generators, churning servers and cooling systems required to keep data centres – the heart of the cloud – operational 24 hours a day, seven days a week, 365 days a year.

But less has been written about those who work inside the machinery of the cloud. Though often forgotten, this community of technicians, engineers and executives is integral to the functioning of our increasingly digitised society. They are the caretakers of the digital, the wardens of our data, and the unsung heroes working tirelessly to sustain an ever-expanding array of digital objects, including our emails, cat videos, maps, non-fungible tokens, metaverse avatars, digital twins and more. The idea of digital caretakers might conjure science fiction images of empty, towering warehouses stacked with racks of automated machines. But these workers are very much flesh and blood. The silicon milieu they’re part of is as human as it is mechanical. From their vantage, the cloud is not merely an infrastructure they maintain, but a way of life, an identity, a culture of stewardship – replete with its own norms, rituals and language.

For the past six years, I have observed, shadowed and interviewed data centre professionals in the United States and Iceland as an anthropologist. During the course of my ethnographic fieldwork, I witnessed and performed many of the tasks that cloud workers engage in on a daily basis: I dined, trained, travelled and bonded with the crews that I had the privilege of joining as an eager intern, guest and social scientist. Along the way, I learned what it meant to be a steward of the cloud. I also learned that the cloud is no monolith, and that the cultures emerging among its workers are far from uniform. Data centres – as workplaces and sites of culture – vary considerably from continent to continent, node to node, or business model to business model. How they operate depends greatly on where they’re located.

Every site has its constraints, which are political (regulatory considerations), economic (total cost of operation, tax-breaks, business model), environmental (climate conditions, risk of natural disasters) and geographic (proximity to power, network, and other natural resources like water). Some companies lease server space or data to other companies, operating shared centres known as ‘colocations’ or ‘colos’. Other companies or entities, such as governments, choose to build their own data centres instead of renting out space in a colocation.

Data centres also differ based on their technological sophistication: there is a tiering system that ranks centres according to their resources, scale of operation, and level of redundancies (fail-safes) that influence their ability to provide uninterrupted service or ‘uptime’. Only about one-third of the world’s data centres resemble the oft-circulated images of Google’s idyllic facilities, glittering with colourful pipes and smiling technicians who get around their workplaces on scooters. The remaining two-thirds of data centres are far less impressive. Some are found in mouldy basements, others in the shells of decaying office buildings or abandoned military installations. Many companies still use outdated, energy-inefficient designs or do not have the resources to invest in cooling or power-optimisation solutions. As such, the workers in these facilities must rely more readily on their experiences and finely tuned instincts to keep their patches of the cloud ‘up’, however imperfectly. They do not see themselves as automatons, as mere cogs in a perfectly optimised machine, but rather as hunters, firefighters or even priests, who must make, find or invent ways to meet the impossible demand of an unremitting cloud.

While not an exhaustive account of an incredibly diverse, global industry, in what follows, I draw on interview transcripts and field notes to recreate my experiences and encounters with cloud workers (to protect their anonymity and the companies they work for, pseudonyms have been used throughout this essay). These are their stories.

August 2015. A data centre in the greater Boston metro area, Massachusetts. It is three in the morning when the alarm starts to blare. Tom’s pale face is flushed. His mane of grey-brown hair is awry from constantly running his fingers through it. I follow him as he navigates a windowless labyrinth of blinking server racks to the site of a thermal anomaly: a chrome rack of high-density ‘blade’ servers. It is warmer here than in other parts of the facility and, as computational heat envelops me, the goosebumps on my skin slightly recede. Tom is silent while his hand hovers over the server’s metallic grill-plate, where air is being suctioned up by tiny ‘muffin’ fans inside the computer to cool its hot innards. The fans are roaring.

‘You hear that?’ Tom says. ‘They’re starving.’ He gestures to the toolbox, and I hand him the proper tool for the job: a handheld metal apparatus with plunger-shaped suction cups attached to its base. He uses the tool to plunge the floor tile beneath the whining server, then lifts and removes it carefully to expose an underfloor plenum below. A burst of cool air rushes up from the cavity, displacing the earlier warmth that I had been grateful for. Data centres are not designed for human comfort. ‘Another hotspot?’ I ask. ‘Unfortunately,’ Tom nods, teeth clenched.

Like a pressurised can, the computer room air conditioner (CRAC) units pump cold air into the plenum – cold that escapes in a controlled way through tiny perforations in floor tiles that help manage the volume of air released into the server aisle. While some data centre managers use sophisticated computer models to map the airflow and thermal profiles of their facilities (computational fluid dynamics models, or CFDs), many rely on their bodily senses to make judgment calls about how much cooling is needed.

‘Hotspots are elusive critters,’ Tom says, ‘no matter what the CFDs say, they always appear somewhere, and it’s our job to put them down.’

‘Or else the data centre overheats?’

‘Yeah, and if that happens, it’s downtime.’

At that point, ‘downtime’ was a word I was already familiar with, referring to a service interruption, a state of unavailability or rupture wherein the client (potentially you) cannot access their data, stream their music, or play their game. But for Tom, ‘downtime’ was charged with foreboding – a word that meant failure, not only in a technological sense, but a personal sense. He was entrusted with the data centre’s wellbeing, and that meant preventing interruptions of any sort....

....MUCH MORE