From the Wall Street Journal via MSN, August 20:
Big tech companies are cobbling together off-grid power systems to match their rushed timelines for AI development. What if these systems are glitchy?
Off-grid power comes with hefty price tags and operating risks. Tech giants are pursuing them anyway. Power providers, some of whom don’t have much data-center experience, are jumping at the chance to sign deals with deep-pocketed customers. But cracks have shown at the few on-site power projects that have started up, highlighting potential costs to both tech companies and their power providers.
In the U.S., there are four operational data centers with off-grid or partially grid-connected power, according to data provider Cleanview. These are xAI’s Colossus and Colossus II, Crusoe’s Stargate Abilene and Vantage Data Centers’ Vantage VA 2. Three of these have already had reported issues.
Earlier this month, Bloomberg reported that some data centers experienced cranks breaking off on combustion engines. The report also said that cracks had formed at gas-fired turbines powering xAI’s Colossus data center in Memphis, Tenn.
In response to the report, Solaris Energy Infrastructure, which operates turbines for the campus, didn’t directly confirm or deny the existence of turbine cracks on its earnings call but said its turbines were “in great shape.” In an emailed statement, Solaris co-CEO Bill Zartler said the company has “not seen any extraordinary operating issues or failure rates on our generation equipment.”
Cracked turbines and broken engine cranks both are major failures that require replacement, according to Nina Sadighi, founder of Eradeh Power Consulting, an electrical engineer who formerly worked at Amazon and Tesla.
In July, Vantage Data Centers experienced a critical power failure at its data-center campus in Loudoun County, Va., forcing the operator to switch over to diesel backup power for about a day, a spokesperson confirmed. Distributed Power Solutions, which Kodiak Gas Services bought earlier this year, operates the on-site power. Crusoe has also run into turbine glitches and other technical problems at Stargate Abilene, according to the Information. Stargate partner Oracle runs servers for OpenAI.
What’s causing these problems? First, AI data centers’ power demands can be incredibly taxing on power systems. AI workloads cause massive power demand spikes during computation, and a drop off as those computation results are gathered, according to a paper from Schneider Electric. As data centers get bigger, oscillations could reach hundreds of megawatts or even gigawatts, according to the paper. Big swings can stress and shorten the life of engines and turbines, and can even cause shaft fractures, according to the energy research and consulting firm Wood Mackenzie.
The same problem has surfaced on power grids, when grid-connected data centers suddenly drop power usage. Grid operators have had to scramble to take power supply offline to make sure the sudden excess of supply doesn’t damage power plants and infrastructure.
Troy Patton, general manager at on-site power provider AlphaStruxure, said in an interview that he spoke to a potential hyperscaler client who required a very short power response time—tens of milliseconds, rather than hundreds of milliseconds as the industry is used to. These are response times that no rotating equipment, or a lithium battery, has ever had to respond to, he said....
....MUCH MORE
