The cost of inference has dropped by over 99.5% in the last three or four years while the price to the end user definitely has not fallen by that much and in fact all-in costs have actually risen. That gsp is the opportunity China is focused on.
From McKinsey & Company, June 25:
AI’s next breakthrough may not be a smarter model but a cheaper token.
The race to establish infrastructure for AI is driving one of the most significant capital mobilizations in history. In 2026, the four leading hyperscalers—Amazon, Google, Meta, and Microsoft—are collectively committing over $700 billion in combined capital expenditure, with a substantial majority directed at AI infrastructure. This figure would have been difficult to contemplate just three years ago. The investments span data center construction, accelerator procurement, and networking buildout on a massive scale, underscoring how compute has become a strategic asset. The new infrastructure is creating extraordinary and sustained demand across the entire semiconductor value chain for chips and other components that enable AI processes.
Two of the most important AI processes are training and inference. Training—either one-time or periodic—is the computationally intensive process of building a model by exposing it to large data sets. Inference is the ongoing process of running a trained model to respond to user queries. A single large language model (LLM),1 once trained, can be queried billions of times per day. The cost of each query is typically small, but it compounds at scale. Historically, training has accounted for most AI compute spending, but the balance is now shifting toward inference.
This dynamic has brought a once-theoretical question to the forefront: How can AI inference be made economically sustainable at the scale demanded by enterprise and consumer applications? Furthermore, how can the energy demand of AI computing be met or reduced? These questions underscore the significant pressures affecting the supply side of the AI economics equation and contributing to spiking AI costs.
Given the magnitude of efficiency gains required, no single breakthrough is likely to deliver the step change needed to achieve positive margins while maintaining frontier-model performance. Instead, meaningful progress will depend on a coordinated wave of innovation across the entire supply chain—from software-level model optimization to advances in silicon architecture, advanced packaging, memory systems, and optical interconnects.
We evaluated 13 of the most promising technology levers related to AI computing and evaluated their potential to reduce inference costs at scale. We also examined two other factors—architecture evolution and chip development timelines—that may affect how quickly and thoroughly companies can reduce inference costs, as well as new compute paradigms that might become an option in 2030 or later.
AI infrastructure is changing fast. In this article, we highlight the technologies most likely to shape AI inference economics and the implications for technologists, investors, and business leaders along the entire supply chain. The discussion that follows is necessarily technical at times because many of the biggest cost-reduction opportunities will come from trade-offs deep within the compute stack....
....MUCH MORE
If for no other reason than to avoid reinventing the wheel, the challenge of thriving in a deflationary environment means someone (yours truly) might have to dust off the economic history books for the period 1873 to 1913 to see how those folks did it.
