From American Affairs Journal, Fall 2025 / Volume IX, Number 3:
The most popular understanding of the U.S. stock market is the S&P 500 Index (SP5) of leading U.S. companies. Introduced in 1957, it is the most widely cited index, and more money is managed to it, by far, than any other benchmark. As such, SP5 has become a proxy for the health of U.S. investors, corporate America, and the overall economy. While SP5’s growth and prominence over the past half-century is well known, few market observers stop to ask basic questions: Why was it created? What need did it fill? What has happened since? And most importantly, is it still fit for purpose? That last question is the focus of this essay.
Nearly seventy years after its creation, SP5 may be fit for purpose, but it is clearly no longer the narrow one of the 1950s. While SP5 was initially a tool for measurement and understanding, it has over time become a central actor in shaping investor behavior. The widespread use of SP5-based products now actively influences the market SP5 was intended merely to observe. Though not a true observer paradox in the quantum physics sense, the feedback loop between measurement and subject has become pervasive. The consequences have been many, but perhaps the most significant one has been the widening gap between investment in the stock market and actual ownership of businesses through the stock market. This phenomenon, in turn, is a distinguishing characteristic of “financialization,” a criticism leveled against Wall Street in recent decades for focusing on generating financial returns regardless of, or even in opposition to, outcomes in the real economy.
The personification of an impersonal system is not unusual in a complex society. In the case of SP5, I would suggest that Dr. Frankenstein’s creature is the right character. Mary Shelley’s story, Frankenstein from 1818, is a tale of how scientific progress can diverge wildly from the plan.1 SP5 tracks that narrative all too well.
Bringing Order to Chaos
It did not start out this way. When SP5 was launched, it was a vast improvement over the numerous industry and early market averages that had appeared in the late nineteenth and early twentieth centuries. Those primitive measures struggled with stock splits, stock dividends, and dividend payments. Over time, they tended to become distorted. As one critic noted in 1957 at the launch of SP5, even if all the constituent elements of those older indices went to zero, the index could still have a positive value. More importantly, these earlier market measures, often associated with media platforms, were mostly indices of price, not size. They did not capture what we now call market capitalization but what was referred to at the time as market value. While indices of value existed, due to calculation limitations, they were neither broad nor timely. The most advanced Standard & Poor’s products prior to 1957 were a value-weighted index of 233 stocks calculated weekly, and one representing ninety companies calculated daily after the market close. Both had been launched in the 1920s.
By contrast, the new index was calculated hourly during the trading day. And with the five hundred largest NYSE stocks (out of nearly 1,100 on the exchange at that time), it covered more than 90 percent of the market’s value. Reflecting the structure of the economy and the market as it existed then, the new index had 425 industrials, twenty-five rails, and fifty utilities. (At that time, banks did not trade on the NYSE. Forty financials were added to the index starting in 1976.)
As the media around the introduction emphasized, SP5 was a technological development, taking advantage of emerging computation capability and communication coordination to provide a comprehensive value-weighted index measure on an hourly basis.2 The result gave investors and the financial media an intraday, easy-to-understand, and non-distorted answer to the question of how the overall stock market was doing in absolute and percentage terms: up, down, sideways. SP5 was designed to and initially did serve as a transparent reflection of the reality of the underlying securities trading on the NYSE, the country’s largest exchange by far.
In its earliest incarnation, SP5 also played a bit role in the emergence of modern finance. Starting in the late nineteenth century, economists had sought to bring to human affairs the measurement, rules, and formulas that had so advanced the natural sciences: hence the rise of what has come to be called the “social sciences.” Human behavior, it was understood, was no different than any other natural system. It was just a matter of getting the right data and figuring out the formulas that explained it. While the science of finance was late to the party, by the middle of the twentieth century, academics were hard at work bringing it up to then-current scientific standards, including the creation of measurement tools, such as SP5. This tale was well told by financial journalist Peter Bernstein in his must-read Capital Ideas (1991). It is countered somewhat—or at least placed in historical context—by my own Getting Back to Business (2018).
To be clear, SP5 has been wildly successful in a traditional sense. It is “the” serious benchmark for measuring and understanding the broad U.S. stock market. (The Dow Jones Industrial Average is older, but it is analytically of little use. That’s one of the reasons SP5 was created.3 The popular Nasdaq-100 is more recent and leans heavily toward the new economy.) Most other major market indices are similar to SP5 in design, including the leading “growth” and “value” benchmarks used by institutional investors and licensed by another index vendor. But SP5 is no longer just a yardstick. It has taken on a life of its own and changed the whole nature of stock investing in ways that most investors have not paused to consider.
“It Lives!”: The Creature Comes Alive
The creature’s first sign of life independent of its components came in the form of estimates for what SP5 itself would “earn.” These new “top-down” estimates took advantage of the growing availability in the postwar decades of greater and more detailed macroeconomic data. That data permitted market participants, especially brokerage strategists, to begin forecasting SP5’s expected change in profits per unit of the index compared to the prior year’s actual results. The move coincided with the emergence of macrolevel market investment strategies.
By the 1970s, Wall Street forecasts for individual companies and the market as a whole became sufficiently ubiquitous that a business sprung up to aggregate the estimates and provide “consensus” forecasts. I/B/E/S (Institutional Brokers’ Estimate System) was founded in 1976. Today, forward estimates for SP5 earnings are common. There were twenty-two on the Bloomberg terminal the last time I checked. Investors and traders can easily track how the consensus based on these estimates rises (and sometimes falls) during the course of the year as strategists revise their numbers. It is worth noting that the two sets of calculations—the top-down ones based on macroeconomic data, and the bottoms-up ones tallied from individual company forecasts—can and regularly do differ. Whether or not the forecasts coincide, the top-down projections for the index have become distinct from the specific prospects of the underlying companies. They may be directionally parallel, but they are analytically separate.
Once SP5 had its own estimates no longer directly linked to the underlying constituents, its evolution to sentient being progressed rapidly when these forecasts were used to calculate the market’s P/E ratio, the basic valuation measure of any stock. Assigning the market a P/E number was akin to Dr. Frankenstein throwing the switch and giving his creation life. If it has a P/E, it breathes. It can be cheap, it can be expensive, it can be fairly priced. It has a character. (That character is not completely transparent. The forward estimates are usually for “operating earnings,” after all the bad stuff is taken out. Actual reported earnings are historically up to 10 percent lower than those operating earnings: that is, forward P/Es are regularly understated.) However analytically useful, the use of valuation metrics like the P/E ratio further blurred the distinction between a constructed aggregate and a distinct investment asset.
Combining earnings estimates and P/E ratios, the creature has also developed hopes and dreams. These are known as “price targets.” As an institutional investor, I struggle with the very idea of a price target, but market strategists spend a lot of time on getting SP5’s bullseye right. What is more important, however, is that the price target exercise is nearly fully attenuated from the current valuation or future prospects of the companies that make up SP5.
Earnings growth from the index—note the semantics: earnings are now attributed to the index itself—has been generally consistent, with a relatively low degree of variance except in times of crisis, such as Covid, the global financial crisis, etc. In contrast, the valuation—the P/E multiple—can vary widely. Not surprisingly, there are now numerous, necessarily top-down models that try to determine the right valuation multiple for SP5. The “Fed model,” created by economist and strategist Ed Yardeni in the 1990s, notes the relationship between the market’s valuation and interest rates, especially the rate on the ten-year Treasury bond. This is not surprising as that bond is the basis for discount rates of future earnings or dividends. There should be a relationship.
Yale’s Robert Shiller assesses the valuation of SP5 earnings in terms of long-term historical P/E ratios adjusted for inflation. The result is the CAPE model, the market’s cyclically adjusted P/E. There are others, such as market capitalization to GDP ratio, a measure favored by Warren Buffett. I have my own much more subjective approach, a very distant derivative of Francis Fukuyama’s underappreciated Trust: The Social Virtues and the Creation of Prosperity (1995) with a heavy dose of Michael Jensen’s agency theory. The resulting simple algorithm is that high-trust societies trade with high multiples. Low-trust societies trade with low multiples. The reader can decide where we currently are or should be on that spectrum. These approaches all share one characteristic: they effectively view SP5 as an independent entity, not just a calculation derived from something else.
In the past five years or so, the direct relationship between SP5 and the underlying securities has become even more distant due to only a handful of companies (first the faangs, and now the Mag7) driving the index. What good is the Yardeni or Shiller model if only a half dozen stocks account for much of the market’s movement? By the time you read this, there may be a new acronym. In this environment, market strategists find themselves valuing—putting a prospective P/E multiple on—something that is supposed to be a broad measure of business activity when it is, in fact, increasingly narrow. SP5 was first an entirely neutral index, then it became a forecastable macroeconomic entity, and now it is the tail of the dog.
Initially just an idea, SP5 first took on corporeal form in August 1976 as an index fund, the Vanguard First Index Investment Trust, that was created specifically to track SP5. The age of so-called passive index investing had begun. Many other products followed suit. In recent decades, these index funds have been joined by ETFs. The well-named SPY (pronounced “spider”) debuted in 1993. Options contracts on SP5 began trading on the CBOE and CME in the early 1980s. The popular mini futures were introduced in 1997. These are real securities, not just a measure of stocks.
S&P Global estimates that, as of the end of 2023, total assets indexed (~$7 trillion) or benchmarked (~$6 trillion) to SP5 amounted to $13 trillion. That equaled just under one-third of the market value of $40 trillion of SP5 at that time. Another ~$3 trillion (notional value) of derivatives were based on SP5.4 All told, up to 40 percent of the index’s value was in SP5-based or SP5-oriented products. More recent figures, through the end of 2024, suggest a similar percentage of index assets are managed to the index itself.5 Think about that. While SP5 still measures the market, models based on it have become the largest single investment in the market.
As a youngster, SP5 was pliant, keeping to its original mission of measuring the market, not moving it. No longer. At its current size, our lumbering giant knocks heavily into the furniture as it moves through our portfolios. It is so large that inclusions and exclusions lead to material share price swings in the affected securities. There is a voluminous academic literature on the impact on security prices when there are changes in the index. Traders keenly participate in the “better to buy” and “better to sell” activity upon those inclusions and removals.6
....MUCH MORE
