"Why the Dow — Quirks and All — Is Beating the S&P 500"

This is a post I'd have written if I had the time. As I said in last month's "What if the Dow Were Market Cap Weighted?":

One of my pet peeves is pseudo-sophisticates who dismiss the Dow Jones Industrial Average out of hand, usually using the word anachronism somewhere in the dismissal.

The DJIA correlates with the S&P 500 92 to 98% of the time which is very close, the index was valuable enough that the CME structured their 2010 joint venture with a distribution to DJ of $607.5 million and finally the top dog on the DJ side, John Prestbo, won the Sharpe Indexing Lifetime Achievement Award (who knew?)

However, over the last 18 months it is not tracking so well...

When I hear someone sneering at the venerable old index I know they haven't studied the correlations between it and the S&P 500 and I am reminded of the line attributed to Churchill's detective-bodyguard:
"I detect a whiff of the parvenu."

Jason Zweig's Intelligent Investor via MoneyBeat:

This coming week, the dowager gets a makeover. The Dow Jones Industrial Average, which turned 117 years old this summer, will get three new members as Nike, Visa and Goldman Sachs Group replace AlcoaAA -0.98%, Hewlett-PackardHPQ +0.50% and Bank of America

Should you care? There will be no immediate impact on the level of the Dow, which finished this week at 15376; the three stocks getting the boot have barely budged in price. But the inner workings of the average, which have changed little since its inception, hold some important lessons for investors—most notably that things on Wall Street are seldom as simple as they seem.

The changes to the Dow “were prompted by the low stock price” of Alcoa, H-P and Bank of America and by the “desire to diversify…the index,” S&P Dow Jones Indices, which oversees the Dow, said in a statement. Dow Jones, publisher of The Wall Street Journal, owns a stake in the index company, which is a unit of McGraw Hill FinancialMHFI -0.47%.

Wall Street professionals, who regard the Dow as antiquated and cumbersome, mostly sneered. “High priced stocks totally skew this horribly constructed index,” tweeted Kid Dynamite, a trader who runs a popular blog. The way the Dow is put together, he later told me by email, “makes no sense at all.”
The Dow consists of 30 stocks; the higher its share price, the more a stock influences the level of the index. At its recent price of $192, International Business MachinesIBM +0.75% has the largest weight in the Dow, at 9.6%—while Alcoa, at $8, makes up just 0.4%. A 1% rise in IBM’s share price would add roughly 15 points to the value of the Dow, while a 1% jump in Alcoa’s price would add only 0.6 point.

That method of building an index, called price weighting, differs from “value weighting,” in which stocks with a greater total market value (regardless of their per-share price) loom larger. The market value of IBM’s shares totals $209 billion — but that gives it only the 12th largest position in the value-weighted S&P 500, at 1.32% of the index. Instead, Apple—total market value, $429 billion — has the biggest weight in the S&P 500, at 2.92%.

Even so, the Dow has delivered. Over the past 20 years, it has returned an annual average of 9.8%, while the S&P 500 has earned 8.6% annually.

A $10,000 investment in the Dow two decades ago would have risen to more than $64,300 by the end of last month, while the same amount invested in the S&P 500 would have amounted to less than $51,800. The Dow also is ahead of the S&P for the past five, 10 and 15 years.....MORE

Here's the post from Bespoke Investment group that got me thinking about index construction:

The Dow Jones Industrial Average is definitely the Rodney Dangerfield of the market these days.  Traders can't wait to take a shot at the index for being a dinosaur, holding too few names, holding the wrong names (this includes us), or being weighted incorrectly.  All of these complaints are mostly true, but if performance is the ultimate arbiter, the Dow wins....MORE

Some time ago I saw a piece from Merrill Lynch Quantitative Analysis who estimated the correlation coefficient relating the S&P 500 and the Dow Jones Industrial Average to be 0.95, and the DJIA.
Mr. Prestbo, mentioned in the intro says it was even higher in 2010 and goes into some detail as to the ins-and-outs of the comparison.
From MarketWatch, Jan. 21, 2011:

S&P 500 and the Dow: Tale of two markets

Commentary: Smaller stocks are a big deal in benchmarks’ returns

By John Prestbo

NEW YORK (MarketWatch) — The Standard & Poor’s 500-stock index and the Dow Jones Industrial Average moved in the same direction 98.5% of the time in 2010. Yet the S&P 500 rose 12.8% while the Dow advanced 11%. How did that happen? 

The short answer is smaller stocks. The long answer illuminates the differences in methodologies that careful investors should keep in mind — always at the back and sometimes in the front — when they use an index to judge the stock market. 

Before getting to the indexes themselves, a small detour is in order. “Correlation” measures the degree of similarity in the direction of movement of two indexes — but not the magnitude of those movements. An index that rises 1% and one that rises 2% in a given period are still 100% correlated.

So, while the Dow DJIA +0.49%  and S&P 500 are correlated 95% over the past 50 years, one or the other comes out on top each year.

Up and down markets

Since 1960, The S&P 500 SPX +0.27%   has outperformed the Dow in 26 calendar years, gaining 13.7% on average. Conversely, the Dow has bested the S&P 500 in 24 different calendar years with an average advance of 4.9%. However, their annualized returns over this half-century are much closer: 8% for the S&P 500 and 7.6% for the Dow. 

The reason is that the S&P 500 tends to beat the Dow in rising markets while the Dow tends to fare better in falling markets. Out of the past 50 years, the S&P 500 rose in 37 to the Dow’s 35; the down-year scoreboard reads Dow 15, S&P 13. That track record is what gives the S&P 500 its slight edge.

Why does this up-market, down-market distinction exist? The primary explanation is that the S&P 500 includes many more smaller stocks than the Dow does. The median market capitalization of the S&P 500 components at year-end was $10.6 billion, compared to $109.9 billion for the Dow. 

The S&P’s biggest 235 stocks are the largest public companies in America, but after that point the components skip down the size scale. The smallest S&P 500 company at year-end ranked 1,172 on the U.S. market capitalization roster (as measured by the Dow Jones U.S. Total Stock Market Index, which includes all listed U.S. stocks). There are 19 stocks in the S&P 500 that Dow Jones classifies as small, and which meet the size qualifications for S&P’s own 400-stock midcap index. 

Smaller stocks tend to do better in the first half of bull runs because their prices are more readily pushed higher by the inflows of investor cash than big stocks are. Last year illustrated this phenomenon neatly when the Russell 2000 trounced the Russell 1000 by 25.3% to 13.9%. (The two were almost tied in the 2009 rebound.)...MORE

So there you go, two of Dow Jones' finest, Zweig and Prestbo, on the Dow Jones Industrial Average.