People on all sides of the recent push for direct replication—a push I find both charming and naive—are angry....and keep his pathetic little job. As the young people used to phrase the rejoinder: L
By the way, that was James Coan, who calls himself Dr. although he apparently didn't have the intellectual horsepower to become a Chiropractor or D.D.S., writing in the Journal Medium.
Rather than the two honorable professions named above he's a freakin' Associate Professor of Clinical Psychology at the University of Virginia.
Jefferson weeps.
See, the thing is, if what one is writing about can't be reproduced, that kind of writing is called 'Literature'.
And, although gentle reader probably doesn't care, yes, I know the difference between replication and reproducibility.
With that note on the current state of the so-called soft sciences here's Barron's Focus on Funds with a much more upbeat post:
You might have heard of the study “Why Most Published Research Findings are False.” But who knew Ph.D.s had this much righteous indignation?
Two months back, we heard the argument that tinkering with investment backtests amounts to fraud.
The latest is a group at Duke University arguing that commonly accepted methods for spotting investment “factors,” i.e., the qualities that quants deem to be sources of investing returns, are too weak. It follows that many of the factors themselves are likely to be bunkum, the group argues.
The most remarkable statistic turned in by Campbell R. Harvey, Yan Liu and Heqing Zhu in their Social Science Research Network paper may be the fact that more than 300 investment factors have been trotted out by researchers, most of them in the last decade.Back to the replication thing, it's pretty important, Nature devoted a special issue to the topic:
Do you suspect that they can’t all be real? I do. That’s more than one factor for every pair of stocks in the S&P 500.
From Harvey, Liu and Zhu:
At least 315 factors have been tested to explain the cross-section of expected returns. Most of these factors have been proposed over the last ten years. Indeed, Cochrane (2011) refers to this as “a zoo of new factors”. Our paper argues that it is a serious mistake to use the usual statistical significance cutoffs (e.g., a t-ratio exceeding 2.0) in asset pricing tests....MORE
Challenges in irreproducible researchBy-the-bye, should Coan decide against the rigor of writing for Medium or appearing on CBS' Sunday Morning there is always The Journal of Irreproducible Results.
No research paper can ever be considered to be the final word, and the replication and corroboration of research results is key to the scientific process. In studying complex entities, especially animals and human beings, the complexity of the system and of the techniques can all too easily lead to results that seem robust in the lab, and valid to editors and referees of journals, but which do not stand the test of further studies. Nature has published a series of articles about the worrying extent to which research results have been found wanting in this respect. The editors of Nature and the Nature life sciences research journals have also taken substantive steps to put our own houses in order, in improving the transparency and robustness of what we publish. Journals, research laboratories and institutions and funders all have an interest in tackling issues of irreproducibility. We hope that the articles contained in this collection will help.
Free full access
HT on Coan, the invaluable Retraction Watch.
And don't get me wrong, if Coan wants to do his thing on his own dime, have at it and have fun. The thing that rankles is his sucking at the taxpayer teat.
And that he's teaching kids.
