If what you're doing isn't reproducible, what you're doing isn't science.
It might be a worthy endeavor, it might important, it might be...ahhh screw it, it's not science.
From Discover Magazine's Neuroskeptic blog, Jan. 2:
In a new paper in the Journal of Experimental & Theoretical Artificial Intelligence, Chris Drummond takes aim at the ‘reproducibility movement’ which has lately risen to prominence in science.
As one of the early advocates for this movement, I was interested to see what Drummond had to say. While I don’t find his argument wholly convincing, he does raise some good points.
Drummond begins by summarizing the case for reproducible research as it sees it. The claim is that reproducibility – the ability of other scientists to exactly reproduce and confirm a given result – is central to science. It is further claimed that we can promote reproducibility by requiring authors to submit their data, and their analysis scripts (code), with each publication and that this will, amongst other benefits, help to prevent scientific fraud.
Against this, Drummond says that
(1) Reproducibility, at least in the form proposed, is not now, nor has it ever been, an essential part of science.To my mind the most interesting part of this paper falls under Drummond’s discussion of point (1), in which he argues that reproducibility is not very important to science, contrary to popular belief. Drummond defines ‘reproducibility’ as the ability to repeat an experiment as exactly as possible and get the same result: “The aim is to minimise the difference from the first experiment including its flaws, to produce independent verification of the result as reported.”...
(2) The idea of a single well-defined scientific method resulting in an incremental, and cumulative, scientific process is, at the very best, moot.
(3) Requiring the submission of data and code will encourage a level of distrust among researchers and promote the acceptance of papers based on narrow technical criteria.
(4) Misconduct has always been part of science with surprisingly little consequence. The public’s distrust is likely more to with the apparent variability of scientific conclusions.
...MORE
*Among our posts on replication and reproducibility::
Reproducibility is pretty much the cornerstone of science. And yet some Bozo can come out and say:
By the way, that was James Coan, who calls himself Dr. although he apparently didn't have the intellectual horsepower to become a Chiropractor or D.D.S., writing in the Journal Medium.
Rather than the two honorable professions named above he's a freakin' Associate Professor of Clinical Psychology at the University of Virginia.
Jefferson weeps.
See, the thing is, if what one is writing about can't be reproduced, that kind of writing is called 'Literature'.
And, although gentle reader probably doesn't care, yes, I know the difference between replication and reproducibility.
People on all sides of the recent push for direct replication—a push I find both charming and naive—are angry....and keep his pathetic little job. As the young people used to phrase the rejoinder: L
By the way, that was James Coan, who calls himself Dr. although he apparently didn't have the intellectual horsepower to become a Chiropractor or D.D.S., writing in the Journal Medium.
Rather than the two honorable professions named above he's a freakin' Associate Professor of Clinical Psychology at the University of Virginia.
Jefferson weeps.
See, the thing is, if what one is writing about can't be reproduced, that kind of writing is called 'Literature'.
And, although gentle reader probably doesn't care, yes, I know the difference between replication and reproducibility.
Serious stuff.
I've mentioned the fact that if you can't can't replicate what you're doing, what you're doing isn't science. It might be metaphysics, it might be pseudoscience, it might be religion, it might be any number of things but it isn't science.
Reproducibility and falsifiability, along with predictive power are pretty much the definition of science.
(here's a quick explanation of the difference between reproducibility and replicability in science)
What Mendeleev did in describing the properties of as-yet undiscovered elements was science. So-called post-normal science is not, it's policymaking gussied up with sciency sounding words.
See Feynman's 1974 Caltech commencement address. Cargo Cult Science for a really smart guy's take on the issue....
