It goes with the territory: weather and economics and markets are all so complex they require modeling just to facilitate comprehension but models are not reality:
October 18, 2012
Modelling vs. Science
A subject near and dear to our jaded hearts, some links below.
If an experiment is not reproducible it is not science.
If an hypothesis is not falsifiable it is not science.
Finally, our two guiding principles regarding models:
"The map is not the territory"
-Alfred Korzybski
"A Non-Aristotelian System and its Necessity for Rigour in Mathematics and Physics"
presented before the American Mathematical Society December 28, 1931
....................................................................................................................................................................
"All models are wrong, but some are useful"
-George E.P. Box
Section heading, page 2 of Box's paper, "Robustness in the Strategy of Scientific Model Building"
(May 1979)
One last point:The current generation of modellers seem never to admit error.
Their hubris and lack of humility are not just ridiculous but dangerous yet they blithely move from academe into government and back without so much as an acknowledgement, much less an apology.
From FT Alphaville's Jemima Kelly:
Is the “science” behind the lockdown any good?
We should all be pretty familiar with the narrative by now. An arrogant, exceptionalist British government was until mid-March pursuing a reckless strategy of herd immunity that would have callously allowed a huge number of old and vulnerable people to die and the health system to be overwhelmed. Then came a “bombshell” from Imperial College London: a “doomsday report” predicting there would be 500,000 deaths if we were to carry on down that road, prompting a sudden government U-turn, and ultimately the decision to lock the country down.....MUCH MORE, come for the exposition, stay for the comments.
Gone was the Machiavellian Dominic Cummings plan of “letting old people die”; in was STAY AT HOME; PROTECT THE NHS; SAVE LIVES. (The notion that it was Cummings who was pushing for the herd immunity idea has since been disputed, while the notion that Cummings was into the staying-at-home idea has also since been, er, disputed.)
But then, after all that, it turned out that the computer code Imperial had relied on to predict the future in that March 16 paper (“Report 9”) was outdated, full of bugs, and based on flimsy, unscientific assumptions. The code was simply totally unreliable. All academic epidemiology should be defunded immediately. The lockdown, surely, could no longer be justified.
As the Telegraph put it in their headline on May 16, this could be “the most devastating software mistake of all time”! It could “supersede the failed Venus space probe” in terms of economic cost and lives lost!
The question is: is any of this true? Did the modelling, as the Daily Mail put it last Saturday, “single-handedly (trigger) a dramatic change in the government’s handling of the outbreak”? If the code is so bad, does that render the modelling useless? And would shoddy modelling remove the justification for the lockdowns in place across much of the globe anyway?
How much did the Imperial paper really matter?
During a virtual panel at the Institute of Art and Ideas over the Bank Holiday weekend, commentator Toby Young argued that states “across the world were completely unjustified in suspending our liberties based on a few weak computer models”. He was referring to the Imperial modelling, which he said had “panicked” governments into shutting down their economies. This came on the back of a couple of posts on Young’s “Lockdown Sceptics” blog, written by a pseudonymous writer, who called the code underpinning the modelling “unusable”.
Actually, though, the Imperial report did not single-handedly trigger a change in government policy at all, as Imperial have themselves made clear. When we spoke to the head of Imperial’s modelling team Neil Ferguson, who had authored the report, in early April, he told us that “a much wider range of scientific advice and modelling advice” had been given to government in the two weeks leading up to the report, all of which pointed in a similar direction.
Research by the London School of Hygiene and Tropical Medicine (LSHTM), for example, whose scientists also sit on the government’s Scientific Advisory Group for Emergencies (SAGE), had come up with similar numbers in their modelling. Indeed the Scientific Pandemic Influenza Group (SPI-M), which reports to SAGE, had already produced a report on March 2, which warned of more than 500,000 deaths in an “unmitigated reasonable worst case scenario”.
And before Imperial had even given their numbers to the government, they had been discussed with the others on SPI-M. As LSHTM’s head modeller John Edmunds made very clear to us on a phone call last week:
There are four groups independently looking at this. I think there’s a misperception here that Imperial are running the country. They’re not. They’re one of many groups and there’s a process whereby they feed their results into SPI-M, SPI-M looks at it, we come to a consensus, it goes up to SAGE, then SAGE looks at it.Why this narrative and this pouncing on the report, then? Partly, it’s because we wanted a narrative to cling onto.
Fleeting heroes and invisible villains....
Someone should also look at the University of Washington Institute for Health Metrics and Evaluation (IHME) model which seems not to even be able to predict the past but it won't be me.
I have to model today's natural gas storage report, model the market's dawning realization that at 24 times sales Beyond Meat's valuation might be a bit rich and model something to eat before manifesting it in my tummy.
